ActAnywhere: Subject-Aware Video Background Generation
Authors: Boxiao Pan
Zhan Xu
Chun-Hao Paul Huang
Krishna Kumar Singh
Yang Zhou
Leonidas Guibas
Jimei Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Background Generation
Video Generation
Video Synthesis
Video Editing
Primary area: machine_vision

Abstract: We study a novel problem to automatically generate video background that tailors to foreground subject motion. It is an important problem for the movie industry and visual effects community, which traditionally requires tedious manual efforts to solve. To this end, we propose ActAnywhere, a video diffusion model that takes as input a sequence of foreground subject segmentation and an image of a novel background and generates a video of the subject interacting in this background. We train our model on a large-scale dataset of 2.4M videos of human-scene interactions. Through extensive evaluation, we show that our model produces videos with realistic foreground-background interaction while strictly following the guidance of the condition image. Our model generalizes to diverse scenarios including non-human subjects, gaming and animation clips, as well as videos with multiple moving subjects. Both quantitative and qualitative comparisons demonstrate that our model significantly outperforms existing methods, which fail to accomplish the studied task. Please visit our project webpage at https://actanywhere.github.io.

ActAnywhere: Subject-Aware Video Background
Generation
Boxiao Pan" Stanford University
Zhan Xu Adobe Research
Chun-Hao Paul Huang
Adobe Research
Krishna Kumar Singh Adobe Research
Yang Zhou Adobe Research
Leonidas J. Guibas Stanford University
Jimei Yang Runway
Abstract
We study novel problem automatically generate video background that tailors to foreground subject motion. It is an important problem for the movie industry and visual effects community, which traditionally requires tedious manual efforts to solve. To this end, we propose ActAnywhere, video diffusion model that takes as input a sequence of foreground subject segmentation together with an image of novel background, and generates a video of the subject interacting in this background. We train our model on large-scale dataset of 2.4M videos of human- scene interactions. Through extensive evaluation, we show that our model produces videos with realistic foreground- backroudd interaction while strictly following the guidance of the condition image. Our model generalizes to diverse scenarios including non-human subjects, gaming and animation clips, as well as vidcos with multiple moving subjects. Both quantitative and qualitative comparisons demonstrate that our model significantly outperforms existing methods, which fail to accomplish the studied task. Please visit our project webpage at https: / /aacaanyeeerreeee. .eterre.gttiii github.io.
1 Introduction
Compositing an acting video of a subject onto novel background central for creative story-telling in filmmaking and visual effects. The key requirement is seamlessly integrating the foreground subject with the background in terms of camera motions, interactions, lighting and shadows, so that the composition looks realistic and vivid as if the subject acts physically in the scene. In movie industry, this process is often conducted by virtual production [1] that requires artists to first create 3D scene and then to film the acting video in LED-walled studio to render the video in 3D engines. This process is not only tedious and expensive, but most importantly, prevents artists from quickly iterating their ideas.
Inspired by this artistic workflow, we study the novel problem of automated subject-aware video background generation. As shown in fig. given a foreground segmentation sequence that provides the subject motion, as well as condition frame that describes a scene, we aim to generate a video that adapts the subject this scene with realistically synthesized foreground-backgroune interaction. This condition frame can be either a background-only image, or composite frame consisting of both background and foreground created via photo editing tools [3] generative image model [32].
This problem, at its core, requires retaining part of the input video while generating the rest to adapt to it. To the best of our knowledge, the closest works to this setting are those video editing and inpainting outpainting. Video editing methods assume a source video as input and make edits
*Wor done during internship at Adobe.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Condition
w/ subject
w/ subject
Input
Condition
w subject
w/o mubject
Figure 1: Given sequence of foreground segmentation as input (top), and one frame that describes the background as the condition (left), ActAnywhere generates coherent video background that adapts the subject motion. We show two subjects here, each with two generated samples. ActAnywhere is able to generate videos consistent with the condition frame with highly realistic details such as splatting water, moving smoke and flame, shadows, duck feet, etc. generalizes to diverse distribution of subjects and backgrounds, including non-human subjects. Our method works with both composited frames and background-only images as the condition.
based on condition signals, such as natural language or image. In comparison, our task poses unique the video needs to be retained
challenges that these methods cannot solve. First, the foreground and sometimes refined on the boundary (as shown in fig. 8 in appendix), while prior works generally perform holistic changes to the entire video [5. 8, 13, 16, 22, 24, 42]. Second, existing methods most commonly condition the editing process on short text prompt [5, 8, 13, 16, 22, 24, 42], which only poses loose constraints that not enough for artists' creative intentions. Those that can condition on an image [13] are not able to generate contents that strictly follow the guidance. We demonstrate such comparisons fig. On the other hand, video inpainting outpainting methods [14, 43, 46, 49) aim to perform context-aware removal / expanding for a video. However, these methods focus pixel harmonization, while our emphasis is on taking additional control signal that indicates the desired background, and synthesizing large and dynamic background region with reasonable interaction with the given foreground.
To this end, we propose video diffusion model that explicitly controls the foreground motion with the foreground segmentation sequence, and additionally, conditions the generation on an image that describes the desired background. We train the model with a designed self-supervised learning procedure that takes the foreground segmentation sequence the input video to predict the original video, conditioned on a randomly sampled frame. This training procedure enables the model to retain the foreground subject; 2) hallucinate missing details from imperfect segmentation; and 3) adhere the guidance of the condition frame while being robust to the subject's pose in it.

---

Moreover, we propose to express the condition signal as the CLIP features of the image, which we show empirically outperforms its alternatives. We also concatenate background mask with the segmentation as the input to better indicate the region to be generated.
We train our model interactions and evaluate both on held-out set as well as on videos from DAVIS [30]. ActAnywhere able generate highly realistic videos that follow the condition frame, and the same time synthesizes video background that conforms to the foreground motion. Notably, despite being trained solely on videos of humans, ActAnywhere generalizes to non-human subjects, such as animals and man made objects, in zero-shot manner.
large-scale dataset [26] that consists of 2.4M videos of human-scene
In summary, contributions
1. We introduce novel problem of automated subject-aware video background generation.
2. We propose ActAnywhere, a video diffusion model to solve this task, and train it on large-scale human-scent scene interaction video dataset in a self-superrised manner.
3. Extensive evaluations demonstrate that model generates coherent videos with realistic subject-scene interactions, camera motions, lighting and shadows, and generalizes to out- of-distribution data including non-human subjects, gaming and animation clips, and videos with multiple moving subjects.
2 Related Work
Video generation. There have been long thread of works on video generation. The core architecture has evolved from GANS [11, 38, 41] to more recent transformers [15, 40, 46, 49] and diffusion models [6. 9, 17, 19, 20, 24, 47]. Below we review the most related diffusion-based works. Most of these works leverage temporal self-attention blocks inside the denoising U-Net order to acquire temporal awareness. On top of that, Text2 correlate the latents in a video. LVDM [19] and Align Your Latents [6] both design hierarchical approach to generate longer-term videos. Align Your Latents additionally fine-tunes a spatial super-resolution model for high-resolutinn video generation. AnimateDiff [17] proposes to train the temporal attention blocks on a large-scale video dataset, which can then be inserted into any text-to-image diffussin models (given that the architecture fits) to turn that into a tcxt-to-video model, in zero-shot manner. VideoCrafterl [9] further uses dual attention to enable joint text and image-conditioned generation. These works focus on unconditional generation or with text or image conditioning, but are not able follow the guidance of additional foreground motion.
[24] introduces additional noise scheduling to
Video editing. Another thread studies the problem of video editing, where a source video is given as input, and edits are performed according to some condition signals. Text2Live [5] uses pre-trained video atlases of the input video, and performs text-guided edits on the foreground or background. Genl [13] leverages depth maps estimated by pre-trained network [33] as additional condition to improve the structural consistency. Tunc-A- A- A-Vidoo [42] proposes to finctune only part of the spatial- attention blocks and all of the temporal-attentinn
blocks on a single input video. TokenFlow [16] uses latent nearest neighbor fields computed from the input video to propagate edited features across all frames. Both VideoControlNet [22] and Control-A-Vides [10] adopt ControlNet [48]-like approach to condition the video diffusion process with additional signals such as depth maps or Canny edges extracted from the input video. As stated above, these works apply holistic changes to the entire video and are not able to retain the subject. Moreover, they either take only loose constraints from text conditioning, or not able strictly follow the image guidance.
One major downside of these works is hence that the generated videos tend to keep the spatial structure from the source video, which limits the edits that the model can perform to stylistic changes. In our work, we propose to condition on the foreground segmentation for the motion, while extract the background information only from one condition frame. In particular, using the masked foreground as input endows a nice separation as in what to preserve and what to generate.
Image and video inpainting. Image / video inpainting aims to fill a missing region, often expressed as a mask. Image inpainting methods either take condition signals such as natural language and image [34, 44, 45), or rely solely on the context outside the masked region [14, 36, 46, 49]. Recent diffusion- image inpainting methods use a combination of masked image and the mask itself, and condition the diffusion process either natural language [34, 44] or an image of the condition
3

---

object [45], or perform unconditional diffusion [36]. For video inpainting, MAGVIT [46] proposes generative video transformer trained through masked token prediction, and is able inpaint small masked regions afterwards. ProPainter [49] designs flow-based method by propagating pixels and features through completed flows. M3DDMM [14] leverages a video diffusion model, and conditions the diffusion process global video features extracted by video encoder. Different from these works, we aim to generate large background regions that strictly follow the condition frame. Moreover, the generated background needs to adapt to the foreground subject motion in a coherent way. This poses significant challenges that previous inpainting methods cannot tackle.
3 Method
We first provide essential preliminary background on latent diffusion in section 3.1. We then formally define our problem in section 3.2 and delve into our model design section 3.3. Finally, we specify the training details in section 3.4.
3.1 Preliminaries Latent Diffusion Models
Diffusion models such as DDPM [21], encapsulate a forward process of adding noise and a backward process of denoising. Given a diffusion time step r, the forward process incrementally introduces Gaussian noises into the data distribution Tn ~ g(2r)) via a Markov chain, following a predefined variance schedule denoted as 3:
(xx11x--1) = -
-,x,-1,
(1)
For the backward process, a U-Net [35] Ep is trained to denoise x, and recover the original data distribution:
px xx) =
Ex(,,)))
(2)
Mg and Eg are parametrized by ca. The discrepancy between the predicted noise and the ground-truid -tuth noise is minimized as the training objective.
Stable Diffusion [34] further proposes to train the diffusion model in the latent space of a VAE [25].
Specifically, an encoder & learns to compress an input image æ into latent representations {(x). and decoder D learns to reconstruct the latents back to pixel space, such that r (z)()). In this way, the diffusion is performed the latent space of the VAE.
3.2 Problem Formulation
Given an input video X' E RTxH*Wx3 featuring a foreground subject, we first deploy a segmentation algorithm, such as Mask R-CNN [18], to obtain subject segmentation sequence, S E IR Hxxxx x3 along with the corresponding masks, M R w'x1, Both S and M serve as input to our model. S contains the segmentation of the foreground subject, with background pixels set to 127 (grey). M has the foreground pixels set to and background to 1. Across all our experiments, H W 256 and T 16.
Additionally, we also incorporate single condition frame c E H x3 describing the background that we want to generate. As shown in fig. 2, is a randomly sampled frame from X' at training time, while can be either a frame showing image at inference time. The goal is thus to generate an output video y with the subject dynamically interacting with the synthesized background. The motivation of using an image not language as the condition is that image is more straightforward media to carry detailed and specific information of the intended background, especially when users already have pre-defined target scene image.
barkkround composition or background
3.3 Subject-Aware Latent Video Diffusion
We build our model based on latent video diffusion models [17]. In our architecture design, we address two main questions: 1) providing the foreground subject sequence to the network to enable proper motion guidance, and 2) injecting the condition signal from the background frame to make the generated video adhere to the condition.

---

Foreground suhject
VAB enoodar
Faregroud Eaauuees
Spatial Temporal attention
Down sample St.
Misks M
Down-somplod masks A
U-No
diffision
at sepp
Ingut video
VAE encoder &
Neisy
VAE decoder 2
Reconsenctel video
daring training V
Condition fremu
OR
Sampled
background during informnce
CLIP image encoder
during training
Genorated video with novel background during inforenee V
Figure Architecture overview. During training, we take a randomly sampled frame from the training video to condition the denoising process. At test time, the condition can be either composited frame of the subject with a novel background, or a background-onl, image.
We present our pipeline in fig. 2. For the foreground segmentation sequence S, we use the pre-trained VAE [34] encoder E to encode the foreground segmentation into latent features S R 16 32xx3xx32>>
We downsample the foreground mask sequence .M times obtain the resized mask sequence M I 16X322x32x to align with the latent features S. To train the denoising network ep, we encode the original frames X' with the same VAE encoder into latent representation Z E R. p16x32x32x4 and add noises at diffusion time step with the forward diffusion process denoted in eq. (1) get noisy latent feature Z,. We subsequently concatenate S, M and Z, along the feature dimension, forming a 9-channel input feature F; IR 16x9xx2x32 to the U-Net. During inference, Zo is initialized as Gaussian noises, and gets auto-regressively denoised for multiple time steps to sample final result, according to the backward diffusion process described in eq. (2). The denoised latents are then decoded a video via the VAE decoder D.
We build our 3D denoising U-Net based on AnimateDiff [17]. AnimateDiff works by inserting a series of motion modules in between the spatial attention layers in the denoising U-Net of pre-trained T21 diffusion model. These motion modules consist of a few feature projection layers followed by ID temporal self-atiention blocks.
For the condition image c, we encode with the CLIP image encoder [31] and take the features from the last hidden layer its encoding F These features then injected into the UNet ca through its cross-attention layers, similar to [26, 34]. We empirically find that this method achieves better temporal consistency compared other alternatives, such using VAE features for either cross-attention or concatenation with other input features. We ablate on this in table 3.
3.4 Training
Training is supervised by simplified diffusion objective, namely predicting the added noise [21]:
-= (F F>)
where e is the ground-trute truth noise added.
Dataset. We train on the large-scale dataset compiled and processed by [26], which we refer to as HiC+. The resulting dataset contains 2.4M videos of human-scene interactions. also provides foreground segmentation and masks. We refer the reader the original paper for more details.
5

---

Pre-trained weights. We initialize the weights of our demoising network eo with the pre-trained weights from the Stable Diffusion image inpainting model [34]', which is fine-tuned on top of the original Stable Diffusion on the text-conditioned image inpainting task. We initialize the weights of the inserted motion modules with DiimateDiff v2
For the CLIP image encoder, we use the haap-vit-large-pacchi features from the last hidden layer have dimension of 1024, while the pre-trained U-Net takes in features of dimension 768 as the condition, which are also in the text feature space. To account for this, we train an additional two-layer MLP to project the features into the desired space.
variant provided by OpenAl, whose
During training, we freeze the shared VAE and the CLIP encoder, and fine-tune the entire U-Net.
Data processing and augmentation. Obtaining perfect segmentation masks from videos is chal- lenging. The masks may be incomplete, missing some parts of the foreground, or be excessive such that they include leaked background near the boundary. To deal with incomplete segmentation, during training, we apply random rectangular cut-outs to the foreground segmentation and masks. We provide more information on this in section 7.2 of the appendix. To reduce information leak from excessive segmentation, we perform image erosion to the segmentation and masks with a uniform kernel of size 5 x 5, both during training and inference.
Random condition dropping. In order to enable classifier-free guidance at test time, we randomly drop the segmentation and the mask, the condition frame, or all of them at 10% probability each during training. In these cases we set them to zeros before passing into the respective encoders.
Other details. We use the AdamW (27] optimizer with a constant learning rate of 3e-5. We train on 8 NVIDIA A100-80GB GPUs with batch size 4, which takes approximately week to fully converge.
Experiments
We start by describing the data used for evaluation. We then show diverse samples generated from our method in section 4.1, both using inpainted frame and background-only frame as the condition. In section 4.2 and section 4.3, we compare with baselines through qualitative and quantitative evaluations, including user study. In section 4.4, provide ablation study results on key design choices. In the appendix, we include additional results on general video inpainting outpainting, generalization to videos from diverse domains, and robustness to inaccurate segmentation, along with further implementation details and a discussion on limitations and potential ethical impacts.
We strongly encourage the reader to check our webpage, where we show extensive videos on video background generation with diverse generated contents and camera motions, and under various condition scenarios. It also contains the video version of the comparison with baselines.
Evaluation data. Following prior works [5, 10, 13, 16, 42], we compare with previous works on videos sampled from the DAVIS [30] dataset. We select videos with both human and non-human subjects. We also evaluate qualitatively and perform ablation study on held-out samples from the HiC+ dataset following the original data splits [26). Samples with our method are generated with 50 denoising steps, with a guidance scale [34] of 5.
4.1 Diverse Generation with ActAnywhere
In fig. 3, we show results on the held-out segmentation sequences from the HiC+ dataset, using an inpainted frame or a background-only frame as condition. ActAnywhere generates highly realistic segground-backgrouse
interactions both at coarse and fine levels. At a coarse level, our model synthesizes road structure, pumpkin field, city views, waves, etc. that align with the subject's motion. While at fine level, our method also generates small moving objects that are in close interaction with the subject, such as the buckets, bed sheets, horses and dune buggies, as well as the dog. Moreover, these generation stay consistent across frames, and tightly follow the guidance in the condition frame. The synthesized backgrounds also exhibit coherent scale, lighting, and shadows (also see fig. 1).
http://wwghnngaaa....ommmmmmmmie http://ggginn..ommooyyywwwannnnt httpss//huggiggfaacccooooommelep latge-patch114

---

Impit
Condition
Condition
ttrstoo
Inpur
Input
Condition
Condition
Input
Input
Condition
Condition
Imput
Inpur
Condition
Condition
Figure 3: Diverse results from our method. The top part shows examples using inpainted frames as condition, while bottom contains examples with background-only conditioning. Foreground sequences are from the held-out set of HiC+.

---

Input (ouns)
Input (ours)
Imput (olbers)
Impur (others)
Condition
Condition
Ours
Cen
Tex2LIVE
drifting on a snowy mountain road
performing motorcycle stunts
Tokenfllow
A car drifting snowy mountain road
A woman performing metorcycle stunts
VrateCraftert Animute DifF Control A- Video
drifting on a snowy mountain roud
performing metorcycle stunts
Figure Comparison with baselines. We provide results on two videos sampled from the DAVIS [30] dataset. For each example, show three representative frames (top) and their corre- sponding condition signal (left). Note that different methods assume different input, conditioning or pre-trained models, specified in section 4.2.
4.2 Qualitative Comparison
Baselines. We first clarify that since we study novel problem, and there is no prior work operating under the exact same setting to the best our knowledge. We hence compare to closest works and adapt some, i.e. AnimateDiff' [17]. if necessary. Nonetheless, emphasize that the formulation and pipeline are the core contribution of this work.
We compare ActAnywhere to a number of baselines, which we classify based on whether they do (fig. 4 middle) do not (fig. 4 bottom) take video as input. For the methods taking a video as input, Genl [13] takes an additional image as condition, and also leverages pre-trained depth-estimatin network [33]. Given pre-trained neural atlases [23], Tex12LIVE [5] assumes a text prompt as condition to synthesize the edited video. TokenFlow [16] also uses text conditioning. Control-A-Vide A- [10] first extracts Canny edges from the input video, then synthesizes the output video conditioned jointly on the edges and text.

---

Method
CLIP cond
CLIP temp 1 FVD 4
Control-A-Vine [7]
0.643
0.942
381.8
Condition
Visual
Temporal
TokenFlow [12]
0.762
0.943
323.0
Gen [9]
0.827
0.943
337.2
97.47%
72.13%
64.36%
Ours
0.862
0.945
313.4
Table User study
Table 1: Quantitative comparison against baselines.
For baselines that do not take video input, the original AnimateDiif [17] only uses text condi- tioning. We use the strategy contributed by public pull request to make take additional image conditioning. Specifically,
time, latent features are first extracted from the condition image with the pre-trained SD VAE encoder [34], which are then merged with the original per-frame Gaussian noises through linear blending. The diffusion process is later conditioned on text prompt too. VideoCrafter [9] provides both text-to-video and an image-to-vides model. We use the latter for closer comparison setting.
Results. The qualitative comparison on two examples from the DAVIS [30] dataset is shown in fig. 4. Our method generates temporally coherent videos that follow the foreground motion with highly realistic details, e-g. falling snow and snow on the car windshield, while strictly following the guidance and constraints given by the condition frame. Baseline methods in the first category generally inherit the structure present in the input video, e.g. road direction, horse, etc., and hence they completely fail when fine-grained edits are desired, e.g. horse changes to motorcycle in the second case. Methods in the second category generate unconstrained motion due to lack of guidance (VideoCrafterl in the second example generates backward motion, which more evident in the video on our webpage).
4.3
Quantitative Comparison
Baselines. We compare with Control-A- Video [10], TokenFlow [16] and Gen1 [13]. We also conduct a user study for preferences in comparison to Genl.
Metries. We evaluate the consistency to the condition image, the temporal consistency of the generated videos, as well as the general generation quality. Specifically, we report the average cosine similarity between the CLIP [31] image embeddings of all generated frames and that of the condition image setP__ond), between all pairs of generated frames (CI/P_temp), and the FVD score [39] against set of real videos.
For the user study, we ask the participants if they prefer results from our model vs. those from Genl on: 1) consistency to condition image; 2) visual quality; and 3) temporal consistency. The results are presented as the percentage of our model being preferred over Genl.
Results. Results on 30 videos from DAVIS are reported in table 1. Our model outperforms baselines across all metrics, particularly by big margin on CLIP_cond, suggesting that our model is able to generate videos that tightly follow the guidance of the condition frame. For the user study, we randomly select 20 videos and ask 16 participants for their preference over Gen [13]. Results shown in table 2. Our method is strongly preferred on these key aspects.
4.4 Ablation Study
We study different choices of conditioning by experimenting with three variants. VAE Concat uses the same VAE encoder to extract features for the condition frame c, and concatenates with the input F; along the feature dimension. VAE Cross-Atm leverages the VAE features E(c) through cross-attention instead of concatenation. No Mask does not concatenate the masks M with the foreground segmentation and noise Z, for the input.
The results on the held-out set of HiC+ are shown in table 3. Providing the condition in the CLIP feature space through cross-attention provides global semantic guidance to the entire diffusion process, while VAE features pose stricter spatial constraints, especially through concatenation, hence making it harder to produce coherent videos that conform to the condition. Providing the mask as

---

Variant
CLIP cond f CLIP temp
FVD
VAE Concat
0.743
0.939
335.2
VAE Cross-Attn
0.823
0.942
326.6
No Mask
0.850
0.943
321.7
Full Model (Ours)
0.857
0.944
315.2
Table 3: Ablation study on key design choices.
additional input further indicates the region that the model should generate thus also improves the overall generation performance.
5
Conclusion
We present ActAnywhere, video diffusion-based model that generates videos with coherent and vivid foreground-backen background interactions, given an input foreground segmentation sequence and a condition frame describing the background. Our model synthesizes highly realistic details such as moving or interacting objects and shadows. The generated videos also exhibit consistent camera scales and lighting effects. We believe our work contributes useful tool for the movie and visual effects community, as well for the general public to realize novel ideas of situating an acting subject in diverse scenes, in simple and efficient way that is not previously possible.
10

---

6
Acknowledgment
We thank the authors of [26] for compiling and processing the dataset HiC+, especially Sumith Kulal for the code and instructions on accessing the data. We also thank Jiahui (Gabriel) Huang from Adobe Research for helping set up the Adobe Firefly GenFill API. Boxiao Pan and Leonidas Guibas are supported by a grant from the Stanford Human-Centered Al Institute (HAI) and a Vannevar Bush Faculty Fellowship.
References
Virtual Production.
[2] Adobe:
Adobe: Photoshop, version http:////ww.e......////////////a
[4] Bain, M., Nagrani, A., Varol, Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV (2021)
[5] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and vidco editing. In: ECCV (2022)
[6] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: CVPR (2023)
[7] Brooks, T., Efros, A.A.: Hallucinating pose-compatible scenes. In: ECCV (2022)
[8] Ceylan, D., Huang, C.H.P., Mitra, NJJ.: Pix2video: Video editing using image diffusion. In: ICCV (2023)
[9] Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q. Wang, X., et al.: Videocrafter1: Open diffusion models for high-quality vidco generation. arXiv preprint arXiv:2310. 195122 (2023)
[10] Chen, W., Wa, J., Xie, P., Wu, H., J., Xia, X., Xiao, X., Lin, L.: Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305. 13840 (2023)
[11] Clark, A., Donahue, J., Simonyan, K.; Adversarial video generation on complex datascts. arXiv preprint arXiv:1907.06577 (2019)
[12] Diba, A., Fayyaz, M., Sharma, V., Paluri, M., Gall, J., Stiefelhagen, R., Van Gool, L.: Large scale holistic video understanding. In: ECCV (2020)
[13] Esser, P.. Chiu, J.. Atighehchian, P., Granskog. J., Germanidis, A.: Structure and content-guided video synthesis with diffiusion models. ICCV (2023)
[14] Fan, F., Guo, C., Gong, L.., Wang. B., Ge, T., Jiang, Y., Luo, C., Zhan, J: Hierarchical masked 3d diffusion model for video outpainting. In: ACM MM (2023)
[15] Ge. S., Hayes, T., Yang, H., Yia, X., Pang. G., Jacobs, D., Huang. J.B., Parikh, D.: Long video generation with time-agnosti vggan and time-sensitive transformer. In: ECCV (2022)
[16] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: Tokenflow: Consistent diffusion features for consistent video editing, arXiv preprint arXiv:2307. 10373 (2023)
[17] Guo, Y., Yang. C., Rao, A., Wang. Y.. Qiao, Y., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.00 .4425 (2023)
[18] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)
[19] He, Y.. Yang. T., Zhang, Y.. Shan, Y. Chen, Q. Latent video diffusion models for figh-fidelity long video generation. arXiv preprint 1xXiv:221..1322 (2023)
[20] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Flect, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint axXiv22200022300 (2022)
[21] Ho. J., Jain. A., Abbeel, P:: Denoising diffusion probabilistic models. In: NeurIPS (2020)

---

[22] Hu, Z., Xu, D.: Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv prepriat arXiv:2307.14073 (2023)
[23] Kasten, Y. Ofri, D., Wang. O., Dekel, T.: Layered neural atlases for consistent video editing, ACM TOG 40(6) (2021)
[24] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang. Z., Navasardyan, S., Shi,
H.: Text2vidco-zero: Text-to-inage diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. 13439 (2023)
[25] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arxiv preprint arXiv: (2013)
[26]
Kulal, S., Brooks, T., Aiken, A., Wu, J. Yang, J. Lu, J., Efros, A.A., Singh, K.K.: Putting people in their place: Affordance-aware human insertion into scenes. In: CVPR (2023)
[27]
Loshchilov, L, Hutter, F: Decoupled weight decay regularization. arXiv preprint arXiv: (2017)
[28] Monfort, M., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S.A., Yan, T., Brown, I.,, Fan, Q., Gutfreund, D., Vondrick, C., ct al.: Moments in time dataset: one million vidcos for event understanding. IEEE TPAMI 42(2) (2019)
[29] OpenAl: ChatGPT (2023), https://chatt www.oipenc...omm
[30] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint 1eXiv:1704.0066 (2017)
[31] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
[32] Ramesh, A., Dhariwal, Nichol, A., Chu, C., Chen, M.: Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 (2022)
[33] Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V:: Towards robust monocular depth estimation: Mixing datasets for zcro-shot cross-dataset transfer. IEEE TPAMI 44(3) (2020)
[34] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
[35] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
[36] Saharia. C., Chan, W., Chang, H., Lee, C., Ho, J.. Salimans, T., Flect, D... Norouzi, M.: Palette: Image-to- image diffusion models. In: ACM SIGGRAPH (2022)
[37] Sigurdsson, G.A., Varol, G., Wang. X., Farhadi, A., Laptev, L., Gupta, A.: Hollywood in homes: Crowd- sourcing data collection for activity understanding. In: ECCV (2016)
[38] Tulyakov, S., Liu, M.Y., Yang, X., Kautz, J.: Mocogan: Decomposing motion and content for vidco generation. In: CVPR (2018)
[39] Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Fvd: A new metric for vidco generation (2019)
[40] Villegas, R., Babacizadeh, M.. Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., Erhan, D.: Phenaki: Variable length video generation from open domain textual description. arXiv preprint a:Xi::2200002399 (2022)
[41] Vondrick, C., Pirsiavash, H., Torralba, A.: Generating videos with scene dynamics. In: NeurIPS (2016)
[42] Wu, J.Z.., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.7.: Tune-a-viden: One-shot tuning of image diffusion models for text-to-vided generation. In: ICCV (2023)
[43] Wu, J., Li, X., Si, C., Zhou, S., Yang, J., Zhang, J., Li, L., Chen, K., Tong, Y., Liu, Z., Loy, C.C.: Towards language-drive driven video inpainting via multimodal large language models. arXiv pre-print (2023)
[44] Xie. S., Zhang, Z., Lin. Z., Hinz, T., Zhang, K.: Smartbrush: Text and shape guided object inpainting with diffusion model. In: CVPR (2023)
[45] Yang, B., Gu, S., Zhang. B., Zhang, T., Chen, X., Sun. X., Chen, D., Wen, F: Paint by example: Exemplar based image editing with diffusion models. In: CVPR (2023)
12

---

[46] Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A.G., Yang, M.H., Hao, Y.. Essa, I., et al:: Magvit: Masked generative video transformer. In: CVPR (2023)
[47] Yu, S., Sohn, K., Kim, S., Shin, J.: Video probabilistic diffusion models in projected latent space. In: CVPR (2023)
[48] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: CVPR (2023)
[49] Zhou, S., Li, C., Chan, K.C., Loy, C.C.: Propainter: Improving propagation and transformer for video inpainting. In: ICCV (2023)
13

---

Figure 5: Zero-shot video inpainting. We show two cases from DAVIS, each with four sampled frames. The yellow regions denote the masked areas to inpainted.
Condition
Inpu
Mask
Output
Condition
Imput
Mask
Output
Figure Zero-shot video outpainting. Both examples are from DAVIS.
Appendix
In this appendix, we first provide additional results and analysis in section 7.1, including inpainting / outpainting capability emerging from our model, generalization to domains different from the training data, robustness to inaccurate foreground segmentation, and inference runtime. We then describe essential processing steps training and evaluation data section 7.2. Next, we show failure cases and discuss limitations of our model in section 7.3. Lastly, we conclude by discussing the ethical impact this work section 7.4.
7.1 Additional Results and Analysis
General video inpainting / outpainting. Interestingly, once trained, certain general video inpainting outpainting capability emerges from our model. We perform preliminary experiments by manually creating mask sequence. and pass those with the foreground sequence the input Two cases are shown in fig. 5, where our model is able to inpaint the missing regions, despite not explicitly trained so. Similarly, our model can also be applied to general video outpainting, whose results are shown in fig. 6. Specifically, we resize the original sequence of frames by 0.75, and pad them with gray boundaries. Associated masks also created to indicate the missing regions. We then randomly select one frame and use Adobe Firefly [2] to outpaint it, with which as condition we outpaint the entire sequence.
model.
These results may suggest that our model learns to approximate the underlying data distribution to certain degree, possibly benefiting from the random condition dropping during training (section 3.4).
14

---

Gaming
Multi-suubeect Animation
Figure 7: Videos from various domains. Gaming video from GTA.
Input
Condition
Figure 8: Our model is robust to inaccurate masks. We show one video sequence from HiC+ with two different condition frames, followed by six generated frames for each.
Videos from various domains. Our model generalizes to various domains such as gaming and animation clips, well as videos with multiple moving subjects. In fig. 7, we show the condition frame with its segmentation, along with three generated frames for one example in each domain.
Robust to inaccurate masks. As stated in section 3.4, masks created or extracted in practice are often imperfect, being either incomplete or excessive. Here we show that our model trained in our designed procedure is robust to imperfect masks. In fig. 8 we showcase example of this. Despite a large region of the guitar missing, our model is able to hallucinate them a reasonable way by taking the global context into account.
Runtime, Generating one video on an NVIDIA A100 GPU takes about 8.5 seconds, enabling much faster idea iteration compared to traditional workflows.
7.2 Data Processing
Training. In Sec. 3.4 of the main manuseript, described our data processing and augmentation strategies for training data. Specifically, to deal with incomplete segmentation, we apply random rectangular cut-outs to the segmentation and masks. We show two examples in fig. 9.
Evaluation. mentioned in Sec. of the main manuscript, at test time, the composited foreground- background frames used as condition can be created with various methods, such photo editing
15

---

Figure 9: Data augmentation. We apply random cut-outs to the person segmentation during training.
Here show two examples of cut-outs with their corresponding original frames.
Input
Input
Condition
Condition
Figure 10: Failure cases. Foreground sequences from DAVIS.
tools (e.g. Adobe Photoshop [3]) automated image outpainting methods (e.g. Dall-E [32]). In our experiments, we adopt ChatGPT 4 [29] and Adobe Firefly [2] to automatically synthesize these test time. Specifically, we first sample the O-th, 8-th, and 15-th frames
composited frames for from the input video, and ask ChatGPT the following question "Based on these three images, can you give description as prompt, less than 10 words, about contextual scenario can put this human in?". We then use Firefly to synthesize the outpainted frames, given the foreground segmentation and the text prompt. We the vistt-vision-poee piriiowwprvvew"" version of ChatGPT 4', and the official Firefly GenFill .*
7.3 Limitations
We show two failure cases of our method in fig. 10. In the first example, the grass-like texture on the dress is excluded from the segmentation, hence the model mistakenly perceives it to be an independent object growing outside the dress. While in the second example, the Firefly-inpainted frame has the broomstick facing towards the wrong direction. Although the model tries to generate something reasonable, it fails to correct this mistake to produce coherent video. Despite certain fault tolerance from our model, providing proper input segmentation and condition signal helps ensure high-quality generation results.
7.4
Data Ethics
The HiC+ dataset [26] includes public datasets HVU [12], Charades [37], Moments [28], and WebVid10M [4]. Readers are encouraged to refer to Section A. in the supplementary material of [7] for the license of these datasets. Moreover, same as [26], more internal videos are incorporated during training and evaluation. We have conducted the necessary legal review process, and can provide more
details of this data upon request.
We present method that can synthesize highly realistic videos that place human and non-human subjects into diverse background scenes. While our work has implications and values for a wide range of fields, we note that our model can be misused to generate malicious content. Similar to Firefly GenFill [2] and Genl [13], the generated content can be watermarked to prevent misuse. Moreover, model inherits the demographic biases presented in the training data. We make our best effort to demonstrate impartiality in all the results shown.
http:://llafoomoopeaii.coom wtwppp/wii...ooooo///eeeeeeeeeee
"https//ffiretly adobe www.///ooooooooogeentnniniannnnn
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of the paper have been truthfully described throughout the abstract and the introduction, and concisely summarized at the end of the introduction. Each of the listed contributions is backed up by experimental results or literature review throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We included a separate section 7.3 to discuss the limitations of the work, including discussions on potential data bias in section 7.4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results, while essential theoretical background with formulas is provided in section 3.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included detailed descriptions of our model architecture in section 3.3, references to publicly available pre-trained weights, data processing procedures, and other training details in section 3.4 and further in section 7.2. We will also opensource our code and model checkpoints upon paper acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: As stated in the response above, we provided detailed instructions on how to replicate our experiment results in the paper. We will release our code and models upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips.cco hcc/publi//guuii http//uuddss////ee/e/eee //t////oodeessseseeeeeee ssiooiooicc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provided detailed instructions on replicating the training and evaluation procedures in section 3.4, section 4, and section 7.2. We did not perform delicate tuning for the hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
19

---

We did not compute error bars due to the lack of adequate computational resources. However, tried our best to provide faithful and reliable quantitative evaluation results, including user study.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We described our compute setup well the training time and inference runtime in section 3.4 and section 7.1, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htappp:////uipppp/ /tt///euurip..// oc hctppullcc/thhcccciieeeeeeeee
Answer: [Yes]
We included separate section 7.4 discuss the ethics-related concerns, including licensing and legal information of the dataset used, and potential societal impacts of our work with possible measures to combat them.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
20

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We included separate section 7.4 that discussed potential societal impacts of our work with possible measures to combat them.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
We mentioned in section 7.4 that the generated content from our model can be watermarked prevent potential misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
21

---

Answer: [Yes]
Justification: As noted in section 7.4, the licenses for the datasets used in this work can be found in the supplement of [7]. All the pre-trained models and tools used are specified in the paper in section 3.4 and section 7.2 with URLs.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi twcoee com condsataeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "s5CEIlVbL1", "reviewer": "Reviewer_Aquh", "summary": "This paper study to automatically generate video background that tailors to foreground subject motion. It proposes ActAnywhere, a video diffusion model that takes as input a sequence of foreground subject segmentation and an image of a novel background and generates a video of the subject interacting in this background. Both quantitative and qualitative comparisons demonstrate that the model significantly outperforms existing methods, which fail to accomplish the studied task.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The task is interesting and the result seems to be very competitive.\n\n2. The paper is written clear.", "weaknesses": "1. Only encoder condition frame by a clip encoder cannot contains details information, especially for high-resolution input images. Please refer AnimtaeAnyone for human image animation. And please compare such a referencenet attention way.\n\n2. Can you provide some video demos. I suppose there may exists blur boundary results.\n\n3. Since the task seems to be very related to human animation task, could you please show some results on TikToK dataset? Please see AnimateAnyone paper for comparision details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback from the reviewer and thank them for acknowledging that our introduced task is interesting and that our proposed model achieved competitive results. We next address the reviewer's individual questions and comments.\n\n> CLIP encoder alone cannot contain detailed information… Compare to the approach of AnimateAnyone using ReferenceNet.\n\nWe thank the reviewer for the suggestion. We would like to note that detail preservation of the condition is still an open question. And because our framework is fairly general and agnostic to the image encoder, we can swap the CLIP encoder with any encoder with ease, e.g., DINO* if object-centric features are more desirable. ReferenceNet is an interesting alternative, but AnimateAnyone is not open sourced, which makes it hard to directly compare. Nonetheless, we are happy to compare if the reviewer insists.\n\n*Emerging Properties in Self-Supervised Vision Transformers. Caron et al. ICCV 2021.\n\n*DINOv2: Learning Robust Visual Features without Supervision. Oquab et al. TMLR, 2024.\n\n> Provide video demos. Blurry boundaries may exist in the videos.\n\nWe already included extensive video results in the easily accessible supplementary webpage, as mentioned in L206-L208 of the main manuscript. We did not observe any particular cases with blurry boundaries.\n\n> Task relevant to human animation. Show results on the TikTok dataset.\n\nWe note a key difference from the human animation task that the goal is in fact the opposite - we are given the foreground motion and try to generate the moving background. The TikTok dataset has a different focus to generate the foreground motion instead. Hence, the background in the dataset is often very simple and does not move much, and very little, if any, interaction happens between the foreground and the background. Thus the dataset is not a good fit for our task. And since AnimateAnyone is not open sourced, it is hard to compare experimentally. Nonetheless, as requested by the reviewer, we tested our model on two samples from the TikTok dataset and included the results in Fig. 1 of the attached PDF. Our model demonstrates good generalization over the tested data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "stGP6ja5ws", "reviewer": "Reviewer_Pq5G", "summary": "This paper studies a new topic: automatic background generation of moving foreground subject. Different from video inpainting/outpainting and other video editing methods, the method in this paper can maintain the consistency of foreground moving subject, and maintain reasonable and realistic interactions, camera motion, lighting and shadows.\n\nIt can also generalizes to diverse scenarios including non-human objects, gaming and animations, as well as videos with multiple moving subjects. Input a foreground segmentation image sequence and a foreground mask sequence, and under the guidance of a background reference image, generate a motion video of the interaction between the foreground and the background. Specifically, the spatial layer parameters of Denoising 3D Unet are loaded from a pre-trained SD-inpainting model, and the temporal layer(motion module) is loaded from the pre trained model of AnimateDiff V2. The input is a feature map of nine channels, which concatenates the nosie latent, the vae encoder feature of the foreground segmentation map, and the downsampled foreground mask in the channel dimension.\n\nThe CLIP feature of the background reference map is injected into the Unet through cross attention.Trained on the HiC+ dataset (2.4M videos of human-scene interactions), Given the error of foreground segmentation, random rectangular cut-outs augmentation is applied to the foreground segmentation sequence and foreground mask sequence", "questions": "Please refere weakness", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "A new task was studied: automatic background generation of moving foreground, and reasonable and realistic foreground/background interaction, camera motion, lighting and shadows were required.\n\nAlthough it is a new task, the results of similar methods were compared under fair conditions as much as possible, which indeed proved the superiority of the method in this paper.", "weaknesses": "The training resolution is low (256x256), and it is unclear whether the method in this paper still has advantages and reliability under high-score conditions?\nAblation experiment: The method of using background reference image for guidance does not seem to be fully considered, such as the method similar to IPAdapter?\n\nThe training details are not clear: is it fine-tuning the spatial layer and the motion module at the same time? Or only fine-tuning the motion module? Or is cotraining still required?\n\nThis article spends a certain amount of space to explain the difference in results between the method proposed in this paper and the SD-inpainting method, but in terms of the algorithm pipeline, except for using the CLIP features of the background reference image for guidance, the rest is almost exactly the same as the SD-inpainting model, and even the spatial layer of Unet directly loads the pre-trained model of SD-inpainting.\n\nThe article does not seem to explain the difference in principle between the method proposed in this paper and the SD-inpainting method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and for acknowledging that our introduced task is novel, and that the experimental comparisons are under a fair setup. We address the reviewer’s particular questions and comments below.\n\n> Training resolution is low… The method of using reference image does not seem fully considered, e.g. the method similar to IP-Adapter?\n\nThere is no fundamental limitation for our method to work under higher-resolution settings, granted a base model under a higher resolution and more compute resources. \n\nCompared to the method of condition encoding in IP-Adapter, we only support image conditioning and thus do not need to map image and text to a joint space, hence we do not need the Decoupled Cross-Attention module as in their framework. Similar to the image encoding method in IP-Adapter though, we also have a linear layer to project the CLIP image features to a lower dimensional feature space, as noted in L179-L182 of the main manuscript.\n\n> Are we finetuning only the spatial layers, only the motion module, or both?\n\nThank you for the question. We finetune both the spatial layers and the motion module layers at the same time. We will clarify this in the final version of the paper.\n\n> Technical framework is very similar to SD-inpainting\n\nOur key contributions include 1) the introduction of the novel problem of automated subject-aware video background generation with a background image as condition, and 2) the proposal of a specific video diffusion-based pipeline that is carefully designed and tailored to solve this problem. \nSD-inpainting works only under the image setting, and it is unclear how to extend it to videos. Our proposed framework, along with the designed self-supervised training pipeline and data augmentation strategies, contribute to our solution altogether.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LCYWLPCI5k", "reviewer": "Reviewer_gAcX", "summary": "This paper introduces ActAnywhere, a video diffusion model designed to generate video backgrounds that adapt to the foreground subject's motion. By utilizing a sequence of foreground subject segmentation and a background image, the model produces realistic videos with coherent foreground-background interactions. Experiments on a large-scale dataset demonstrate the model's effectiveness, outperforming existing methods in generating realistic and dynamic backgrounds.", "questions": "Q1: I wonder about the impact of the quality of the foreground segmentation. It seems that the model relies heavily on the quality of the foreground segmentation.\n\nQ2:  How scalable is your model for generating longer video sequences? Have you tested its performance in generating videos of varying lengths, and what are the results?", "limitations": "I think there is no potential negative societal impact and the authors have addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1: The paper introduces a novel problem of automated subject-aware video background generation.\n\nS2: The methodology shows improvements in generating coherent videos with realistic subject-background interactions.\n\nS4: The contributions are significant, particularly for applications in the movie industry and visual effects.\n\nS4: The paper is comprehensive and well-written.", "weaknesses": "W1: The paper lacks sufficient comparison with a broader range of existing methods, particularly those leveraging recent advancements in video generation and editing (though they are not for background generation, they can do).\n\nW2: It seems that the model relies heavily on the quality of the foreground video segmentation masks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback from the reviewer and thank them for acknowledging that our introduced problem is novel, the proposed method is effective and makes a significant contribution to the movie and VFX industries, and that our paper is well-written. We address the reviewer's individual questions and comments below.\n\n> Lacks sufficient comparison with recent works on video generation and editing\n\nTo the best of our knowledge, the work with the closest setting to ours is AVID*, which can perform a task termed “Environment Swap” by editing the background region with a text condition. However, AVID does not take image conditioning, and hence cannot make the generated video follow the strict guidance specified by an image as we do. Moreover, from the results the authors showed on the website, they do not manifest realistic foreground-background interactions as ours (e.g. the sand does not deform according to the woman / tiger’s movement), and fail to generate correct shadows (the shadows are simply carried over from the input). We also note that this work was published recently in CVPR which was after the NeurIPS submission deadline, and is not open sourced yet which makes it hard to compare with.\n\nApart from this work, we believe others study different settings from ours and are generally non-trivial to extend to work under our setting. We are happy to compare any specific methods that the reviewer may suggest. \n \n*AVID: Any-Length Video Inpainting with Diffusion Model. Zhang et al. CVPR 2024.\n\n\n> It seems that the model relies heavily on the quality of the foreground segmentation\n\nAs referenced in L204 of the main manuscript and discussed in Appendix 6.1, our model is in fact quite robust to inaccurate masks (Fig. 8), thanks to our designed data augmentation and processing procedures (i.e. random cut-outs and image erosion to the segmentation and mask) as noted in Sec. 3.4 of the main manuscript and Appendix 6.2.\n\n> How scalable is the model for generating longer / variable-length video sequences? \n\nDue to the modular nature of our framework, we can easily scale up to generating longer videos if we swap to a different base model. The choice of generating 16-frame videos aligns with many previous works (e.g. [9, 17] for 16-frame generations, and [24] for 8-frame generations) and is primarily limited by the compute resource we had. Generating longer videos is out of the scope of this work, and we will explore this in future work with the latest DiT backbone.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zas0vGUOxX", "reviewer": "Reviewer_ghM9", "summary": "The task addressed by this paper is, given the the appearance and segmentation mask of a foreground subject in a video, for example a human running, to synthesize video backgrounds that are plausible and realistic, both in content and motion. For example, for a person running, if the ground is wet there should be splashes corresponding to footfalls, and the background scene should move along with the runner's movement.\n\nThe authors address this problem as a generative video outpainting task. They model it using a latent video diffusion model. The foreground subject appearances (encoded) and masks are concatenated with the latent feature vector, while the conditioning image (what the background should look like) is encoded via CLIP and is provided via cross-attention to the U-net.\n\nA variety of qualitative results are included, demonstrating the ability to out-paint plausible scenes, including background motion, effects (splashing water) and even objects that the subject interacts with. Quantitative evaluations include ablations validating the main components of the model, as well as human rater studies comparing the quality of the output to competing methods.", "questions": "Have you tried unconditional generation, i.e. not providing any signal as to what the desired background should be? This could be quite interesting, since it would challenge the model to understand exactly what the foreground subject is doing and to invent plausible backgrounds. For example, this child seems to be interacting with an object, what could that object be? This woman is riding something… what fits?\n\n\nWould you like to discuss the appearance of the Shutterstock watermarks in the generated backgrounds, particularly in figure 3? This could be an interesting example of the model using foreground context informing the background generation, as per my previous question.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality & Significance.\nThe paper addresses a minor new variant on the outpainting/inpainting problem, which introduces some new challenges such as picking up on foreground contextual cues and using that to inform the background.\n\nQuality & Clarity:\nThe paper is clearly written, seems correct, and demonstrates visually appealing results with a variety of different foreground subjects (person, car, duck, etc.). The supplemental videos help illustrate the model's capabilities well. The empirical results also support the quality of the generative model.", "weaknesses": "The main weakness of this paper is very limited novelty, which compromises the claimed contributions of the paper.\n\nVideo inpainting/outpainting is a fairly well-known challenge. The main novelty here is that the masks are simply inverted: instead of deleting an unwanted person and generating the missing background pixels, here we retain only the desired foreground person and generate the missing pixels. I am not convinced that this difference is substantively novel.\n\nSimilarly, the modeling approach bears considerable similarity to [13] \"Structure and Content-Guided Video Synthesis with Diffusion Models\". In particular, compare Fig.2 from the two papers. The primary difference between the two is that this work conditions on foreground appearances and masks, while [13] conditions on estimated depth images. Again, the novelty is minor and unsurprising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and for acknowledging that the introduced problem possesses originality, the high quality of the results demonstrates the effectiveness of the proposed model, and that the paper is clearly written. We next address the reviewer’s questions and comments.\n\n> Video inpainting / outpainting is a fairly well-known challenge…\n\nAs discussed in L40-L44 and L100-L112 of the submission, the key challenge of our task compared to general video inpainting / outpainting lies in **generating dynamic interactions** and **large background regions** that **follow the image condition**. This is acknowledged by reviewers Pq5G and gAcX. It is nontrivial how to extend the video inpainting / outpainting frameworks, which generally tackle a pixel harmonization problem, to solve these challenges. Also, in Appendix 6.1, we showed that our model, once trained, exhibits general video inpainting / outpainting capabilities, while general inpainting / outpainting methods are not able to solve our proposed task.\n\n> The modeling approach is very similar to Gen1 [13]\n\nOur key contributions include 1) introducing the novel problem of automated subject-aware video background generation with a background image as condition, and 2) proposing a specific video diffusion-based pipeline to solve this problem. Prior works such as Gen1 [13] are not able to solve the introduced problem, as demonstrated qualitatively and quantitatively in Sections 4.2 and 4.3 of the main manuscript. We found empirically that our proposed framework can effectively tackle the problem.\n\n> Unconditional generation with the model\n\nThank you for the question. We have performed the requested experiment and included the results in Fig. 2 of the attached PDF. Specifically, we set the condition to zeros, same as the random condition dropping described in L192-L194 of the submission. We sampled two subjects from the HiC+ dataset, and for each we ran our model with three different seeds. From each of the three generated videos, we selected one frame to show its input segmentation along with the generation. We observed that the model can generate reasonable random backgrounds that fit the foreground. \n\n> Watermark in the generated backgrounds\n\nWebvid* videos are a major source of the HiC+ dataset, which all contain the “shutterstock” watermark. Training on them provides a dataset bias such that when conditioned on segmentations or a condition frame with the watermark, the generated results will also contain and complete (if the watermark appears partially in the input) the watermark. As the reviewer pointed out, this is an example of the model using foreground context to inform the background generation.\n\nDespite that this issue is orthogonal to the focus of this work, we are aware of methods that can alleviate watermarks appearance in the generation, e.g. the “Domain Adapter” in AnimateDiff [17]. We leave the integration of such methods in our framework to future work.\n\n*Frozen in Time: A Joint Video and Image Encoder for End to End Paper. Bain et al. ICCV 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to sincerely thank the AC and the reviewers for their hard work and time in reviewing our submission. We appreciate the positive feedback and recognition of the novelty and significance of the introduced problem, the high quality of the results and the effectiveness of the proposed method, as well as the clear paper writing. We also appreciate the insightful suggestions for further improving our work.

We also want to emphasize that the key technical contributions of this paper are 1) the introduction of the novel problem of automated subject-aware video background generation, and 2) the proposal of a video diffusion-based pipeline that is carefully designed and tailored to solve this problem. And as we have shown in Sec. 4 of the main manuscript, no baseline was able to effectively solve the introduced problem (and with reasonable adaptations), and our proposed method achieved significantly better performance than the baselines both qualitatively and quantitatively. This demonstrates the challenge of the introduced problem and the effectiveness of the proposed solution.

In the comments below, we have addressed all specific issues / concerns / questions raised by the four reviewers by replying to each individually. We also attached a PDF which contains the additional results requested by Reviewers **Aquh** and **ghM9**.

Thank you again for your time and efforts throughout the review process -- they are greatly appreciated!

Meta-review: The significance of this paper is unanimously recognized by all the reviewers, including the novelty of the proposed task, its impact in other applications (e.g., movie industry), thorough experimental evaluation and comparisons with other methods, and clear writing of the paper. The AC thus recommends to accept this paper as a Poster.
Decision: Accept (poster)