MagicVFX: Visual Effects Synthesis in Just Minutes

Authors: jiaqi  guo
junchen  zhu
jiaxin  zhang
siyang  li
jingkuan  song
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:04:08

Abstract: Visual effects synthesis is crucial in the film and television industry, which aims at enhancing raw footage with virtual elements for greater expressiveness. 
 As the demand for detailed and realistic effects escalates in modern production, professionals are compelled to allocate substantial time and resources to this endeavor. 
 Thus, there is an urgent need to explore more convenient and less resource-intensive methods, such as incorporating the burgeoning Artificial Intelligence Generated Content (AIGC) technology. However, research into this potential integration has yet to be conducted. 
 As the first work to establish a connection between visual effects synthesis and AIGC technology, we start by carefully setting up two paradigms according to the need for pre-produced effects or not: synthesis with reference effects and synthesis without reference effects. 
 Following this, we compile a dataset by processing a collection of effects videos and scene videos, which contains a wide variety of effect categories and scenarios, adequately covering the common effects seen in films and television industry.
 Furthermore, we explore the capabilities of a pre-trained text-to-video model to synthesize visual effects within these two paradigms. 
 The experimental results demonstrate that the pipeline we established can effectively produce impressive visual effects synthesis outcomes, thereby evidencing the significant potential of existing AIGC technology for application in visual effects synthesis tasks. Our dataset can be found in https://github.com/ruffiann/MagicVFX.

Main paper: 
 introduction: 
 Visual effects are indispensable components of films, television, and even social media videos. Detailed and realistic visual effects often provide audiences with the ultimate visual pleasure. For example, in disaster films, the tsunamis or fires processed with visual effects will be more immersive for the audience, enabling audiences to feel as though they are part of the scene.As the film and television industry advances, audience expectations for visual effects have increased dramatically, covering both realistic phenomena that are challenging to capture (such as explosions and car accidents) and fantastical effects that do not exist in reality (such as magical shields and teleportation circles). Meeting these expectations requires significant investment in highperformance hardware and professional talent, making the process costly and complex. Therefore, there is an immediate need to improve current approaches and workflows of visual effects synthesis. In light of this, we propose leveraging the rapidly evolving Artificial Intelligence Generated Content (AIGC) technology for visual effects synthesis, aiming to improve the efficiency in the current film and television production industry. To the best of our knowledge, we are the first to attempting to bridge these two areas, which means we will be exploring the feasibility of this idea from scratch.First, we relate AIGC to visual effects synthesis and propose the task definition. In film and television production, creating visual effects often involves capturing a base video and then adding designated effects to specific locations or objects within it. These effects generally fall into two categories: those that do not require customization and can be described using natural language, such as flames and lightning, and those that require customization, such as the magical shields seen in Doctor Strange. The latter are typically pre-produced by professionals as samples before being integrated into the scene. Consequently, we empirically divide AIGC-based visual effects synthesis into two paradigms: synthesis with reference effects (SRE) and synthesis without reference effects (SNRE), as shown in Figure 1 . Both paradigms utilize a base video as inputs, with SRE additionally requires a customized visual effects video.Secondly, we compile a dataset for evaluation purposes. Our data acquisition approach is shown in Figure 3 . This dataset comprises two types of videos: pre-produced effects videos and base videos. The effects videos are collected from online sources, with each effect being manually annotated with a brief description. For the base videos, we select clips appropriate for effects augmentation from both online sources and public video dataset [24] , each accompanied by detailed textual description crafted manually. Utilizing these two distinct video types as a foundation, we further processed them to create samples tailored for both SRE and SNRE paradigms. Our dataset encompasses a wide range of visual effects commonly encountered in the film and television industry, enabling a comprehensive assessment of the capabilities of synthesis methods.In the third phase, utilizing a pre-trained text-to-video generator, VideoCrafter2 [6] , we design a simple yet effective pipeline in Figure 4 capable of achieving visual effects synthesis under both paradigms. Specifically, users are required to designate an area for effects on a base video using a mask. Should a reference effect be available, it is pasted to the marked area, creating an initial video; if not, the original base video remains unchanged. Next, the initial video undergoes the addition of low-level noise, leading to a process of denoising to generate the final output. Throughout this procedure, we enhance the effects synthesis by using descriptions without effects as negative prompts and employing user masks to replace cross-attention maps. The experimental results validate the effectiveness of our pipeline in accomplishing visual effects synthesis, also highlighting the research potential of AIGC-based visual effects synthesis tasks.In summary, our contributions are as follows:• To the best of our knowledge, we are the first to introduce AIGC techniques to visual effect synthesis which only takes few minutes to generate complex and high-quality special effects in videos.  
related work: 
 In the realm of visual effects, traditional methodologies have predominantly relied on software-based approaches [27] and conventional techniques [22] . These methods only optimize the software at a certain step and do not improve the overall workflow. In contrast, our work pioneers the application of AIGC technology to the synthesis of visual effects, revolutionizing the workflow and making it possible to synthesize visual effects in just minutes. Video Composition. In the realm of video composition, recent advancements have primarily concentrated on achieving temporal consistency [7] and seamless blending [12, 16, 21, 28] . But video composition concentrates on real entities such as humans and animals, neglecting the compositing of virtual objects such as visual effects.Text-driven Video Editing. Text-driven video editing [4, 8, 11, 17-19, 25, 26, 31, 32] involves modifying the video conditional on the text, which is similar to our proposed paradigm of synthesis without reference effects. However, as illustrated in Figure 2 , the focus of these two tasks is different. The goal of visual effects synthesis is to enhance the original input with additional visual elements, whereas video editing tasks [10, 17, 20, 32] focus on modifying certain aspects of the visual content in the original input.Text-to-video Models. Advancements in diffusion models have led to breakthroughs in text-to-video (T2V) generation [5, 33, 35] and multi-modal models [1] [2] [3] 9] . Several startups, like Pika Labs1 , Moonvalley2 , and Genmo3 , have released impressive text-to-video generation services. However, these rely on proprietary datasets and models, hindering broader research and development. Among open-source models, some [5, [13] [14] [15] 29] often yield videos of lower resolution. While I2VGen-XL [34] can generate high-resolution outputs, it falls short in text-to-video tasks, which are essential to our study. Therefore, we have selected VideoCrafter2 [6] for our work, which is the SOTA open-source text-to-video generator.3 Problem formulation and datasets 
problem formulation: 
 In this study, we explore integrating AIGC technology with synthesizing visual effects, aiming to augment the production productivity. Following an in-depth investigation of the industrial workflows and techniques utilized in creating visual effects within the cinematic sphere, we propose categorizing AIGC-based visual effects synthesis into two distinct paradigms: synthesis with reference videos (SRE) and synthesis without reference videos (SNRE). These two paradigms validly cover the spectrum of synthesis scenarios encountered in practical applications.Synthesis with Reference Effects (SRE). In the film industry, fantasy and science fiction genres command a significant market share, featuring an abundance of scenes rich in imaginative special effects. Such movies often delve into concepts like interstellar travel, time travel, and magical battles. These scenarios necessitate a wide array of custom special effects, meticulously pre-designed by visual effects designers and seamlessly integrated during post-production to bring these visionary concepts to life on screen. We describe this type of need for pre-produced effects as synthesis with reference effects (SRE). SRE process can be formulated as:EQUATIONwhere 𝑉 𝑏𝑎𝑠𝑒 is the base video, 𝑉 𝑟𝑒 𝑓 is the reference effect video, and 𝑀 𝑢𝑠𝑒𝑟 represents the designated location for effect integration. The output 𝑉 𝑜 is the composite video with the reference effect seamlessly and harmoniously integrated.Beyond the imaginative and bespoke effects, there exists a spectrum of common effects primarily crafted to simulate scenarios challenging to capture in the real world. These include weather phenomena, natural disasters, and smoke, as well as effects related to water and fire dynamics. Such effects do not require pre-production design but can be directly adapted to fit within scenes. We refer to the synthesis of these types of effects as synthesis without reference effects (SNRE). SNRE can be represented by:EQUATIONwhere 𝑇 should be a textual description that is sufficiently descriptive of the effect, or a description of the expected output. 𝑉 𝑜 , 𝑉 𝑏𝑎𝑠𝑒 and 𝑀 𝑢𝑠𝑒𝑟 have the same meaning as in SRE.After formally defining these two paradigms, we can use the AIGC model to fit and solve the problem, thus meeting the need for efficiency and automation in the visual effects field and speeding up the production process.Since AIGC-based visual effects synthesis is a novel work, there is no publicly available dataset that evaluates how well the method performs on this task. Hence, we collect and construct a dataset named VFX-307 to fill this gap. Our data acquisition method is shown in Figure 3 . Data Collection. VFX-307 contains two types of data, base videos and effect videos. We gather an assortment of base videos from two sources: open-source video datasets (e.g. DAVIS-2017 [24] ) and online sources. We meticulously curate scenes suitable for Original Caption: Earth from space. There is a rocket is floating in the air.Effect Caption: Earth from space. There is a rocket is floating in the air. There is constant lightning surrounding the rocket.Original Caption: An overhead video of a city with tall buildings in the early morning hours.Effect Caption: An overhead video of a city with tall buildings. Suddenly there is an explosion in the center of the city, with flames shooting up into the sky.Original Caption: There are white clouds in the sky and a girl is contacting Kung Fu alone.Effect Caption: There are white clouds in the sky and a girl is contacting Kung Fu alone. She punched out a flame in the shape of a yellow crescent moon. adding visual effects, prioritizing those with stable camera motion, clear settings, and distinct actions. We manually trim the videos to retain only segments shorter than 300 frames and standardize the resolution to 1080𝑝 with a 16 : 9 aspect ratio. This process resulted in resulted in a collection of 117 base videos. For effect videos, we compile 190 pre-produced standalone effects videos uploaded by professionals on public networks. Despite varying resolution and duration, they all have a 16 : 9 aspect ratio. This compilation contains a wide variety of effect categories, adequately covering the common effects seen in films and the TV industry. Upon collecting the two types of videos, we manually provide each base video with a detailed textual description as the original caption and each effect video with a brief description. Samples Creation. Based on these two categories of videos, we further process them to create sample sets for SRE and SNRE. Specifically, we manually pair each base video with an effect video to form valid and logical combinations. For each combination, we merge the original caption of the base video with the brief description of the effect video to generate a new effect caption. This results in a sample in the form of a quadruplet comprising the original caption, the effect caption, the base video, and the effect video. We created 75 such samples for SRE. For SNRE, we manually add a special effect description to the original caption of each base video to formulate an effect caption. This yields a triplet consisting of the original caption, the effect caption, and the base video. Each base video is associated with 1 to 4 such triplets, culminating in a total of 175 samples. 
method: 
 In this section, we explore to implement the task of visual effects synthesis using a pre-trained text-to-video generation model. We first give an overview of our synthesis pipeline in Section 4.1 followed by detailing the implementation of utilizing this pipeline for SRE and SNRE, in Section 4.2 and 4.3, respectively.Our pipeline offers a straightforward yet potent approach to synthesizing visual effects, as shown in Figure 4 . First, users are required to define a mask 𝑀 𝑢𝑠𝑒𝑟 to specify where effects should be applied to a base video 𝑉 𝑏𝑎𝑠𝑒 . In SRE, where a reference effect 𝑉 𝑟𝑒 𝑓 is available, it is pasted onto the designated area to create an initial video 𝑉 𝑖𝑛𝑖𝑡 ; in SNRE, the original 𝑉 𝑏𝑎𝑠𝑒 is utilized directly as 𝑉 𝑖𝑛𝑖𝑡 . Following this, low-level noise is added to the 𝑉 𝑖𝑛𝑖𝑡 , which then undergoes a process of denoising to generate the final output.Throughout the SNRE procedure, we employ two key techniques to enhance effect synthesis: 1. utilizing original captions of the base videos as negative prompts to ensure the generated results do not remain unchanged. 2. utilizing user masks as cross-attention maps in specific UNet layers, thereby strengthening the connection between the effect text token and its intended location.Next, we detail the design of our pipeline for both paradigms and provide an in-depth explanation of our key techniques.In the scenario of visual effects synthesis with reference effects, our goal is to reasonably integrate a pre-produced effect video 𝑉 𝑟𝑒 𝑓 into a specific location within the base video 𝑉 𝑏𝑎𝑠𝑒 . Professional engineers often adjust parameters like color and contrast to harmonize the overall visual effect and may alter elements in the original footage to ensure the composite video's coherent. For example, when facing destructive effects like explosions, it may be necessary not only to adjust the effects themselves but also to modify elements within the original scene to depict the destruction.This complex and time-consuming manual process can be greatly simplified by employing a pre-trained video generation model with advanced semantic understanding and generalization capabilities. [10, 12, 23] showed that introducing an appropriate level of noise into the input image, and using this noisy image as the starting point for a denoising generation process, can effectively make the output more harmonious while preserving the structural information of the input.Inspired by this finding, we first paste the reference effect video 𝑉 𝑟𝑒 𝑓 , at a certain value of transparency discussed in Section 5.2, onto the designated location by 𝑀 𝑢𝑠𝑒𝑟 of the base video 𝑉 𝑏𝑎𝑠𝑒 to create an initial composite video 𝑉 𝑖𝑛𝑖𝑡 . Then, we add low-level noise to 𝑉 𝑖𝑛𝑖𝑡 to serve as a starting point for generation. During the denoising generation process, we use the textual description of the scene containing the special effect as a conditional guidance. Unlike methods like [23, 30] , which generate realistic images and videos by adding noise to sketches, our purpose in adding noise is not to provide a rough prior, but to facilitate adding effects.In the scenario of synthesis without reference effects, we rely on only textual descriptions to guide the model in generating the desired effects. First, we attempt the same method utilized for SRE, which involves directly using the base video as the initial video, introducing low-level noise to establish a generation starting point, and then guiding the generation process with text description containing the desired effects.However, lacking the information provided by reference effects, the model suffers from two problems: the inability to produce visible effects and the inability to synthesis the effects to the correct location. Therefore, we treat this method as a baseline and employ the following two techniques to improve our approach in order to solve these problems. Original Captions as Negative Prompts. In conditional diffusion model with classifier-free guidance, the model typically predicts noises 𝜖 𝑡 and 𝜖 𝑡 𝑛𝑒𝑔 at each step 𝑡 conditioned separately on positive and negative prompts. These two noises are then linearly combined to produce the final noise ε𝑡 for the current timestep 𝑡:EQUATIONwhere 𝜔 is the classifier-free guidance scale. This approach guides the model to avoid generating elements or themes mentioned in the negative prompts within the output image. Here, to emphasize visual effects in the output video, our method uses original textual descriptions of the base video that do not contain effects text directly as negative prompts. User Masks as Attention Maps. As illustrated in subfigure (b) of Figure 9 , the result of synthesizing 'flames' effect guided only by text description. The desired flames do not appear on the car as expected, but on the grass next to it. We deduce that this issue stemmed from the model's inability to adequately focus the attention of the token 'flames' on the location we expected. Thus, we force a connection between the the 'flames' and the location we expected by introducing a user mask and replacing the cross-attention map in the UNet. And as discussed in Section 5.2, we modify the value of non-zero pixels in the mask and the action position of the mask to get better results. 
experiments: 
 In this section, we first show the results of our method in Section 5.1 for both SRE and SNRE paradigms. Subsequently, we report a series of ablation experiments in Section 5.2 to demonstrate the reasons for some of the design choices in our method.Baseline. Since we are the first work to explore AIGC-based visual effects synthesis, there is no applicable methodology for usA young boy kicking a water ball on the grass.An elderly woman running in a restaurant, with flowers flying around.A young woman is seen twirling in a field with people in the background, waving out colorful bubbles.Drift car performance on a race track, surrounded by fires.A man in a tank top and shorts with boxing gloves is practicing his boxing against a sandbag. His fists are surrounded by flames.An astronaut stands next to a huge rock with brown earth in front of him. Lightning falls from the sky beside the astronaut.A long-haired woman in a white dress is walking toward the sea. In front of the rocks there is a blue, spinning, magic portal made of water.A gray kitten was lying on the couch , looking around. A circle of flames surrounded its face.Figure 5 : Qualitative results for synthesis with reference effects (SRE).A young boy kicking a water ball on the grass.An elderly woman running in a restaurant, with flowers flying around.A young woman is seen twirling in a field with people in the background, waving out colorful bubbles.Drift car performance on a race track, surrounded by fires.A man in a tank top and shorts with boxing gloves is practicing his boxing against a sandbag. His fists are surrounded by flames.An astronaut stands next to a huge rock with brown earth in front of him. Lightning falls from the sky beside the astronaut.A long-haired woman in a white dress is walking toward the sea. In front of the rocks there is a blue, spinning, magic portal made of water.A gray kitten was lying on the couch , looking around. A circle of flames surrounded its face.Figure 6 : Qualitative results for synthesis without reference effects (SNRE) of baseline and our improved method.to compare. For SRE, we conduct experiments using the pipeline described in Section 4.2 and report qualitative results below. For SNRE, we qualitatively and quantitatively compare the baseline and improved methods described in section 4.3, respectively.Metrics. In the context of synthesizing visual effects without reference, we employ two CLIP scores metrics to evaluate the performance of our method against baseline: Text: This metric assesses the alignment between the synthesized video and the effect prompt, measuring how well the video matches the described effects. Content: This score determines the similarity between the synthesized video and the base video, aiming to measure how much the output video modifies the input.Synthesis with Reference Effects (SRE). Figure 5 illustrates the qualitative results of our pipeline for SRE. The base videos represent diverse scenarios where visual effects are to be incorporated. The effect videos demonstrate the specific visual effects used as references, and the output columns display the final synthesized videos. For instance, in the first row, the boxer's fists are enhanced with realistic flames, highlighting our method's ability to adapt effects to various contexts and scales.Synthesis without Reference Effects (SNRE). Figure 6 presents a comparison between the baseline and our improved approach for SNRE. In the first sequence, the baseline method only alters the color of the soccer ball, while our method transforms it into a water ball, complete with a realistic bursting effect that closely matches the textual description. Similarly, in the fourth scene, the baseline only generates smoke around the drifting car, lacking the dynamic fire effect described in the text. Our method, however, successfully synthesizes the flames, adding intensity and excitement to the scene.The second and third rows further highlight the shortcomings of the baseline. It produces minimal floral elements in the restaurant and barely noticeable bubbles in the park, resulting in effects that are not prominent and lack the striking impact required by the scene's description. In the stark contrast, our outputs are rich with blossoms and bubbles, enveloping the subjects in a vibrant and lively manner that greatly aligns with the textual prompts.The quantitative results are presented in Table 1 , which indicate that our method effectively modifies the base video and outperforms the baseline method in generating a video that more closely matches the effect description.In addition, we conduct a subjective user study to further compare these methods. Specifically, we randomly selected 75 synthesized videos and made a questionnaire, inviting 25 users to vote in terms of text matching, video quality and temporal consistency respectively. The voting results are displayed in Table 1 . Our method has a great advantage in text matching degree and is comparable to baseline in temporal consistency. The video quality score measures visual quality without regard to text alignment. Since we use the original caption of the base video as a negative prompt to enhance the expression of visual effects, it inevitably makes the synthesized video lose some details compared to the base video, thus lowering the video quality. In contrast, the baseline scores higher because it tends to ignore visual effects and reconstruct the input.Influence of Paste Transparency. In SRE, we first paste the effects video into the specified position on the base video, and then use the video generator to re-generate the video in order to fuse the two in a more rational way. However, direct pasting would make the effects in the outputs very stiff and abrupt, as shown in subfigure (c) in Figure 7 , so we adjust the transparency 𝛼 of the effect video. That is, the pixels of the base video are linearly combined with the pixels of the effect video when pasting by the following formula:EQUATIONwhere 𝑃 denotes the pixel value at the specified paste position. The experimental results are shown in Figure 7 . Lowering the transparency 𝛼 does result in a better integration of the special effect with the base video, but too low 𝛼 value will cause the model ignore the presence of the reference effect.Influence of Using Original Captions as Negative Prompts.We employ quantitative metrics to investigate the impact of utilizing the original captions of the base videos as negative prompts in the generation process, as illustrated in Table 1 . We use 𝑁 𝑃 in the table to denote this method and calculate metrics for different values of 𝜔. The results shows that implementing original captions as negative prompts improves the alignment between the output video and the effect prompt, and this alignment increases with the escalation of 𝜔. Figure 8 corroborates this trend. Additionally, while this approach amplifies the divergence between the output and the base video, it ensures that the quality remains at an elevated level.Influence of Using User Masks as Attention Maps. First, we explore the impact of different pixel values of non-zero region in the user mask. When the value is 1, the result is shown in subfigure (c) of Figure 9 , where the user mask is in the lower left corner. The result indicates that it leads to the model adding an excessive amount of flame effects in the specified area, causing complete distortion of the image in that region. Consequently, we further reduced the pixel values of the non-zero areas in the mask image. The outcomes of this adjustment are displayed in Figure 9 . In conclusion, we chose 0.3 as the pixel value for the non-zero region. There is an explosion in front of a house, ultra realistic.A gray car tightly surrounded by flames is driving normally on a mountain street.er on the table. There is an explosion in front of a house, ultra realistic.A gray car tightly surrounded by flames is driving normally on a mountain street.ter on the table. Furthermore, we experimentally investigate the impact of using user masks at different positions within the UNet. Specifically, we divide the UNet into four layers from top to bottom, from layer 1 to layer 4, according to the feature dimensions in descending order within the UNet architecture. We replace the attention maps with the user mask at various positions within these four layers, and the results are illustrated in Figure 10 . As can be seen, the use of masks in the bottom layers is not as effective as in the top layers, especially when replacing only in the fourth layer, the replacement of the attention map is almost ineffective. We finally chose to use the user mask in layers1-2. 
discussions: 
 Our experiments have demonstrated the feasibility and potential of integrating AIGC with visual effects synthesis. In this section, we will discuss some issues identified during our research that warrant further extensive consideration. Scale of Dataset. In this paper, we compiled an effects dataset from the internet, covering a wide range of effect categories commonly seen in the film and television industry. However, limited by our lack of professional experience in effects production, we could not produce or collect a more extensive array of effects. This prevents our dataset from supporting training or large-scale validation.Moreover, for SNRE, our data acquisition method involved manually captioning base videos and modifying these to include descriptions of effects. This process also restricted the scale of our dataset. Our experiments demonstrated the competitive potential of AIGC technology in generating fluid and particle effects. However, we also observed that the model lacked experience in generating soft-body and rigid-body effects, particularly with rigid-body effects. As shown in Figure 11 , even in simple scenarios, our method fails to achieve effects such as "break". This limitation might be attributed to the capabilities of our backbone model Videocrafter2. Indeed, we noticed that even the most impressive text-to-video model to date, Sora, struggles with understanding the concept of a "cup shattering". This suggests that generating rigid bodies is not only a challenge for AIGC-based visual effects synthesis but also a broader issue faced by the pursuit of a "world simulator" in current research. 
conclusion: 
 In this work, we explore the integration of visual effects synthesis with the burgeoning field of AIGC. Our experimental findings demonstrate the substantial potential of this integration, where pretrained video generator can seamlessly blend effects with scenes and adjust scene elements logically. We believe that advancing research in AIGC-based visual effects synthesis will transform film and television production by enabling streamlined processes and innovative applications, while also allowing non-professional users to create high-quality visual effects videos, opening new creative opportunities for a broader audience. 
visual special effects synthesis without reference effects: 
 Suddenly a meteorite falls from the sky and explodes in front of them.In front of the woman was a spinning, sparking magic portal.A young man is riding his bicycle, with bubbles appearing surrounding.Drift car performance on a turn of the race track, emitting a magical light.Figure 1 : Two paradigms of AIGC-based visual effects synthesis: synthesis with reference effects(SRE) and synthesis without reference effects(SNRE). Both paradigms take a base video and a textual description as input, and the paradigm with reference effects additionally has a visual effects video as input. Top: Our results for the paradigm with reference effects. Bottom: Our results for the paradigm without reference effects.greater expressiveness. As the demand for detailed and realistic effects escalates in modern production, professionals are compelled to allocate substantial time and resources to this endeavor. Thus, there is an urgent need to explore more convenient and less resourceintensive methods, such as incorporating the burgeoning Artificial Intelligence Generated Content (AIGC) technology. However, research into this potential integration has yet to be conducted. As the first work to establish a connection between visual effects synthesis and AIGC technology, we start by carefully setting up two paradigms according to the need for pre-produced effects or not: synthesis with reference effects and synthesis without reference effects. Following this, we compile a dataset by processing a collection of effects videos and scene videos, which contains a wide variety of effect categories and scenarios, adequately covering the common effects seen in films and television industry. Furthermore, we explore the capabilities of a pre-trained text-to-video model to synthesize visual effects within these two paradigms. The experimental results demonstrate that the pipeline we established can effectively produce impressive visual effects synthesis outcomes, thereby evidencing the significant potential of existing AIGC technology for application in visual effects synthesis tasks. Our dataset can be found in https://github.com/ruffiann/MagicVFX. 
 
 Appendix: https://www.pikalabs.com/https://moonvalley.ai/https://www.genmo.ai/ 


Decision: The paper introduces AIGC techniques for visual effects, which generate complex special effects in videos in just a few minutes.  For that, the paper proposes a new dataset and a pipeline that uses text-to-video diffusion.  The method can generate effects with or without reference.  The core problem the paper addresses is very interesting and has a potential broad impact on the VFX industry.  The proposed method is technically sound, and the presented results are very interesting.  The reviewers raised concerns about the initial state of the paper, e.g., the user study being done with only 9 users, the lack some implementation details, and asked the authors for additional details regarding the subjective study.  During the rebuttal phase, the authors extended the study to include 16 additional participants and answered most of the reviewers' concerns.

As final scores, three reviewers recommend the acceptance (1 Accept and 2 Weak Accept), while one reviewer rates it as borderline reject.  Given the importance of the topic, the exciting results, and the fact that the proposed dataset can contribute to advancing similar research in the area, the recommendation is to accept the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces AIGC techniques for visual effect synthesis, which generate complex, high-quality special effects in videos in only a few minutes. Moreover, an experiment was conducted to demonstrate that the pipeline proposed by this paper could produce impressive visual effects synthesis outcomes.
strength_weakness: strength_weakness: The paper collects a basic dataset encompassing common visual effects for assessing the ability of visual effects synthesis methods. It builds a training-free base model with extensive experiments and demonstrates the potential of the pre-trained video diffusion model in visual effect synthesis. The paper, in general, looks into a relevant problem for MM.
limitations: limitations: 1. This paper implements the task of visual effects synthesis using a pre-trained text-to-video generation model. However, the model’s name (VideoCrafter2) is only mentioned in Section 1&2. It is unclear why the authors selected VideoCrafter2 as their backbone model. Is there any reason why VideoCrafter2 is better than other pre-trained text-to-video generation model?

Besides, the method this paper proposed is somewhat straightforward. The pipeline for synthesizing visual effects seems to mostly rely on the backbone pre-trained model, which is a previous contribution of another work. Therefore, this paper's contribution seems not outstanding.

2. This paper claims that it is the first work to explore AIGC-based visual effects synthesis, and there is no applicable methodology to compare. Please verify this argument carefully. If it is the first work, it is more convincing to transfer other similar works to this field and compare this method with them. Besides, since this paper uses VideoCrafter2 as the pre-trained model, some other pre-trained models can be chosen as baselines to compare with (Page 5, Section 5.1).

3. This paper invites nine users to conduct a user study and compare different methods. The number of users is relatively small, which diminishes the study's contribution.

4. A conclusion section seems necessary for this paper. However, the authors combined it with the discussion, making the paper less easy to read. 

5. According to the results of the subjective user study, the method proposed in this paper has a much lower video quality score compared to the baseline. Although the authors explained it as the synthesized video losing some details, is this the only reason? More explanations and discussions are needed.

6. Considering the reproducibility of this study, please add the questionnaires used in the subjective user study as support material or appendix. 

Minor comments
Page 7, Section 5.1, Line 720. There should be a space between the period and "Our".
Page 5, Section 4.3, Line 544. The “predict” should be “predicts”.
Page 8, Section 6, Line 899. “visual effects” should be “Visual effects”.
paper_summary: paper_summary: This paper proposes MagicVFX, a new framework to integrate generative AI, specifically text-to-video models, into the VFX production pipeline. The goal is to increase controllability and reduce manual costs in VFX systems. To address this, the paper introduces a new dataset consisting of reference-based and reference-free visual effect videos with captions. The authors also present a pipeline that uses a text-to-video diffusion model to integrate visual effects (either with or without reference effects) into a base video. Using the proposed dataset and pipeline, the authors provide experimental results demonstrating the effectiveness of MagicVFX in successfully adding visual effects to the original videos.
strength_weakness: strength_weakness: * The core problem addressed in the paper is both new and potentially has significant implications for the VFX industry.

* The proposed dataset can be used for future evaluation of AIGC-based VFX systems.

* The proposed method is simple to implement, making it accessible to a broader audience.

* The depth of the experimental results and ablations is reasonable, covering the important aspects and design choices of the method.

* The writing of the paper is clear and easy to read for the most part.
limitations: limitations: - The comparison with current video editing systems is missing. I acknowledge that the authors briefly mention this in the related work section and Figure 2, but I am not yet convinced why existing video editing systems cannot tackle this task. If the authors could provide more experimental results on this, as well as detail the limitations of current video editing systems compared to the proposed method with visual examples (e.g., cases where using existing editing methods fail but the proposed method succeeds in adding the visual effect to the base video), it would significantly enhance the overall contribution of the paper.

- The authors use negative prompting to enhance the quality of visual effects, but since they use the base caption as the negative part, it reduces the quality of the generated video compared to the baseline. I am curious to see an example where two CFG signals are used (with the null prompt for the negative part): one that uses the base caption and one that uses the visual effect part of the caption. i.e., using $ \hat{\epsilon} = \epsilon(x_t, t, c_{base}) + w_1 (\epsilon(x_t, t, c_{base}) - \epsilon(x_t, t, c_{null})) +  w_2 (\epsilon(x_t, t, c_{VFX}) - \epsilon(x_t, t, c_{null})).$ It would be interesting to see if this modification results in both high-quality generation and visual effects with proper $w_1$ and $w_2$.

#### **Minor Comments**

- Please provide more implementation details regarding the noise level, sampler, and details of the attention-based editing in the appendix for better reproducibility of the results.

- As similar ideas have been used in other image and video editing papers, please provide a more detailed analysis of the related work to further distinguish the novelty of the method. For instance, you should cite the SDEdit [1] paper for the idea of adding noise and denoising, and also compare to [2, 3].

[1] Meng, Chenlin, et al. "Sdedit: Image synthesis and editing with stochastic differential equations." arXiv preprint arXiv:2108.01073 (2021).

[2] Esser, Patrick, et al. "Structure and content-guided video synthesis with diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Wang, Xiang, et al. "Videocomposer: Compositional video synthesis with motion controllability." Advances in Neural Information Processing Systems 36 (2024).

#### **Final Thought**

I will be happy to adjust my score if the authors provide more explanation on the limitations during the rebuttal.
paper_summary: paper_summary: This paper presents a visual effects synthesis study. A dataset and a model are proposed. Specifically, the proposed method can achieve two effects, including visual special effects synthesis with reference effects and visual special effects synthesis with texts.
strength_weakness: strength_weakness: 1. The results are impressive, especially for the text-controlled effect generation.
2. The method supports both video effect integration and text-guided integration.
3. The paper is well written and well organized.
limitations: limitations: 1. In Fig. 4, it is clear how to add video effect to the base videos. However, the method of adding effect using text is not clear.
2. The pipeline of text editing is not clear. Can we add the effect to anywhere described in the text? How to realize it.
3. How to train the whole framework, since the ground truth seems unknow.
4. The references are too few, seem insufficient
paper_summary: paper_summary: The authors compile an effects dataset from the internet, encompassing a wide range of effect categories. Furthermore, they propose an approach for synthesising visual effects for two paradigms - with reference effects and without reference effects.
strength_weakness: strength_weakness: - Paper is well written and structured, and the figures are very useful to understand the approach.
- Authors have compiled a novel dataset for visual effects in videos, which is beneficial for further research in this field.
- Proposed pipeline for visual effects synthesis is straightforward and sound.
- Good evaluation using qualitative scores (human preferences from a user study) .
- Ablation study is well done.
limitations: limitations: - The user study is done only with 9 users. That's quite a small sample for a user study, at least twice as much users would be good.
- Method performs worse in video quality than the baseline.