AutoSFX: Automatic Sound Effect Generation for Videos

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 11:21:03

Abstract: Sound Effect (SFX) generation, primarily aims to automatically produce sound waves for sounding visual objects in images or videos. Rather than learning an automatic solution to this task, we aim to propose a much broader system, AutoSFX, significantly applicable and less time-consuming, \ie automating sound design for videos. Our key insight is that ensuring consistency between auditory and visual information, performing seamless transitions between sound clips, and harmoniously mixing sounds playing simultaneously, is crucial for creating a unified audiovisual experience. AutoSFX capitalizes on this concept by aggregating multimodal representations by cross-attention and leverages a diffusion model to generate sound with visual information embedded. AutoSFX also optimizes the generated sounds to render the entire soundtrack for the input video, leading to a more immersive and engaging multimedia experience. We have developed a user-friendly interface for AutoSFX enabling users to interactively engage in the SFX generation for their videos with particular needs. To validate the capability of our vision-to-sound generation, we conducted comprehensive experiments and analyses using the widely recognized VEGAS and VGGSound test sets, yielding promising results. We also conducted a user study to evaluate the performance of the optimized soundtrack and the usability of the interface. Overall, the results revealed that our AutoSFX provides a viable sound landscape solution for making attractive videos.

Main paper: 
 introduction: 
 Throughout the history of video making, sound effects (SFX) have played a crucial role in enhancing storytelling and creating immersive experiences for audiences [37] . The production of SFX involves identifying the essential visual objects and expressing the corresponding semantics with sounds. This process requires a deep understanding of audiovisual consistency as through exposure to a tremendous amount of visual-audio combinations, humans subconsciously learn the intricate correlations between visual and audio stimuli [14] .In recent years, significant advancements have been made in the automated SFX generation, realizing sounds for a single object in images or videos, such as dog barking, baby crying, and the explosion sounds of fireworks. Thanks to deep learning techniques and large-scale datasets (e.g., AudioSet [15] ), many models applied RNN, GAN, VQ-VAE, diffusion model, etc., to address the SFX generation task. However, relying solely on these strategies proves unsatisfactory for practical scenarios, e.g., sound design for videos and VR games. We then pose the question -can computers further imitate the sound effect generation process like a sound designer, converting a video from concrete visual information to auditory signals, i.e. elevating the SFX generation task to a comprehensive and practical sound design (SD) task? For example, a child playing in a garden should be synchronized with laughter, footsteps, bird chirping, etc., with all these sounds blending harmoniously and transiting seamlessly.In this paper, we propose to compile the rich spatially and temporally audiovisual correlations into the prevalent sound generation regime. Similar to how humans construct a perceptual space by extracting information from visual objects, grouping objects by distance, and incorporating mixed sounds, our key insight is that visual information can further serve as strong learning signals for the SD task. For instance, by leveraging segmented visual objects, we can identify and localize the sound events; according to the different distances of visual objects, we can layer the mixed sounds.Building on this insight, we introduce AutoSFX, a new SD pipeline that aims to achieve improved performance of sound generation and automated practical sound design for videos (cf. Fig. 1 ). Specifically, we first extract compact features with pixel-wise audiovisual features based on the Segment Anything Model (SAM) [24] , which has recently proven to be highly effective in image segmentation tasks. We then employ adapters and cross-attention mechanisms to learn the correlation between audio and visual information, i.e. fusing pixel-wise audiovisual representations. For SFX generation, we leverage a spectrogram autoencoder to predict self-supervised auditory representation and a diffusion model to map visual information to latent representations. To further refine AutoSFX 's applicability to sound design tasks, we devise two modules: i) Transition Module, facilitates seamless transitions of generated sounds in alignment with changes in visual objects or scenes; ii) Hierarchy Module, aims to blend the sounds that playback simultaneously according to their corresponding depths. These modules are designed to circumvent the labor-intensive processes typically associated with text-prompted sound effect generation applications, such as Pika 1 . With AutoSFX, we are not only able to automate sound generation but also contribute to the field of sound design for videos.To evaluate the effectiveness of our approach, we first undertook experiments on VEGAS [53] and VGGSound [4] to evaluate AutoSFX 's sound generation capabilities. We also conducted user studies to qualitatively evaluate the generated sound effects and the implemented interface. Experimental results demonstrate that our AutoSFX achieves a new state-of-the-art performance of sound generation and could provide promising results for videos featuring diverse objects and scenes. Our main contributions are:• We propose AutoSFX, a deep learning-based system to tackle the problem of automatically generating sound effects for videos with seamless transition and harmonious mixing.  
related work: 
 In this section, we first give a concise overview of the sound design industry. Then, we give a brief discussion on vision-to-audio techniques and the segment anything model (SAM) that pertains closely to our work.The use of sound in theater dates back centuries, however, the position of a sound designer emerged approximately 50 years ago with the advancement of audio and recording technology 2 . Sound design is the art of creating and manipulating sound for various media productions, such as film, advertising, and interactive learning tools [48] , which is a broad and evolving field that requires both technical skills and sensitive vision [34] . It requires significant human effort to leverage sound effects to enhance the mood, atmosphere, realism, and storytelling of different visual media [8] .Typically, sound design involves understanding the visual content, i.e. the semantics of objects and scenes; recording sounds from various sources, such as field recording, sampling, synthesis, or using sound libraries, for initialization of the soundtrack for the visual media; editing and manipulating sounds to make them fit the desired visual context; mixing and balancing sounds to create a coherent and immersive audio experience 3 . The cost of professional sound design is high. For example, adding sound effects to a game project requires 48 hours of sound design work, with an average sound designer rate of $30 to $100 per hour 4 ; the sound design costs are expected to fall in the range from $1,440 to $4,800.On the other hand, when an amateur aspires to create a video enriched with sound effects, he/she may encounter constraints, such as having to choose from a limited repository of sound effects offered by platforms (e.g., TikTok), without the ability to generate sounds that are optimal to their specific recordings. Alternatively, they could use text-prompted sound effects generation applications (e.g., Pika), which may require multiple attempts with different descriptions and subsequently editing the results, e.g.trimming the waveform to have an appropriate segment to match the video content. These processes can be cumbersome and limit creative freedom. Therefore, we desire to explore the possibility of devising a computational approach to sound design.With advancements in technology and research in multimedia, human-computer interaction, computer vision, and graphics, the concept of converting visual information into sound has been implemented in recent years. Previous works on this topic can be divided into two categories: natural sound generation and music generation. As music is commonly used to enhance the atmosphere of the visual content [43] and our goal is to generate authentic sound effects for video, our discussions in this section will focus on natural sound generation endeavors.For physics-based techniques, they aimed to improve the audiovisual consistency of computer-animated phenomena, i.e. synthesizing natural sounds for rain [28] , fire [46] , rigid bodies [9, 42] , etc. Moreover, image sonification techniques synthesize auditory interpretations of visual stimuli, e.g., mapping pixel value to various sound parameters (e.g., pitch and loudness), thus enhancing scientific discovery [48] and visual accessibility to people who have visual impairments [22] . A similar idea to our AutoSFX is Audible Panorama [20] , where suitable sound effects are chosen from a precollected dataset for objects in static panorama images, considering object depth and audio source placement.On the other hand, several promising works have emerged, leveraging deep learning techniques to generate waveforms for specific objects [5, 6, 23, 29, 35, 51] and soundscapes of wild scenes [21, 50, 53] . More recently, diffusion-based models have achieved state-ofthe-art sample quality in the field of audio generation. For example, Luo et al. [31] proposed to train a latent diffusion model with temporally and semantically aligned features on spectrogram latent space. Liu et al. [26] , Yang et al. [45] , and Ghosal et al. [16] leveraged diffusion models to generate sound conditioned on a text prompt. While these methods strive to provide precise representations of real-world soundscapes, their application is constrained to short video clips, rather than the diverse and vast videos uploaded online.Additionally, the realm of text-to-audio generation has captured significant interest [2, 7, 25] . However, the challenge of automatic sound design differs from these works -it involves not only encoding visual content and producing a corresponding soundtrack with multiple waveforms, but also performing seamless transitions and harmonious blending.Segment-Anything Model (SAM) [24] , a large foundation model trained on one billion masks from 11M images, supports flexible and interactive prompts in real time to achieve image segmentation. SAM represents a revolutionary method for image segmentation, demonstrating remarkable generalization capabilities when applied to different datasets. There have been recent proposals for extensive applications based on SAM [49] , such as image editing [47] and style transfer [30] . Such visual segmentation techniques have inspired researchers to integrate SAM into audiovisual segmentation pipelines. For example, Mo et al. [32] and Wang et al. [44] proposed to extract visual features by utilizing the pre-trained image encoder in SAM and aggregating them with auditory features, i.e. following the encoder-fusion-prompt-decoder paradigm. Liu et al. [27] simply employed adapters to inject the audio information into the pre-trained SAM, achieving deep audio-visual fusion in the encoding stage. Following the bidirectional generation strategy, Hao et al. [17] proposed to use the generated segmentation masks to reconstruct audio features and minimize auditory reconstruction errors during training. Inspired by these studies, we propose to leverage SAM as the fundamental feature extraction model to contribute to audiovisual representation learning and further design a diffusion-based model to generate sound effects for visual objects. 
overview: 
 As shown in Fig. 1 , AutoSFX contains two main components: A Sound Generation module, which formulates the visual-guides sound generation task as a collaborative parallel generation problem for both auditory and visual channels. In the training Dataset 𝐷 = {𝑋 𝑖 , 𝑦 𝑖 }, 𝑋 = {𝑥 𝑣 , 𝑥 𝑎 } represents the video with continuous frames (𝑥 𝑣 ) and the corresponding audio clip's spectrogram (𝑥 𝑎 ), and 𝑦 𝑖 represents the ground truth visual segmentation masks. Given the input data 𝑋 , the goal of the Vision-to-Sound module is to generate the audio spectrogram with the guidance from 𝑥 𝑣 . Moreover, 𝑖 represents the sample index for a clearer explanation.A Sound Optimization module, designed with constraints for seamless transition and harmonious mixing, aims to synchronize the generated sounds with video frames to create an immersive audiovisual experience. Let 𝐴 𝑘 = {𝑎 1 , 𝑎 2 , ..., 𝑎 𝑘 } denote the generated sounds for the sounding object 𝑘 in the video clip. We choose the optimal 𝑎 that minimizes the cost:EQUATIONwhere 𝐶 tran (•) is the transition cost term, which measures the transition difficulty between two continuous sounds, constraining the pitch and tempo. 𝐶 mix (•) is the cost term for evaluating consistency between the visual information and the generated results based on the estimation of depth and emotion expression. 𝜆 is a regularization factor to balance these two terms. 
sound generation: 
 As shown in Fig. 2 , the Sound Generation module has three parts: First, we encode the visual and auditory representations from two branches and fuse them in an attention manner; Second, we resort to the diffusion model to map the audiovisual information to latent representation and simultaneously to help model the conditional distribution; Finally, spectrograms could be generated with the projection of the visual prompt into the auditory space.For the visual branch, we sampled the input videos at intervals of 1 second to obtain frames 𝑥 𝑣 ∈ R 𝑇 𝑣 ×3×𝐻 ×𝑊 , where 𝑇 𝑣 represents the number of frames. We extract visual features from the ImageNet pre-trained SAM backbones, which is based on ViT [12] and consists of 12 transformer layers. We also apply the visual encoder with bottleneck adapters [19] and obtain the visual feature𝐹 𝑣 ∈ R 𝑑 𝑣 ×𝐻 ×𝑊 .On the other hand, as a spectrogram (i.e. 1-channel 2D images) autoencoder with reconstruction objective as self-supervision has demonstrated the effectiveness of image-to-audio generation [44] , we transfer the audio mono-waveforms (with a sampling rate of 16kHz) into a sequence of mel-spectrogram sample 𝑥 𝑎 ∈ [0, 1] 𝐶 𝑎 ×𝑇 𝑎 , where 𝐶 𝑎 denotes the mel channels and 𝑇 𝑎 is the number of frames. Then, we leverage the VGGish [18] , a model designed for capturing both temporal and spectral information, to extract auditory features 𝐹 𝑎 ∈ R 𝑇 𝑎 ×𝑑 𝑎 . 𝑑 𝑎 is 128 as the default.𝐹 𝑎 includes auditory guidance for sound objects that is crucial for image segmentation, and 𝐹 𝑣 contains important visual context information for consistent audiovisual sound generation. However, 𝐹 𝑎 and 𝐹 𝑣 extracted from different branches do not align well. To address this, we apply an audiovisual cross-modal attention module, shown in Fig. 3 , to align the audio sources and visual locations by treating them as a joint attention space. The attention module takes 𝐹 𝑣 and 𝐹 𝑎 as inputs. In the joint attention setting, attention operates simultaneously over time and space. Moreover, we utilize the multi-head attention (MHA) layer [41] to capture different aspects of the input, allowing for a more comprehensive and nuanced representation of the relationships between 𝐹 𝑣 and 𝐹 𝑎 . At the 𝑙-th layer, when considering a auditory query, the directional attention operations can be described as:𝑎 (𝑙 ) 1 = MHA(𝐹 (𝑙 -1) 𝑣 , 𝐹 (𝑙 -1) 𝑎 ), 𝑎 (𝑙 ) 2 = LN(𝑎 (𝑙 ) 1 + 𝐹 (𝑙 -1) 𝑎 ), 𝐹 (𝑙 ) 𝑎 = LN(𝑓 (Dropout(𝑎 (𝑙 ) 2 )) + 𝐹 (𝑙 -1) 𝑎 )).For the auditory direction, we modulate the visual features 𝑣 (𝑙 ) using the auditory features 𝑎 (𝑙 ) by swapping 𝐹 𝑣 and 𝐹 𝑎 in the above equation.Diffusion-based models, first transform a given data distribution into unstructured noise (usually Gaussian noise) and then proceed to learn how to reverse the forward process to recover the original data distribution. In this paper, we deploy an AV-Diffusion Model to bidirectionally generate we further propose a bidirectional multimodal latent diffusion model, shown in the middle part of Fig. 2 . Our goal is to recover two consistent modalities within one diffusion process. Specifically, the forward diffusion maps both audio and video data to noise independently, while the reverse process gradually reconstructs the original multimodal contents using a unified model. With the paired visual and auditory feature (𝑎, 𝑣) (i.e. the simplification of 𝐹 𝑙 𝑎 and 𝐹 𝑙 𝑣 ), the forward processes of each modality are independent. Taking the auditory feature 𝑥 𝑎 as an example, the corresponding forward process is defined as:𝑞(𝑎 𝑡 |𝑎 𝑡 -1 ) = N 𝑎 (𝑎 𝑡 ; √︁ 1 -𝛽 𝑡 𝑎 𝑡 -1 , 𝛽 𝑡 I).𝑡 ∈ [1,𝑇 ] is the time step. The forward process for visual represents 𝑣 a similar formulation. Let 𝑎 𝑡 as the sample fitting standard Gaussian distribution and is independent from 𝑎 0 using the Markovian forward process. We can calculate any 𝑎 𝑡 through:𝑞(𝑎 1:𝑇 |𝑎 0 ) = 𝑇 ∑︁ 𝑡 =1 𝑞(𝑎 𝑡 |𝑎 𝑡 -1 ).On the other hand, we utilize the joint reconstruction of audiovisual pairs from independent Gaussian distributions [36] . The coupled U-Net [36] 𝜃 𝑎𝑣 takes both auditory and visual information as inputs and reinforces generation quality for each other. Specifically, the reverse process 𝑝 𝜃 𝑎𝑣 (𝑣 𝑡 -1 |(𝑣 𝑡 , 𝑎 𝑡 )) for obtaining 𝑣 𝑡 -1 in visual domain is defined as:𝑝 𝜃 𝑎𝑣 (𝑎 𝑡 -1 |(𝑣 𝑡 , 𝑎 𝑡 )) = N (𝑎 𝑡 -1 ; 𝜇 𝜃 𝑎𝑣 (𝑣 𝑡 ,𝑎 𝑡 ,𝑡 ) ),where 𝑎 𝑡 -1 is generated from a Gaussian distribution jointly determined by both 𝑣 𝑡 and 𝑎 𝑡 . The reverse process for visual represents 𝑣 a similar formulation.Visual-Promted Sound Generation. As declared in previous works [35] , decoding in the auditory space with visual prompts can enhance the model's generalization ability. Our visual-promoting module was designed to prompt the audio decoder to generate sounds that are consistent with the objects/events from the visual space. As shown in the third part of Fig. 2 , we first feed 𝑣, i.e. the output of the visual branch from the coupled U-Nets, into an MLP. We then leverage a bottleneck adapter ColA(•) to model the audiovisual correlation, yielding the updated visual feature 𝐹 𝑐 ∈ R 𝑑 𝑣 :𝐹 𝑐 = ColA(MLP(𝑣)) + MLP(𝑣).As the visual prompt, 𝐹 𝑐 is further fed into the audio decoder. We adopt WaveNet [40] as the decoder to convert the synthesized spectrogram into waveform 𝐴. Simultaneously, we also obtain the final visual output, i.e. mask embedding 𝐹 𝑚 ∈ R 𝑑 𝑣 ×𝐻 ×𝑊 , which is then upscaled by convolutional blocks.Learning Objectives. AutoSFX highlights the potential opportunities arising from the visual segmentation into the audio generation pipelines. The total loss has three terms: i) Reconstruction Loss, i.e. the L2 error for measuring the generated spectrogram 𝑠 ′ and the ground truth 𝑠:L 𝑟𝑒𝑐𝑜𝑛 = ||𝑠 ′ -𝑠 || 2 2 ;ii) Segementaion Loss, we use the binary cross-entropy 𝐵𝐶𝐸 (•) loss to measure the difference between the predicted mask 𝑀 𝑝 and the ground truth 𝑀 𝑔 :L 𝑠𝑒𝑔 = BCE(𝑀 𝑝 , 𝑀 𝑔 );iii) Generation Loss, is defined as the mean squared error in the noise space 𝜖 ∼ N (0, I). We leverage 𝜖-prediction to optimize the network, i.e.L 𝜃 𝑎𝑣 = E 𝜖∼N { 0,I} 𝜆 𝑡 || ε𝜃 (𝑎 𝑡 , 𝑣 𝑡 , 𝑡) -𝜖 || 2 2 , 𝑡 ∈ [0,𝑇 ],where 𝑡 represents a random term for stochastic gradient descent and 𝜆 𝑡 is an optional weighting function.Therefore, the final loss function isEQUATIONDuring the training phase, we only fix the parameters of both visual and auditory encoders, and update others, e.g., the adapters, mask decoder, and spectrogram decoder.Dataset. We leverage AVSBench [52] , a recently released video segmentation dataset, providing masks for sounding objects with audio signals, to train our Sound Generation module. It covers 23 object categories, e.g., animal and human-related sound events. Specifically, the single-source subset, includes 4932 videos with 10,852 annotated frames. 
sound optimization: 
 When a soundtrack is out of sync with the visual content, such as lagging behind or rushing ahead of the visual elements, it will result in a discordant, disjointed, or confusing viewing experience [38] . As a result, sounds should be both temporally and content-wise aligned with visual content, i.e. maintaining consistent changes when visual content changes and sounds played at the same time should be mixed properly.To improve the consistency between the generated audio and input video, we formulate the synchronization process as an optimization with several constraints. As depicted in equation 1, we design two cost terms, i.e. transition cost and mixing cost.A key observation of our work is that a transition from sound effect 𝑎 𝑘 to another one 𝑎 𝑘+1 sounds natural when pitch and tempo are similar [43] ; and if the pitch is harmonious progressed. We compute the transition cost C tran (•) by combining these two cost terms:C tran (𝐴 𝑘 , 𝐴 𝑘+1 ) = 𝛼 1 C tem (𝐴 𝑘 , 𝐴 𝑘+1 ) + 𝛼 2 C pit (𝐴 𝑘 , 𝐴 𝑘+1 ),where C tem (•) is tempo cost, evaluating the BPM (beats per minute) difference between two sounds. For pitch cost, C pit (•), we estimate pitch differences by computing chroma features [13] for each beat in a sound clip, respectively. Then we compute the average cosine distances between 𝐴 𝑘 and 𝐴 𝑘+1 .Driving content-wise audiovisual consistency, is not merely aligning sounds with visual elements; it is more about creating an immersive experience that resonates with the audience on a deeper level [6] . So we delve into mixing techniques commonly used in the sound design process, meticulously analyzing and integrating them to form the mixing cost C mix (𝐴 𝑘 , 𝑥 𝑣 ). Below we describe how mixing guidelines are formulated and adapted to our setting.Rhythm. Video with a greater rhythm-matched soundscape is preferred in the viewing experience. We design a cost term, C r (𝐴 𝑘 , 𝑥 𝑣 ), to find the best part of 𝐴 𝑘 that matches the visual content progression. This cost has two elements: i) Speed/Tempo, denoted by C s (𝐴 𝑘 , 𝑥 𝑣 ), influences the sense of urgency, excitement, or contemplation, etc.: the faster the speed, the higher the content tension. We apply the visual beats extraction approach [11] to obtain the visual tempo of 𝑥 𝑣 , and calculate the absolute distance between it and the 𝐴 𝑘 's BPM as C s (•).ii) Movement, whether it's the action of subjects or the camera's motion, the consistent sound effects can significantly influence the rhythm of the video. We believe that changes in motion should consistently match the peak changes in sound effects. For each segmented object, we first compute the homography transformation matrix linking consecutive frames 𝑓 and 𝑓 ′ . We then use the estimated homography matrics 𝐻 (𝑓 , 𝑓 ′ ) and 𝐻 (𝑓 ′ , 𝑓 ′′ ) from three consecutive frames to compute the movement 𝑚𝑜𝑣 (𝑓 ) for frame 𝑓 :𝑚𝑜𝑣 𝑣 (𝑓 ) = 1 4 4 ∑︁ 𝑛=1 ||𝐻 (𝑓 ′ , 𝑓 ′′ )𝑝 𝑓 ′ (𝑛)-𝑝 𝑓 ′ (𝑛)-(𝐻 (𝑓 , 𝑓 ′ )𝑝 𝑓 (𝑛)-𝑝 𝑓 (𝑛))|| 2 ,where 𝑝 𝑓 (𝑖), 𝑖 ∈ [1, 4] represents the 2D positions of the four corners of the sounding objects measured in pixels. For the sound effect, we find 𝐴 𝑘 's peak (a long-pitch plateau forming a flat turn) 𝑚𝑜𝑣 𝑠 (𝑓 ) by binary search. Consequently, the movement cost term C m (𝐴 𝑘 , 𝑥 𝑣 ) is defined as the Kullback-Leibler divergence between 𝑚𝑜𝑣 (𝑓 ) and 𝑚𝑜𝑣 (𝑝), i.e.EQUATIONAs shown in Fig. 4 , we demonstrate the comparison between video keyframes and the corresponding sound effect's peak. Emotion. To address the significant gap between the visual domain and the audio space, we leverage the color tone and frequency distribution representing visual and auditory information, respectively. These attributes have been demonstrated to be tightly related to emotion expression in video creation [3, 39] . For example, low-frequency sounds might be more likely to induce sadness, while high-frequency sounds can invoke happiness or excitement. We first use color histograms to obtain the saturation and brightness of frames and further compute the ratio 𝜉 𝑣 (𝑓 ) of pixels within high saturation and brightness regions, i.e. above 70% of the maximum value (i.e. 255). Similarly, we compute the ratio 𝜉 𝑎 (𝑓 ) of high-frequency regions, i.e. above 30% of the maximum value of 𝐴 𝑘 . Therefore, C e (𝐴 𝑘 , 𝑥 𝑣 ) is defined as:C e (•) = ||𝜉 (𝑓 ) -𝜉 (𝑓 )|| 2 ,Distance. When mixing multiple sound effects that playback overlapped, the relative depth should be considered -resulting in a layered soundscape that enhances the perception of distance and space within the audiovisual content [20] . We apply the approach proposed by Dai et al. [10] to estimate the segmented objects' three translations in real time and calculate Euclidean distance between each pair of sounding objects, yielding 𝑑 𝑣 (𝑝, 𝑝 + 1), 𝑝 ∈ [1, 𝑁 ]. For sound effects, we compute the corresponding loudness distance, i.e. 𝑑 𝑎 (𝐴 𝑝 , 𝐴 𝑝+1 ). We normalize 𝑑 𝑣 and 𝑑 𝑎 into [-1, 1], and define the distance cost as:C d (•) = - 𝑑 𝑣 (𝑝, 𝑝 + 1) 𝑑 𝑎 (𝐴 𝑝 , 𝐴 𝑝+1 ) ,The goal of harmonious mixing is to find a sequence of generated sound effects 𝐴 𝑘 by selecting and ordering a subset of candidate sounds, cutting within them, performing low/high-frequency pass, and adjusting the loudness. The total mixing cost isC mix (𝐴 𝑘 , 𝑥 𝑣 ) = 𝛽 1 C r (𝐴 𝑘 , 𝑥 𝑣 ) + 𝛽 2 C e (𝐴 𝑘 , 𝑥 𝑣 ) + 𝛽 3 C d (𝐴 𝑘 , 𝑥 𝑣 ), C r (•) = C s (𝐴 𝑘 , 𝑥 𝑣 ) + 𝛾C m (𝐴 𝑘 , 𝑥 𝑣 ).𝛽 and 𝛾 are regularization factors to balance these cost terms.Since our optimization problem is combinatorial and the number of combination items can vary, we adopt the Reversible Jump MCMC (RJMCMC) framework to explore the space of possible soundtrack extensively. To efficiently explore the solution space, we apply the simulated annealing process in the optimization process. We define a Boltzmann-like objective function:EQUATION)where 𝑡 is the temperature of the simulated annealing process, which decreases gradually throughout the optimization. There are three types of moves that can be selected by the optimizer:(1) Modify Sound, including modifying the tempo, pitch, and loudness of the current sound effect. Take tempo as an example, the modification is defined as 𝑡𝑒𝑚 ′ = 𝑡𝑒𝑚 0 + △𝑡𝑒𝑚 0 , where △𝑡𝑒𝑚 0 is sampled from a Gaussian distribution whose mean is zero and variance is 0.1𝑡𝑒𝑚 0 ; (2) Change Onset, i.e. randomly change the timestamp of playing the current sound effect; (3) Swap Sound, i.e. randomly change to another sound clip from the generated sounds. The selection probabilities of the moves are 𝑝 𝑚 , 𝑝 𝑐 , and 𝑝 𝑠 , which are set as 0.35, 0.3, and 0.35. Compared to existing sound effect generation approaches discussed in § 2.2, this hybrid optimization approach not only ensures the diversity of sound styles, but also preserves seamless transition between sounds and harmonious mixing consistent with the visual content. 
autosfx interface: 
 As shown in Fig. 1 , we develop an interface for video creators or sound designers to facilitate automatic sound generation for videos. Specifically, The Operation Panel offers users three options: "Chapters", displaying keyframes of video clips and allowing for quick skipping; "Objects", showing the segmentation heatmap of each video clip and enabling users to specify their desired sounding objects; "Sound Effects", presenting our generated and optimized results that match the visual content. Please refer to the demo for detailed operation of our AutoSFX. 
experiments: 
 In this section, we first introduce the experiment setup ( § 7.1), including datasets, evaluation metrics, and implementation details. Then, we evaluate the performance of our AutoSFX in comparison to state-of-the-art methods for vision-to-sound ( § 7.2). We also provide mass generated results and conduct user studies to evaluate our generated sounds and the interface ( § 7.3).Dataset. We leveraged two datasets to evaluate the performance of the sound generation of our proposed AutoSFX.i) VEGAS [53] , containing 28,109 videos with both visual and auditory information, is derived from AudioSet [15] . It covers 10 categories, such as dog barking, baby crying, and water flowing.ii) VGGSound [4] , contains over 200k clips for 309 different sound categories, which are all downloaded from YouTube. Each clip lasts at least 10 seconds and the corresponding label is flat (i.e. no hierarchy architecture among labels).Evaluation Metrics. We quantified the performance by adopting audio generation metrics outlined in [6, 29] .i) Quantitative evaluation involves: i) missing sound (Miss. S.), i.e. when an event should produce sound but the model fails to generate it; ii) redundant sound (Red. S.), the opposite of missing sound, i.e. the model generates sound without any corresponding visual stimuli; iii) mismatched sound (Mism. S.), i.e. the generated sound does not match the video content; iv) objective similarity grade (OSG), i.e. measuring the distance between ground truth and the generated audio at the acoustic feature level.ii) User study, primarily focuses on participants' audiovisual experience in user-uploaded YouTube videos with soundtracks designed by different approaches, including our AutoSFX, online tools, and professional sound design.We also conducted comparisons between our AutoSFX and audiovisual segmentation approaches, demonstrating a comparable capability on the task of video segmentation. Please refer to supplementary materials for more details.Implementation Details. We resized all video frames to a size of 1024 × 1024 and extracted the log Mel-Spectrogram using 64 mel filter banks over 1 second of audio data sampled at 16,000 kHz on the training set of VEGAS and VGGSound. The weight for balancing the three losses in Equation 2 was empirically set to 1. We employed Adam optimizer to optimize the model parameters with an initial learning rate of 10 -4 with cosine decay. The batch size was defined as 8 and we trained for 40 epochs. Additionally, we utilized PyTorch for model training on an NVIDIA A800 GPU.To fairly compared our AutoSFX and other vision-to-sound models, we trained our diffusion-based Sound Generation module under two settings: i) When comparing to models [6, 29] , we utilized 80% of the data from the VEGAS dataset to perform the training process. The remaining 20% data was used for evaluation. Each video clip has three corresponding sound effects generated by different models; ii) When comparing to models [21, 31] , we trained all models on both VEGAS and VGGSound datasets. The same splitting strategy was used for training and evaluation. Note that, some of the compared models are not official implementations; however, we have made efforts to reproduce their pipeline.For the metrics of missing sound, redundant sound, and mismatched sound, we recruited 10 participants to evaluate the generated results of different models. The objective similarity grade (OSG), consisting of maximum modulation spectrum (MMS), mean spectral flux (MSF), and root mean square (RMS), is defined as:EQUATIONwhere 𝑟 1 = 0.57, 𝑟 2 = 0.52, 𝑟 3 = 0.45, as suggested in [29] .Results and Analysis. As shown in Table 1 , we demonstrate the evaluation results of different models under the two settings. The lowest the value, the higher the performance. Overall, our AutoSFX attains the lowest error for all metrics under the first setting. Similar to the statistical results demonstrated in [6] , the sound generation of "sneeze" and "Cough" demonstrated relatively lower performance, leading to the suboptimal results in our experiments.For the results of the second setting, the model proposed by Iashin et al. [21] , which generates sound effects with a fixed length of 10 seconds, yielded significantly lower performance than that of the other two approaches, particularly for the mismatched sound. On the other hand, our results are comparable to those of Diff-Foley (i.e. Lou et al. [31] ) -for example, AutoSFX outperformed in animal sounds, while Diff-Foley excelled in instrument sounds. We believe that such results could be improved with our Sound Optimization module. In Fig. 5 , we illustrate some sound effects generated by different approaches along with the alignment cases, i.e. missing sound, redundant sound, and mismatched sound. We also provided some sounds generated by different approaches in our demo.In addition to the quantitative evaluation, we further evaluated the effectiveness of our AutoSFX for practical sound design scenarios. Different Approaches. We compared three approaches:• AutoSFX, our generated and optimized sound effects;• Pika5 , an online tool for users to generate sound effects based on generated videos or text prompts; • professionally designed soundtrack by sound designers.Dataset. We randomly selected 10 videos on YouTube from the "Sound Design Tutorial" category. These videos cover different objects, styles, and themes, typically uploaded by professional sound designers. Therefore, we leveraged the provided soundtrack as the professional results. We then applied AutoSFX and Pika to automatically generate sound effects for these videos. Please refer to the demo for videos featuring different soundtracks "designed" by different approaches.Participants. We recruited 30 participants aged between 30 and 32, who reported normal or corrected-to-normal vision, with no color blindness and normal hearing. 10 of them are video creators who usually record, edit, and upload videos at least once a week (referred to as Creators); and the rest are viewers of video streaming platforms, like YouTube and TikTok (referred to as Viewers).Procedures. The goal of this study is to evaluate how well the soundtrack matches the visual content. We asked participants to rate the authenticity of the sounds, audiovisual temporal consistency, sound quality, and overall viewing experience, using a 1-5 Likert scale, where 1 indicates poor performance and 5 indicates the opposite. For each video, the three corresponding soundtracks were presented randomly so as to avoid bias. Participants were allowed to view each video and play the corresponding soundtrack an unlimited number of times. Finally, all participants were invited to use our implemented interface. We then conducted a semi-structured interview about their experience to explore nuanced insights into our system design. Results and Analysis. As shown in Fig. 6 , we demonstrate the ratings by different participant groups, i.e. Creators and Viewers.Authenticity and Sound Quality, i.e. whether the sounds authentically and plausibly express the video content. Overall, ratings for all three approaches average around 4, with the professional results exceeding 4.9. Our AutoSFX received an average rating of 𝑀 = 3.92, 𝑆𝐷 = 0.76 over all participants, i.e. 𝑀 = 3.68, 𝑆𝐷 = 0.86 for Creators and 𝑀 = 3.95, 𝑆𝐷 = 0.79 for Viewers. According to participants' feedback, over 75% of our generated sound effects for visual objects were largely successful in tricking humans into thinking that these sounds were real. However, the ratings for Pika's results are relatively lower, yielding an average of 𝑀 = 3.41, 𝑆𝐷 = 0.92. Such results could be explained by the fact that sound effects used by professional designers are recorded or created using foley techniques; whereas the other two kinds of sounds are generated. On the other hand, when multiple objects appear in the video, Pika's results sometimes sound noisy, not to mention a soundscape with depth distinctions; while our AutoSFX could generate soundS for each segmented object and mix them through the optimization.Similar ratings were received for the ratings of sound quality. When we asked participants about the "fake" sounds or low-quality ones, we found there were two main reasons for their judgments:• Sounds can potentially convey significant information. Take the baby crying as an example; it often indicates that the baby has a need or discomfort that must be addressed. However, the generated sounds may not show diverse variations. • Sound quality significantly influences the perception of authenticity -whether for music, animal sounds, or even human speeches.Audiovisual Temporal Consistency. The professionally designed sounds received the highest ratings, i.e. an average of 𝑀 = 4.89, 𝑆𝐷 = 0.10, followed by the results by our system (𝑀 = 4.16, 𝑆𝐷 = 0.97). On the contrary, Pika's results showed relatively lower performance (𝑀 = 3.94, 𝑆𝐷 = 1.05). Note that, the length of sound generated by Pika is longer than that of the video, so we trimmed sounds to start and stop with the video. Additionally, the ratings of Creators are significantly lower than those of Viewers. Some Creators reported being more rigorous on temporal consistency. We also observed that these participants usually repeatedly played video segments to check the alignment.Overall Experience. To verify the above three factors contributed to the overall experience, we computed Bivariate (Pearson) correlation coefficients between their ratings and overall ratings, respectively. As a result, we obtained positive correlations, i.e. 𝑟 = .51, 𝑝 < .05 (authenticity), 𝑟 = .44, 𝑝 < .05 (temporal consistency), and 𝑟 = .39, 𝑝 < .05 (sound quality). This further supports the effectiveness of our adopted Sound Optimization module. According to the participants' feedback, we observe that while some rated authenticity, synchronization, and quality highly, they gave lower ratings for overall consistency. They explained that sometimes the high-frequency part of the generated sound is unclear, leading to poor viewing experience.About the Interface. After participants used our interface to generate sound effects for videos, we asked them about their experience by giving a rating of "Not Helpful", "Somewhat Helpful", and "Very Helpful". 80% (16 out of 20) of the Viewers rated it as "Very Helpful", noting its significant helpfulness for amateur creators to quickly produce videos with an appealing soundscape. 30% of Creators considered the system "Very Helpful" as they appreciated the tool's ability to automate the complex process of sound selection, editing, and mixing; 60% found it "Somewhat Helpful," reducing some efforts to produce videos; only two participants rated is as "Not Helpful" due to the sound quality issues. All participants expressed a desire for a more comprehensive system like the Adobe series to make such automation widely applicable. 
conclusion: 
 Conditioned on visual content, automatic sound design is challenging but can streamline the video-making process, benefiting video creators and sound designers. In this paper, we propose a computational approach to automatically generate sound effects for videos considering audiovisual consistency. Our AutoSFX is the first attempt to integrate sound generation techniques into practical sound design scenarios.Limitation and Future Work. Videos, recorded by different creators may have different intended uses, themes, and styles featuring different backgrounds, contextual storytelling, lighting, etc. Due to the difficulty of learning effective representations of these factors by computer vision techniques and obtaining accurate annotations, we only generate sound effects for audiovisual consistent objects and involve quantifiable cost terms during optimization. Moreover, when dealing with intricate scenarios, e.g., a crowded concert with numerous individuals and instruments, our Sound Generation module may encounter challenges in processing such visual information. We also provide some failure examples of such issues in supplementary materials.Currently, our AutoSFX primarily focuses on generating sounds for realistically sounding objects. However, sound design often utilizes abstract sounds to enhance the atmosphere, emphasize emotions, and convey subtle nuances. For example, a "woosh" sound could be used to describe the camera movement. Inspired by textprompt techniques, e.g., text-guided music generation [1, 33] , future work could further incorporate text-prompt during the training process of the generation module. We also believe there is room for improvement in video understanding and in particular, generating abstract, indescribable, and context-ware sound effects. 
 
 Appendix: https://www.nfi.edu/sound-design/https://www.coursera.org/courses?query=sound%20designhttps://www.visartech.com/blog/sound-effects-in-games-development/https://pika.art/ 


Decision: The paper is written in a clear manner and well structured. All 3 reviewers recommend acceptance, 2 weak and 1 borderline accept. The authors have provided a rebuttal that improved the overall rating from 4.0 to 4.67.
One reviewer (R3) remains somewhat more critical after rebuttal. Here, there are some contradictions in the review, in "strengths" mentioning the approach to be a breakthrough, and in "limitations" stating that "AutoSFX heavily relies on existing models and techniques, such as the Segment Anything Model (SAM) and diffusion models, which raises concerns about the originality of the methodology." 
All reviewers appreciate the novel approach that combines SAM and diffusion modelling in the context of audiovisual fusion. The technical approach and depth of presentation were appreciated by all, too. A further strong point mentioned by all is the systematic and extensive evaluation including also a user test.

The limitations mentioned by the reviewers were mostly well addressed in rebuttal.
Some that cannot directly be solved but were also openly reported by the authors relate to the sound quality, which may be limited in certain cases and may affect overall viewing experience (R1). Also, limitations for abstract sounds were mentioned, and for complex scenes, which cannot fully be overcome. Some improvement can be achieved with the optimization component.

Overall, I recommend accepting this paper given the ratings and especially universal appreciation of the merits, as well as a good rebuttal.
If space permits, I recommend "oral" presentation, but a "poster" could also be good.

In case of final acceptance of the paper, authors are strongly encouraged to make sure that all points mentioned in the rebuttal and further reviewer comments are considered for the camera-ready version of the paper.

- modification 08 July 2024 - updated to "accept as poster" after interaction with SAC -

Reviews: review: paper_summary: paper_summary: This paper presents AutoSFX, a novel system designed to automatically generate sound effects (SFX) for videos, ensuring audiovisual consistency, seamless transitions, and harmonious mixing. AutoSFX utilizes deep learning techniques, including the Segment Anything Model (SAM) for image segmentation and a diffusion model for sound generation. The system is composed of two main modules: Sound Generation and Sound Optimization. The Sound Generation module integrates audiovisual features to produce sound spectrograms, while the Sound Optimization module ensures the generated sounds align well with the visual content. The paper reports promising results from experiments conducted on the VEGAS and VGGSound datasets and a user study evaluating the usability and effectiveness of the system.
strength_weakness: strength_weakness: 1. Novelty: The integration of SAM for audiovisual feature extraction and the use of a diffusion model for sound generation represent innovative approaches in the field of automatic sound effect generation.
2. Technical Approach: The paper provides a detailed description of the system architecture, including the use of cross-modal attention and diffusion-based models, which are technically sound and state-of-the-art.
3. Evaluation: The system's performance is evaluated through extensive experiments on well-known datasets (VEGAS and VGGSound) and a user study, providing both quantitative and qualitative insights into its effectiveness.
4. Applications: The developed user interface and the potential applications in video creation and VR games demonstrate the practical relevance and impact of AutoSFX.
limitations: limitations: 1. Complex Scenarios: The system may struggle with complex scenarios involving multiple overlapping sound sources, such as crowded events or concerts. This limitation is acknowledged but not fully addressed in the paper.
2. Abstract Sound Generation: While the system excels at generating realistic sounds, it does not handle abstract sound effects used to enhance the atmosphere or convey emotions, which are often used in professional sound design.
3. Sound Quality: Some generated sounds, particularly high-frequency components, may lack clarity, affecting the overall viewing experience. This issue was highlighted in the user study feedback.
4. Limited Contextual Understanding: The system primarily focuses on audiovisual consistent objects and might not fully capture the broader context or thematic elements of the video, which could be crucial for a holistic sound design.
paper_summary: paper_summary: This paper proposed a sound effect generation system, AutoSFX, to automatically generate sounds guided by visual information. The proposed framework integrates multimodal representations via cross-attention and leverages a diffusion model to generate sound conditioned on visual cues. A user-friendly interface is also developed for the application. Experiments on the VEGAS and VGGSound test set verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Considering segmentation characteristics, such as leveraging SAM as the visual backbone, introducing segmentation loss, and training on the AVSBench dataset is interesting in the music/sound generation research community.

2. The authors conduct comprehensive qualitative and quantitative results to show the proposed algorithm outperforms previous methods. The user-friendly interface also makes the system applicable.

3. The writing is clear and easy to follow.
limitations: limitations: 1. Technical contribution is limited. Leveraging a latent diffusion model for generation tasks is not novel, and the audiovisual fusion/cross-attention module is also utilized in many previous audio-visual methods.

2. For the visual encoder, the authors select the recent advance, SAM, yet VGGish is adopted as the audio encoder. Why use VGGish rather than some more recent works such as Beats or AST? Is there any ablation? Besides, for the visual encoder, is there any ablation to show its effectiveness?

3. The authors analyze the audiovisual consistency, yet some existing music-video corresponding metrics need to be compared and discussed, such as beat hit rate, and beat coverage rate proposed in [1, 2]. For the visual rhythms, some previous works apart from [4] are also in need of discussion [2, 3].


Reference:  

[1] Zhu, Ye, et al. "Discrete contrastive diffusion for cross-modal music and image generation." arXiv preprint arXiv:2206.07771 (2022).  
[2] Yu, Jiashuo, et al. "Long-term rhythmic video soundtracker." International Conference on Machine Learning. PMLR, 2023.  
[3] Su, Kun, Xiulong Liu, and Eli Shlizerman. "How does it sound?." Advances in Neural Information Processing Systems 34 (2021): 29258-29273.  
[4] Davis, Abe, and Maneesh Agrawala. "Visual rhythm and beat." ACM Transactions on Graphics (TOG) 37.4 (2018): 1-11.
paper_summary: paper_summary: The proposed system, AutoSFX, aims to significantly streamline and broaden the applicability of automating sound design for videos. The key insight behind AutoSFX is that ensuring consistency between auditory and visual information, performing seamless transitions between sound clips, and harmoniously mixing sounds playing simultaneously is crucial for creating a unified audiovisual experience. AutoSFX capitalizes on this concept by aggregating multimodal representations through cross-attention and leveraging a diffusion model to generate sound with embedded visual information. Additionally, AutoSFX optimizes the generated sounds to render the entire soundtrack for the input video, leading to a more immersive and engaging multimedia experience.
strength_weakness: strength_weakness: AutoSFX introduces a new method for automated sound design in videos. It combines visual and auditory information using cross-attention and a diffusion model. This innovative approach leverages the Segment Anything Model (SAM) for audiovisual fusion, marking a significant breakthrough in the field. AutoSFX is thoroughly evaluated using both objective and subjective methods. The system's performance is validated on widely recognized datasets such as VEGAS and VGGSound, demonstrating its capability to generate high-quality sound effects. The paper clearly articulates the motivation, methodology, and contributions of AutoSFX. Detailed descriptions of the system architecture, including the Sound Generation and Sound Optimization modules, provide clarity on how the system functions.
limitations: limitations: The paper introduces innovative sound generation techniques for practical sound design. However, it heavily relies on existing models and techniques, such as the Segment Anything Model (SAM) and diffusion models, which raises concerns about the originality of the methodology.