Title: Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video-to-Audio Generation; Contrastive Audio-Visual Pretraining; Latent Diffusion Model;
Abstract: The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and adaptability via customized downstream finetuning. Project Page: https://diff-foley.github.io/

DIFF-FOLEY: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Abstract

The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present DIFFFOLEY, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CA VP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a crossattention module. We further significantly improve sample quality with 'double guidance'. DIFF-FOLEY achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning. Project Page: https://diff-foley.github.io/

Silent

Videos / Movies

AI Generated Audio

小

AI

1 Introduction

Recent advances in diffusion models [42; 45; 16] have accelerated the development of AI-generated contents, e.g . Text-to-Image (T2I) generation [37; 39; 38], Text-to-Audio (T2A) generation [48; 27], and Text-to-Video (T2V) generation [15]. This paper focuses on Video-to-Audio (V2A) generation, which has practical applications in video/film production. During live shooting, due to the presence of excessive background noise and challenges in the audio collection in complex scenarios, the majority

of the sounds recorded in the film need to be recreated during post-production. This process of adding synchronized and realistic sound effects to videos is known as Foley [1]. We compare traditional Foley performed in the studio and our neural Foley in Figure 1. Traditional Foley is a laborious and time-consuming process, that involves skilled artists manipulating objects to create hours of authentic physical sounds. In contrast, Neural Foley offers an appealing alternative. High-quality synchronized audio generated by AI can greatly accelerate video production and alleviate the human workload. Different from Neural Dubber [19], which synthesizes speech from text scripts and video frames, Neural Foley focus on generating a broad range of audio solely based on the video content, a task that poses a considerably higher level of difficulty.

Video-based audio generation offers two natural advantages over text-based generation while performing Foley. First, T2A requires lots of hard-to-collect text-audio pairs for training, in contrast, audio-video pairs are readily available on the Internet, e.g . millions of new videos are uploaded to YouTube daily. Second, along with the semantics of the generated audio, V2A can further control the temporal synchronization between the Foley audio and video.

Semantic content matching and temporal synchronization are two major goals in V2A. While some progress [35; 50; 3; 21; 41] have been made recently on V2A, most methods of audio generation focus only on the content relevance, neglecting crucial aspect of audio-visual synchronization. For example, given a video of playing drums, existing methods can only generate drums sound, but cannot ensure the sounds match exactly with what's happening in the video, i.e . hitting the snare drum or the crash cymbal at the right time.

RegNet [3] uses a pretrained (RGB+Flow) network as conditional inputs to GAN for synthesizing sounds. Meanwhile, SpecVQGAN [21] uses a Transformer [47] autoregressive model conditioned on pretrained ResNet50 [14] or (RGB+Flow) visual features for better sample quality. These methods have limitations in generating audio that is both synchronized and relevant to video content as pretrained image and flow features cannot capture the nuanced correlation between audio and video.

We present DIFF-FOLEY, a novel Neural Foley framework based on Latent Diffusion Model (LDM) [38] that synthesizes realistic and synchronized audio with strong audio-visual relevance. DIFFFOLEY first learns temporally and semantically aligned features via CAVP. By maximizing the similarity of visual and audio features in the same video, it captures subtle audio-visual connections. Then, an LDM conditioned on CAVP visual features is trained on the spectral latent space. CAVPaligned visual features help LDM in capturing audio-visual relationships. To further improve sample quality, we propose 'double guidance' , using classifier-free and alignment classifier guidance jointly to guide the reverse process. DIFF-FOLEY achieves state-of-the-art performance on large-scale V2A dataset VGGSound [2] with IS of 62.37, outperforming SpecVQGAN [21] baseline (IS of 30.01) by a large margin. We also demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning, validating the potential of V2A pretrained generative models.

2 Related Work

Video-to-Audio Generation Generating audio from silent videos has potential in video production, which can greatly improve post-production efficiency. Despite recent progress, open-domain V2A remains a challenge. FoleyGAN [11] uses a class-conditioned GAN for synchronized audio synthesis, but it requires a specific action class label and is tested on a limited dataset. RegNet [3] extracts video features by using a pretrained (RGB+Flow) network and serves as a conditional input for GAN to synthesize sounds. SpecVQGAN [21] uses more powerful Transformer-based autoregressive models for sound synthesis with extracted ResNet50 or RGB+Flow features. Im2Wav [41], current state-of-the-art, adopts a two-transformer model, using CLIP [36] features as the condition, yet suffers from slow inference speed, requiring thousands of inference steps. Existing V2A methods struggle with synchronization as they lack audio-related information in pretrained visual features, limiting the capture of intricate audio-visual correlations. A notable subfield, video-to-speech, has gained attention with works like vid2speech [7], Neural Dubber [19]. However, the foley task aims to generate complex and synchronized audio from video, presenting much higher challenge.

Contrastive Pretraining Contrastive pretraining [34; 13; 4] has shown potential in various generative tasks. CLIP aligns text-image representations using contrastive pretraining. For Text-to-Image, CLIP is integrated with Stable Diffusion [38] to extract text prompt features for realistic image generation. For Text-to-Audio, CLAP [6] aligns text and audio representations, and the resulting audio features

Video

Encoder

Audio

Encoder

P(

·

)

P(

·

)

Contrastive Objective

(Semantic + Temporal)

Contrastive Audio-Visual Pretraining (CAVP)

CAVP Aligned

Visual Feature

Audio Spectrogram

Video Frames

Audio Spectrogram

ℰ

Latent

𝑧

!

𝒟

Generated Spectrogram

Diffusion

Process

Reverse

Process

Denoising Objective

𝑧

"

~𝑁(0,𝐼)

Training

Inference

LDM Training with Aligned Visual Representation

𝜏

#

Latent Diffusion

Q

KV

Q

KV

are used in AudioLDM [27] for audio generation. Recognizing the importance of modality alignment in multimodal generation, we propose contrastive audio-visual pretraining (CA VP) to learn temporally and semantically aligned features initially and use them for subsequent generation tasks.

Latent Diffusion Model Diffusion Models [43; 16] have achieved remarkable success in various generative tasks including image generation [43; 16; 32; 37; 38], audio generation [25; 48; 27], and video generation [18; 15; 8]. Latent diffusion models, like Stable Diffusion (SD) [38] perform forward and reverse processes in data latent space, resulting in efficient computation and accelerated inference. SD is capable of running inference on personal laptop GPUs, making it a practical choice for a wide range of users.

3 Method

DIFF-FOLEY consists of two stages: Stage1 CAVP and Stage2 LDM training. Our model overview is shown in Figure 2. Audio and visual components in a video exhibit strong correlation and complementarity. Unfortunately, existing image or optical flow backbones (ResNet, CLIP etc .) struggle to reflect the strong alignment relationship between audio and visual. To address this, we propose Contrastive Audio-Visual Pretraining (CAVP) to align audio-visual features at the outset. Then, an LDM conditioned on CAVP-aligned visual features is trained on the spectral latent space.

3.1 Contrastive Audio-Visual Pretraining

Given a audio-video pair ( x a , x v ) , where x a ∈ R T ′ × M is a Mel-Spec. with M mel basis and T ′ is the time axis. x v ∈ R T ′′ × 3 × H × W is a video clip with T ′′ frames. An audio encoder f A ( · ) and a video encoder f V ( · ) are used to extract audio feature E a ∈ R T × C and video feature E v ∈ R T × C with same temporal dim T . We adopt the design of the audio encoder from PANNs [24], and SlowOnly [9] architecture for the video encoder. Using temporal pooling layer P ( · ) , we obtain temporal-pooled audio/video features, ¯ E a = P ( E a ) ∈ R C , ¯ E v = P ( E v ) ∈ R C . We then use a similar contrastive objective in CLIP [36] to contrast ¯ E a and ¯ E v . To improve the semantic and temporal alignment of audio-video features, we use two objectives: Semantic contrast L S and Temporal contrast L T .

For L S , we maximize the similarity of audio-visual pairs from the same video and minimize the similarity of audio-visual pairs from different videos. It encourages learning semantic alignment for audio-visual pairs across different videos. In specific, we extract audio-visual features pairs from different videos, B S = { ( ¯ E i a , ¯ E i v ) } N S i =1 , where N S is the number of different videos. We define the per-sample pair semantic contrast objective: L S ( i,j ) , where sim ( · ) is the cosine similarity.

For L T , we sample video clips at different times within the same video. It aims to maximize the similarity of audio-visual pairs from the same time segment and minimize the similarity of audiovisual pairs across different time segments. In details, we sample different time segments in the

same video to extract audio-visual feature pairs. B T = { ( ¯ E i a , ¯ E i v ) } N T i =1 , where N T is the number of sampled video clip within the same video. We define the per-sample pair temporal contrast objective:

The final objective is the weighted sum of semantic and temporal objective: L = L S + λ L T , where λ = 1 . After training, CA VP encodes an audio-video pair into embedding pair: ( x a , x v ) → ( E a , E v ) where ( E a , E v ) are highly aligned, with the visual features E v containing rich information for audio. The aligned and strongly correlated features E v and E a facilitate subsequent audio generation.

3.2 LDMwith Aligned Visual Representation

LDMs[38] are probabilistic models that fit the data distribution p ( x ) by denoising on data latent space. LDMs first encode the origin high-dim data x into low-dim latent z = E ( x ) for efficient training. The forward and reverse process are performed in the compressed latent space. In V2A generation, our goal is to generate synchronized audio x a given video clip x v . Using similar latent encoder E θ in [38], we compress Mel-Spec x a into a low-dim latent z 0 = E θ ( x a ) ∈ R C ′ × T ′ r × M r , where r is the compress rate. With the pretrained CAVP model to align audio-visual features, the visual features E v contain rich audio-related information. This enables synthesis of highly synchronized and relevant audio using LDMs conditioned on E v . We adopt a projection and positional encoding layer τ θ to project E v to the appropriate dim. In the forward process, origin data distribution transforms into standard Gaussian distribution by adding noise gradually with a fixed schedule α 1 , . . . , α T , where T is the total timesteps, and ¯ α t = ∏ t i =1 α i .

The goal of LDM is to mirror score matching by optimizing the denoisng objective [46; 16; 44]:

After LDM is trained, we generate audio latent by sampling through the reverse process with z T ∼ N (0 , I ) , conditioned on the given visual-features E v , with the following reverse dynamics:

Finally, the Mel-Spec. ˆ x a is obtained by decoding the generated latent z 0 with a decoder D , ˆ x a = D ( z 0 ) . In the case of DIFF-FOLEY, it generates audio samples with a duration of 8 seconds.

3.3 Temporal Split & Merge Augmentation

Using large-scale text-image pairs datasets like LAION-5B [40] is crucial for the success of current T2I models [38]. However, for V2A generation task, large-scale and high-quality datasets are still lacking. Further, we expect V2A model to generate highly synchronized audio based on visual content, such temporal audio-visual correspondence requires a large amount of audio-visual pairs for training. To overcome this limitation, we propose using Temporal Split & Merge Augmentation , a MixUp [49] like augmentation strategy to facilitate model training by incorporating prior knowledge for temporal alignment into the training process. During training, we randomly extract video clips of different time lengths from two videos ( Split ), denoted as ( x 1 a , x 1 v ) , ( x 2 a , x 2 v ) , and extract visual features E 1 v , E 2 v with pretrained CAVP model. We then create a new audio-visual feature pair for LDM training with:

where [ · ; · ] represent temporal concatenation ( Merge ). Split and merge augmentation greatly increase the number of audio-visual pairs, preventing overfitting and facilitating LDM to learn temporal correspondence. We validate the effectiveness of this augmentation method in Sec 4.1.2.

3.4 Double Guidance

Guidance techniques is widely used in diffusion model reverse process for controllable generation. There are currently two main types of guidance techniques: classifier guidance [5] (CG), and classifierfree guidance [17] (CFG). For CG, it additionally trains a classifier (e.g class-label classifier) to guide the reverse process at each timestep with gradient of class label loglikelihood ∇ x t log p ϕ ( y | x t ) . For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates [17], where the c is the condition and ω is the guidance scale. In stable diffusion [38], the CFG is implemented as:

When ω = 1 , CFG degenerates to conditional score estimates. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label. In V2A setting, the desired property refers to semantic and temporal alignment. Moreover, we discover that these two methods are not mutually exclusive. We propose a double guidance technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specific, for CG we train an alignment classifier P ϕ ( y | z t , t, E v ) that predicts whether an audiovisual pair is a real pair in terms of semantic and temporal alignment. For CFG, during training, we randomly drop condition E v with prob. 20%, to train conditional and unconditional likelihood ϵ θ ( z t , t, E v ) , ϵ θ ( z t , t, ∅ ) . Then double guidance is achieved by improved noise estimation:

where ω, γ refer to CFG, CG guidance scale respectively. We provide further analysis of the intuition and mechanism behind double guidance in Appendix C.

4 Experiments

Datasets We use two datasets VGGSound [2] and AudioSet [10]. VGGSound consists of ∼ 200K 10-second videos. We follow the original VGGSound train/test splits. AudioSet comprises 2.1M videos with 527 sound classes, but it is highly imbalanced, with most of the videos labeled as Music and Speech. Since generating meaningful speech directly from video is not expected in V2A tasks (not necessary either), we download a subset of the Music tag data and all other tags except Speech, resulting in a new dataset named AudioSet-V2A with about 80K music-tagged videos and 310K

Input Silent Video: Playing Golf

0s

1s

2s

3s

4s

5s

6s

7s

（

Hitting golf ball

）

Ground Truth

（

Hitting golf ball

）

Diff-Foley (Ours)

Im2Wav

Fails to generate

✘

Fails to generate

✘

✓

✓

other tagged videos. We use VGGSound and AudioSet-V2A for Stage1 CAVP, while for Stage2 LDM training and evaluation, we only use VGGSound, which is consistent with the baseline.

Data Pre-Processing For DIFF-FOLEY training, videos in VGGSound and AudioSet-V2A are sampled at 4 FPS, which already yields significantly better alignment results compared to the baseline method [21; 41] using 21.5 or 30 FPS. Each 10-second video sample consists of 40 frames are resized to 224 × 224 . For audio data, the original audio is sampled at 16kHz and transformed into Mel-spectrograms (Mel Basis M = 128 ). For Stage1 Pretraining, we use hop size 250 for better audio-visual data temporal dimension alignment, while we use hop size 256 for Stage2 LDM training.

Model Configuration and Inference For CAVP, we use a pretrained audio encoder from PANNs [24] and a pretrained SlowOnly [9] based video encoder. For training, we randomly extract 4-second audiovideo frames pairs from 10-second samples, resulting in x a ∈ R 256 × 128 and x v ∈ R 16 × 3 × 224 × 224 . We use temporal contrast L T with N T = 3 and a minimum time difference of 2 seconds between each pair. For LDM training, we utilize the pretrained Stable Diffusion-V1.4 (SD-V1.4) [38] as a powerful denoising prior model. Leveraging the pretrained SD-V1.4 greatly reduces training time and improves generation quality (discussed in Sec 4.3). Frozen pretrained latent encoder E and decoder D from SD-V1.4 are used. Interestingly, despite being trained on image datasets (LAION-5B), we found E , D can encode and decode Mel-Spec well. For inference, we use DPM-Solver [30] sampler with 25 sampling steps, unless otherwise stated. More training details for each model are in Appendix. A.

Evaluation Metrics For evaluation, we use Inception Score (IS), Frechet Distance (FID) and Mean KL Divergence (MKL) from [21]. IS assesses sample quality and diversity, FID measures distributionlevel similarity, and MKL measures paired sample-level similarity. We introduce Alignment Accuracy (Align Acc) as a new metric to assess synchronization and audio-visual relevance. We train an alignment classifier to predict real audio-visual pairs. To train the classifier, we use three types of pairs: 50% of the pairs are real audio-visual pairs ( true pair ) labeled as 1, 25% are audio-visual pairs from the same video but temporally shifted ( temporal shift pair ) labeled as 0, and the last 25% are audio-visual pairs from different videos ( wrong pair ) labeled as 0. Our alignment classifier reaches 88 . 31% accuracy on test set. The detailed analysis of alignment classifier is provided in Appendix. A.1. We prioritize IS and Align Acc as the primary metrics to evaluate sample quality. For evaluation, we generate ∼ 145K audio samples (10 samples per video in the test set). We emphasize the difference between the noisy alignment classifier F DG ϕ used for double guidance techniques and the alignment classifier F sync θ used for Align Acc metric evaluation in Appendix A.1.

d. ResNet50 Visual Feature

b. CLIP Visual Feature

c. CAVP Visual Feature (Ours)

1s

2s

3s

4s

5s

6s

7s

Baseline For comparison, we use two current state-of-the-art V2A models: SpecVQGAN [21] and Im2Wav [41]. SpecVQGAN offers two model settings based on different visual features (RGB+Flow and ResNet50), while Im2Wav generates semantically relevant audios using CLIP features. We use the pretrained baseline models for evaluation, which were both trained on VGGSound.

4.1 Video-to-Audio Generation Results

Table 1 presents the quantitative results on VGGSound test set and model inference time. DIFFFOLEY outperforms the baseline method significantly in IS and Align Acc (primary metrics), while maintaining comparable performance on MKL/FID. DIFF-FOLEY achieves twice the performance of baseline on IS ( 62.37 v.s ∼ 30 ) and an impressive 94.05% Align Acc. Im2Wav's slow inference speed hinders its practicality despite its advantages in KL metrics. In contrast, DIFF-FOLEY utilizes the state-of-the-art diffusion sampler DPM-Solver [30], generating 64 samples per batch at an average of 0.38 seconds per sample with 25 inference steps (compared to Im2Wav 8192 autoregressive steps). We included the KL and FID metrics for comprehensive analysis despite their unreliability and potential misalignment with human perceptions. The suboptimal results on these metrics might be due to using the pretrained autoencoder in SD-v1.4 [38], leading to reconstruction errors. See Appendix. D for more details. We conducted a human evaluation in Table 2. The generated audios are rated based on content relevance and synchronization on a scale of 1 (bad) to 5 (excellent), then multiplied by 20. The detailed human evaluation procedure is provided in Appendix. A.4. The results in Table 2 highlight DIFF-FOLEY's excellence in audio-visual synchronization and relevance. We also observe that content relevance and synchronization assessments from our classifier closely align with those from human evaluators, confirming the effectiveness of our sync classifier.

Figure 3 presents the qualitative results. DIFF-FOLEY demonstrates a remarkable ability to generate highly synchronized audio compared with baseline methods. Given a silent golf video, DIFF-FOLEY successfully generates the corresponding hitting sound at the 4th second, while baseline methods fail to generate sound at that specific second. This demonstrates the superior synchronization ability of DIFF-FOLEY. We emphasize that our aim is to generate audios that align well with human perception, even if they differ from the ground truth audio. More generated results can be found at this link. 2 .

4.1.1 Visual Features Analysis

CAVP uses semantic and temporal contrast objectives to align audio and visual features. We evaluate the effectiveness of CAVP visual features by comparing them with other visual features in LDM training. Results in Table 3 show that CAVP visual features significantly improve synchronization and audio-visual relevance (Align Acc), verifying the effectiveness of CA VP features. Despite CLIP

Cutting Things with Knife

Ground Truth Audio

Generated Audio (Ours)

0s

1s

2s

3s

4s

5s

6s

7s

0s

1s

2s

3s

4s

5s

6s

7s

Washing Dishes

Ground Truth Audio

（

Cutting sound

）

（

Cutting sound

）

（

Water sound stops

）

（

Water sound stops

）

（

Plate clinking

）

（

Plate clinking

）

Generated Audio (Ours)

features exhibiting advantages in IS and KL, it does not accurately reflect synchronization capability. We expect such a gap can be bridged by expanding CAVP datasets to a larger scale. In the drumming video example shown in Figure 4, different visual features are utilized to generate audio. In this example, the drum is hit four times (clear spikes in the ground truth, see a.), while ResNet and CLIP features fail to generate synchronized audio, CA VP features successfully learn the relationship between drumming moments and drum sounds, generating audio with 4 clear spikes (see c.), showing excellent synchronization capability.

4.1.2 Temporal Augmentation Analysis

We evaluate the impact of Temporal Split & Merge Augmentation (discussed in Sec 3.3) in Table 4. Our results demonstrate improvements across all metrics using Temporal Split & Merge Augmentation , particularly for Align Acc metric. This augmentation technique leverages prior knowledge of audiovisual temporal alignment, enhancing the model's synchronization capability and increasing the number of audio-visual training pairs, thus mitigating overfitting.

(b). Different Samplers

SI

Align Acc (%)

55

+InceptionScore(IS)

95

AlignmentAccuracy(AlignAcc)%

50

90

0

40

85

86

35

25

20

30

80

70

75

35

30

PLMS

DPMSolver

DDIM

B4

82

80

78

PLMS

DPMSolver

DDIM

1

2

3

4

CFG Scale

5

6

7

8

9

2

3

4

CFG Scale

5

6

7

8

9

10 2550

100

Steps

FID

250

10 2550

100

SteDs

KL

250

17

t

FrechetDistance(FiD)

6.6

KL Divergence (KL)

20

22

DDIM

PLMS

6.9

7.0

DDIM

PLMS

15

6

6.4

$'9

DPMSolver

6.8

DPMSolver

14

6.3

16

6.6

13

12

Z9

14

S'9

11

12

6.3

2

3

CFG Scale

5

6

8

3

4

CFG Scale

6

7

10 2550

100

Steps

250

10 2550

100

250

Steps

4.2 Downstream Finetuning

Background Foundation generative models like Stable Diffusion [38], trained on billions of data, excel at creating realistic images, but struggle with generating specific styles or personalized images. Finetuning techniques help overcome such limitations, aiding in specific-style synthesis and enhancing adaptability. Similarly, DIFF-FOLEY is expected to adapt to other datasets via downstream finetuning to synthesize specific types of sound, as evaluated in this section.

Finetuning Details We finetuned DIFF-FOLEY on EPIC-Kitchens [20], a high quality egocentric video dataset( ∼ 100 hours) about object interactions in kitchen with minimal sound noises. EPICKitchens is notably distinct from VGGSound. This experiment validates the adaptability of finetuned DIFF-FOLEY. We used a pretrained CAVP model to extract visual features, and then finetuned Stage2 LDM (trained on VGGSound) for 200 epochs. See more finetuning details in Appendix. A.

Generation Results We show qualitative results of finetuned LDM on EPIC-Kitchens in Figure 5. DIFF-FOLEY generates highly synchronized audio with original video, effectively capturing the timing of sounds, e.g . knife cutting, water flow, and plate clinking. Best viewed in demo link 2.

4.3 Ablation Study

Model Size & Param Init. We explore the impact of model size and initialization with pretrained Stable Diffusion-V1.4 (SD-V1.4) on DIFF-FOLEY. We trained three models of varying sizes: DIFFFOLEY-S (335M Param.), DIFF-FOLEY-M (553M Param.), and DIFF-FOLEY-L (859M Param. , the same architecture as SD-V1.4). We report detailed architectures of these models in Appendix. A Our results in Table 5 show that increasing model size improves performance across all metrics,

demonstrating DIFF-FOLEY's scaling effect. Moreover, initializing with SD-V1.4 weights (pretrained on billions of images), significantly improves performance and reduces training time due to its strong denoising prior in image/Mel-spectrogram latent space (see the last row in Table 5).

Guidance Techniques & Pretrained Dataset Scale We conduct extensive ablation study on DIFFFOLEY in Table 6, exploring the scale of Stage1 CA VP datasets and guidance techniques. We see that: 1). More CAVP pretraining data enhances downstream LDM generation performance. 2). Guidance techniques greatly improve all metrics except KL 3). Double guidance techniques achieve the best performance on IS and Align Acc (the primary metrics).

CFG Scale & Different Sampler We studied the impact of CFG scale ω and different samplers on DIFF-FOLEY. CFG scale influences the trade-off between sample quality and diversity. A large CFG scale typically enhances sample quality but may have a negative effect on diversity (FID, KL). Beyond a certain threshold, increased ω can decrease quality due to sample distortion, resulting in U-Shaped curves in Figure 6 (a). Best results in IS and Align Acc were achieved at ω = 4 . 5 , with minor compromise in FID and KL. Faster Sampling for diffusion models, reducing inference steps from thousands to tens, is a trending research area. We evaluated the usability of our DIFF-FOLEY using current accelerated diffusion samplers. We compare three different samplers, DDIM [43], PLMS [28], DPM-Solver [30] with different inference steps in Figure 6 (b). All three samplers converge at 250 steps, with DPM-Solver converging in only 25 inference steps, while others require more steps to converge. DPM-Solver allows for faster DIFF-FOLEY inference, making it more accessible.

5 Limitations and Broader Impact

Limitations DIFF-FOLEY has shown great audio-visual synchronization on VGGSound and EPICKitchens, however its scalability on super large (billion-scale) datasets remains untested due to limited data computation resources. Diffusion models are also slower than GANs due to the iterative reverse sampling process.

Broader Impact The advancements in V2A models, such as DIFF-FOLEY, have the potential to significantly expedite video production processes, offering efficiency gains in the entertainment and media sectors. However, as with most powerful technologies, there's a flip side. There's an underlying risk of these models being misused to create misleading or false content. As such, while the potential benefits are vast, it is imperative for developers, users, and regulators to exercise caution.

6 Conclusion

We introduce DIFF-FOLEY, a V2A approach for generating highly synchronized audio with strong audio-visual relevance. We empirically demonstrate the superiority of our method in terms of generation quality. Moreover, we show that using double guidance technique to guide the reverse process in LDM can further improve the audio-visual alignment of generated audio samples. We demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning. Finally, we conduct an ablation study, analyzing the effect of pretrained dataset size, various guidance techniques, and different diffusion samplers.

References

Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation . Routledge, 2021.

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721-725. IEEE, 2020.

Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing , 29:8292-8302, 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Ariel Ephrat and Shmuel Peleg. Vid2speech: speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5095-5099. IEEE, 2017.

Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 , 2023.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 62026211, 2019.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 776-780. IEEE, 2017.

Sanchita Ghose and John J Prevost. Foleygan: Visually guided generative adversarial network-based synchronous sound generation in silent videos. IEEE Transactions on Multimedia , 2022.

Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acoustics, speech, and signal processing , 32(2):236-243, 1984.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Chenxu Hu, Qiao Tian, Tingle Li, Wang Yuping, Yuxuan Wang, and Hang Zhao. Neural dubber: dubbing for videos according to scripts. Advances in neural information processing systems , 34:16582-16595, 2021.

Jaesung Huh, Jacob Chalk, Evangelos Kazakos, Dima Damen, and Andrew Zisserman. EPIC-SOUNDS: A Large-Scale Dataset of Actions that Sound. In IEEE International Conference on Acoustics, Speech, Signal Processing (ICASSP) , 2023.

Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arXiv:2110.08791 , 2021.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems , 33:17022-17033, 2020.

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2880-2894, 2020.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020.

Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems , 32, 2019.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 , 2023.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

ALYSSA MAIO. What is a foley artist - how to bring movies to life with sound, 2022. https: //www.studiobinder.com/blog/what-is-a-foley-artist/ .

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2405-2413, 2016.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. arXiv preprint arXiv:2211.03089 , 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems , 34:1415-1428, 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3550-3558, 2018.

Appendix

The Appendix is organized as follows:

Sec. A elaborates the implementation details of DIFF-FOLEY, including model architectures, data preprocessing and training/evaluation details for different stages models and tasks.

Sec. B presents additional visualization results of Video-to-Audio generation on the VGGSound [2] dataset and downstream finetuning results on EPIC-Kitchens [20] dataset, further demonstrating the superior performance of DIFF-FOLEY in synchronized audio generation.

Sec. C provides theoretical and intuitive perspectives on the mechanism of Double Guidance techniques discussed in Sec. 3.4.

Sec. D examines the reconstruction capability of the Stable Diffusion [38] frozen pretrained Encoder E and Decoder D on Mel-spectrograms.

A Implementation Details

A.1 Model Architectures

Our proposed DIFF-FOLEY employs a two-step procedure for synchronized Video-to-Audio generation. The first stage, Contrastive Audio-Visual Pretraining (CA VP), aims to learn temporally and semantically aligned audio-visual features to capture the subtle connection between the audio and visual modalities. The second stage involves training latent diffusion models (LDMs) conditioned on the CAVP-aligned visual features in the Mel-spectrogram latent space. In this section, we present detailed model architectures for each of the model components in DIFF-FOLEY.

Stage1 CAVP. CAVP adopts a similar two-stream encoder design as CLIP [36], consisting of an audio encoder f A ( · ) and a video encoder f V ( · ) . For the audio encoder f A ( · ) , we adopt the Mel-spectrogram encoder design from PANNs [24], which is a 2D convolution-based encoder with ∼ 81M parameters. For the video encoder f V ( · ) , we use the SlowOnly branch design from the SlowFast [9] architecture, which is a 3D convolution-based backbone with ∼ 30M parameters. During CAVP training, we incorporate the pretrained weights of the audio encoder from PANNs [24], which were initially trained on the large-scale AudioSet [10] dataset. Additionally, we adopt the pretrained weights of the video encoder from SlowFast [9], which were trained on the Kinetics-400 [22] dataset. For audio and video input pairs ( x a , x v ) , where x a ∈ R T ′ × M is a Mel-spectrogram with M mel basis and T ′ is the time axis of the spectrogram, and x v ∈ R T × 3 × H × W is a video clip with T frames. The audio encoder f A ( · ) and video encoder f V ( · ) are used to extract audio feature E a ∈ R T × C and video feature E v ∈ R T × C . Specifically, the audio encoder f A ( · ) encodes the spectrogram to match the time dimension of the video frames. In CAVP, we set the feature dimension C = 512 . By using a Temporal Max-Pooling Layer P ( · ) , we obtain temporal-pooled audio and visual features ¯ E a = P ( E a ) ∈ R C , ¯ E v = P ( E v ) ∈ R C .

Stage2 LDM. For LDM, we use the same UNet backbone architecture in Stable Diffusion-V1.4 (SD-V1.4) [38]. Using the frozen pretrained latent encoder E , we encode a spectogram into a low dimensional latent z 0 = E θ ( x a ) ∈ R C ′ × T ′ r × M r , where the channel number of C ′ = 4 and a compression rate of r = 8 . To successfully encode a single channel spectrogram x a into a latent z 0 with the pretrained frozen 3-channel image encoder E in SD-V1.4, we first repeat the spectrogram channel into 3 ( x a → ˜ x a ∈ R 3 × T ′ × M ), and then obtain z 0 as z 0 = E θ (˜ x a ) . To decode a spectrogram from latent z 0 with the frozen image decoder D in SD-V1.4, we simply use the first channel from the decoded spectrogram ( ˆ x a = D ( z 0 )[0 , : , :] ∈ R T ′ × M , where D ( z 0 ) ∈ R 3 × T ′ × M , [: , : , :] is the tensor slicing operation). The reconstruction capability of the pretrained frozen image latent encoder E and decoder D on Mel-spectrograms is discussed in Sec. D. With the pretrained CAVP model to align audio-visual features, the visual aligned features E v ∈ R T × C are adopted in LDM training. To better learn the temporal relationship between audio and visual modality, we adopt a positional encoding and projection layer τ θ to project E v to appropriate dimension with ˜ E v = τ θ ( E v ) = MLP ( E v + PE ) ∈ R T × C ′ , where MLP is the projection layer and PE represents positional encoding. We set C ′ = 768 for all experiments. We use the same denoising schedule α 1 , · · · , α T in SD-V1.4 [38], with T = 1000 . To learn the audio-visual correlation, we adopt the cross-attention

module from SD-V1.4. Specifically, we apply cross-attention on latents with down-sampling rates of [1 , 2 , 4] .

In Sec. 4.3 of the main paper, we use three different model settings of varying sizes: DIFF-FOLEY-S (335M parameters), DIFF-FOLEY-M (553M parameters), and DIFF-FOLEY-L (859M parameters). All three models use the UNet backbone with four encoder blocks, a middle block, and four decoder blocks. For DIFF-FOLEY-S, with a basic channel number of c = 192 , the channel dimensions of the encoder blocks are [ c, 2 c, 3 c, 4 c ] , and the design of the decoder blocks are basically the reverse of the encoder blocks. For DIFF-FOLEY-M, we set c = 256 , and the channel dimensions of the encoder blocks are [ c, 2 c, 4 c, 4 c ] . Finally, for DIFF-FOLEY-L, we adopt the same architecture as in SD-V1.4 [38], resulting in c = 320 and channel dimensions of the encoder blocks [ c, 2 c, 4 c, 4 c ] . We have observed a significant improvement in performance in Sec. 4.3 when using the pretrained weights from SD-V1.4 for parameter initialization. Therefore, we default to using the DIFF-FOLEY-L model and the pretrained weights from SD-V1.4 unless otherwise specified.

Vocoder. Vocoder is used to transform Mel-spectrogram into waveform signal. There are two categories of vocoders: Traditional algorithm-based vocoders such as the Griffin-Lim algorithm [12], which uses an iterative refining strategy to estimate the original sound signal, and deep learning-based vocoders such as WaveNet [33], MelGAN [26], and HiFi-GAN [23], which are widely used for speech waveform generation. Although our method is fully compatible with deep learning-based vocoders, we choose to use the Griffin-Lim algorithm for audio waveform reconstruction in this paper due to its simplicity.

Alignment Classifier for Align Acc Metric. To evaluate the generated audio synchronization and audio-visual relevance, we introduce Alignment Accuracy (Align Acc.) as a new metric in Sec. 4. In specific, we train an alignment classifier F θ to predict real audio-visual pairs. To train the classifier, we use three types of audio-visual pairs. (1). 50% of the pairs are the real audio-visual pairs from the same video and the same time ( true pair ), labeled as 1; (2). 25% are audio-visual pairs from the same video but temporally shifted, resulting in an out-of-sync audio-video pair ( temporal shift pair ), labeled as 0. (3). The remaining 25% are audio-visual pairs from different videos, resulting in completely mismatched audio-visual pairs ( wrong pair ), labeled as 0. This dataset comprises of 50% aligned audio-visual pairs and 50% unaligned audio-visual pairs (temporal and semantic). By training a classifier on this audio-visual pairs dataset, we can evaluate both audio-visual synchronization and audio-visual relevance using the alignment probability predicted by the alignment classifier F θ . We use the VGGSound [2] datasets, and our alignment classifier F θ achieves 88 . 31% accuracy on test set. The detailed metrics of our alignment classifier is Recall: 84 . 92% , Precision: 91 . 32% , and Accuracy: 88 . 31% . For the model architecture of the alignment classifier F sync θ , we adopt the lightweight U-Net encoder block design in SD-V1.4. The basic channel number is set to c = 128 , and the channel dimensions of the encoder blocks are [ c, 2 c, 2 c ] . Cross-attention is applied to latents with down-sampling rates of [2 , 4] and the model has ∼ 12M parameters. The alignment classifier F sync θ takes two input. The encoded spectrogram latent z 0 , and the visual aligned features E v from Stage1 CAVP. The predicted alignment label ˆ y is computed as follows: ˆ y = F θ ( z 0 , E v ) .

Here, we emphasize that the the alignment classifier F sync θ for align acc metric is different from the sync classifier F DG ϕ used in double guidance techniques. When utilizing the double guidance techniques, we did not access the alignment classifier that used for accuracy measurement. In specific, the classifier F DG ϕ used in double guidance is a noisy classifier, and it takes the noisy latent z t , time embedding t , and visual aligned features E v as input, the predicted alignment label ˆ y is computed as follows: ˆ y = F DG ϕ ( z t , t, E v ) . While the sync classifier F sync θ for accuracy measurement, it only takes two input, latent z 0 and visual aligned features E v to predict the synchronization label with ˆ y = F sync θ ( z 0 , E v ) .

A.2 Data Processing

In this section, we provide additional information on data processing for reproducibility.

Video-to-Audio Generation. To train Stage1 CAVP, we use the VGGSound [2] and AudioSet-V2A [10] datasets. The videos are sampled at 4 FPS, resulting in 40 frames for each 10-second video sample. The frames are then resized to 224 × 224 . For audio data, the audios are sampled at 16KHz and then transformed into Mel-spectrograms, using FFT Num 1024, mel basis Num 128 and hop size 250. This results in a 10-second spectrogram with the size of 640 × 128 ( T ′ × M ) . Weuse a hop size of

250 in CAVP for better audio-visual data temporal dimension alignment. To train CA VP, we randomly extract 4-second audio-video frame pairs from the 10-second samples, resulting in x a ∈ R 256 × 128 and x v ∈ R 16 × 3 × 224 × 224 . We then pass the audio-visual pairs through the audio encoder f A ( · ) and video encoder f V ( · ) respectively. This results in E a ∈ R 16 × 512 and E v ∈ R 16 × 512 . In Stage2 LDM training, we only use the VGGSound datasets, which is consistent with the baseline setting. We use ∼ 8 second audio and video pairs for training LDM. In specfic, we use FFT Num 1024, mel basis Num 128, and hop size 256 to transform the ∼ 8 seconds audios into Mel-spectrograms with the size 512 × 128 ( T ′ × M ) , resulting in x a ∈ R 512 × 128 and x v ∈ R 32 × 3 × 224 × 224 . The LDMs takes the encoded spectrogram noisy latent z t ∈ R 4 × 16 × 64 , time embedding t , and CAVP-aligned visual features E v ∈ R 32 × 512 as input: ϵ θ ( z t , t, E v ) .

Downstream Finetuning. For downstream finetuning, we utilized the high-quality egocentric video dataset, EPIC-Kitchens [20], which contains approximately 100 hours of daily activities and object interactions in kitchens with minimal sound noise. The sound categories in EPIC-Kitchens are significantly different from those in VGGSound [2]. For dataset processing, we randomly divided the original video dataset into training and testing sets, with 90% of the total duration assigned to the training set and 10% to the testing set. We split the original videos into 10-second segments for each dataset, resulting in about 26k training samples and 2.9k testing samples. It's important to note that the video clips in the training and testing sets were sourced from different videos and do not overlap. The processing steps for the datasets were the same as those described in Sec. A.2. Notably, we did not finetune the Stage1 CAVP models, but instead directly adopted the pretrained CAVP model to extract aligned visual features for EPIC-Kitchens videos. Subsequently, we finetune the Stage2 LDM, which was originally trained on VGGSound, using pairs of extracted CAVP visual features and audio from EPIC-Kitchens datasets.

A.3 Training Details

CAVP Training We train the CAVP model for 1.4M steps on 8 A100 GPUs, with a total batch size of 720 using automatic mixed-precision training (AMP). We used the AdamW [29] optimizer with a learning rate of 8e-4 and 200 steps of warmup. The total training time for the CA VP model was approximately 80 hours.

LDMTraining We initialize the LDM model with pretrained weights from SD-V1.4 [38] and train LDM for 24.4K steps on 8 A100 GPUs with a total batch size of 1760. We used the AdamW [29] optimizer with a learning rate of 1e-4 and 1000 steps of warmup. The total training time for DIFF-FOLEY-L is approximately 60 hours.

Downstream Finetuning For downstream finetuning, we finetune the DIFF-FOLEY-L model (trained on VGGSound) on EPIC-Kitchens for 3.2K steps on 8 A100 GPUs with a total batchsize 1760. We use the AdamW [29] optimizer with the learning rate 1e-4 and 1000 steps of warmup. The total downstream finetuning time is around 9 hours.

A.4 Evaluation Details

For metric evaluation , we use Inception Score (IS), Frechet Distance (FID) and Mean Kullback-Leibler Divergence (MKL) from [21]. For comparison, we use two current state-of-the-art V2A models: SpecVQGAN [21] and Im2Wav [41]. We use the pre-trained models provided by the authors, and these models are both trained on VGGSound [2] datasets. SpecVQGAN [21] utilizes videos with 21.5 FPS to generate 10-second audio samples conditioned on 10-second video features, while Im2Wav [41] utilizes videos with 30 FPS to generate 4-second audio samples conditioned on 4-second video features. In contrast, our DIFF-FOLEY generates 8-second audio conditioned on 8-second video features. To ensure a fair comparison, for SpecVQGAN, we generate 10s audio samples and truncate them to the first 8 seconds. For Im2Wav, we generate the first 4 seconds of audio based on the features extracted from the first 4 seconds of the video, followed by generating the (4s-8s) audio conditioned on the (4s-8s) video features. We then merge the two audio segments to obtain the final 8 seconds audio output. Lastly, we compare the generated 8-second audio samples with the ground truth Mel-spectrogram. Since our generated Mel-spectrogram has a Mel basis num of 128, while the baseline ground truth Mel-spectrogram has a Mel basis num of 80, we transform our generated Mel-spectrogram into 80 Mel basis num for a fair comparison. For each video sample in the test set, we generate 10 audio samples for evaluation, resulting in around 145 K generated audio

samples. For human evaluation , we conducted a human evaluation by randomly selecting 60 videos from the VGGSound test set and having different models to generate corresponding audio samples. The output and ground truth audios were anonymized and rated by 30 people unfamiliar with the project. Each sample received scores from at least 5 raters, ranging from 1 (bad) to 5 (excellent) for content relevance and synchronization. The scores were then scaled by a factor of 20.

B More Generation Results on Video-to-Audio Generation

B.1 V2A Generation Results on VGGSound.

We present additional generation results for Video-to-Audio generation on VGGSound dataset in Figure 7. We observe that DIFF-FOLEY effectively generates the corresponding sounds (e.g., gunshots and underwater bubbling) at the appropriate time, which matches the ground truth audio. In contrast, other methods fail to generate highly synchronized audio that aligns with the video content. This supports the superior performance of our approach. For more details, please visit our demo websites.

B.2 Downstream Finetuning Results on EPIC-Kitchens.

We have included additional downstream finetuning results on the EPIC-Kitchens dataset in Figure 8. Our observations indicate that DIFF-FOLEY generates highly synchronized audio that effectively captures the timing of sounds with the original video. In the first video, DIFF-FOLEY produces the corresponding sound of opening drawer at both the 1st, 3rd seconds. In the second video, it successfully generates the sound of plates clinking and placing at both the 1st, 4th seconds. For more details, please visit our demo websites. These results demonstrate the potential of DIFF-FOLEY to adapt to other video datasets, making it easier and more promising for future users to generate specific types of audio data through downstream finetuning.

C Guidance Techniques

C.1 Guidance Techniques Overview

Guidance techniques are commonly utilized in the reverse process of the diffusion model to achieve controllable generation. Currently, there are two main types of guidance techniques: Classifier Guidance [5] (CG) and Classifier-Free Guidance [17] (CFG). For CG, it additionally trains a classifier p ϕ ( e.g . class-label y classifier) to guide the reverse process at each timestep using the gradient of the class label loglikelihood ∇ x t log p ϕ ( y | x t ) , resulting in the conditional score estimates [5]:

, where γ is the CG scale. For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates [17], where the c is the condition and ω is the CFG scale.

When ω = 1 , CFG degenerates to conditional score estimate. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label y . For instance, if we want to generate images with specific attributes ( e.g . a girl with yellow hair) from a pretrained image diffusion model, we can simply train a 'girl with yellow hairs' classifier to guide the generation process. In V2A setting, the desired property refers to audio semantic and temporal alignment.

In Sec. 3.4, we discover that the CG and CFG methods are not mutually exclusive. We then propose a double guidance (DG) technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specfic, for CG we train an noisy alignment classifier P ϕ ( y | z t , t, E v ) that predict whether an audio-visual pair is a real pair in terms of semantic and temporal alignment. The noisy alignment classifier P ϕ ( y | z t , t, E v ) takes noisy latent z t , time embedding t , and the CAVP-aligned visual features E v as input. We follow similar training process in Sec. A.1 (see Align Acc.) to train the noisy alignment classifier P ϕ ( y | z t , t, E v ) . For CFG, during training, we randomly drop condition E v with probability 20% to estimate the conditional

Input Silent Video: Gun Shooting

Ground Truth

Im2Wav

Diff-Foley (Ours)

（

Gun Shot

）

Fails to generate

✘

✓

Fails to generate

✘

（

Gun Shot

）

✓

0s

1s

2s

3s

4s

5s

6s

7s

Ground Truth

Im2Wav

Diff-Foley (Ours)

（

Under Water

）

✓

（

Under Water

）

✓

Fails to generate

✘

Fails to generate

✘

0s

1s

2s

3s

4s

5s

6s

7s

score ϵ θ ( z t , t, E v ) and unconditional score ϵ θ ( z t , t, ∅ ) . Then double guidance is achieved by the improved noise estimation:

where ω, γ refer to CFG, CG guidance scale respectively.

Open Drawer

Ground Truth Audio

Generated Audio (Ours)

（

Open Drawer

）

（

Open Drawer

）

（

Open Drawer

）

（

Open Drawer

）

0s

1s

2s

3s

4s

5s

6s

7s

（

Plate clincking

）

（

Plate clincking

）

（

Drop the plate

）

（

Drop the plate

）

Ground Truth Audio

Generated Audio (Ours)

0s

1s

2s

3s

4s

5s

6s

7s

C.2 Theoretical Perspective of Double Guidance

Although the approach of double guidance may appear heuristic, we provide a theoretical perspective for explaining the underlying mechanism behind this techniques. The goal of guidance techniques is to generate sample from the conditional distribution: p ( z t | y ) , where z t is the generated noisy sample at each reverse process timesteps and the y is the condition or specific attribute. In diffusion models, the ϵ θ ( z t , y, t ) estimate the conditional score function, which is ∇ z t log p ( z t | y ) .

For CG , we first derive the mechanism behind CG [5] method to show how CG help in controllable generation. With the bayes's formula, we have:

, where the ∇ z t log p ( y | z t ) is actually the classifier guidance (CG) term. To enhance the conditional singal y , we scale the ∇ z t log p ( y | z t ) term by a factor γ , also known as CG scale, then we obtain the CG improved score estimation in Equation 10.

For CFG , we also derive the mechanism behind CFG [17] method. To improve the conditional signal c , CFG use the mixture of conditional score ∇ z t log p ( z t | c ) and unconditional score ∇ z t log p ( z t ) to enhance the term ∇ z t log p ( c | z t ) . Using the bayes formula, we have:

, then we scale the ∇ z t log p ( c | z t ) with a factor of ω (CFG scale), and substitute to Equation 14, then we obtain the CFG improved score estimation in Equation 11.

For double guidance (DG) , we construct a more fine-grained and controllable conditional distribution p ( z t | c, y ) that takes into account two conditions, c and y , where c is the general condition and y is another data attribute label. Specifically, in DIFF-FOLEY, c refers to the CAVP-aligned visual features E v ( general condition ), while y refers to a more fine-grained and desired property, such as semantic and temporal alignment ( data attribute ). DIFF-FOLEY aims to generate high-quality synchronized audio with strong audio-visual relevance from this distribution z t ∼ p ( z t | c, y ) . To simplify the derivation, we assume that p ( y | z t ) and p ( c | z t ) are independent conditioned on z t . Using the Bayes formula, we have:

, we then use CG scale γ and CFG scale ω to scale the CG term ∇ z t log p ( y | z t ) and CFG term ∇ z t log p ( c | z t ) respectively:

Finally, we obtain the double guidance (DG) improved score estimation in Equation 12.

( DG ) : ˆ ϵ θ ( z t , t, E v ) ← ωϵ θ ( z t , t, E v ) + (1 -ω ) ϵ θ ( z t , t, ∅ ) -γ √ 1 -¯ α t ∇ z t log P ϕ ( y | z t , t, E v ) (21) , where c is the CAVP-aligned visual features E v , y is the semantic and temporal alignment label, ω

is CFG scale and γ is the CG scale.

C.3 Intuition Perspective of Double Guidance

We offer another intuitive perspective to explain the technique of double guidance . In practice, the CFG [17] method tends to outperform the CG [5] method. This is because the classifier in the CG method may learn shortcuts for classification, leading to good classification results, but when its loglikelihood gradient is used to guide the reverse process, it can generate incorrect samples. In contrast, the double guidance technique addresses this issue by leveraging both the CFG and CG terms. The CFG term provides a robust and reliable noise direction for each timestep in the reverse process, while the CG term refines the CFG term's direction towards the desired data attribute, resulting in better sample quality. The effectiveness of the double guidance technique is validated by the evaluation results in main paper Table. 1, 6.

C.4 The Effect of Guidance Techniques

We present the generated results using double guidance techniques in Figure 9. The first row showcases the ground truth Mel-spectrogram and video frames, depicting a man smashing a window. Subsequent rows show the generated Mel-spectrogram results, arranged from left to right with progressively larger CFG scales ω and from top to bottom with increasingly larger CG scales γ . Moving from left to right, we observed a significant improvement in audio quality and synchronization with increasing CFG scale ω . Additionally, with increasing CG scale γ , the generation results of the original CFG scales were progressively refined, leading to higher levels of audio-visual alignment in the generated results.

Ground Truth Spec.

Ground Truth Video

CFG

𝜔

= 1.0

CG

𝛾

= 0

CFG

𝜔

= 2.0

CFG

𝜔

= 3.0

CFG

𝜔

= 4.0

CG

𝛾

= 15

CG

𝛾

= 30

CG

𝛾

= 45

…

Ground Truth

Reconstruction

(a).

(b).

D Discussion of Stable Diffusion Encoder and Decoder

Reconstruction Capability of SD Encoder & Decoder In DIFF-FOLEY, we directly use the frozen pretrained latent encoder E and decoder D in SD-V1.4 [38] to encode the Mel-spectrogram to a low-dimensional latent (mentioned in Sec. A.1). The reason we choose not to retrain encoder E and decoder D on the Mel-spectrogram data is that using the pretrained SD-V1.4 weight for parameter initalization can significantly reduce model training time and improve generation quality (Already discussed in main paper Table. 4). However, it is necessary to use the corresponding frozen latent encoders E and decoders D when using the pre-trained SD-V1.4 model. Interestingly, despite being trained on large scale image datasets (LAION-5B [40]), we found that latent encoder E and decoder D demonstrate good reconstruction capability on Mel-spectrogram data. We report the reconstruction metrics results of SD latent encoder E and decoder D in Table. 7. It can be noticed that the latent autoencoder ( E , D ) has relatively good reconstruction capability, as demonstrated by the KL and MSE metrics in Table. 7. However, the FID score is slightly higher, which may explain why DIFF-FOLEY does not exhibit a significant advantage over other baselines in terms of FID, which is likely due to the limitation in the frozen pretrained autoencoder reconstruction capabilities. We also visualize the reconstructed Mel-spectrogram results in Figure 10, showing that frozen pretrained E and D have excellent reconstruction capability on Mel-spectrogram data.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for performing the task of \"Foley\", which entails the creation of sounds that match a video, for example, making an audio clip in a studio that matches a character's footsteps in a film, as a post-production task. The authors proposed to do this automatically via deep learning, which has been proposed in other works, notably Im2Wav from last year. They propose 4 main contributions: audio-visual contrastive pre-training, latent diffusion modelling, an augmentation technique based on cropping different clips together, and the combination of classifier guidance and classifier-free guidance. Through a set of ablations, the authors show the importance of these four contributions. They also show superior results compared to 2 previous works on 2/4 metrics, and on inference time. Many qualitative results are shown and the appendix includes a large amount of discussion around many of the central topics.\n\nSTRENGTHS:\nFundamentally, the paper has enough contributions, in my opinion. Out of the 4, the first is very substantial, and the other 3 are reasonably novel when applied to this field. The writing is quite consistent with little to no mistakes, and in general the clarity of the discussions is remarkable. The paper does well in introducing readers to the task of video foley, which is not particularly well known amongst most readers, and also does a good job at presenting a lot of the modeling techniques used. The ablations satisfy the vast majority of questions the reader may have, and the more fine-grained ones are a nice addition. Quantitative results are explicit and well-presented. The lengthy appendix is welcome with some fruitful discussions. The demos are really impressive when compared to other works.\n\nWEAKNESSES:\nFrom the demos and quantitative results, the previous approaches seem to feature quite weak reproductions of the original sound. The authors only compare with two methods, which leaves me wondering if there could be comparisons with more works. I have found, for example, that FoleyGAN (2021) seems to perform the same task, but no comparison (or citation) is present. I believe the results would be a lot more convincing with more comparisons. Also, Diff-Foley is outperformed on 2/4 metrics, which is also not very reassuring, although they are FID and KL which are known to be somewhat unreliable.\n\nWhile the authors clearly show that adding Classifier guidance helps compare to only CFG, I would like to see an ablation where only Classifier guidance is used as well, as a matter of completeness, to show that both components of the double guidance are truly important. \n\nFigure 2 is good for the most part, but the latent diffusion section is, in my opinion, too vague - instead of just having a big box labelled \"latent diffusion\", it would be nice to have more detail to explain exactly what is happening here. Particularly for readers who are not already familiar with LDMs.\n\nThe third contribution \"Temporal Split & Merge Augmentation\" is similar to augmentations that have long been used in other domains. Namely cropping parts of different data points together to form a new one. For example, \"MixUp\" does exactly this by mixing two different images and labels together. Some discussion around previous approaches similar to this contribution with adequate referencing is necessary.\n\nIn table 1, if inference time is mentioned, then I believe number of parameters and training time, if possible, should also be presented for a more fair comparison.\n\nIn the related work, there is a section on \"Video-to-Audio Generation\", but existing works on a specific subset of video-to-audio which has received a lot of attention, video-to-speech, are not mentioned at all.\n\nQUESTIONS:\nAs mentioned above, are there any other methods that this work could be compared with in the results section?\n\nWill the code be released upon acceptance? If so, will this include training code, pre-trained models, etc.? These contributions to the open-source community would be very welcome.\n\nLIMITATIONS:\nAlthough I realize there are space limitations, the broader impact is really insufficient. Please put a bit more substance into this discussion. For example, you can talk about how this technology could be used to create fake videos more accurately, which could help spread false information. Or perhaps you can talk about the consequences of using producing an inaccurate reproduction of the audio, which may change the meaning of the video and mislead viewers. This may also be used to improve video surveillance, etc.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach to generate and align audio with an existing video track, using a diffusion process. This is useful in video post-processing, where frequently sound effects have to be aligned with existing video footage (and not just match semantically). The authors demonstrate the quality of their approach, which relies on a \"double guidance\" technique to improve the quality of the reverse process in LDM, and improve the alignment. Authors perform fine-tuning and ablation studies, to further understand the properties of the proposed approach.\n\nSTRENGTHS:\nOriginality\n\nThe paper presents a novel approach to learning audio-visual alignment, using a CLIP-style loss, taking within-clip and across-clip contrastive pairs from audio and video. This CVAP is a novel contribution, and at the core of the proposed work. LDMs are still a non-standard use in audio, although they are finding applications.\n\nQuality, Significance\n\nGiven the paper's improvements (mostly given through the CVAP formulation and its application to the LDM), I think the paper makes a significant contribution, although the topic is somewhat \"niche\"\n\nClarity\n\nThe paper is mostly well written and covering a reasonable set of experiments, it describes the main argument.\n\nWEAKNESSES:\nI don't understand the effect of temporal split & merge, section 3.3 (also see the question below) -- IIUC, the model could be trained on any audio-video pair, so if AudioSet+VGGSound is not enough, why not simply crawl more data from the Internet? It is clear that temporal augmentation improves results, specifically around the alignment metric, but Table 5 does not seem to indicate that more data always improves results?\n\nQUESTIONS:\n- In Section 3.3, you say \"We validate the effectiveness of this augmentation method in Sec 3.3.\" - please explain?\n- In the abstract (and in other places): \"we demonstrate DIFF-FOLEY practical applicability and generalization capabilities via downstream finetuning.\" -- how does fine-tuning, which is kind of a specialization of the model, demonstrate generalization capability?\n- Figure 5 - I am not sure I understood what is presented. The model makes mistakes for the first example - are these being resolved through fine-tuning the diffusion model, or do they still present?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present DIFF-FOLEY, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates audio with improved synchronization and audio-visual relevance. The method adopts contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. During inference, the method adopts a combination of classifier-free guidance and classifier guidance based on a synchronization classifier. The proposed model outperforms existing methods in audio-visual synchronicity and inception score.\n\nSTRENGTHS:\nThe proposed model achieves greater audio-visual synchronicity compared to several baselines. Through analysis, the audio-visual pre-training module indeed brings significant gains to the synchronicity. The paper is overall well-written and the authors have conducted a thorough analysis on the effectiveness of different modules, including the effect of two guidances, the used features, effect of pre-training.\n\nWEAKNESSES:\nDespite its advantages in audio-visual synchronicity and inception score, the proposed method falls below the baselines in other metrics such as FID and KL divergence. There is no human evaluation metrics in the comparison among the methods. Thus it is unclear overall how realistic the generated audios are compared to other methods. \nRegarding the synchronicity classifier, the authors have not thoroughly analyzed the synchronicity classifier, which is the key model to measure the main metric used in the paper. The paper mentioned it achieves 90% accuracy on the test set. However, it is unclear how the test set is constructed. For example, how is the negative-sample set constructed? What is the precision and recall respectively? How well does the accuracy match the human perception of audio-visual synchronicity?\nIn terms of modeling, the generative model is a typical LDM model used in audio generation and lacks general novelty.\n\nQUESTIONS:\n* What is the audio-visual synchronicity classifier used in the model? Is it the same with the classifier for accuracy measurement?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe focus of the paper is audio synthesis. More specifically, it focuses on video to audio synthesis and in particular on synchronized synthesis of audio. It relies on Latent Diffusion models for the synthesis and proposes an aligned audio-visual representation learning approach to improve synchronization of synthesized audio and input video. CAVP - Contrastive Audio-Visual Pre-training – tries to learn these features through contrastive learning. Experiments are conducted on the VGGSound dataset and both quantitative and qualitative results are shown in the dataset. In quantitative terms, the proposed DIff-foley method outperforms prior works by a considerable margin with respect to some of the metrics.\n\nSTRENGTHS:\n-- The paper addresses a key problem in audio/video synthesis. Generating synchronized audio for videos is challenging and most of the current struggle at. \n\n-- Learning good audio-visual representation is essential to solving this problem. The approach taken by this paper makes sense. \n\n-- The gain in IS metric and inference time using the proposed method is large. Moreover, the demo and the qualitative results does show superior and more synchronized\n\nWEAKNESSES:\n-- In Eq 1, aren’t the two terms same ? Why have they been separated ? Same is true for Eq 2. \n\n-- Analysis and discussion on duration as a factor is completely missing. It appears that all empirical results are given for fixed 8 seconds audio. Some results for variable length video inputs would be helpful to understand the duration factor. More importantly, producing synchronized video on longer form audio/video would be more challenging than relatively short audio/video. How does the method work out in those cases ? \n\n-- WHile the Diff-foley method does well (compared to others in Table 1) in terms of IS and synchro. Acc. it does not do well in terms of FID and KL. Some discussion on why that might be happening is desirable. Diff-foley is significantly inferior to other methods on these metrics. In Table 2, on impact of visual features - CAVP ends up doing better than others in terms of FID but not IS and KL. Overall, these results and lack of discussion in what might be happening leads to unclear understanding of the performances. \n\n-- Audio/speech synthesis works in my opinion should include some form of subjective tests. Objective metrics (especially when some of them are basically imperfect pre-trained models) do not paint a clear picture. This is also necessary because the generated sounds themselves are often not corresponding to the visual objects. The frequency content of the generated sound in Fig 3 and Fig 4 is clearly far from the ground truth. From the demo also, in many cases the generated sound does not correspond to the visual object. \n\n\n--- updated score after rebuttal ---\n\nQUESTIONS:\nPlease respond to the points outlined in the weakness section.\n\nLIMITATIONS:\nThe authors discuss limitations of the method in terms of scalability. A bit more discussion on how generative AI for audio can have societal impact might be good.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}