SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation Metrics For evaluation, we use Inception Score (IS), Frechet Distance (FID) and Mean KL Divergence (MKL) from [21]. IS assesses sample quality and diversity, FID measures distribution-level similarity, and MKL measures paired sample-level similarity. We introduce Alignment Accuracy (Align Acc) as a new metric to assess synchronization and audio-visual relevance. We train an alignment classifier to predict real audio-visual pairs. To train the classifier, we use three types of pairs:  $50\%$  of the pairs are real audio-visual pairs (true pair) labeled as 1,  $25\%$  are audio-visual pairs from the same video but temporally shifted (temporal shift pair) labeled as 0, and the last  $25\%$  are audio-visual pairs from different videos (wrong pair) labeled as 0. Our alignment classifier reaches  $88.31\%$  accuracy on test set. The detailed analysis of alignment classifier is provided in Appendix. A.1. We prioritize IS and Align Acc as the primary metrics to evaluate sample quality. For evaluation, we generate  $\sim 145\mathrm{K}$  audio samples (10 samples per video in the test set). We emphasize the difference between the noisy alignment classifier  $F_{\phi}^{DG}$  used for double guidance techniques and the alignment classifier  $F_{\theta}^{sync}$  used for Align Acc metric evaluation in Appendix A.1.
Figure 4: Visualization of generated sample with different visual features: It can be see that while other visual features fail to generate synchronized audio based on drum video, CAVP visual features successfully generate drum sound with 4 clear spikes (c.), matching the ground truth spectrogram (a.).
Baseline For comparison, we use two current state-of-the-art V2A models: SpecVQGAN [21] and Im2Wav [41]. SpecVQGAN offers two model settings based on different visual features (RGB+Flow and ResNet50), while Im2Wav generates semantically relevant audios using CLIP features. We use the pretrained baseline models for evaluation, which were both trained on VGGSound.
4.1 Video-to-Audio Generation Results
Table 1 presents the quantitative results on VGGSound test set and model inference time. DIFF-FOLEY outperforms the baseline method significantly in IS and Align Acc (primary metrics), while maintaining comparable performance on MKL/FID. DIFF-FOLEY achieves twice the performance of baseline on IS  $(62.37\mathrm{v.s}\sim 30)$  and an impressive  $94.05\%$  Align Acc. Im2Wav's slow inference speed hinders its practicality despite its advantages in KL metrics. In contrast, DIFF-FOLEY utilizes the state-of-the-art diffusion sampler DPM-Solver [30], generating 64 samples per batch at an average of 0.38 seconds per sample with 25 inference steps (compared to Im2Wav 8192 autoregressive steps). We included the KL and FID metrics for comprehensive analysis despite their unreliability and potential misalignment with human perceptions. The suboptimal results on these metrics might be due to using the pretrained autoencoder in SD-v1.4 [38], leading to reconstruction errors. See Appendix. D for more details. We conducted a human evaluation in Table 2. The generated audios are rated based on content relevance and synchronization on a scale of 1 (bad) to 5 (excellent), then multiplied by 20. The detailed human evaluation procedure is provided in Appendix. A.4. The results in Table 2 highlight DIFF-FOLEY's excellence in audio-visual synchronization and relevance. We also observe that content relevance and synchronization assessments from our classifier closely align with those from human evaluators, confirming the effectiveness of our sync classifier.
Figure 3 presents the qualitative results. DIFF-FOLEY demonstrates a remarkable ability to generate highly synchronized audio compared with baseline methods. Given a silent golf video, DIFF-FOLEY successfully generates the corresponding hitting sound at the 4th second, while baseline methods fail to generate sound at that specific second. This demonstrates the superior synchronization ability of DIFF-FOLEY. We emphasize that our aim is to generate audios that align well with human perception, even if they differ from the ground truth audio. More generated results can be found at this link.2.
4.1.1 Visual Features Analysis
CAVP uses semantic and temporal contrast objectives to align audio and visual features. We evaluate the effectiveness of CAVP visual features by comparing them with other visual features in LDM training. Results in Table 3 show that CAVP visual features significantly improve synchronization and audio-visual relevance (Align Acc), verifying the effectiveness of CAVP features. Despite CLIP
2https://diff-foley.github.io/, best accessed via Chrome.
Cutting Things with Knife
Figure 5: Downstream finetuning results on EPIC-Kitchens: DIFF-FOLEY achieves highly synchronized audio generation with the original video. The generated sounds closely match the ground truth, especially in terms of timing, such as knife cutting, water flow, and plate clinking (refer to Generated Audio and Ground Truth Audio).
features exhibiting advantages in IS and KL, it does not accurately reflect synchronization capability. We expect such a gap can be bridged by expanding CAVP datasets to a larger scale. In the drumming video example shown in Figure 4, different visual features are utilized to generate audio. In this example, the drum is hit four times (clear spikes in the ground truth, see a.), while ResNet and CLIP features fail to generate synchronized audio, CAVP features successfully learn the relationship between drumming moments and drum sounds, generating audio with 4 clear spikes (see c.), showing excellent synchronization capability.
MODEL	VISUAL FEATURES	METRICS
IS ↑	FID ↓	KL ↓	ALIGN ACC (%)↑
DIFF-FOLEY (Ours)	ResNet50	29.48	17.34	7.34	54.88
CLIP	57.43	13.09	5.88	79.23
CAVP (Ours)	52.07	11.61	6.33	92.35
Table 3: The effect of adopting different visual features in Stage2 Video-to-Audio training, using only Classifier-Free Guidance (CFG) with  $\omega = {4.5}$  and DDIM [43] Sampler with 250 inference steps. CAVP visual features demonstrate a notable advantage in terms of Align Acc.
MODEL	TEMPORAL AUG.	METRICS
IS ↑	FID ↓	KL ↓	ALIGN ACC (%)↑
DIFF-FOLEY (Ours)	X	50.62	12.65	6.38	79.79
✓	52.07	11.61	6.33	92.35
Table 4: Evaluation results w./w.o Temporal Split & Merge Augmentation, using only CFG with  $\omega = 4.5$  and DDIM [43] Sampler (250 steps). Temporal Aug. refers to Temporal Split & Merge Augmentation in Sec 3.3.
4.1.2 Temporal Augmentation Analysis
We evaluate the impact of Temporal Split & Merge Augmentation (discussed in Sec 3.3) in Table 4. Our results demonstrate improvements across all metrics using Temporal Split & Merge Augmentation, particularly for Align Acc metric. This augmentation technique leverages prior knowledge of audio-visual temporal alignment, enhancing the model's synchronization capability and increasing the number of audio-visual training pairs, thus mitigating overfitting.
(a).Guidance Scale
(b). Different Samplers
at Samplers Align Acc  $(\%)$
Figure 6: CFG Scale & Different Samplers: Enhancing CFG scale  $\omega$  improves sample quality up to a point, beyond which it decreases, creating U-shaped curves in (a). DDIM[43], PLMS[28], DPM-Solver[30] samplers are compared in (b). DPM-Solver, converging in only 25 steps, greatly improving DIFF-FOLEY inference speed and its accessibility. For fast evaluation, here we only generate 1 audio sample per video in the test set.
4.2 Downstream Finetuning
Background Foundation generative models like Stable Diffusion [38], trained on billions of data, excel at creating realistic images, but struggle with generating specific styles or personalized images. Finetuning techniques help overcome such limitations, aiding in specific-style synthesis and enhancing adaptability. Similarly, DIFF-FOLEY is expected to adapt to other datasets via downstream finetuning to synthesize specific types of sound, as evaluated in this section.
Finetuning Details We finetuned DIFF-FOLEY on EPIC-Kitchens [20], a high quality egocentric video dataset( $\sim$ 100 hours) about object interactions in kitchen with minimal sound noises. EPIC-Kitchens is notably distinct from VGGSound. This experiment validates the adaptability of finetuned DIFF-FOLEY. We used a pretrained CAVP model to extract visual features, and then finetuned Stage2 LDM (trained on VGGSound) for 200 epochs. See more finetuning details in Appendix. A.
Generation Results We show qualitative results of finetuned LDM on EPIC-Kitchens in Figure 5. DIFF-FOLEY generates highly synchronized audio with original video, effectively capturing the timing of sounds, e.g. knife cutting, water flow, and plate clinking. Best viewed in demo link 2.
MODEL	# PARAM.	PRETRAINED	METRICS
IS ↑	FID ↓	KL ↓	ALIGN ACC (%)↑
DIFF-FOLEY-S	335M	X	16.96	27.66	7.10	80.18
DIFF-FOLEY-M	553M	X	18.48	29.10	6.94	80.74
DIFF-FOLEY-L	859M	X	24.91	16.98	6.05	92.61
DIFF-FOLEY-L	859M	✓	52.07	11.61	6.33	92.35
Table 5: Model Scaling Effect: Only use CFG with  $\omega = {4.5}$  and DDIM [43] Sampler (250 steps). PRETRAINED refers to using pretrained SD-V1.4 [38] model as an effective parameter initialization for training LDM.
MODEL	STAGE1 CVAP DATASET	CFG	CG	METRICS
IS ↑	FID ↓	KL ↓	ALIGN ACC (%)↑
DIFF-FOLEY (Ours)	VGGSound	X	X	19.86	18.45	6.41	67.59
VGGSound	X	✓	16.58	20.20	6.81	62.24
VGGSound	✓	X	51.42	11.48	6.48	85.88
VGGSound	✓	✓	53.45	10.67	6.54	89.08
VGGSound + AudioSet-V2A	X	X	22.07	18.20	6.52	69.41
VGGSound + AudioSet-V2A	X	✓	17.57	20.87	6.69	66.05
VGGSound + AudioSet-V2A	✓	X	52.07	11.61	6.33	92.35
VGGSound + AudioSet-V2A	✓	✓	60.39	10.73	6.42	94.78
Table 6: Abalation Study: Evaluating the impact of Stage1 CAVP pretrained datasets scale and guidance techniques with CFG scale  $\omega = 4.5$ , and CG scale  $\gamma = 50$ , using DDIM [43] Sampler with 250 inference steps.
Model Size & Param Init. We explore the impact of model size and initialization with pretrained Stable Diffusion-V1.4 (SD-V1.4) on DIFF-FOLEY. We trained three models of varying sizes: DIFF-FOLEY-S (335M Param.), DIFF-FOLEY-M (553M Param.), and DIFF-FOLEY-L (859M Param., the same architecture as SD-V1.4). We report detailed architectures of these models in Appendix. A Our results in Table 5 show that increasing model size improves performance across all metrics,
demonstrating DIFF-FOLEY's scaling effect. Moreover, initializing with SD-V1.4 weights (pretrained on billions of images), significantly improves performance and reduces training time due to its strong denoising prior in image/Mel-spectrogram latent space (see the last row in Table 5).
Guidance Techniques & Pretrained Dataset Scale We conduct extensive ablation study on DIFF-FOLEY in Table 6, exploring the scale of Stage1 CAVP datasets and guidance techniques. We see that: 1). More CAVP pretraining data enhances downstream LDM generation performance. 2). Guidance techniques greatly improve all metrics except KL 3). Double guidance techniques achieve the best performance on IS and Align Acc (the primary metrics).
CFG Scale & Different Sampler We studied the impact of CFG scale  $\omega$  and different samplers on DIFF-FOLEY. CFG scale influences the trade-off between sample quality and diversity. A large CFG scale typically enhances sample quality but may have a negative effect on diversity (FID, KL). Beyond a certain threshold, increased  $\omega$  can decrease quality due to sample distortion, resulting in  $U$ -Shaped curves in Figure 6 (a). Best results in IS and Align Acc were achieved at  $\omega = 4.5$ , with minor compromise in FID and KL. Faster Sampling for diffusion models, reducing inference steps from thousands to tens, is a trending research area. We evaluated the usability of our DIFF-FOLEY using current accelerated diffusion samplers. We compare three different samplers, DDIM [43], PLMS [28], DPM-Solver [30] with different inference steps in Figure 6 (b). All three samplers converge at 250 steps, with DPM-Solver converging in only 25 inference steps, while others require more steps to converge. DPM-Solver allows for faster DIFF-FOLEY inference, making it more accessible.
Limitations Diff-FOLEY has shown great audio-visual synchronization on VGGSound and EPIC-Kitchens, however its scalability on super large (billion-scale) datasets remains untested due to limited data computation resources. Diffusion models are also slower than GANs due to the iterative reverse sampling process.
Broader Impact The advancements in V2A models, such as DIFF-FOLEY, have the potential to significantly expedite video production processes, offering efficiency gains in the entertainment and media sectors. However, as with most powerful technologies, there's a flip side. There's an underlying risk of these models being misused to create misleading or false content. As such, while the potential benefits are vast, it is imperative for developers, users, and regulators to exercise caution.
We introduce DIFF-FOLEY, a V2A approach for generating highly synchronized audio with strong audio-visual relevance. We empirically demonstrate the superiority of our method in terms of generation quality. Moreover, we show that using double guidance technique to guide the reverse process in LDM can further improve the audio-visual alignment of generated audio samples. We demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning. Finally, we conduct an ablation study, analyzing the effect of pretrained dataset size, various guidance techniques, and different diffusion samplers.
[1] Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation. Routledge, 2021.
[2] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[3] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 29:8292-8302, 2020.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[44] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
[45] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[50] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550-3558, 2018.
- Sec. A elaborates the implementation details of DIFF-FOLEY, including model architectures, data preprocessing and training/evaluation details for different stages models and tasks.
- Sec. B presents additional visualization results of Video-to-Audio generation on the VG-GSound [2] dataset and downstream finetuning results on EPIC-Kitchens [20] dataset, further demonstrating the superior performance of DIFF-FOLEY in synchronized audio generation.
- Sec. C provides theoretical and intuitive perspectives on the mechanism of Double Guidance techniques discussed in Sec. 3.4.
- Sec. D examines the reconstruction capability of the Stable Diffusion [38] frozen pretrained Encoder  $\mathcal{E}$  and Decoder  $\mathcal{D}$  on Mel-spectrograms.
A.1 Model Architectures
Our proposed DIFF-FOLEY employs a two-step procedure for synchronized Video-to-Audio generation. The first stage, Contrastive Audio-Visual Pretraining (CAVP), aims to learn temporally and semantically aligned audio-visual features to capture the subtle connection between the audio and visual modalities. The second stage involves training latent diffusion models (LDMs) conditioned on the CAVP-aligned visual features in the Mel-spectrogram latent space. In this section, we present detailed model architectures for each of the model components in DIFF-FOLEY.
Stage1 CAVP. CAVP adopts a similar two-stream encoder design as CLIP [36], consisting of an audio encoder  $f_{A}(\cdot)$  and a video encoder  $f_{V}(\cdot)$ . For the audio encoder  $f_{A}(\cdot)$ , we adopt the Mel-spectrogram encoder design from PANNs [24], which is a 2D convolution-based encoder with  $\sim 81\mathrm{M}$  parameters. For the video encoder  $f_{V}(\cdot)$ , we use the SlowOnly branch design from the SlowFast [9] architecture, which is a 3D convolution-based backbone with  $\sim 30\mathrm{M}$  parameters. During CAVP training, we incorporate the pretrained weights of the audio encoder from PANNs [24], which were initially trained on the large-scale AudioSet [10] dataset. Additionally, we adopt the pretrained weights of the video encoder from SlowFast [9], which were trained on the Kinetics-400 [22] dataset. For audio and video input pairs  $(x_{a}, x_{v})$ , where  $x_{a} \in \mathbb{R}^{T' \times M}$  is a Mel-spectrogram with  $M$  mel basis and  $T'$  is the time axis of the spectrogram, and  $x_{v} \in \mathbb{R}^{T \times 3 \times H \times W}$  is a video clip with  $T$  frames. The audio encoder  $f_{A}(\cdot)$  and video encoder  $f_{V}(\cdot)$  are used to extract audio feature  $E_{a} \in \mathbb{R}^{T \times C}$  and video feature  $E_{v} \in \mathbb{R}^{T \times C}$ . Specifically, the audio encoder  $f_{A}(\cdot)$  encodes the spectrogram to match the time dimension of the video frames. In CAVP, we set the feature dimension  $C = 512$ . By using a Temporal Max-Pooling Layer  $P(\cdot)$ , we obtain temporal-pooled audio and visual features  $\bar{E}_{a} = P(E_{a}) \in \mathbb{R}^{C}$ ,  $\bar{E}_{v} = P(E_{v}) \in \mathbb{R}^{C}$ .
Stage2 LDM. For LDM, we use the same UNet backbone architecture in Stable Diffusion-V1.4 (SD-V1.4) [38]. Using the frozen pretrained latent encoder  $\mathcal{E}$ , we encode a spectrogram into a low dimensional latent  $z_0 = \mathcal{E}_{\theta}(x_a) \in \mathbb{R}^{C' \times \frac{T'}{r} \times \frac{M}{r}}$ , where the channel number of  $C' = 4$  and a compression rate of  $r = 8$ . To successfully encode a single channel spectrogram  $x_a$  into a latent  $z_0$  with the pretrained frozen 3-channel image encoder  $\mathcal{E}$  in SD-V1.4, we first repeat the spectrogram channel into 3 ( $x_a \to \tilde{x}_a \in \mathbb{R}^{3 \times T' \times M}$ ), and then obtain  $z_0$  as  $z_0 = \mathcal{E}_{\theta}(\tilde{x}_a)$ . To decode a spectrogram from latent  $z_0$  with the frozen image decoder  $\mathcal{D}$  in SD-V1.4, we simply use the first channel from the decoded spectrogram ( $\hat{x}_a = \mathcal{D}(z_0)[0, :, :] \in \mathbb{R}^{T' \times M}$ , where  $\mathcal{D}(z_0) \in \mathbb{R}^{3 \times T' \times M}, [:, :, :]$  is the tensor slicing operation). The reconstruction capability of the pretrained frozen image latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  on Mel-spectrograms is discussed in Sec. D. With the pretrained CAVP model to align audio-visual features, the visual aligned features  $E_v \in \mathbb{R}^{T \times C}$  are adopted in LDM training. To better learn the temporal relationship between audio and visual modality, we adopt a positional encoding and projection layer  $\tau_{\theta}$  to project  $E_v$  to appropriate dimension with  $\tilde{E}_v = \tau_{\theta}(E_v) = MLP(E_v + PE) \in \mathbb{R}^{T \times C'}$ , where MLP is the projection layer and PE represents positional encoding. We set  $C' = 768$  for all experiments. We use the same denoising schedule  $\alpha_1, \dots, \alpha_T$  in SD-V1.4 [38], with  $T = 1000$ . To learn the audio-visual correlation, we adopt the cross-attention
module from SD-V1.4. Specifically, we apply cross-attention on latents with down-sampling rates of [1, 2, 4].
In Sec. 4.3 of the main paper, we use three different model settings of varying sizes: DIFF-FOLEY-S (335M parameters), DIFF-FOLEY-M (553M parameters), and DIFF-FOLEY-L (859M parameters). All three models use the UNet backbone with four encoder blocks, a middle block, and four decoder blocks. For DIFF-FOLEY-S, with a basic channel number of  $c = 192$ , the channel dimensions of the encoder blocks are  $[c, 2c, 3c, 4c]$ , and the design of the decoder blocks are basically the reverse of the encoder blocks. For DIFF-FOLEY-M, we set  $c = 256$ , and the channel dimensions of the encoder blocks are  $[c, 2c, 4c, 4c]$ . Finally, for DIFF-FOLEY-L, we adopt the same architecture as in SD-V1.4 [38], resulting in  $c = 320$  and channel dimensions of the encoder blocks  $[c, 2c, 4c, 4c]$ . We have observed a significant improvement in performance in Sec. 4.3 when using the pretrained weights from SD-V1.4 for parameter initialization. Therefore, we default to using the DIFF-FOLEY-L model and the pretrained weights from SD-V1.4 unless otherwise specified.
Vocoder. Vocoder is used to transform Mel-spectrogram into waveform signal. There are two categories of vocoders: Traditional algorithm-based vocoders such as the Griffin-Lim algorithm [12], which uses an iterative refining strategy to estimate the original sound signal, and deep learning-based vocoders such as WaveNet [33], MelGAN [26], and HiFi-GAN [23], which are widely used for speech waveform generation. Although our method is fully compatible with deep learning-based vocoders, we choose to use the Griffin-Lim algorithm for audio waveform reconstruction in this paper due to its simplicity.