Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Authors: Simian Luo
Chuanhao Yan
Chenxu Hu
Hang Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Video-to-Audio Generation; Contrastive Audio-Visual Pretraining; Latent Diffusion Model;
TL;DR: Diff-Foley is a synchronized Video-to-Audio synthesis method with a latent diffusion model that generates high quality, highly synchronized audio with improved audio-visual relevance. It outperforms previous methods by a large margin.

Abstract: The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and adaptability via customized downstream finetuning. Project Page: https://diff-foley.github.io/

DIFE-FOLEY: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Simian Luo
Chuanhao Yan' Chenxu Hu' Hang Z
'II, Tsinghua University Shanghai Qi Zhi Institute (luosm22, yanch21, hu-cx21 aa-cx22)mmaals emails tsinghua. erigguuaedu..... cn hangzhao@mail, attechaommmi..sinnhnnee tsinghua edu. cn
Abstract
The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in vidco/film production. However, previous methods in V2A have limited generation quality in terms temporal synchronization and audio-visual relevance. We present DIFF- FOLEY, synchronized Video-to-Aus Audio synthesis method with latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross- attention module. We further significantly improve sample quality with 'double guidance'. DIFF-FOLEY achieves state-of-ther oh-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning, Project Page: https://dift foley. github.io/
X
Traditional Foley
Artists Generated Audio
Silent
Video Production
6
Neural Foley
Figure 1: Traditional Foley v.s Neural Foley: Traditional Foley, a time-consuming and labor- iebor-intensive process involving skilled artists manipulating objects create hours of physical sounds for sound recording (31) In contrast, Neural Foley presents appealing alternative utilizing neural networks synthesize high-quality synchronized audio, accelerating video production and alleviating human workload.
Introduction
Recent advances in diffusion models 42
45 [6] have accelerated the development of Al-generated
contents, e.g. Text-to-Image (T2I) generation 37
39
38, Text-to-Audio (T2A) generation [48 27
and Text-to- Video (T2V) generation [15]. This paper focuses Video-to-Audio (V2A) generation, which has practical applications video/film production. During live shooting, due to the presence of excessive background noise and challenges in the audio collection in complex scenarios, the majority
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of the sounds recorded in the film need to be recreated during post produurro This process of adding synchronized and realistic sound effects to videos is known as Foley [. We compare traditional Foley performed in the studio and our neural Foley in Figure [ Traditional Foley is laborious and time-consuming process, that involves skilled artists manipulating objects to create hours of authentic physical sounds. In contrast, Neural Foley offers an appealing alternative. High-quality synchronized audio generated by AI can greatly accelerate video production and alleviate the human workload. Different from Neural Dubber [9], which synthesizes speech from text scripts and video frames, Neural Foley focus on generating a broad range of audio solely based on the video content, a task that poses considerably higher level of difficulty.
Video-based audio generation offers two natural advantages over text- baxt-based generation while per- forming Foley. First, T2A requires lots of hard-to-collect text-audio pairs for training. in contrast, audio-video pairs are readily available on the Internet, egg. millions of new videos are uploaded to YouTuub daily. Second, along with the semantics of the generated audio, V2A can further control the temporal synchronization between the Foley audio and video.
Semantic content matching and temporal synchronization are two major goals in V2A. While some progress 35 50 21 [1] have been made recently on V2A, most methods of audio generation focus only on the content relevance, neglecting crucial aspect of audio-visual synchronization. For example, given a video of playing drums, existing methods can only generate drums sound, but cannot ensure the sounds match exactly with what's 's happening in the video, i.e. hitting the snare drum or the crash cymbal at the right time.
RegNet 3 uses a pretrained (RGB+Flow) network as conditional inputs to GAN for synthesizing sounds. Meanwhile, Spec (1 uses Transformer 47 autoregressive model conditioned on pretrained ResNet50 [14] or (RGB+Flow) visual features for better sample quality. These methods have limitations in generating audio that is both synchronized and relevant to video content as pretrained image and flow features cannot capture the nuanced correlation between audio and video.
We present FOLE-, novel Neural Foley framework based on Latent Diffusion Model (LDM) [38] that synthesizes realistic and synchronized audio with strong audio-visual relevance. DIFF- FOLEY first learns temporally and semantically aligned features via CAVP. By maximizing the similarity of visual and audio features in the same video, it captures subtle audio-visual connections. Then, an LDM conditioned CAVP visual features is trained on the spectral latent space. CAVP- aligned visual features help LDM in capturing audio-visual relationships. To further improve sample quality, propose 'double guidance', , using classifier-fre and alignment classifier guidance jointly to guide the reverse process, DIFF-FOLEY achieves state-of-the-art performance on large-scale V2A dataset VGGSound [2] with IS of 62.37, outperforming Spec VQGAN (2I) baseline (IS of 30.01) by large margin. We also demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning, validating the potential of V2A pretrained generative models.
2 Related Work
Video-to-Audio Generation Generating audio from silent videos has potential in video production, which can greatly improve post- postuproduttion efficiency. Despite recent progress, open-domain V2A remains a challenge. FoleyGAN [II] uses a class conditoond GAN for synchronized audio synthesis, but it requires specific action class label and is tested on a limited dataset. RegNet extracts video features by using pretrained (RGB+Flow) network and serves as conditional input for GAN to synthesize sounds. SpecVQGAN 21 uses more powerful Transformer-based autoregressive models for sound synthesis with extracted ResNet50 or RGB+Flow features. Im2Wav [4], current ofate-of-the-art, adopts a two-transformer model, using CLIP (36) features as the condition, yet suffers from slow inference speed, requiring thousands of inference steps. Existing V2A methods struggle with synchronization as they lack audio-related information in pretrained visual features, limiting the capture of intricate audio-visual correlations. A notable subfield, video-to-speech has gained attention with works like vid2speech 77. Neural Dubber [19]. However, the foley task aims to generate complex and synchronized audio from video, presenting much higher challenge.
has shown potential various generative tasks. CLIP aligns text-image representations using contrastive pretraining. For Texte CLIP integrated with Stable Diffusion (38) to extract text prompt features for realistic image generation. For text-to-Audio, CLAP 66 aligns text and audio representations, and the resulting audio features
Contrastive Pretraining Contrastive pretraining 34
2

---

Training
Inference
Leion.e
Audia
PD
Specinogrmn
Spactegrors
Dulitaiaa
Revane
Cusinative Ohiacivv Tenpors))
Phooses
Latent Diffisionn
Vidas Enooder
KV
Ki
Denaieing Objective
Valeo Fiarcss
CAVPAlgee
Vanaal Fenture
Sy- () Cuassian Noos
Contrastive Audio-Viess Pretraining (CAVP)
LDM Training with Aligned Visual Representation
Figure 2: Overview of DIFF-FOLEY: First, it learns more semantically and temporally aligned audio-visual features by CAVP, capturing the subtle connection between audio-visual modality. Second, a LDM conditioned on the aligned CAVP visual features is trained on the spectrogram latent space. DIFF-FOLEEY can synthesize highly synchronized audio with strong audio-visual relevance. P-)) denotes temporal pooling layer.
are used in AudioLDM [77 for audio generation. Recognizing the importance of modality alignment multimodal generation, we propose contrastive audio-visual pretraining (CAVP) to learn temporally and semantically aligned features initially and use them for subsequent generation tasks.
Latent Diffusion Model Diffusion Models 43 T6 have achieved remarkable success in various
generative tasks including image generation 43 16: 32, 37 38). audio generation [25 48 27).
and video generation [8 15 8] Latent diffusion models, like Stable Diffusion (SD) [38 perform
forward and reverse processes in data latent space, resulting in efficient computation and accelerated inference. SD is capable of running inference on personal laptop GPUs, making practical choice
for wide range of users.
3 Method
DIFF-FOLES consists of two stages: Stagel CAVP and Stage2 LDM training. Our model overview shown in Figure 2 Audio and visual components in a video exhibit strong correlation and complementarity. Unfortunately, existing image optical flow backbones (ResNet, CLIP etc.) struggle to reflect the strong alignment relationship between audio and visual. To address this, propose Contrastive Audio- Visual Pretraining (CAVP) to align audio-visual features at the outset. Then, an LDM conditioned on CAVP-aligned visual features trained on the spectral latent space.
3.1 Contrastive Audio-Visiee Pretraining
Given audio-video pair (Ia, N.), where va E X.'x is a Mel-Spec. with M mel basis and T" is the time axis. Ry E RT Hx is a video clip with 7 frames. An audio encoder (-).) and a video encoder [v(() are used to extract audio feature E., RTxC and video feature E,, E RTC with same temporal dim T. We adopt the design of the audio encoder from PANNs (44, and SlowOnly [ architecture for the video encoder. Using temporal pooling layer P(.). we obtain cemporal-pooled audio/video features, E. P(E.) RC E. = P(E.) E RC, We then use similar contrastive objective CLIP (36] contrast E. and E.. To improve the semantic and temporal alignment of audio-video features, we use two objectives: Semantic contrast Ls and Temporal contrast Ly.
For Ls. maximize the similarity of audio-visun pairs from the same video and minimize the similarity of audio-visual pairs from different videos. encourages learning semantic alignment for audio-visual pairs across different videos. In specifie, we extract audio-visual features pairs from different videos, B5- {(Ft) ) where Ns is the number of different videos. We define the per-sample pair semantic contrast objective: LS
(f-5), where sim(() is the cosine similarity.
(s = - log (xima Fi,, RB)/T)
exp exp(sim(FF) (Ei, Rb)/T) log
(I)
INe exp (sim(E), Et, Bt)/T) 2 ΣNe exp (Eim(E,))) E2)/T)
For LT, we sample video clips at different times within the same video. It aims to maximize the similarity of audio-visual pairs from the same time segment and minimize the similarity of audio- visual pairs across different time segments. In details, sample different time segments in the

---

same video to extract audio-visual feature pairs. By = {(E) E where Nr is the number of sampled video clip within the same video. We define the per-sample pair temporal contrast objective:
exp (sim( (E,, E()/T) CT log
1
exp eximsin(E (Ea, E()/T)
log
E exp (sim( Pa. P+)/T) 2
EE exp (sim (Pti F2./TT)
The final objective is the weighted sum of semantic and temporal objective: L Ls + XLT, where À = 1. After training. CAVP encodes an audio-video pair into embedding pair: (Ia, k.,) (Ea: Ev) where (En, Ex) are highly aligned, with the visual features E,, containing rich information for audio. The aligned and strongly correlated features E, and E, facilitate subsequent audio generation.
3.2 LDM with Aligned Visual Representation
L.Ms 38 are probabilistic models that fit the data distribution p(x) by denoising on data latent space. LDMs first encode the origin high-dim data x into low-dim latent = {xx) for efficient training. The forward and reverse process are performed in the compressed latent space. In V2A generation, our goal is to generate synchronized audio To given video clip Z.. Using similar latent encoder &g in [8 we compress Mel-Spec & into a low-dim latent 2n = (x.,) E xx x where r is the compress rate. With the pretrained CAVP model to align audio-visual features, the visual features E, contain rich audio-related information. This enables synthesis of highly synchronized and relevant audio using LDMs conditioned on E,, We adopt projection and positional encoding layer to project E, to the appropriate dim. In the forward process, origin data distribution transforms into standard Gaussian distribution by adding noise gradually with fixed schedule a1,...,, ar, where T is the total timesteps, and I[ Ozz-
(<<12211) (26 vaiz-1, (1 II)
(<<1220) =M(z; varz (1-
(3)
The goal of LDM is to mirror score matching by optimizing the denoisng objective 46 16
LLDM = E.,.,,elle c0(2,, E
(4)
After LDM is trained, we generate audio latent by sampling through the reverse process with 2T ~ (0, conditioned on the given visual-features E,, with the following reverse dynamics:
po(#1-1223 = -N Ev), o
(5)
Ha (2r, 1, Ev)
2t
1 az
- En) of
- ar)
(6)
Von
VI - āt
at
Finally, the Mel-Spec. &a is obtained by decoding the generated latent 20 with decoder D, f.- = D(z)). In the case of DIFF-FOLEY, it generates audio samples with a duration of 8 seconds.
3.3 Temporal Split & Merge Augmentation
Using large-scale text-image pairs datasets like LAION-SB 40 is crucial for the success of current T2I models (38]. However, for V2A generation task, large-scale and high-quality datasets are still lacking. Further, expect V2A model to generate highly synchronized audio based on visual content, such temporal audio-visual correspondence requires a large amount of audio-visual pairs for training. To overcome this limitation, propose using Temporal Split & Merge Augmentation, MixUp (49 like augmentation strategy to facilitate model training by incorporating prior knowledge for temporal alignment into the training process. During training, of different time lengths from two videos (Split), denoted as (m), =1),, (x²2 2 and extract visual features Ey, E3 with pretrained CAVP model. We then create new audio-visue feature pair for LDM training with:
randomly extract video clips
Eo =:1)
Enew
E
where [: represent temporal concatenation (Merge). Split and merge augmentation greatly increase the number of audio-visual pairs, preventing overfitting and facilitating LDM to learn temporal correspondence. We validate the effectiveness of this augmentation method in See 4.1.2

---

METRICS
MODEL
VISUAL FEATURE FPS GUIDANCE
INFER. TIME
IS t
FID
K Acc (%)
SpeeVQGAN [21
RGB + Flow
21.5
X
30.01
8.93
6.93
52.94
5.47s
SpeeVQGAN 221
ResNet50
21.5
X
30.80
9.70
7.03
49.19
5.47s
Im2Wav 4
CLIP
30
CFG (
39.30
11.44
5.20
67.40
6,41s
DIFF-FOLEY (Ours)
CAVP
4
CFG
53.34
11.22
6.36
92.67
0.38s
DIFF-FOLEY (Ours)
CAVP
Double
62.37
9.87
6.43
94.05
0.38s
Table I: Video-to-Ause generation evaluation results with CFG scale w = 4.5, CG scale Y = 50, using DPM-Solver [30] Sampler with 25 inference steps. DIFF-FOLEY achieves impressive temporal synchronization and audio-visual relevance (Align Acc) with only using 4 FPS video, compared with baseline method using 21.5/30 FPS. CFG represents Classifier-Frre Guidance, Double represents Double Guidance, and ACC. refers Align Acc. INFER. TIME denotes the average inference time per sample when generating 64 samples in batch.
HUMAN EVAL. METRICS
MODEL
VISUAL FEATURE
FPS
GUIDANCE
Content Relevance
Synchronization
SpecVQGAN 121
ResNc150
21.5
X
46.20
45.20
Im2wav 41
CLIP
30
CFG
62.13
57.73
Diff-Foley (Ours)
CAVP
4
CFG
71.73
71.00
Diff-Foley (Ours)
CAVP
Double
74.53
74.93
Ground Truth
84.80
84.20
Table 2: Tideo-to-Audi generation human evaluation results with CFG scale w 4.5, CG scale 7 = 50.
Raters are required to score the generated audio based on content relevance and synchronization. DIFE-FOLEY with double guidance shows superiority performance compared with baseline methods.
3.4 Double Guidance
Guidance techniques widely used in diffusion model reverse process for controllable generation. There are currently two main types of guidance techniques: classifier guidance (5) (CG), and classifier- free guidance (17) (CFG). For CG, it additionally trains a classifier (e.g class-label classifier) to guide the reverse process each timestep with gradient of class label loglikelihood Va, log log (zz). For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates [17], where the e the condition and w is the guidance scale. In stable diffusion (8), the CFG is implemented as:
Co c) + wco(2e, (,c) + (1 co,, t,, 2).
(8)
When = 1, CFG degenerates to conditional score estimates. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label. In V2A setting, the desired property refers to semantic and temporal alignment. Moreover, we discover that these two methods not mutually exclusive. We propose a double guidance technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specific, for CG we train an alignment classifier (v 24, E..) that predicts whether audio- visual pair is a real pair in terms of semantic and temporal alignment. For CFG, during training, we randomly drop condition E,, with prob. 20%, train conditional and unconditional likelihood cz(z,, Ev), (z(2,, 1 Ø). Then double guidance is achieved by improved noise estimation:
60 t, E.) + wee (ze, t, E.) + (1- w)e 2) >VI V., log Po(ylze, t, Et),
(9)
where w, y refer to CFG, CG guidance scale respectively. We provide further analysis of the intuition and mechanism behind double guidance in Appendix
4 Experiments
Datasets We use two datasets VGGSound 2 and AudioSet [0). VGGSound consists of 200K 10-second videos. We follow the original VGGSound train/test splits. AudioSet comprises 2. IM videos with 527 sound classes, but it is highly imbalanced, with most of the videos labeled as Music and Speech. Since generating meaningful speech directly from video is not expected in V2A tasks (not necessary either), we download subset of the Music tag data and all other tags except Speech, resulting in a new dataset named AudioSet-V22 V2A with about 80K music-tagged videos and 310K
5

---

Input Silent Video: Playing Goll
Ground Truth
[Hitiing gall hall)
Spee VOGAN
Fails to generate
Im2Wav
F Fails generate
DifT-Foley (Ours)
(Hitting golf ball)
Figure 3: Video-to-Audio generation results on VGGSound: Given silent playing golf video, only the DIFE-FOLEY successfully generates the corresponding hitting sound at the 4th second timestamp, showcasing its remarkable ability to generate synchronized audio based video content.
other tagged videos. We use VGGSound and AudioSet- for Stagel CAVP, while for Stage2 LDM training and evaluation, we only use VGGSound, which is consistent with the baseline.
Data Pre-Processing For DIFF-FOLEY training, videos in VGGSound and AudioSet-V22 sampled at FPS, which already yields significantly better alignment results compared to the baseline method 21 41 using 21.5 30 FPS. Each 10-second video sample consists of 40 frames resized to 224 x 224. For audio data, the original audio is sampled at 16kHz and transformed into Mel-spectroograss (Mel Basis M 128). For Stagel Pretraining, audio-visual data temporal dimension alignment, while we use hop size 256 for Stage2 LDM training.
hop size 250 for better
Model Configuration and Inference For CAVP, and a pretrained SlowOnly 99 based video encoder. For training, we randomly extract 4-second audio- video frames pairs from 10-second samples, resulting in F. 2566x128 and x, E IR 16xxxx24x224
pretrained audio encoder from PANNs (4
We use temporal contrast Ly with N₂r = 3 and a minimum time difference of seconds between utilize the pretrained Stable Diffusion- V 1.4 (SD-V1.4) [38]
each pair. For LDM training, powerful denoising prior model. Leveraging the pretrained SD-V1.4 greatly reduces training time and improves generation quality (discussed in Sec(4.3). Frozen pretrained latent encoder & and decoder D from SD-V1. .4 are used. Interestingly, despite being trained on image datasets (LAION-5B). we found E, 'D can encode and decode Mel-Spec well. For inference, we use DPM-Solver 25 sampling steps, unless otherwise stated. More training details for each model are in Appendix. A
30 sampler with
Inception Score (IS), Frechet Distance (FID) and Mean KL Divergence (MKL) from [2I]. assesses sample quality and diversity, FID measures distribution- level similarity, and MKL measures paired sample-leved similarity. We introduce Alignment Accuracy (Align Acc) as a new metric to assess synchronization and audio-visual relevance. We train an alignment classifier to predict real audio-visual pairs. To train the classifier, pairs: 50% of the pairs are real audio-visual pairs (Irue pair) labeled as 1, 25% are audio-visual pairs from the same video but temporally shifted (temporal shift pair) labeled as O, and the last 25% are audio-visual pairs from different videos (wrong pair) labeled as O. Our alignment classifier reaches 88.31% accuracy on test set. The detailed analysis of alignment classifier is provided in Appendix. A.I We prioritize IS and Align Acc as the primary metrics to evaluate sample quality. For evaluation, we generate 145K audio samples (10 samples per video in the test set). We emphasize the difference between the noisy alignment classifier PDG used for double guidance techniques and
Evaluation
Metrics For evaluation,
three types of
the alignment classifier payne used for Align Ace metric evaluation in Appendix A.I

---

Ground Truth
CT Visual Feature
CAVP Visual Feature (Ours)
d. Net50 Visual Feature
Figure 4: Visualization of generated sample with different visual features: It can be that while other visual features fail to generate synchronized audio based on drum video, CAVP visual features successfully generate drum sound with 4 clear spikes (c.). matching the ground truth spectrogram (a.).
Baseline For comparison, we use two current state-of-the-art V2A models: SpecVQGAN [2] and Im2Wav 41). SpecVQGAN offers two model settings based on different visual features (RGB+Flow and ResNet50), while Im2Wav generates semantically relevant audios using CLIP features. We use the pretrained baseline models for evaluation, which were both trained on VGGSound.
4.1 Video-to-Audio Generation Results
Table presents the quantitative results on VGGSound test set and model inference time. DIFF- FOLEY outperforms the baseline method significantly in IS and Align Ace (primary metrics), while maintaining comparable performance on MKL/FID, DIFF- DIFF-FOLEY achieves twice the performance of baseline on IS (62.37 v.5 ~30) and impressive 94.05% Align Acc. Im2Wav's slow inference speed hinders its practicality despite its advantages in KL. metrics. In contrast, DIFF-FOLEY utilizes the state-of-the-an diffusion sampler DPM-Solver B0), generating 64 samples per batch average of 0.38 seconds per sample with 25 inference steps (compared to Im2Wav 8192 autoregressive steps). We included the KL and FID metries for comprehensive analysis despite their unreliability and potential misalignment with human perceptions. The suboptimal results on these metrics might be due using the pretrained autoencoder in SD-v1.4 (38, leading to reconstruction errors. See Appendix. Dooor more details. We conducted human evaluation in Table 2 The generated audios are rated based on content relevance and synchronization on scale of (bad) to 5 (excellent), then multiplied by 20. The detailed human evaluation procedure is provided in Appendix. [A.4 The results in Table 2 highlight DIFF-FOLER s excellence audio-visual synchronization and relevance, We also observe that content relevance and synchronization assessments from our classifier closely align
with those from human evaluators, confirming the effectiveness of our sync classifier.
Figure 3 presents the qualitative results. DIFE-FOLEY demonstrates a remarkable ability to generate highly synchronized audio compared with baseline methods. Given silent golf video, DIFF-FOLEY successfully generates the corresponding hitting sound at the 4th second, while baseline methods fail to generate sound at that specific second. This demonstrates the superior synchronization ability of We emphasize that our aim is to generate audios that align well with human perception, even if they differ from the ground truth audio. More generated results can be found this link
4.1.1 Visual Features Analysis
CAVP uses semantic and temporal contrast objectives to align audio and visual features. We evaluate the effectiveness of CAVP visual features by comparing them with other visual features in LDM training. Results in Table Ssow that CAVP visual features significantly improve synchronization and audio-visuull relevance (Align Acc), verifying the effectiveness of CAVP features. Despite CLIP
httts://diffffee github......... 1 best accessed via Chrome.
7

---

Cutting Things with Kmife
Cutring sound)
[Curting sound)
Generated Audio (Ours)
Washing Dishes
Ground Truth Audio
[Water sound stops)
(Plate clinking)
Generated Audio (Ours)
(Water sound stops)
(Plate clinking)
Figure 5: Downstream finetuning results EPIC-Kitchens: DIFF-FOLEY achieves highly synchronized audio generation with the original video. The generated sounds closely match the ground truth, especially in terms of timing, such as knife cutting, water flow, and plate clinking (refer Generated Audio and Ground Truth Audio).
features exhibiting advantages in and KL, it does accurately reflect synchronization capability. We expect such gap can be bridged by expanding CAVP datasets to larger scale. In the drumming video example shown in Figure 4 different visual features are utilized to generate audio. In this example, the drum is hit four times (clear spikes in the ground truth, see a.), while ResNet and CLIP features fail generate synchronized audio. CAVP features successfully learn the relationship between drumming moments and drum sounds, generating audio with 4 clear spikes (see c.), showing excellent synchronization capability.
MUTRICS KL.
MODEL
VISUAL FEATURES
1S
FID
ALION ACC
ResNet50
29.48
17.34
7.34
54.88
DIFE-FOLEY (Ouns)
CLIP
5,88
CAVP (Ouns)
52.07
11.61
6.33
92.35
Table 3: The effect of adopting different visual features in Stage2 Vidco-to-Audio training, using only Classifier-
Free Guidance (CFG) with 4.5 and DDIM [ Sampler with 250 inference steps. CAVP visual features demonstrate a notable advantage in terms Align Acc.
METRICS KL
MODEL
TEMPORAL AUG.
1ST
FID
ALIGN ACC (%J
DIFF DIFF-FOLEY FOLEY (Ours)
50.62
12.65
6.38
79.79
Table 4: Evaluation results w/wwo Temporal Split & Merge Augmentation, using only CFG with w = 4.5 and Sampler (250 steps). Temporal Aug, refers to Temporal Split & Merge Augmentation in Sec 3.3
DDIM 43
4.1.2 Temporal Augmentation Analysis
We evaluate the impact of Temporal Split & Merge Augmentation (discussed in See 3.3) in Table 4 Our results demonstrate improvements across all metrics using Temporal Split & Merge Augmentation, particularly for Align Acc metric. This augmentation technique leverages prior knowledge of audio- visual temporal alignment, enhancing the model's synchronization capability and increasing the number of vidial training pairs, thus mitigating overfitting.
8

---

(4), Guidance Scale
(b). Different Samplers
Iroption Soore 031
A Agummntt Accuracy dAige And
TJH DPVisoher
Paacher Cinrance 713)
Eiargence
DCM PJRS
D0M NJ
briishee
DrVGolvrr
Figure 6: CFG Scale & Different Samplers: Enhancing CFG scale w improves sample quality up to point, beyond which it decreases, creating U-shaped curves in (a). DDIMER), PLMSS DPM-Solver 30) samplers are compared in (b). DPM-Solver, converging in only 25 steps, greatly improving DIFF-FOLEY inference speed and accessibility. For evaluation, here we only generate audio sample per video the test set.
4.2 Downstream Finetuning
Background Foundation generative models like Stable Diffusion [8], trained on billions of data, excel at creating realistic images, but struggle with generating specific styles or personalized images. Finetuning techniques help overcome such limitations, aiding in specific-style synthesis and enhancing adaptability. Similarly, DIFE-FOLEY is expected to adapt to other datasets via downstream finetuning to synthesize specific types of sound, evaluated in this section.
Finetuning Details We finetuned DIFE-FOLEY on EPIC-Kitchens video dataset(~100 hours) about object interactions in kitchen with minimal sound noises. EPIC- Kitchens notably distinct from VGGSound. This experiment validates the adaptability of finetuned DIFE-FOLEY. We used a pretrained CAVP model to extract visual features, and then finetuned Stage2 LDM (trained on VGGSound) for 200 epochs. See more finetuning details in Appendix. A
high quality egocentric
Generation Results We show qualitative results of finetuned LDM on EPIC-Kitchens in Figure 5 DIFF-FOLEY generates highly synchronized audio with original video, effectively capturing the timing of sounds, egg. knife cutting, water flow, and plate clinking, Best viewed in demo link 2
MODEL
PARAM.
PRETRAINED
IS 1
FID
KL
ALIGN ACC (%-)
DIR--OLBB-S
335M
X
16.96
27.06
7.10
80.18
DIF--OLEY-M
553M
X
18.48
29.10
6.94
80.74
DIFF-FLLY-LL
DIFF- L.
859M
V
52.07
11.61
6.33
92.35
Table 5: Model Scaling Effect: Only use CFG with w
4.5 and DDIM 43
Sampler (250 steps). PRETRAINED
refers to using pretrained SD-VI V1.4
(38 model as an effective parameter initialization for training LDM.
METRICS
MODEL
STAGEI CVAP DATASET
CFG
CG
f
FID
KL
(%)
VGGSound
19.86
18.45
5.41
67.59
VGGSound
16.58
20.20
6.81
62.24
VGGSound
51.42
11.48
6.48
85.8%
DUV-FOLBY (Ours)
VGGSound
VGGSound AudioSet V2A
X
X
22.07
18.20
6.52
69.41
VGGSound AudioSet-V244
17.57
20.87
6.69
66.05
VGGSoumd AudioSet- VZA
52.07
11.61
6.33
92.35
VGGSound AudioSet V2A
60.39
10.73
6.42
94.78
Table 6: Abalation Study: Evaluating the impact of Stagel CAVP pretrained datasets scale and guidance Sampler with 250 inference steps.
techniques with CFG scale 4.5, and CG scale = 50, using DDIM 43
4.3 Ablation Study
Model Size & Param Init. We explore the impact of model size and initialization with pretrained Stable Diffusion-V V1.4 (SD-V1.4) on FILE-FOLEY. We trained three models of varying sizes: DIFF- FOLEY-S (335M Param.), DIFF-FOLEY-M (553M Param.), and DIFF-FOLEY-L. (859M Param. the same architecture as SD-V1.4). We report detailed architectures of these models in Appendix, A Our results in Table 5 show that increasing model size improves performance across all metries,

---

demonstrating DIFF-FOLEY's scaling effect. Moreover, initializing with SD-V1.4 weights (pretrained on billions of images), sigmificantly improves performance and reduces training time due its strong denoising prior in image/Mellsjeectrogrann latent space (see the last row in Table
Guidance Techniques & Pretrained Dataset Scale We conduct extensive ablation study on DIFF-
FOLEY in Table 6 exploring the scale of Stagel I CAVP datasets and guidance techniques. We see that: 1). More CAVP pretraining data enhances downstream LDM generation performance. 2). Guidance techniques greatly improve all metrics except KL 3). Double guidance techniques achieve the best performance on IS and Align Acc (the primary metries).
CFG Scale & Different Sampler We studied the impact of CFG scale w and different samplers on DIFF-FOLEY CFG scale influences the trade-off between sample quality and diversity. A large CFG scale typically enhances sample quality but may have a negative effect on diversity (FID, KI.). Beyond certain threshold, increased
decrease quality due to sample distortion, resulting in U-Shaped curves in Figure 6 (a). Best results in IS and Align Acc were achieved at w = 4.5, with minor compromise FID and KL. Faster Sampling for diffusion models, reducing inference steps from thousands to tens, is a trending research area. We evaluated the usability of our DIFF-FOLD using current accelerated diffusion samplers. We compare three different samplers, DDIM 4K] PLMS (88). DPM-Solver [30] with different inference steps in Figure O (b). All three samplers converge at 250 steps, with APM-Solver converging in only 25 inference steps, while others require more steps to converge. DPM-Solver allows for faster DIFF-FOLDE inference, making it more accessible.
5 Limitations and Broader Impact
Limitations DIFF-FOLEY has shown great audio-visual synchronization VGGSound and EPIC- Kitchens, however its scalability on super large (billion-scale) datasets remains untested due to limited data computation resources. Diffusion models are also slower than GANs due the iterative reverse sampling process.
Broader Impact The advancements in V2A models, such as DIFF-FOLEY, have the potential to significantly expedite video production processes, offering efficiency gains the entertainment and media sectors. However, with most powerful technologies, there's flip side. There's 's underlying risk of these models being misused to create mislcading or false content. As such, while the potential benefits are vast, it is imperative for developers, users, and regulators to exercise caution.
Conclusion
We introduce SIFF-FOLEY, a V2A approach for generating highly synchronized audio with strong audio-visual relevance. We empirically demonstrate the superiority of our method in terms of genera- tion quality. Moreover, we show that using double guidance technique to guide the reverse process in LDM can further improve the audio-visual alignment of generated audio samples. We demon- strate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning. Finally, we conduct an ablation study, analyzing the effect of pretrained dataset size, various guidance techniques, and different diffusion samplers.
References
[1] Vanessa Theme Ament. The Foley grail: The art of performing sound film, games, and animation. Routledge, 2021.
[2] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[3] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions Image Processing, 29:8292 8302, 2020.
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
10

---

[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advunces in Neural Information Processing Systems, 34:8780-8794, 2021.
[6] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2203 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. TEFE, 2023.
[7] Ariel Ephrat and Shmuel Peleg. Vid2speech: speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5099, IEEE, 2017.
[8] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis, Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2332.03011, 2023.
[9] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202- 6211, 2019.
[10] Jort F Gemmeke, Daniel PW Ellis. Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[11] Sanchita Ghose and John J Prevost. Foleygan: Visually guided generative adversarial network-based synchronous sound generation in silent videos. IEEE Transactions on Multimedia, 2022.
[12] Daniel Griffin and Jae Lim, Signal estimation from modified short-time fourier transform. IEEE Transac- tions on acoustics, speech, and signal processing, 32(2):236 22222333333322 1984.
[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-776 2016.
[15] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2txXx:2220002000 2022.
[16] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[17] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
[18] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint (tt.::2024004455. 2022.
[19] Chenxu Hu. Qiao Tian, Tingle Li, Wang Yuping, Yuxuan Wang, and Hang Zhao. Neural dubber: dubbing for videos according to scripts. Advances in neural information processing systems, 34:16582-16595, 2021.
[20] Jaesung Huh, Jacob Chalk, Evangelos Kazakos, Dima Damen, and Andrew Zisserman, EPIC-SOUNDS: A Large-Scale Dataset of Actions that Sound. In IEFE International Conference on Acoustics, Speech, Signal Processing (ICASSP), 2023.
[21] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint ax111:11100000020070
2021.
[22] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video datasct. arXiv preprint arXiv: 2017.
[23] Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022 17033, 2020.

---

[24] Qiugiang Kong. Yin Cao. Turab lqbal. Yuxuan Wang. Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 18:2880-2894. 2020.
[25] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976. 2020.
[26] Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis, Advances in neural information processing systems, 32, 2019.
[27] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang. and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint 1rXiv:2301.12503, 2023.
[28] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pscudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 2022.
[29] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.
[30] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm solver++: Fast solver for guided sampling of diffusion probabilistic models, arXiv preprint arXiv:2211.01095, 2022.
[31] ALYSSA MAIO. What is foley artist how bring movies life with sound, 2022.
https
//www. //////stuee stw.iobiidddd..... www.oooo///..o.ooooooooooooooo. what is- a- folyy-oolyrr-ttttt
[32] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
[33] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 10X.v9160..03499 2016.
[34] Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.0374, 2018.
[35] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405 2413, 2016.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamola Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204, 06125,, 2022.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36477 2022.
[40] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08400 2022.
[41] Roy Sheffer and Yossi Adi. hear your truc colors: Image guided audio generation. arXiv preprint urXiv:2211.00 2022.
[42] Jascha Sohll-Dcckstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermoyynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
12

---

[43] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint urXiv:2010.02502, 2020.
[44] Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415- 14288, 2021.
[45] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rXiv:2011.33456. 2020.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin. Attention you need. Advances neural information processing systems. 30, 2017.
[48] Dongchao Yang, Jianwei Yu, Helin Wang. Wen Wang, Chao Weng. Yuexian Zou, and Dong Diffsound: Discrete diffusion model for text-to-sound generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[49] Hongyi Zhang, Moustapha Cisse. Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
[50] Yipin Zhou, Zhaowen Wang, Chen Fang. Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550 3558, 2018.
13

---

Appendix
The Appendix is organized as follows:
Sec. Ajeabbraates the implementation details of DIFF-FOLEY, including model architectures, data preprocessing and training/evaluation details for different stages models and tasks.
Sec. B presents additional visualization results of Video-to-Audio generation on the VG- GSound 2
dataset and downstream finetuning results on EPIC-Kitchens (20) dataset, further demonstrating the superior performance of DIFF-FOLEY in synchronized audio generation.
Sec. [provides theoretical and intuitive perspectives on the mechanism of Double Guidance techniques discussed in Sec. 3.4
Sec. Deexamines the reconstruction capability of the Stable Diffusion 38 frozen pretrained Encoder & and Decoder D Mel-spectrograms.
A Implementation Details
A.1 Model Architectures
Our proposed DIFE-FOLEY employs two-step procedure for synchronized Video-to-Audio gener- ation. The first stage, Contrastive Visual Pretraining (CAVP), aims to learn temporally and semantically aligned audio-visual features to capture the subtle connection between the audio and visual modalities. The second stage involves training latent diffusion models (LDMs) conditioned on the CAVP-ale CAVP-aligned visual features in the Mel-spectrogram latent space. In this section, we present detailed model architectures for each of the model components in DIFF-FOLEY.
Stagel CAVP. CAVP adopts similar two-stream encoder design CLIP [36). consisting of an audio encoder sa(-) and a video encoder [v(.). For the audio encoder (-). we adopt the Mel-spectrogram
encoder design from PANNs (24,, which is a 2D convolution- based encoder with ~81 parameters.
For the video encoder (vv we use the SlowOnly branch design from the SlowFast [9] architecture, which is 3D convolution- based backbone with ~30M parameters. During CAVP training, incorporate the pretrained weights of the audio encoder from PANNS [24), which were initially trained on the large-scale AudioSet [10] dataset. Additionally,
adopt the pretrained weights of the video encoder from SlowFast [9), which were trained on the Kinetics-400 [22] dataset. For audio and video input pairs (Maa T.), where Ta E R XM a Mel-spectrogram with M mel basis and 7' is the time axis of the spectrogram, and 2hp E RTx3* xW is video clip with T frames. The audio encoder f-(-) and video encoder (-) are used to extract audio feature Eg E R and video feature E. E TxC Specifically, the audio encoder fa(-) encodes the spectrogram to match the time dimension of the video frames. In CAVP, we set the feature dimension = 512. By using Temporal Max-Pooling Layer P(.), we obtain temporal-podd audio and visual features Eg = -P{,a) E TRC E. = P(E.) ) E C
Stage2 LDM. For LDM, we use the same UNet backbone architecture in Stable Diffusion-Vi V1.4 (SD-V1.4) 38 Using the frozen pretrained latent encoder E, we encode spectogram into a low dimensional latent 2 Eg(xa) C² J x where the channel number of C' and compression rate of 8. To successfully encode single channel spectrogram Ta into a latent 2o with the pretrained frozen 3-channel image encoder E in SD- -1.4, we first repeat the spectrogram channel into (r., - S. E 3R3 and then obtain 2g as 2n (at.) To decode spectrogram from latent 2g with the frozen image decoder in SD-V11.4 14, we simply use the first channel from the [0, E RT XMM where D(=a) E R3xT' xM [ is the tensor slicing operation). The reconstruction capability of the pretrained frozen image latent encoder & and decoder D on Mel-spectrograms is discussed in Sec. D With the pretrained CAVP model to align audio-visual features, the visual aligned features E. T'C are adopted LDM training. To better learn the temporal relationship between audio and visual modality, we adopt a positional
decoded spectrogram (a..
encoding and projection layer ты to project Es to appropriate dimension with E, = (e(E.)) = MLP(E, + PE)
TTxC", where MLP is the projection layer and PE represents positional encoding. We set C" = 768 for all experiments. We use the same denoising schedule on,, or in SD-V1.4 [38 with T 1000. To learn the audio-visual correlation, adopt the cross-attention
14

---

module from SD-V1.4. Specifically, we apply cross- cross-attention on latents with down-sampling rates of [1,2,
In Sec. 4.3 of the main paper, we use three different model settings of varying sizes: DIFE-FOLRY-S
(335M parameters), DIFF-FOLEY-M (553M parameters), and DIFF-FOLEY-L. (859M parameters). All three models use the UNet backbone with four encoder blocks, a middle block, and four decoder blocks. For DIFF-FOLEY-S, with basic channel number of c = 192, the channel dimensions of the encoder blocks are [c, 2c, 3c, 4c), and the design of the decoder blocks are basically the reverse of the encoder blocks. For DIFF-FOLEY-M. M, we set c 256, and the channel dimensions of the encoder blocks are [c. 2c, 4c, 4c]. Finally, for DIFF-FOLEY-L, we adopt the same architecture as in SD-V1A4 [38], resulting in 320 and channel dimensions of the encoder blocks [c. 2c, 4c, Ac]. We have observed a significant improvement in performance in Sec. 4.3) when using the pretrained weights from VID-V1.4 for parameter initialization. Therefore, we default to using the DIFF-FOLEY-L. model and the pretrained weights from SDD-V1.4 unless otherwise specified.
Vocoder. Vocoder used to transform Mel-spectrogram into waveform signal. There are two categories of vocoders: Traditional algorithm-bass vocoders such as the Griffin-Lim algorithm [12]. which uses an iterative refining strategy to estimate the original sound signal, and deep learning-based vocoders such as WaveNet [33]. MelGAN [26), and HiFi-GAN [23), which are widely used for speech waveform generation. Although our method is fully compatible with deep learning- based vocoders, we choose to use the Griffin- algorithm for audio waveform reconstruction in this paper due to its simplicity.
A Alignment Classifier for Align Acc Metric. To evaluate the generated audio synchronization and audio-visual relevance, introduce Alignment Accuracy (Align Acc.) new metric in Sec. 4 In specific, we train an alignment classifier I's to predict real audio-visual pairs. To train the classifier, we use three types of audio-visual pairs. (1). 50% of the pairs are the real audio-visual pairs from the same video and the same time (Irue pair), labeled as 1; (2). 25% are audio-visual pairs from the same video but temporally shifted, resulting in an out-of-syne audio-video pair (temporal shift pair), labeled as O. (3). The remaining 25% are audio-visuda visual pairs from different videos, resulting in completely mismatched audio-visual pairs (wrong pair), labeled as O. This dataset comprises of 50% aligned audio-visual pairs and 50% unaligned audio-visual pairs (temporal and semantic). By training a classifier on this audio-visual pairs dataset, we can evaluate both audio-visual s synchronization and audio-visual relevance using the alignment probability predicted by the alignment classifier Fo. We use the VGGSound 2 datasets, and our alignment classifier Fg achieves 88. 18.31% accuracy on test set. The detailed metrics of our alignment classifier is Recall: 84.92%. Precision: 91.32%, and Accuracy: 88.31%. For the model architecture of the alignment classifier F Eeaymee we adopt the lightweight U-Net encoder block design in SD-V1.4. The basic channel number is set to e 128, and the channel dimensions of the encoder blocks are [c, 2c, 2c]. Cross-attention is applied to latents with down- samplingg rates of [2, 4] and the model has ~ 122 parameters. The alignment classifier F takes two input. The encoded spectrogram latent 2p. and the visual aligned features E, from Stagel CAVP. The predicted alignment label ý is computed as follows:
Fo E.).
Here, we emphasize that the the alignment classifier F for align acc metric is different from the sync classifier FPG used in double guidance techniques. When utilizing the double guidance techniques, we did not access the alignment classifier that used for accuracy measurement, In specific, the classifier F FD used in double guidance is noisy classifier, and it takes the noisy latent 2,, time embedding f, and visual aligned features E, as input, the predicted alignment label ý is computed as follows: 인 = FFDG (ze, E.). While the sync classifier Eaayee for accuracy measurement, it only takes two input, latent 20 and visual aligned features E, to predict the synchronization label with i = Fr (zo, E.).
A.2 Data Processing
In this section, we provide additional information on data processing for reproducibility.
Video-to-Audio Generation. To train Stage 1 CAVP, we use the VGGSound [2] and AudioSet-V22 10 datasets. The videos are sampled at 4 FPS, resulting in 40 frames for each 10-second video sample. The frames are then resized to 224 x 224. For audio data, the audios are sampled at 16KHz and then transformed into Mel-spectrograms, using FFT Num 1024, mel basis Num 128 and hop size
250. This results in 10-second spectrogram with the size of 640x x 128 (T' xM) ). We use hop size of
15

---

250 in CAVP for better audio-visual data temporal dimension alignment. To train CAVP, we randomly extract 4-second audio-video frame pairs from the 10-second samples, resulting in Ia E 2526xx and 2h E TR16X3X24XX22 We then pass the audio-visual pairs through the audio encoder (a(+) and video encoder (+)+) respectively. This results in Ea E R16x612 and E. E R16x512 In Stage? LDM training, we only use the VGGSound datasets, which is consistent with the baseline setting. We use ~ second audio and video pairs for training LDM. In specfic, we use FFT Num 1024, mel basis Num 128, and hop size 256 to transform the ~ 8 seconds audios into Mel-spectrogramss with the size 512 x 128 (T" x M), resulting in Ta E IR-12x122 and Zo E 12-2x3x224x22 The LDMs takes the encoded spectrogram noisy latent 2 R 16x64 time embedding and CAVP- visual features Eg E 32 as input: c0(2,, t, E.).
Downstream Finetuning, For downstream finetuning, we utilized the high-quaitty egocentric video dataset, EPIC-Kitchens [20]. which contains approximately 100 hours of daily activities and object interactions in kitchens with minimal sound noise. The sound categories in EPIC-Kitchens are significantly different from those in VGGSound 2] For dataset processing, we randomly divided the original video dataset into training and testing sets, with 90% of the total duration assigned to the training set and 10% to the testing set. We split the original videos into 10-second segments for each dataset, resulting in about 26k training samples and 2.9k testing samples. It's important to note that the video clips in the training and testing sets were sourced from different videos and do not overlap. The processing steps for the datasets were the same as those described in Sec. [A.2] Notably, not finetune the Stagel CAVP models, but instead directly adopted the pretrained CAVP model to extract aligned visual features for EPIC-Kitchen. videos. Subsequently, we finetune the Stage2 LDM, which was originally trained on VGGSound, using pairs of extracted CAVP visual features and audio from
did
datasets.
A.3 Training Details
CAVP Training We train the CAVP model for 1.4M steps on 8 A100 GPUs, with total batch size of 720 using automatic mixed-precision training (AMP). We used the Adam (29] optimizer with learning rate of 8e-4 and 200 steps of warmup. The total training time for the CAVP model was approximately 80 hours.
LDM Training We initialize the LDM model with pretrained weights from SD-V11.4 38 and train LDM for 24.4K steps on 8 A100 GPUs with a total batch size of 1760. We used the AdamW (2 optimizer with a learning rate of le-4 and 1000 steps of warmup. The total training time for DIFF-FoLEY-L approximately 60 hours.
Downstream Finetuning For downstream finetuning, we finetune the DIFF-FOLEY-L model (trained on VGGSound) on EPIC-Kitchens for 3.2K steps 8 A100 GPUs with a total batchsize 1760. We use the Adam (29] optimizer with the learning rate lc-4 and 1000 steps of warmup. The total downstream finetuning time is around 9 hours.
A.4 Evaluation Details
For metric evaluation, we use Inception Score (IS), Frechet Distance (FID) and Mean Kull- back-Leibler Divergence (MKL) from [2 For comparison, we use two current state-of-the V2A models: Spec 21 and Im2Wav [4I]. We use the pre-trained models provided by the authors, and these models are both trained on VGGSound 2 datasets. SpecVQGAN (1 utilizes videos with 21.5 FPS to generate 10-second audio samples conditioned 10-second video features, while Wm2 [4 utilizes videos with 30 FPS to generate 4-second audio samples conditioned on 4-second video features. In contrast, our DIFF-FOLEY generates 8-second audio conditioned on 8-second video features. To ensure fair comparison, for SpeeVQGAN, we generate 10s audio samples and truncate them to the first 8 seconds. For Im2Wav, we generate the first seconds of audio based on the features extracted from the first 4 seconds of the video, followed by generating the (4s-8s) audio conditioned on the (4s-8s) video features. We then merge the two audio segments to obtain the final 8 seconds audio output. Lastly, we compare the generated 8-second audio samples with the ground truth Mel-spectrogram, Since our generated Mel-spectrogram has a Mel basis num of 128, while the baseline ground truth Mel-spectrogram has a Mel basis num of 80, we transform our generated Mel-spectrogram into 80 Mel basis num for a fair comparison. For each video sample in the test set, generate 10 audio samples for evaluation, resulting in around 145K generated audio
16

---

samples. For human evaluation, we conducted human evaluation by randomly selecting 60 videos from the VGGSound test set and having different models generate corresponding audio samples. The output and ground truth audios were anonymized and rated by 30 people unfamiliar with the project. Each sample received scores from at least raters, ranging from (bad) to (excellent) for content relevance and synchronization. The scores were then scaled by a factor of 20.
B More Generation Results on Video-to-Audio Generation
B.1
V2A Generation Results on VGGSound.
We present additional generation results for Video-to-Audia generation on VGGSound dataset in Figure 7 We observe that DIFF-FOLEY effectively generates the corresponding sounds (e.g.. gunshots and underwater bubbling) at the appropriate time, which matches the ground truth audio. In contrast, other methods fail to generate highly synchronized audio that aligns with the video content. This supports the superior performance of our approach. For more details, please visit our demo websites.
B.2 Downstream Finetuning Results EPIC-Kitchens.
We have included additional downstream finetuning results on the EPIC-Kitchens dataset in Figure Our observations indicate that DIFF-FOLEY generates highly synchronized audio that effectively captures the timing of sounds with the original video. In the first video, DIFF-FOLEY produces the corresponding sound of opening drawer at both the 1st, 3rd seconds. In the second video, successfully generates the sound of plates clinking and placing at both the 1st, 4th seconds. For more details, please visit our demo websites. These results demonstrate the potential of DIFE-FOLEY to adapt to other video datasets, making it easier and more promising for future users to generate specific types of audio data through downstream finetuning.
C Guidance Techniques
C.1 Guidance Techniques Overview
Guidance techniques are commonly utilized in the reverse process of the diffusion model to achieve controllable generation. Currently, there are two main types of guidance techniques: Classifier Guidance [5] (CG) and Classifier-Fre. Guidance [17] (CFG). For CG, it additionally trains a classifier Ps (e.g. class-label y classifier) to guide the reverse process at each timestep using the gradient of the class label loglikelihood V., log Po (y) resulting in the conditional score estimates [5]
CG:
(o(, <- c0 1) v1 7 V x, log (y(rrs)
(10)
where is the CG scale. For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates (17). where the e is the condition and w is the CFG scale.
CFG: : E)(x,, c) + weo(2,, t, c) + - w)oo
(11)
When w 1, CFG degenerates to conditional score estimate. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label y. For instance, if we want to generate images with specific attributes (e.g. a girl with yellow hair) from pretrained image diffusion model, we can simply train a 'girl with yellow hairs' classifier to guide the generation process. In V2A setting, the desired property refers audio semantic and temporal alignment.
In Sec. 3.4 we discover that the CG and CFG methods are not mutually exclusive. We then propose a double guidance (DG) technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specfic, for CG we train an noisy alignment classifier Po(y)2e, f, E.)) that predict whether an audio-visual pair is a real pair in terms of semantic and temporal alignment. The noisy alignment classifier Po Fr, L, E.)) takes noisy latent 2t, time embedding I, and the CAVP-aligned visual features E,, as input. We follow similar training process in Sec. A. (see Align Acc.) to train the noisy alignment classifier Po 24 4, F.). For CFG, during training, we randomly drop condition E. with probability 20% to estimate the conditional
17

---

Input Silent Video: Gun Shooting
Ground Truth
[Gen Shot
SpeeVOGAN OGAN
Fails penerate
Im2Wav
Difi-Foley (Ours)
(Gun Shot)
Input Silent Video: Underwater Bubbling
Ground Truth
QOAN
Im2Wav
Diff-Foley (Ours)
Under Water
that only the DIFF-FOLEY generate the corresponding gun shooting sound the appropriate time given the video content (see the 4th, second.), while other methods fails to generate the corresponding gun shooting sound. underwater bubbling video, DIFF-FOLEY produces highly realistic audio that captures the transition from above to underwater (see the 4th second), while other methods fail to capture this transition. For more details, please visit our demo websites.
Figure 7: More V2A results on VGGSound: In the first gun shooting video,
the second
score c0(21, f, Ev) and unconditional score co(2,, Ø). Then double guidance is achieved by the improved noise estimation:
Eo(2,, F..)
F.)) < weo(s, 1, F.) + (1 (,0) 04 V., log Po(y 1, F.o) (12)
where w,7 refer to CFG, CG guidance scale respectively.
18

---

Open Drawer
Open Drawer)
(Open Drawer)
Ground Truth Audio
(Open Drawer)
(Open Drawer)
Generated Audio (Ours)
Get Plate from the Cupboard
5s
68
7s
(Plate clincking)
(Drop the plate)
Ground Truth Audio
(Plate clincking)
(Drop the plate)
Generated Audio (Ours)
Figure 8: More downstream finetuning results on EPIC-Kitchens: DIFF-FOLEY accurately generates the sound of opening drawers at the st,, 3rd seconds in the first video, and produces the sound of plates clinking and placing the 1st, 4th seconds in the second video, showing the potential of DIFF-FOLEY on downstream finetuning. more details, please visit our demo websites.
C.2 Theoretical Perspective of Double Guidance
Although the approach of double guidance may appear heuristic, we provide theoretical perspective for explaining the underlying mechanism behind this techniques. The goal of guidance techniques is generate sample from the conditional distribution: p(zel)) ly), where 2; is the generated noisy sample at each reverse process timesteps and the is the condition or specific attribute. In diffusion models, the (z y. t) estimate the conditional score function, which is Vx, log ((2
For CG, we first derive the mechanism behind CG [5] method show how CG help in controllable generation. With the bayes's formula, we have:
p(z p(ylz)) p(2))
p(y)
(13)
log pgp(2 (2) log log x)) + log p(2)) - log p(y)
V., loggol = V., log ((y) 2.) V., logp((z))
where the Zz, logp(y) 2x) is actually the classifier guidance (CG) term. enhance the conditional
singal y. we scale the V., log 1lgggppyla p(yl2x) term by factor y, also known as CG scale, then we obtain the CG improved score estimation in Equation I0]
V., log log = V., log p())) + + (oy)()))
(14)
(CG): (0(=..11) + 50(e, () - v1 a Vay log log
(15)
where Vz, log (() = - -i-a. co (2(,
16
19

---

For CFG, we also derive the mechanism behind CFG [7 method. To improve the conditional signal c, CFG use the mixture of conditional score =., log p(2 ()) and unconditional score V log p(2+) to
enhance the term V log Using the bayes formula, we have:
p(c(2)) p(zz(c) p(c)
p(2()
(16)
logp(c(2)) log p(z((c) + logp(c) logp(zz)
v., log((22))) =., logp(z)(c) x. log p(z))
then we scale the V. as log p(c]2) with a factor of w (CFG scale), and substitute to Equation 14 then obtain the CFG improved score estimation in Equation
21, (og Vs, logp(=)) w( logp(z((c) vs logp(z)))
(17)
(CFG): Co(2,, t wro t, c) + w)co(ze 2))
(18)
where Ve, logg(=+) = -- VI-a.
En to (2,, and v., log = - c0 l).
For double guidance (DG), we construct more fine-grained and controllable conditional distribution p(zelc,)) v) that takes into account two conditions, and y, where is the general condition and another data attribute label. Specifically, DIFF-FOLEY, e refers to the CAVP-aligned visual features E, (general condition), while y refers more fine- grained and desired property, such as semantic and temporal alignment (data attribute). synchronized audio with strong audio-visuse relevance from this distribution 2 p(z//c, y). To simplify the derivation, we assume that p(yl and p(c(zz) are independent conditioned on 2,- Using the Bayes formula, have:
aims to generate high-quality
p(a(c,)) p(c, v12)) p(z.) p(cl2e) p(y 21) p(z))
p(c,
p(c,y)
(19)
logp(z)l,,)) v) log ((gg(c))))) + logp(ulz)) + p(z) log p(c,
Ve, logp(z/le v., logu(clæ)) + V.. logp(y) [x) v., log
we then use CG scale and CFG scale w to scale the CG term V log logp())))) and CFG term
v., log p(cl(() respectively:
2A, logPu, = V., logp(c/z) + vV, log + v., logp(zt)
,V log v., log logp(zz)) V. logp(y|zz) + V., logp(z))
= -w log log 23 + (1 - V log p(z) + v lugl())z)
(20)
Finally, we obtain the double guidance (DG) improved score estimation in Equation
(DG): Eo(=e,t, t, Fee) weo(=e, t, F.) (1 w) >VT V. log Polulee, t, Fe) (21)
where is the CAVP-aligned visual features Ev, y is the semantic and temporal alignment label, w CFG scale and 7 is the CG scale.
C.3 Intuition Perspective of Double Guidance
We offer another intuitive perspective to explain the technique of double guidance. In practice, the CFG [17] method tends outperform the CG (5) method. This is because the classifier in the CG method may learn shortcuts for classification, leading to good classification results, but when its loglikelihood gradient is used to guide the reverse process, can generate incorrect samples. In contrast, the double guidance technique addresses this issue by leveraging both the CFG and CG terms. The CFG term provides a robust and reliable noise direction for each timestep the reverse process, while the CG term refines the CFG term's 's direction towards the desired data attribute, resulting better sample quality. The effectiveness of the double guidance technique is validated by the evaluation results in main paper Table. 6
20

---

C.4 The Effect of Guidance Techniques
We present the generated results using double guidance techniques in Figure 9 The first row showcases the ground truth Melsspecto and video frames, depicting man smashing window. Subsequent rows show the generated Mel-spectrogram results, arranged from left right with progressively larger CFG scales w and from top to bottom with increasingly larger CG scales Moving from left to right, we observed significant improvement in audio quality and synchronization with increasing CFG scale w. Additionally, with increasing CG scale 7 the generation results of the original CFG scales were progressively refnned leading higher levels of audio-visual alignment in the generated results.
Ground Truth Spec.
Ground Truth Video
CFG ==
CFG 2.0
CFG 3.0
CFG @ 4.0
CG y=0
CG 1-
CG yy
CG CG y-45
Figure 9: Results of double guidance techniques: The first rows shows the Ground Truth Mel-spectrogram and
video frames. The subsequent rows exhibit generated results with progrssively larger CFG scales w and CG scales
21

---

Reconstruction Metrics
Model
FID
KL
MSE І
DIFF- FOLEY (Encoder & Decoder D)
9.20
1.44
0.00264
Table 7: Reconstruction results of SD-V1.4 frozen latent encoder and decoder D. MSE represents the square error the reconstructed sel-sppctrogram. The frozen latent encoder and decoder D exhibit strong
capability in reconstructing the Mel-spectrogram.
Ground Truth
Reconstruction
(a).
(b).
Figure 10: Visualization results of Mel-spectrogram reconstruction. The frozen latent encoder & and decoder
D from SD-V1.4 3 demonstrate exceptional capability in reconstructing the Mel-spectrogram.
D Discussion of Stable Diffusion Encoder and Decoder
Reconstruction Capability of SD Encoder & Decoder In DIFF-FOLEY, directly the frozen pretrained latent encoder and decoder 2> in SD-V1.4 [38] to encode the Mel-spectrogram to low-dimensional latent (mentioned in Sec. A.I]. The reason we choose not retrain encoder & and decoder 'D on the Mel-spectrogram data is that using the pretrained SD-VI. weight for parameter initalization can significantly reduce model training time and improve generation quality (Already discussed in main paper Table. 4). However, is necessary to use the corresponding frozen latent encoders and decoders D when using the pre- pre-trained SD-VI. V1.4 model. Interestingly, despite being trained on large scale image datasets (LAION-5B 440), we found that latent encoder & and decoder D demonstrate good reconstruction capability on Mel-spectrogram data. We report the reconstruction metrics results of SD latent encoder & and decoder ' in Table. 7 It can be noticed that the latent autoencoder (E, D) has relatively good reconstruction capability, as demonstrated by the KI. and MSE metrics in Table. / However, the FID score is slightly higher, which may explain why DIFE-FOLEY does not exhibit significant advantage over other baselines in terms of FID, which is likely due to the limitation in the frozen pretrained autoencoder reconstruction capabilities. We also visualize the reconstructed Mel-spectrogram results in Figure I0 showing that frozen pretrained and D have excellent reconstruction capability on Mel-spectrogram data.
22


Reviews:
[{"review_id": "SEHX3zM9a4", "reviewer": "Reviewer_k7sd", "summary": "The focus of the paper is audio synthesis. More specifically, it focuses on video to audio synthesis and in particular on synchronized synthesis of audio. It relies on Latent Diffusion models for the synthesis and proposes an aligned audio-visual representation learning approach to improve synchronization of synthesized audio and input video. CAVP - Contrastive Audio-Visual Pre-training – tries to learn these features through contrastive learning. Experiments are conducted on the VGGSound dataset and both quantitative and qualitative results are shown in the dataset. In quantitative terms, the proposed DIff-foley method outperforms prior works by a considerable margin with respect to some of the metrics. ", "questions": "\nPlease respond to the points outlined in the weakness section. \n", "limitations": "The authors discuss limitations of the method in terms of scalability. A bit more discussion on how generative AI for audio can have societal impact might be good. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- The paper addresses a key problem in audio/video synthesis. Generating synchronized audio for videos is challenging and most of the current struggle at. \n\n-- Learning good audio-visual representation is essential to solving this problem. The approach taken by this paper makes sense. \n\n-- The gain in IS metric and inference time using the proposed method is large. Moreover, the demo and the qualitative results does show superior and more synchronized \n", "weaknesses": "\n-- In Eq 1, aren’t the two terms same ? Why have they been separated ? Same is true for Eq 2. \n\n-- Analysis and discussion on duration as a factor is completely missing. It appears that all empirical results are given for fixed 8 seconds audio. Some results for variable length video inputs would be helpful to understand the duration factor. More importantly, producing synchronized video on longer form audio/video would be more challenging than relatively short audio/video. How does the method work out in those cases ? \n\n-- WHile the Diff-foley method does well (compared to others in Table 1) in terms of IS and synchro. Acc. it does not do well in terms of FID and KL. Some discussion on why that might be happening is desirable. Diff-foley is significantly inferior to other methods on these metrics. In Table 2, on impact of visual features - CAVP ends up doing better than others in terms of FID but not IS and KL. Overall, these results and lack of discussion in what might be happening leads to unclear understanding of the performances. \n\n-- Audio/speech synthesis works in my opinion should include some form of subjective tests. Objective metrics (especially when some of them are basically imperfect pre-trained models) do not paint a clear picture. This is also necessary because the generated sounds themselves are often not corresponding to the visual objects. The frequency content of the generated sound in Fig 3 and Fig 4 is clearly far from the ground truth. From the demo also, in many cases the generated sound does not correspond to the visual object. \n\n\n--- updated score after rebuttal --- ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the positive feedback from you. We respond to the weakness and questions below.\n\n### 1. Formula Clarification.\n\n> *1.1: In Eq 1, aren’t the two terms same ? Why have they been separated ? Same is true for Eq 2.*\n\nIn Eq 1, the two terms are distinguished by the normalized value in the denominator. The first denominator term, $\\sum_{k=1}^{N_S}\\exp{(sim(\\bar{E}_a^i, \\bar{E}_v^k)/\\tau)}$,  fixes the i-th audio embedding $\\bar{E}_a^i$ and sums over the exponential similarity score with different video embeddings $\\bar{E}_v^k$ in a batch. \n\nThe second denominator term, $\\sum_{k=1}^{N_S}\\exp{(sim(\\bar{E}_a^k, \\bar{E}_v^j)/\\tau)}$, does the opposite, fixing the j-th video embedding $\\bar{E}_v^j$ and summing over the exponential similarity score with different audio embeddings $\\bar{E}_a^k$ in a batch. This loss function is almost the same as the one used in CLIP model. A similar explanation applies to Eq 2.\n\n### 2. About variable length.\n\n> *2.1:  Analysis and discussion on duration as a factor is completely missing.*\n\nThanks for pointing out. While our experiments have thoroughly verified the effectiveness of different modules, we haven't specifically addressed the duration factor, as this wasn't discussed and solved in previous works like SpecVQGAN and Im2Wav. Diff-Foley is currently designed to support audio generation up to 8 seconds in duration. For shorter videos, we extend them to 8 seconds using zero-padding, generating the full audio and trimming as needed. For longer videos, we segment them into 8-second chunks and concatenate the resulting audio, although this may lead to discontinuities. This maximum duration constraint is a common challenge, shared with previous works such as SpecVQGAN and Im2Wav, and presents an area for future exploration and improvement.\n\n> *2.2: How does the method work out in longer form audio/video cases?*\n\nWe recognize the importance of handling longer form audio/video cases. Currently, Diff-Foley, like SpecVQGAN and Im2Wav, has a maximum duration constraint. To handle longer content, we divide the video into 8-second segments, process them individually, and concatenate the resulting audio. This can lead to inconsistencies between segments. While we've considered potential solutions such as: **(1). Transforming Diff-Foley into the form of auto-regressive generation**. **(2). Training Diff-Foley on longer audio-video pairs**. Still, we believe our current contributions are substantial. The validation and implementation of these potential solutions present exciting avenues for future work.\n\n### 3. Discussion on Metrics.\n\n> *3.1: Diff-Foley does not do well in terms of FID and KL. Some discussions on why is desirable.*\n\nThanks for pointing out. Please refer to the **Global Response 2**.\n\n> *3.2: Ablation study on visual features CAVP ends up doing better on FID but not IS and KL.*\n\nOur ablation study shows that CAVP features significantly enhance audio-visual relevance and synchronization, as evidenced by the substantial improvement in Align Acc metrics. Also, as discussed in Q3.1, the rich semantics in CLIP feature (training with billions of text-image pair) indeed contribute to better KL metrics,  We expect such a gap can be bridged by expanding CAVP datasets to similar scale compared with CLIP.\n\n### 4. Subjective tests and clarification for demos.\n\n> *4.1: Audio synthesis should include subjective tests.*\n\nThanks for your valuable advice. Please refer to the **Global Response 1**.\n\n> *4.2: Frequency content of generated sound in Fig 3,4 is far from the ground truth.*\n\nThanks, we would like to emphasize that the goal of audio generative model is not to perfectly reconstruct the audio based on video content - a task that is, in fact, impossible also unnecessary. Our aim is to **generate audios that align well with human perception**, even if they differ from the ground-truth audio. For instance, in Figure 3, the audio generated by Diff-Foley remains nearly silent until the golf ball is struck, a result in tune with human perception, regardless of variations in the ground-truth audio, such as wind noise, resulting differences in spectrogram. Diff-Foley precisely generates the sound at the moment the golf ball is hit, showcasing its superiority in creating synchronized audio and significantly outperforming other methods. Audios in Figure 3 and Figure 4.c might appear different from the ground-truth spectrogram, yet they're sensible and well-aligned with human perception. Feel free to revisit these examples on our website. We believe you'll find they are reasonable and align well with human perception.\n\n> *4.3: Some generated sound in demos does not correspond the visual object.*\n\nOur human evaluation results on content relevance metrics in **Global Response 1** and other evaluation metrics in Table 1 effectively support our points that Diff-Foley show superiority on generating highly synchronized audio with strong audio-visual relevance compared with other methods. We acknowledge that some extremely challenging cases in the demos may yield unsatisfactory results across all methods.\n\n### 5. Limitations.\n\n> *5.1: A bit more discussion on how generative AI for audio can have societal impact might be good.*\n\nThanks for pointing out. We will incorporate further discussions on the societal impact of generative AI for audio in the final version of our paper. \n\n\n\n\n\n\n\n"}, {"review_id": "0b8mzGc7Qe", "reviewer": "Reviewer_tzzm", "summary": "This paper present DIFF-FOLEY, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates audio with improved synchronization and audio-visual relevance. The method adopts contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. During inference, the method adopts a combination of classifier-free guidance and classifier guidance based on a synchronization classifier. The proposed model outperforms existing methods in audio-visual synchronicity and inception score.\n", "questions": "* What is the audio-visual synchronicity classifier used in the model? Is it the same with the classifier for accuracy measurement? \n", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed model achieves greater audio-visual synchronicity compared to several baselines. Through analysis, the audio-visual pre-training module indeed brings significant gains to the synchronicity. The paper is overall well-written and the authors have conducted a thorough analysis on the effectiveness of different modules, including the effect of two guidances, the used features, effect of pre-training.\n\n", "weaknesses": "Despite its advantages in audio-visual synchronicity and inception score, the proposed method falls below the baselines in other metrics such as FID and KL divergence. There is no human evaluation metrics in the comparison among the methods. Thus it is unclear overall how realistic the generated audios are compared to other methods. \nRegarding the synchronicity classifier, the authors have not thoroughly analyzed the synchronicity classifier, which is the key model to measure the main metric used in the paper. The paper mentioned it achieves 90% accuracy on the test set. However, it is unclear how the test set is constructed. For example, how is the negative-sample set constructed? What is the precision and recall respectively? How well does the accuracy match the human perception of audio-visual synchronicity?\nIn terms of modeling, the generative model is a typical LDM model used in audio generation and lacks general novelty. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the constructive feedback from you. We address your concerns below.\n\n### 1. Discussion on metrics.\n\n> *1.1: The proposed method falls below the baselines in other metrics such as FID and KL divergence.*\n\nThanks, please refer to **Global Response 2**.\n\n### 2. Human evaluation.\n\n> *2.1: There is no human evaluation metrics in the comparison among the methods.*\n\nThanks for your valuable advice, please refer to **Global Response 1**. \n\n### 3. About classifier for accuracy measurement.\n\n> *3.1: Unclear how the test set is constructed? Recall and Precision? Human perception of audio-visual synchronicity?*\n\nThanks for asking the details of classifier for accuracy measurement. While we have a detailed discussion in supplementary Section A.1, we're adding further discussion here to fully address your concern.\n\n**(1). Test set construction:** As mentioned in Line 188-191, we train the classifier with three types of pairs: true (label 1), temporal shift (label 0), and wrong (label 0). The test set is constructed similarly. Using a fixed random seed, each original audio-video pair in the VGGSound test set is either left unchanged with prob. 50%  (true pair, label 1), temporally shifted with prob. 25%  (temporal shift pair, label 0), or mismatched with another video's audio with prob. 25\\% (wrong pair, label 0). The total test set is around 14K samples, with 50% true pairs **(positive sample set)**; and 25% temporal shift pairs + 25% wrong pairs **(negative sample set)**.\n\n**(2). Precision and Recall:** \n\nDetailed classifier's metrics are provided here. **Recall: 84.92%**, **Precision: 91.32%**, and **Accuracy: 88.31%**.\n\n**(3). Human Perception Alignment:** As discussed in Q2, human evaluation results on content relevance and synchronization are provided. We observe that **content relevance and synchronization assessments from our classifier closely align with those from human evaluators**, confirming the effectiveness of our sync classifier.\n\nWe will add the above details to the supplementary. Thanks for your advice again.\n\n> *3.2: What is the audio-visual synchronicity classifier used in the model? Is it the same with the classifier for accuracy measurement?*\n\nFirst of all, it's important to note that these are two distinct classifiers and are trained in different ways. When utilizing the double guidance techniques, we did not access the alignment classifier that used for accuracy measurement. In specific, the classifier used in double guidance is a noisy classifier, named $F_\\theta^{DG}$, and it takes the noisy latent $z_t$, time embedding $t$, and visual aligned features $E_v$ as input, the predicted alignment label $\\hat{y}$ is computed as follows: $\\hat{y}=F_\\theta^{DG}(z_t, t, E_v)$. While the sync classifier for accuracy measurement, named $F_\\phi^{sync}$, it only takes two input, latent $z_0$ and visual aligned features $E_v$ to predict the synchronization label with $\\hat{y}=F_\\phi^{sync}(z_0, E_v)$.\n\n### 4. About model novelty\n\n> *4.1: The generative model is a typical LDM model used in audio generation and lacks general novelty.*\n\nThank you for pointing out. The LDM model is recognized as the current state-of-the-art generative models, having achieved significant progress in fields such as image generation (e.g Stable Diffusion). Its effectiveness make it a natural choice for our work, aligning with our focus on **exploring its application rather than designing an entirely new generative model**. \n\nHowever, we emphasize that our work isn't just a straightforward application of the LDM model. We've introduced significant innovations for audio generation using LDM, including novel techniques like CAVP features, double guidance and temporal split and merge augmentation. These enhancements were rigorously verified to confirm their effectiveness. In this context, the contributions of our article go beyond the mere application of a known model. **We firmly believe that our models present significant contributions and novelty to the field**.\n\n\n\n\n\n\n"}, {"review_id": "JYgmUif6jZ", "reviewer": "Reviewer_55K5", "summary": "The paper presents an approach to generate and align audio with an existing video track, using a diffusion process. This is useful in video post-processing, where frequently sound effects have to be aligned with existing video footage (and not just match semantically). The authors demonstrate the quality of their approach, which relies on a \"double guidance\" technique to improve the quality of the reverse process in LDM, and improve the alignment. Authors perform fine-tuning and ablation studies, to further understand the properties of the proposed approach.", "questions": "- In Section 3.3, you say \"We validate the effectiveness of this augmentation method in Sec 3.3.\" - please explain?\n- In the abstract (and in other places): \"we demonstrate DIFF-FOLEY practical applicability and generalization capabilities via downstream finetuning.\" -- how does fine-tuning, which is kind of a specialization of the model, demonstrate generalization capability?\n- Figure 5 - I am not sure I understood what is presented. The model makes mistakes for the first example - are these being resolved through fine-tuning the diffusion model, or do they still present?", "limitations": "n/a", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality\n\nThe paper presents a novel approach to learning audio-visual alignment, using a CLIP-style loss, taking within-clip and across-clip contrastive pairs from audio and video. This CVAP is a novel contribution, and at the core of the proposed work. LDMs are still a non-standard use in audio, although they are finding applications.\n\nQuality, Significance\n\nGiven the paper's improvements (mostly given through the CVAP formulation and its application to the LDM), I think the paper makes a significant contribution, although the topic is somewhat \"niche\"\n\nClarity\n\nThe paper is mostly well written and covering a reasonable set of experiments, it describes the main argument.", "weaknesses": "I don't understand the effect of temporal split & merge, section 3.3 (also see the question below) -- IIUC, the model could be trained on any audio-video pair, so if AudioSet+VGGSound is not enough, why not simply crawl more data from the Internet? It is clear that temporal augmentation improves results, specifically around the alignment metric, but Table 5 does not seem to indicate that more data always improves results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the constructive feedback from you. We address your concerns below.\n\n### 1. About Temporal Split & Merge Augmentation.\n\n> *1.1: Explain for sentences: \"We validate the effectiveness of this augmentation method in Sec 3.3.\"*\n\nMy apologies for the confusion. The correct sentences should read: \"We validate the effectiveness of this augmentation method in Sec 4.1.2.\" The ablation study in Sec 4.1.2 verifies this temporal augmentation's effectiveness. This error will be corrected in our final version paper.\n\n> *1.2: Model can be trained on any audio-video pair? If AudioSet+VGGSound is not enough, why not simply crawl more data from the Internet?*\n\nYes, you're correct that our model can be trained on any audio-video pair, and more data might lessen the need for Temporal Split and Merge Aug. However, Diff-Foley demands high-quality and natural audio-video pairs with strong audio-visual correlation. Simply crawling more data from the Internet often leads to low-quality pairs. Most online videos contain elements like human speech, unrelated audio, and noise, which can seriously degrade performance. **Cleaning and filtering this data require considerable time and effort**. This augmentation method offers an elegant solution to this problem. By incorporating the temporal alignment prior into the training process, it enables the model to be trained on relatively smaller but carefully filtered, high-quality datasets like VGGSound.\n\n> *1.3: Table 5 does not seem to indicate that more data always improves results?* \n\nIn Table 5, all the metrics has improved except for FID and KL when using more data for Stage1 CAVP. As reviewer MA6i mentioned that FID and KL are known to be somewhat unreliable, however, we included them for the sake of completeness. In general, IS and Align Acc are more convincing metrics that truly reflect audio quality. The results in Table 5 illustrate the potential for enhanced performance by scaling up the dataset to super large scale for Stage1 CAVP, akin to CLIP model.\n\n### 2. About downstream fine-tuning.\n\n> *2.1: How does fine-tuning, which is kind of a specialization of the model, demonstrate generalization capability?*\n\nThanks for pointing out the ambiguity. What we originally meant is that by fine-tuning the pre-trained Diff-Foley model, we can skillfully adapt it to specific sound synthesis tasks, much like fine-tuning Stable Diffusion models for personalized images. This adaptation not only promotes broader applications but also demonstrates the strong generative capabilities of the origin pre-trained model, showing certain extent of generalization capabilities. Thank for your valuable feedback, we'll place greater emphasis on the broad applicability and specialization of downstream fine-tuning to clarify this aspect and make it less ambiguous in final paper.\n\n### 3. Illustration of Figure 5.\n\n> *3.1: The model makes mistakes for the first example.*\n\nThanks for asking. Figure 5 shows the generative results after fine-tuning on the Kitchen dataset. It's important to note that our audio generative model isn't aimed at perfectly reconstructing audio from video content—a task that is indeed impossible. Rather, we seek to **generate various audios that align with human perception, even if they differ from the ground-truth audio**. In Figure 5, Diff-Foley capably generate the corresponding cutting sounds, an outcome that aligns with human perception, despite noise and other variations in the ground-truth audio, which leads to spectrogram differences. Feel free to revisit this sample on our website, you'll find they are reasonable. Despite the subtlety of the cutting movement, which is extremely challenging, Diff-Foley manages to create synchronized audio that aligns convincingly with human perception.\n\n### 4. Update on human evaluation.\n\nFor your information, as suggested by reviewer k7sd and tzzm, we've conducted human evaluation, please refer to **Global Response 1**. This further demonstrate the superiority of our method in generating highly synchronized audio with strong audio-visual relevance.\n\n\n\n\n\n\n"}, {"review_id": "Hu5jNchvpq", "reviewer": "Reviewer_MA6i", "summary": "This paper proposes a new method for performing the task of \"Foley\", which entails the creation of sounds that match a video, for example, making an audio clip in a studio that matches a character's footsteps in a film, as a post-production task. The authors proposed to do this automatically via deep learning, which has been proposed in other works, notably Im2Wav from last year. They propose 4 main contributions: audio-visual contrastive pre-training, latent diffusion modelling, an augmentation technique based on cropping different clips together, and the combination of classifier guidance and classifier-free guidance. Through a set of ablations, the authors show the importance of these four contributions. They also show superior results compared to 2 previous works on 2/4 metrics, and on inference time. Many qualitative results are shown and the appendix includes a large amount of discussion around many of the central topics.", "questions": "As mentioned above, are there any other methods that this work could be compared with in the results section?\n\nWill the code be released upon acceptance? If so, will this include training code, pre-trained models, etc.? These contributions to the open-source community would be very welcome.", "limitations": "Although I realize there are space limitations, the broader impact is really insufficient. Please put a bit more substance into this discussion. For example, you can talk about how this technology could be used to create fake videos more accurately, which could help spread false information. Or perhaps you can talk about the consequences of using producing an inaccurate reproduction of the audio, which may change the meaning of the video and mislead viewers. This may also be used to improve video surveillance, etc.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Fundamentally, the paper has enough contributions, in my opinion. Out of the 4, the first is very substantial, and the other 3 are reasonably novel when applied to this field. The writing is quite consistent with little to no mistakes, and in general the clarity of the discussions is remarkable. The paper does well in introducing readers to the task of video foley, which is not particularly well known amongst most readers, and also does a good job at presenting a lot of the modeling techniques used. The ablations satisfy the vast majority of questions the reader may have, and the more fine-grained ones are a nice addition. Quantitative results are explicit and well-presented. The lengthy appendix is welcome with some fruitful discussions. The demos are really impressive when compared to other works.", "weaknesses": "From the demos and quantitative results, the previous approaches seem to feature quite weak reproductions of the original sound. The authors only compare with two methods, which leaves me wondering if there could be comparisons with more works. I have found, for example, that FoleyGAN (2021) seems to perform the same task, but no comparison (or citation) is present. I believe the results would be a lot more convincing with more comparisons. Also, Diff-Foley is outperformed on 2/4 metrics, which is also not very reassuring, although they are FID and KL which are known to be somewhat unreliable.\n\nWhile the authors clearly show that adding Classifier guidance helps compare to only CFG, I would like to see an ablation where only Classifier guidance is used as well, as a matter of completeness, to show that both components of the double guidance are truly important. \n\nFigure 2 is good for the most part, but the latent diffusion section is, in my opinion, too vague - instead of just having a big box labelled \"latent diffusion\", it would be nice to have more detail to explain exactly what is happening here. Particularly for readers who are not already familiar with LDMs.\n\nThe third contribution \"Temporal Split & Merge Augmentation\" is similar to augmentations that have long been used in other domains. Namely cropping parts of different data points together to form a new one. For example, \"MixUp\" does exactly this by mixing two different images and labels together. Some discussion around previous approaches similar to this contribution with adequate referencing is necessary.\n\nIn table 1, if inference time is mentioned, then I believe number of parameters and training time, if possible, should also be presented for a more fair comparison.\n\nIn the related work, there is a section on \"Video-to-Audio Generation\", but existing works on a specific subset of video-to-audio which has received a lot of attention, video-to-speech, are not mentioned at all.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the positive feedback from you. We respond to the weakness and questions below.\n\n### 1. About more baselines.\n\n> *1.1: FoleyGAN (2021) seems to perform the same task, but no comparison (or citation) is present.*\n\nThanks for pointing out, we are glad to cite the references in our final paper. Regarding FoleyGAN (2021), we couldn't reproduce their experiment as their code is not publicly available. Additionally, FoleyGAN has only been tested on a small dataset of 28K samples, and its efficacy on large-scale audio-video datasets like VGGSound remains unverified.\n\n### 2. Discussion of metrics.\n\n> 2.1: Diff-Foley is outperformed on 2/4 metrics, which is also not very reassuring.\n\n**(1) Update on Human Eval:** As suggested by reviewer k7sd and tzzm, we have conducted a human evaluation, which further demonstrate the superiority of our method in generating highly synchronized audio with strong audio-visual relevance. Please refer to **Global Response 1**. \n\n**(2) About FID and KL:** As you mentioned that FID and KL are known to be somewhat unreliable, we agree with your point. As for the detailed discussion, please refer to **Global Response 2**.\n\n### 3. Ablation on using only classifier guidance.\n\nSure. Results are provided in the following table for completeness. We see that only adopt the classifier guidance leads to inferior results. Only using the gradient of classifier for guidance is unstable during inference, but combining CFG and CG together results in better performance (as discussed in supplementary C.3,4). This further verify the effectiveness of our proposed double guidance techniques.  \n\n| Model             | Stage1 CVAP Dataset     | CFG | CG  |           |                     |      Metrics              |                             |\n| :---------------- | :---------------------: | :-: | :-: | :--------------: | :-----------------: | :----------------: | :-------------------------: |\n|                   |                         |     |     | IS $\\uparrow$ | FID $\\downarrow$ | KL $\\downarrow~$ | Align Acc  (%) $\\uparrow$ |\n|                   | VGGSound                |  &cross;  | &cross;  | 19.86           | 18.45              | 6.41              | 67.59                      |\n|                   | VGGSound                |  &cross;  | &check; | 16.58           | 20.20              | 6.81              | 62.24                      |\n|                   | VGGSound                |  &check; | &cross;  | 51.42           | 11.48              | 6.48              | 85.88                      |\n|  Diff-Foley (Ours)  | VGGSound     | &check;  | &check;  | 53.45           | **10.67**              | 6.54              | 89.08                      |\n|                   | VGGSound + AudioSet-V2A | &cross;  | &cross;  | 22.07           | 18.20              | 6.52              | 69.41                      |\n|                   | VGGSound + AudioSet-V2A | &cross;  | &check;  | 17.57           | 20.87              | 6.69              | 66.05                      |\n|                   | VGGSound + AudioSet-V2A | &check;  | &cross;  | 52.07           | 11.61              | **6.33**              | 92.35                      |\n|                   | VGGSound + AudioSet-V2A | &check;  | &check;  | **60.39**           | 10.73              | 6.42              | **94.78**                      |\n\nWe will add these additional results in the final version of our paper for completeness, thanks for the suggestion again.\n\n### 4. About paper revision.\n\n> *4.1: Figure 2, latent diffusion models too vague.*\n\nThanks for pointing out, we will draw more details of latent diffusion module in Figure 2 in the final version of our paper.\n\n> *4.2: Reference for Temporal Split \\& Merge Aug.*\n\nThanks, we will add more discussions and cite relevant papers like Mixup in the final version of our paper.\n\n> *4.3: Show training time and number of params.*\n\nThanks for mentioning it. In the realm of generative models, our focus lies primarily on inference time, which has larger impact on pratical need. Diffusion model, current state-of-the-art generative model, is known to require a larger number of parameters and longer training time to achieve remarkable generation performance, like Stable Diffusion.  Detailed training time and number of parameters are provided in the supplementary.\n\n> *4.4: Video-to-Speech are not mentioned.*\n\nThanks for pointing out. We will add more discussions and references for video-to-speech in the final version of our paper.\n\n> *4.5: The broader impact is really insufficient.*\n\nThanks for your advice, we will add more substance in the discussion of broader impact in the final version.\n\n### 5. Other Questions.\n\n> *5.1: Are there any other methods that this work could be compared with in the results section?*\n\nThank you for raising this point. We have compared our work with SpecVQGAN and Im2wav, the latest two models that have been applied to the VGGSound large-scale dataset for video-to-audio tasks. Other related works may either be outdated or have not been validated on large-scale datasets, which limits their relevance for comparison with our method. We believe that our comparison with the baseline methods **provides a comprehensive evaluation and stands as relatively complete.**\n\n> *5.2: Will the code be released upon acceptance? If so, will this include training code, pre-trained models, etc.?*\n\nYes, we recognize the value of open-source. We will release the code and the pre-trained model once the paper has been accepted.\n\n\n\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
## Global Response 1: Human Evaluation Results

As suggested by reviewer k7sd and tzzm. We've conducted a human evaluation by randomly selecting 60 videos from the VGGSound test set and having different models to generate corresponding audio samples. The output and groundtruth audios were anonymized and rated by 30 people unfamiliar with the project. Each sample received scores from at least 5 raters, ranging from 1 (bad) to 5 (excellent) for content relevance and synchronization. The scores were then scaled by a factor of 20. The human evaluation results is shown below, which effectively demonstrate the superiority of Diff-Foley in generating audios with strong audio-visual relevance and synchronization. The human evaluation results and details will be added to the main paper and supplementary, respectively.

| Model                | Guidance | Content Relevance | Synchronization |
| :------------------- | :------: | :---------------: | :-------------: |
| SpecVQGAN (ResNet50) | -        | 46\.20            | 45\.20          |
| Im2wav               | CFG      | 62\.13            | 57\.73          |
| Diff-Foley (Ours)    | CFG      | 71\.73            | 71\.00          |
| Diff-Foley (Ours)    | Double   | **74\.53**            | **74\.93**          |
| Groundtruth          |          | 84\.80            | 84\.27          |
||||

## Global Response 2: Discussion on inferior metrics on FID and KL.

Thanks for mentioning it. As acknowledged by reviewer MA6i, FID and KL measurements are known to be somewhat unreliable, but we included them to provide a comprehensive analysis. FID and KL may not consistently reflect human subjective perceptions. In our study, they seem less representative of audio quality than metrics such as IS and Align Acc, which have shown a stronger correlation with the perceived quality of audio in human eval results and experiments. 

Still, some discussions on why FID and KL are relatively inferior to other method are provided. 

(1). KL: Diff-Foley ranks second to Im2Wav [2], possibly due to Im2Wav's use of the CLIP feature. We found that incorporating CLIP features in Diff-Foley also improves KL results (see Table 2.). The richer video semantic features in CLIP may contribute to this improvement, and this gap is expected to be bridged by scaling the CAVP pretraining to a scale of billions, akin to CLIP.

(2). FID: Diff-Foley outperforms Im2Wav [2] but falls short of SpecVQGAN [1]. This might be related to our use of the frozen Stable Diffusion latent encoder and decoder. FID seems very sensitive to reconstruction quality. As shown in supplementary Table 1, the groundtruth spectrogram reconstruction FID of $9.20$ represents Diff-Foley's FID lower bound. Improving the reconstruction quality of frozen latent encoder and decoder is left for future work.

(3). To enhance perceptual quality, we adjusted the CFG scale to 4.5, further sacrificing Diff-Foley's FID and KL performance. Figure 6 illustrates a U-shape curve for FID and KL, indicating optimal results around a CFG scale of $2.5\sim3$.

 

### Reference:
[1] Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arXiv: 2110.08791, 2021.

[2] Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. arXiv preprint arXiv:2211.03089, 2022.



Meta-review: All reviewers agree that this paper addresses a key problem in audio-video synthesis. The proposed method achieves good results compared to the evaluated baselines. The authors provide a thorough analysis, and the paper is clearly written. Additionally, the authors engaged with the reviewers in the discussion and provided additional results per the reviewers' request (e.g., human study). Hence, I recommend acceptance.
Decision: Accept (poster)