FOLEYCRAFTER: BRING SILENT VIDEOS TO LIFE WITH LIFELIKE AND SYNCHRONIZED SOUNDS

Authors: Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen
Primary area: generative models
Keywords: Diffusion Model, Audio Generation, Video to Audio Generation

Abstract:
We study Neural Foley, the automatic generation of high-quality sound effects
synchronizing with videos, enabling an immersive audio-visual experience. Despite
its wide range of applications, existing approaches encounter limitations
when it comes to simultaneously synthesizing high-quality and video-aligned
(i.e.,semantic relevant and temporal synchronized) sounds. To overcome these
limitations, we propose FoleyCrafter, a novel framework that leverages a pretrained
text-to-audio model to ensure high-quality audio generation. FoleyCrafter
comprises two key components: a semantic adapter for semantic alignment and a
temporal adapter for precise audio-video synchronization. The semantic adapter
utilizes parallel cross-attention layers to condition audio generation on video features,
producing realistic sound effects that are semantically relevant to the visual
content. Meanwhile, the temporal adapter estimates time-varying signals from
the videos and subsequently synchronizes audio generation with those estimates,
leading to enhanced temporal alignment between audio and video. One notable
advantage of FoleyCrafter is its compatibility with text prompts, enabling the use
of text descriptions to achieve controllable and diverse video-to-audio generation
according to user intents. We conduct extensive quantitative and qualitative experiments
on standard benchmarks to verify the effectiveness of FoleyCrafter. Models
and codes will be available.

Decision: reject
Reviewer scores: 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1730275563474

thread_type:
official_review

<thread_object>
creator: reviewer_p9qh

<content>
summary:
In this paper, the authors proposed a framework called FoleyCrafter to synthesize high-qulity audio with text prompt, which contains two key components as follows:
1.Semantic adapter condition generated audio conditioned on video features, rendering more semantically relevance.
2.Temporal adapter estimates time signals, synchronizing with audio.
The authors carried experiments on two datasets and achieved better performance compared with current powerful models.

strengths:
1.Originality: The authors proposed two adapters to improve the audio synthesis. However, the structure inside originates from other works.
2.Quality: Although the method proposed is effective compared to others, it lacks rigorous mathematical proof.
3.Clarity: Semantic adapter has not been clarified clearly, especially the cross-attention component.
4.Significance: The significance of the method is relatively high comparing to existing methods. However, parameters to be trained is relatively high compared to others.

weaknesses:
1.Lack of Innovation: In this article, there are two key components. However, the semantic adapter is derived from the IP-adapter[1], while the temporal adapter originates from ControlNet[2]. This article lacks substantial original contributions.
2.Inference Latency Concerns: In the articles mentioned above, the authors only add a single adapter to the original model. However, in this article, the proposed method includes two separate adapters, which may result in higher inference latency, potentially impeding efficiency and scalability.
3.Insufficient Analysis of Text Prompts: In this article, there are text prompts and video prompts for audio generation. However, The authors provide only a qualitative description of the text prompt's capabilities, without comparing it to other models.

[1] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.
[2] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023a.

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1730305559296

thread_type:
official_review

<thread_object>
creator: reviewer_qwiq

<content>
summary:
This paper introduces FoleyCrafter, a framework designed for automatically generating realistic and synchronized sound effects for silent videos. FoleyCrafter leverages a pre-trained text-to-audio model, incorporating a “semantic adapter” and “temporal adapter” to ensure that the generated audio is semantically aligned with video content and precisely synchronized over time. Additionally, it supports customizable audio generation through text prompts. The primary contributions include: 1) presenting a novel neural Foley framework for high-quality, video-aligned sound generation, 2) designing semantic and temporal adapters to improve audio-video alignment, and 3) achieving state-of-the-art performance on benchmarks through comprehensive quantitative and qualitative evaluations.

strengths:
1.	Originality: This paper introduces an innovative framework, FoleyCrafter, which stands out in the field of sound generation for silent videos. By combining a pre-trained text-to-audio model with novel adapter designs (semantic and temporal adapters), it effectively addresses the limitations of existing methods in terms of audio quality and video synchronization, showcasing unique and original thinking.
2.	Quality: The paper demonstrates high research quality through comprehensive experimental design and implementation. It includes extensive quantitative and qualitative experiments, validating the effectiveness of FoleyCrafter on standard benchmark datasets. The results show that this method surpasses several state-of-the-art approaches in both audio quality and synchronization performance. Additionally, the availability of code and models facilitates future replication and research.
3.	Clarity: The paper is well-structured, with clear explanations of concepts and model design, allowing readers to easily understand how FoleyCrafter operates. The figures and results in the experimental section are also well-presented, enabling readers to intuitively grasp the method’s performance and advantages.
4.	Significance: FoleyCrafter holds substantial application potential in the field of video-to-audio generation. This approach not only enhances the realism and synchronization of sound effects but also offers controllability and diversity through text-based prompts. Such innovations have broad applicability in multimedia production, including film and gaming, and further advance cross-modal generation technology in the audio-visual domain.

weaknesses:
The paper’s originality appears limited. The whole model system exploits many present models, such as Freesound Project and Auffusion.
Although the part of Quantitative Comparison includes evaluations in terms of semantic alignment, audio quality and temporal synchronization, the comparison of audio generation speed has not been expressed.
The lack of some ablation experiments for Semantic Adapter and Temporal Controller weakens persuasiveness. The Semantic Adapter could be entirely removed to observe the system’s performance without visual semantic information. The Onset Detector and Timestamp-Based Adapter could be individually removed to investigate their roles in temporal alignment and onset detection. In addition, it would be more persuasive if ablation experiments for Parallel Cross-Attention with different λ had been done.

questions:
Refer to Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690268402

thread_type:
official_review

<thread_object>
creator: reviewer_bd3g

<content>
summary:
This paper presents a new video-to-audio model, featured by the semantic adapter and temporal adapter. The proposed model uses the [Auffusion](https://arxiv.org/abs/2401.01044) model as a baseline, and not only video-audio paired data but also text-audio paired data are used for training its sub-modules for connecting between the visual encoder and Auffusion. The temporal adapter, trained with the BCE loss or MSE loss to estimate the energy map of audio from video, enhances the synchronization between video and audio. The authors conducted both quantitative and qualitative comparisons with previous video-to-audio models to demonstrate that the proposed model outperforms them. They also conducted ablation studies to show that their proposed semantic and temporal adapters are effective.

strengths:
1. Although there is room for improvement in writing style, the paper itself is well-written enough to make readers understand their motivation, the proposed method, and the experimental results.
2. The proposed video-to-audio model is well-designed to address the issue of synchronization between video and audio. There may be other designs for resolving the issue, but they conducted ablation studies to demonstrate that their designed model works well.
3. The authors quantitatively evaluated their model on the commonly used benchmarks and qualitatively analyzed the audio signals generated from the proposed and previous models for comparison. These experimental results show that the proposed model outperforms the previous models.

weaknesses:
- L.365: "We employed several evaluation metrics to assess semantic alignment and audio quality, namely Mean KL Divergence (MKL) (Iashin and Rahtu, 2021), CLIP similarity, and Frechet Distance (FID) (Heusel et al., 2017), following the methodology of previous studies (Luo et al., 2023; Wang et al., 2024; Xing et al., 2024). MKL measures paired sample-level similarity"
  - The application of FID to audio quality evaluations is proposed by [Iashin and Rahtu (2021)](https://www.bmvc2021-virtualconference.com/conference/papers/paper_1213.html), and [Luo et al. (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/hash/98c50f47a37f63477c01558600dd225a-Abstract-Conference.html) followed them. However, [Wang et al. (2024)](https://ojs.aaai.org/index.php/AAAI/article/view/29475) and [Xing et al. (2024)](https://openaccess.thecvf.com/content/CVPR2024/html/Xing_Seeing_and_Hearing_Open-domain_Visual-Audio_Generation_with_Diffusion_Latent_Aligners_CVPR_2024_paper.html) use different metrics, FD ([Liu et al., 2023](https://proceedings.mlr.press/v202/liu23f.html)) and FAD ([Kilgour et al., 2019](https://www.isca-archive.org/interspeech_2019/kilgour19_interspeech.html)). I recommend the authors additionally evaluate their proposed model with these metrics for several reasons. The FID and FAD are calculated from spectrograms and do not consider phase information of audio signals. The FD is based on the PANN network ([Kong et al., 2020](https://ieeexplore.ieee.org/document/9229505)), which takes audio waveforms and achieves better performance in classification tasks than VGGish. Plus, recent papers use FAD or FD more frequently. The evaluation on these metrics will be informative to readers, which means the authors can contribute more to the community.

questions:
I would appreciate the authors' response to my comments in "Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732087494924

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank all reviewers for your time and invaluable feedback. 

We appreciate the recognition of FoleyCrafter as a **well-designed model** (Bd3G) with **unique and original thinking** (Qwiq) and the acknowledgement of the **comprehensiveness of our experimental results** (Bd3G, Qwiq) which shows it a **effective method** for video-to-audio generation (p9QH).

We address the questions and concerns for each reviewer in the rebuttal sessions individually.   As mentioned in our responses, we will incorporate clarifications in the final version of our paper. Please let us know if there are any questions or further clarifications/discussions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a framework: FoleyCrafter for adding foley sound effects to videos. The key innovations are: i) to enhance an audio latent diffusion model that is conditioned on text to also include video semantics via incorporating video features using cross-attention layers, and ii) temporal synchronization of the audio with the time-varying video content, using either manually-provided time-stamps or using an energy map estimated from mel spectrograms, followed by using ControlNet for synchronization. Experiments on VGGSound and AVSync15 show promising results.