Tell What You Hear From What You See - Video to Audio Generation Through Text
Authors: Xiulong Liu
Kun Su
Eli Shlizerman
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal learning
audio-visual learning
multi-modal large-language-model
text-guided video-to-audio generation
video-to-audio captioning
Primary area: speech_and_audio
TL;DR: A novel multi-modal generation framework for text guided video-to-audio generation and video-to-audio captioning.

Abstract: The content of visual and audio scenes is multi-faceted such that a video stream can
be paired with various audio streams and vice-versa. Thereby, in video-to-audio
generation task, it is imperative to introduce steering approaches for controlling the
generated audio. While Video-to-Audio generation is a well-established generative
task, existing methods lack such controllability. In this work, we propose VATT, a
multi-modal generative framework that takes a video and an optional text prompt
as input, and generates audio and optional textual description (caption) of the
audio. Such a framework has two unique advantages: i) Video-to-Audio generation
process can be refined and controlled via text which complements the context
of the visual information, and ii) The model can suggest what audio to generate
for the video by generating audio captions. VATT consists of two key modules:
VATT Converter, which is an LLM that has been fine-tuned for instructions and
includes a projection layer that maps video features to the LLM vector space, and
VATT Audio, a bi-directional transformer that generates audio tokens from visual
frames and from optional text prompt using iterative parallel decoding. The audio
tokens and the text prompt are used by a pretrained neural codec to convert them
into a waveform. Our experiments show that when VATT is compared to existing
video-to-audio generation methods in objective metrics, such as VGGSound audiovisual dataset, it achieves competitive performance when the audio caption is
not provided. When the audio caption is provided as a prompt, VATT achieves
even more refined performance (with lowest KLD score of 1.41). Furthermore,
subjective studies asking participants to choose the most compatible generated
audio for a given silent video, show that VATT Audio has been chosen on average
as a preferred generated audio than the audio generated by existing methods. VATT
enables controllable video-to-audio generation through text as well as suggesting
text prompts for videos through audio captions, unlocking novel applications such
as text-guided video-to-audio generation and video-to-audio captioning.

Tell What You Hear From What You See- Video to Audio Generation Through Text
Xiulong Liu
Kun Su
Eli Shlizerman 14
Abstract
The content of visual and audio scenes is multi-faccted such that a video stream can be paired with various audio streams and vice-versa Thereby, in video-to-audee generation task, is imperative to introduce steering approaches for controlling the generated audio. While Video-to-Audi generation well-established generative task, existing methods lack such controllability. In this work, we propose VATT, a multi-modal generative framework that takes video and an optional text prompt as input, and generates audio and optional textual description (caption) of the audio. Such framework has two unique advantages: i) Video-to-Audio generation process can be refined and controlled via text which complements the context of the visual information, and ii) The model can suggest what audio generate for the vidco by generating audio captions. VATT consists of two key modules: VATT Converter, which is an LLM that has been fine-tuned for instructions and includes projection layer that maps video features to the LLM vector space, and VATT Audio, a bi-directional transformer that generates audio tokens from visual frames and from optional text prompt using iterative parallel decoding. The audio tokens and the text prompt are used by pretrained neural codec to convert them into waveform. Our experiments show that when VATT is compared to existing video-to-audio generation methods in objective metries, such as VGGSound audio- visual dataset, it achieves competitive performance when the audio caption is not provided. When the audio caption is provided as a prompt, VATT achieves even more refined performance (with lowest KLD score of 1.41). Furthermore, subjective studies asking participants to choose the most compatible generated audio for given silent video, show that VATT Audio has been chosen average as a preferred generated audio than the audio generated by existing methods. VATT enables controllable video-to-audio generation through text as well suggesting text prompts for videos through audio captions, unlocking novel applications such text-guided video-to-audio generation and iidoott--udiio captioning.
Introduction
The combination of human perception and cognition represents a "multi-modal" way of processing and interpreting scenes. For example, when we are presented with a silent video of fountain show attended by crowd of people gathered around the spectacle visual scene into an auditory experience, where the visuals are semantically processed and transformed into a corresponding sound narrative in our mind. Thus, we may associate audio that mixes sounds of splashing water accompanied by people talking and laughing with possibly background music in sync with the fountain.
interpretation might translate the
'Department of Electrical & Computer Engineering, University of Washington, Seattle, USA. 1 Departmeen of Applied Mathematics, University of Washington, Seattle, USA "Correpoondigg author: shlizce@uw.odit
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Water Splashing sounds are heard.
Generated Audio Caption (If No Text Prompt)
Generated Audio (No Text Prompt)
VATT
Generated Audio (with Text Prompt)
Video Frames
Flow w/o Text Prompt
Live music along with water splashing sounds.
Goneration Flow Text Prompt
Text Prompt (Optional)
Figure 1: VATT is flexible audio generative model capable of generating audio in two modes: i) When a silent video is the sole input, the model generates the audio along with a caption describing the possible audio that could match the video. ii) When in addition the video, text prompt is provided, the model generates audio aligned with both the video and the given text prompt.
As generative AI continues to progress, the incorporation of the aforementioned aspects into generative platforms presents itself as the future desirable capability. In particular, the goal of an ideal video-to- audio generative model would be to generate sounds that seamlessly match the video temporally and fully capture the semantics. Moreover, it desirable to control such a generation process towards the themes and sounds that match user preference. Recent state-of- of-the-art approaches have adopted two types of generative modeling techniques: auto-regressive token-based modeling and diffusion-basee modeling. These methods enable end-to-end video-to-audio generation and are applicable across wide variety video and audio categories. However, while these methods are capable of capturing the general semantics of sound sources in videos, they often overlook the subtleties of the context. For example, a video depicting two cats in territorial dispute, the model might produce a calm, amiable meowing sound, which contradicts the contentious nature of the scene. This discrepancy mainly stems from the limitations of the vision encoder, which struggles to distinguish between varying sound properties emitted by identical sound sources across different contexts, due to an incomplete understanding of the entire scene. Second, these methods lack controllability since the generation is conditioned only visual frames, without taking into account the context and interpretation of the sounds. While text-to-audio models could explicitly control the context of the sounds, such models are based on text only without incorporating the rich and dynamic context of visuals, which could significantly inform video and audio alignment. Indeed, text only generative outcomes often result unmatched audio with the visual (c.g.. temporal misalignment or semantic loss).
To solve the above challenges, we propose novel framework, Video-to-Audio Through Text (VATT), that is able to generate audio from both video frames and an optional text prompt describing the expected sounds. VATT consists of two modeling stages: i) Aideo-to-Caption stage, which converts video features into an audio caption through a pretrained large language model (LLM) with learnable projection layer. Through this cross-modal conversion, visual features that are relevant audio concepts are extracted. These features are closely connected to audio-related tasks such as audio captioning and audio generation. ii) Video Text to Audio stage, that generates audio conditioned on the hidden states extracted from the L.MM in the prior modeling stage. At its core, the proposed model in this stage is bi-directional transformer decoder that generates audio using a token-basee based representation similar to [1 (2). To obtain the conditioning on the hidden states of the preceding component, the projected video features along with the optional text prompts are concatenated together and fed into the LLM in stage i), with the hidden states from the last layer extracted and attached the audio tokens for the decoder. The decoder is trained using masked token modeling, where the objective is to predict masked audio tokens from unmasked ones at varying masking ratios. During inference, starting from all tokens being masked, an efficient parallel decoding algorithm implemented which gradually unmasks multiple tokens in parallel based video and text inputs until a stop condition is met. Finally, the generated tokens are converted into audio waveforms through neural audio codec decoder.

---

We perform experiments with the proposed framework on existing large-scale audio-visual datasets such as VGGSound [3] and Audioset-2MM, [ To facilitate training and evaluation with text, we created "V2A Instruction", a large-scale synthetic audio captions corpus, by prompting ITTU13B, an existing Audio L.LM 5), to generate audio deseriptions for both datasets. Our experiments demonstrate that the proposed model and its training method achieve competitive performance in comparison to previous video-to-audio methods on both objective and subjective metrics. Furthermore, it is designed to enable controllable generation that adheres to both the video inputs and the text prompts. Indeed when a text prompt is provided, our experiments show significant improvement audio metrics that measure the match of the generated sounds to the video. In addition, when the text prompt is not provided, our method can generate reasonable audio captions, which can be utilized for potential description of the video or classification of sounds for a given video. These capabilities hence make VATT a multifaceted single model able to perform both text-guided video-to-audio generation and video-to-audio captioning. To summarize our contributions:
To our best knowledge, we propose mist-of-its-kind framework that enables both text-guided video-to-audio generation and video-to-audin captioning through the integration of LLM.
We create a large-scale synthetic audio captions dataset that facilitates text-conditional training and generation.
Our method achieves state-of-the-art video-to-audio generation performance when compared with existing methods and enables text-controllable generation. In particular, our text-guided model surpasses existing SOTA in terms of KLD score (with lowest KLD score of 1.41) by a significant margin.
VATT generates audio in an efficient way an order of magnitude faster than existing methods.
2
Related Works
2.1 Visual-to-Audio Generation
Visual-to-Audi Generation task has drawn significant attention since generative frameworks such as diffusion and transformer- based architectures have been developed. Existing Visual-to-Audio generation approaches can be divided into two branches of studies based on audio categories: visual- to-music generation and visual-to-natane earlier studies explored Midi or spectrogram generation from human body movements by studying methods have been proposed to generate music waveforms directly from videos [Л]. In visual-to-natural sound generation, earlier efforts pioneered the generation of sounds linked to various objects and materials 12 Later works proposed an audio generation approach based on SampleRNN [ 14] that could generate several types of natural sounds from in-the-wild videos. While these approaches showcase promising results, they are often limited to specific audio categories. Neural codec [55 [7] [8] and autoregressive transformer architectures [19 20 addressed these limitations and as they have evolved. generative models now effectively generalize across a broader range of sounds 23 24 Similar advances have been shown with diffusion techniques such as (25 26 However, these methods often lack detailed sound control and their inference time turns out to be consuming. Our work aims to address these limitations by introducing a text-guided framework to improve controllability and efficiency in vidco-to-audii generation. While there are several concurrent works that aim to achieve partially similar goals to our proposed method (27 28 29 our work is different since it is designed to achieve these capabilities within a single unified framework.
sound generation. In visual-to-music generation domain,
the temporal and semantics alignment (6, 7 8 9 10]. More recently, diffusion-basee
or music, leveraging compressed latent spaces [2] 22
2.2 Text-to-Audio Generation
As an alternative to the generation of audio from video, text can be used as an input to guide audio generation. When text is the input, audio generation becomes more controllable semantically. Existing approaches such as Make-An-Audio [30] AudioLDM [I], AudioLDM-22 [32] and others [33 34| 34]
35 enable general text-to-audio (or music) generation by adapting latent diffusion techniques, first developed in [36]. Concurrenty,, methods such as AudioGen [37] MusicGen [38) Audiol.M (9)
MusicL.M . 40), SoundStorm 2 VampNet [41 leverage transformer- architectures and token-
based modeling techniques to produce audio tokens, that are then decoded into waveforms using
3

---

neural codecs like Encodec [8] and SoundStream 142]. Notably. SoundStorm and VampNet use an efficient technique known as masked token-based modeling which speeds up generation with parallel unmasking in the decoder. In our work, we consider similar approach. While these models deliver high-quality audio with strong relevance to the text, they do not necessarily align with visual dynamics when adapted to video-to-audio generation. This is expected since such models have not been trained to attend to visual inputs. Our work addresses this by integrating a pretrained large language model (LLM) as multi-modal encoder that processes both visual and textual inputs such that the generated audio considers both visual and text information.
2.3 Multi-modal Large Language Models
Multi-modal Large Language Models (MLLMs), have been able to attain significant progress. With the advent of open source, pretrained and instruction-tunet LLMs such as LLama [43. Alpaca [4), Vicuna [4)] In particular, when extending these LLMs into MLLMs, a pretrained modality-speciff encoder extracts the features and then a projection layer maps these features into vectors of the same dimension text embeddings of the corresponding LLM. This approach led to developments in visual LLMs [4 47] audio LLMs [5] 488. audio-visual LLMs 49 and showed improvement in multi-moda understanding tasks such captioning (50) and question-answering [51 (2). Recent efforts have also focused on tasks such as multi-modal retrieval (53] multi-modal embodied navigation [54] [55] leveraging LLM'' strong reasoning capabilities
interpret improve the results. In terms of generation, several works [56] 57] aimed at achieving any-to-any modality generation using LLMs as a central medium. While these methods have been successful in general podallty-to-moddal generation, they do not achieve particular end-to-end video-to-audio generation, with or without text guidance, which is the unique direction our work focuses on.
3 Methods
VATT is a flexible vision-to-audio generative framework that can process both visual and textual inputs and generate both audio waveforms and captions of audio. To achieve this, VATT consists of two modeling stages: i) Video-to-Caption This stage utilizes Large Language Model (LLM) with a learnable projection layer that converts video features into embeddings compatible with the LLM. The model receives instruction to generate audio captions from video inputs. ii) Video Text to Audio: This stage incorporates an encoder-decoder architecture. The encoder uses the finetuned LLM from Video-to-Caption stage with frozen weights. The decoder is a bi-directional transformer trained to generate audio tokens using masked token modeling techniques in training. The training pipeline of VATT system is shown in Figure 2 During inference, VATT generates audio tokens from video and optional text prompts through iterative parallel decoding. These tokens are then converted into audio waveforms using Encodec
3.1 Video-to-Caption Stage
VATT Converter is designed to integrate visual and textual prompts for audio generation as well as audio captioning. The core component, VATT Projector, is an embedding layer that maps video features into the text embedding space of the LLM. Given visual features extracted from frame- level vision encoders V/- = { (11, U2 ... vr Linear layer is applied to project each feature from its original dimension dhp to the LLM text embedding dimension dim, producing sequence of transformed features Vm = V,W, + bt, where Wi and by are learnable parameters of the linear projection.
The key functionality of VATT Converter is to extract from visual stream semantic features relevant to audio. Drawn on the success of multi-modal LLMs, such as visual-LLM 46) and audio-LLM [5), we employ multi-modal instruction tuning align the visual inputs of videos with the ground truth audio captioning of the same videos. Given prompt instruction, T: = (t11, t.2. ... J. such as "Describe the audio that the video could generate:" and the projected visual features Von as inputs, we model conditional distribution of audio descriptions T, (La1, la2,..., LaN), ), as (T, Vmm) by fine-tuning an instruction- -uuded LLM, e.g., Vicuna- 7B 45 Unlike typical instruction-tuning that maps signal into textual concepts within the same modality, our method bridges the concepts from visual to audio modality, unifying the representation for text-guided video-to-aie iidoo-to-audio generation task that we describe in section 3.2 For training efficiency,
V2A Instruction Tuning:

---

Stage 1: Video-to-Caption
VATT Converter
Freeze Params
Trainsble Parems
Visual Enpoder
VAFT Projecto
Video Fremnes
Proj Viceo
LLM Decoder
4A pigeon repaatedty while apweke nearby
Irougine
Tokenizer
LoRA
Generated Audio Captions
Embeddinge
VisA Instruction
Stage 2: Video + Text to Audio
M-f
Fremes
VATT Converter (Stage
VATT Audio (Masked Parallel Decoding)
(inoand Example: "magii pooambee audo for
Precucted Audo Toxens
Cond Example: Keds speakin wiile a grgenn in orotng V.S Instruction
Original Taken
Encodec Enzoder
Random
Masking
Mask Taken
Audio Weveform
Predicted Token!
Masked Audio
Tokens
Two stages of VATT system training pipeline: (1) Video-to-Caption stage that maps video features into an audio caption through LLM. (2) Video + Text to Audio stage that learns to generate audio tokens through masked tokens prediction conditioned on Stage (1) features.
Figure 2:
we fine tune the LLM with VATT Projector by integrating LoRA [58] adaptors while keeping the original L.L weights frozen. We minimize the negative log-likelihood of audio caption tokens conditioned on visual inputs and prompt instruction
D
Ta
Tú, Vim)
Σ
log [o (fat tat T3 Vim))]
(1)
where fat is the 1-th text token in the ground truth audio description Ta, and @ is the set of trainable weights including VATT Projector and LoRA adaptor. Further details of the constructions of text prompts and synthesis of audio captions are described in Section and Appendix C
3.2 Video Text to Audio Stage
Once the audio-related visual features are aligned with the text features in the L.M embedding space, the LLM effectively encodes multi-modal information that serves representation for text generation and audio generation. Indeed, in the second stage of VATT, there are two generation modes to generate audio: i) When no conditional text prompt is provided, the video features along with standard template prompt (e.g.. "Describe possible audio that the video could infer.") are fed as inputs to VATT Converter. ii) When an audio caption is provided as the text prompt, the video features and the audio caption are fed together into VATT Converter. In such a case, the provided audio caption helps guide the video-to-an -ocedooo generation process and overrides the need for generated audio caption.
3.2.1 Audio Token Decoder
To generate audio, we design an audio token-based decoder, VATT Audio, conditioned on the encoded features from VATT Converter. In contrast to existing methods, which typically use auto-regressive token modeling 37 38, (33 or latent diffusion techniques [3132] we adopt a novel coken-based modeling technique based on masking tokens. The method, originally derived in image generation tasks I and recently adapted to text-to-audio generation
2 41 is capable of achieving competitive generation quality while improving efficiency through an iterative parallel decoding algorithm during inference.
Tasen-based Representation for Audio To represent audio waveforms using discrete tokens, we adopt pretrained audio neural codec, Encodec [17, similarly to FoleyGen [23]. Encodec is multi-level residual vector-quantized (RVQ) autoencoder trained with waveform reconstruction and
5

---

VATT Audio
Linear Head
Bidirectional Multi-Head Self-Attention
Figure 3: Audio Tokens Decoder: VATT Audio is bi-directional transformer that models the audio tokens and the conditioning inputs (LLM hidden states) jointly. We extract the part that corresponds the audio features and apply Linear layers in parallel perform classification on masked tokens at each codebook layer.
adversarial objectives, capable of high-fidelity reconstruction from compressed tokens. Specifically, Encodec uses = 4 codebooks of tokens to represent the audio. Lower-level codebooks encode coarse semantic information, while higher-level codebooks capture fine fine-grained details. We adopt an open source Encodec model pretrained using audio waveforms at Sr = 16kH2 sampling rate. The model compresses waveform into tokens at Sr, 50Hz sampling rate, leading to Sr= 320 waveform samples per token. For any waveform Awar R R1xT=, extract
Ttw 3r,
corresponding audio tokens representation Atok E N (Tc = Tm from Encodec encoder part. Once the model generates Atok. the embedding vectors of levels of tokens at each time step are summed up before being sent Encodec decoder obtain the waveform.
Masked Audio Token Generative Modeling We model the distribution of audio token matrix Atok E NTXT. by developing token masking strategy which learns the joint distribution of the audio tokens full parallelism. This is different than using "delayed patterns" proposed in [38] which enables parallelism but only on the level of codebook dimension. At each time step of Atok- embedding vectors of I, tokens are summed up to represent audio waveform at the corresponding segment. In order to perform masking operation at any position, introduce an additional learnable <MASK> token each codebook. By randomly replacing some of the tokens entries in the Atok with <MASK> corresponding codebook obtain the masked audio token matrix A, E N4xT.. We obtain E.M Rdem x1. by summation of the embedding vectors of each token in A.d. along the level axis.
Conditional generative modeling is implemented as follows. We extract the hidden states of the last layer Hmm pd XTim (before the LLM prediction head) from VATT Converter as the conditional inputs into the audio token decoder. We use linear layer to project Him to Eim. Bdde XTm with same feature dimension as the masked audio embeddings F. straightforward way model the relationship between E.M and Eim is to use an interleaving self-attention and cross-attention block as proposed in Vanilla Transformer architecture [59]. However, we find that such interleaved interaction between audio and multi-modal input condition does not capture the fine-grained correspondence between them. Therefore, we propose to use a bi-directional self-attention architecture to fuse the features.
Specifically, we concatenate Eim with E.M along the temporal axis to obtain the fused features Emo Concat( Eam, F..... The decoder consists of I.mm layers of self-attention blocks, as shown in Fig. The output hidden states in the last layer of the decoder, Ho Dec (Emm). represent fused audio and conditions features. We only extract the part of the hidden states corre- sponding the audio tokens, Ho RdmmXT, and pass it through L, Linear layers in parallel to perform classification on masked tokens at each level of the codebooks. For each masked audio token matrix A, we calculate the cross-entropy loss between the predicted token âeok and the ground truth token agg formulated as
LV
Σ
I(arok = <MASK>> log [Po = aooh
AM Htm))]]
where ò is the set of trainable parameters in the audio token decoder, and ll is the indicator function.

---

3.2.2 Masking Design and Iterative Parallel Decoding
Masking Distribution Design Inspired by 2). we incorporate variable random masking. In particular, it was shown that masking ratio plays an important role in audio token decoder to generate meaningful signals. While in 2 arc-cosine masking distributions is used by default, here we study several masking strategies that include distributions along with different hyper-parameters to find the strategy that reaches more optimal generation quality (see Appendix [A] for further details). Our study shows that normal distribution with mean of 0.75 and standard deviation of 0.25, truncated from 0.5 1.0 is such optimal strategy. The general interpretation of this strategy is that relatively high range masking ratio enables models to generate better initial tokens when most of the entries in the token matrix are masked. This essential for future decoding steps to generate meaningful tokens.
Iterative Parallel Decoding Scheduling of masking plays a key role as well. During inference, we follow the cosine scheduling scheme proposed in [1 to gradually resolve the audio tokens. The iterative sampling procedure starts with all <MASK> in the audio token matrix. At a step I, the model takes the audio token matrix 1--1 from the previous step along with the conditions inputs and samples a new audio token matrix Å, in parallel with all tokens unmasked. Based on the confidence at each entry of Ag only tokens with top-k confidence are kept while the remaining entries are re-filled with <MASK>, resulting in A,. The cosine scheduling scheme determines the ratio of re-masked tokens by rt = cos (5 4). Notably, to resolve the confidence of each entry in the matrix, we
adopt the "gumbel-top-k trick" (60) with temperature that varies, i.e., G log(p.) + G, where G ~ Gumbel(0, 1) and p. denotes the output probability of the sampled token at the entry i. This is equivalent sampling k values from multinomial distribution from the softmax probabilities without replacement. The temperature I controls the degree of stochasticity. We use T = To (1 - +) with linear decay during generation, where To the initial temperature. Similarly to 212), our method achieves optimal quality and fast speed within a few decoding steps (typically 10 - 20).
Experiments
Datasets:
We use common benchmarks datasets VGGSound [3] and AudioSet-2M 44 for training and evaluation. VGGSound is large-scale audio-visual dataset sourced from YouTube, containing 192k videos from 309 audio-visual categories, with 177k / 15k train-test video splits. AudioSet-2M is larger audio-visual database with around 2M You Tube videos, with only 1.6M available online. In Stage 1, we train VATT Converter with both datasets and test on VGGSound only. In Stage 2, for fair comparison against existing video-to-audio generation methods, we train and evaluate on VGGSound dataset only.
To train VATT with text, we synthesize a large-scale audio caption dataset, "V2A Instruction". using LTU (5), an existing audio LLM. We obtain audio captions by prompting the pretrained LTU-113 model with the inputs of audio waveform along with the instruction "### Instruction: Close-ended question: Write an audio caption describing the sound. ### Response:" " For AudioSet [4] and VGGSound 3 we generate single audio caption per each video for a total of 1.77M videos.
To ensure the quality of captions, we first manually verified the validity of LTU -TU-gennrated captions prior to using them as synthetic ground-truth (GT) and then performed an experiment to further evaluate captions quality. In particular, we randomly selected 100 videos from VGGSound test set with stratified sampling according to video categories to conduct human study. We used 1-5 point MOS Thean-Opinion-Score -Scor)) scale (the higher the better) to measure correctness of the captions. We provide pairs of videos and the corresponding captions to the raters, asking "How accurately the provided caption reflects the sound events happening in the video? 1. Inaccurate and irrelevant. 2. Relevant but inaccurate with many mistakes. 3. Partially accurate but missing details and with mistakes. 4. Mostly accurate with some minor mistakes. 5. Accurate and complete." " We used the MTurk platform to perform the evaluation and collected total of 300 responses. The generated captions have a high MOS of mean 4.72 and std 0.37, providing an additional indication for the validity of the synthetic ground truth.
For visual inputs, we use eva-CLIP 61 image encoder to extract mean- pooled visual features from video frames at 5fps rate, which result in 50 x 768 visual sequence for 10s video. To represent audio, we extract audio tokens from a pretrained Encodec- 16kHz. For each 10s audio waveform, we represent it with Atok 5y05000 token matrix.
Implementation Details:
7

---

Table 1: Quantitative results against video-to-audio generation methods VGGSound test set. TTT' refers to model with text prompts.
Methods
KLD 4
FAD L
Align Ace f
Speed (8) !
SpecVQGAN
21
3.78
6.63
48.79
7.2
1M2WAV 22
2.54
6.32
74.31
289.5
Dift-Foley
25
3.15
6.40
82.47
4.4
FoleyGen
26
2.89
2.59
73.83
6.9
V2A-Mapper
26
2.78
0.99
74.37
11.54
VATT-LLama (Ours)
2.39
2.38
80.32
1.1
VATT-Gemma (Ours)
2.25
2.35
82.81
0.65
VATT-LLama-T (Ours)
1.41
2.54
80.16
1.2
VATT-Gemma-I (Ours)
1.66
2.98
81.48
0.76
For LLM, we explore two open-source models, Gemma-2B 662 and LLama-2-7B 45 using instruction-tunel checkpoints. The LLM hidden size of Gemma-2B is 2048 and 4096 for LLama-7B: For both LLMs, we train VATT Converter using LoRA parameter-efficient fine-tuning technique while keeping the LLM weights frozen. We rank 16 and a 32 with 0.1 dropout rate for LoRA configuration.
VATT Audio is bi-directional transformer with 24 layers, each with hidden size 1024 with 16 attention heads. To differentiate the conditioning inputs and audio tokens, we add two learnable -ppelity- embeddings with respect to the corresponding inputs(see further implementation details in AppendixD).
To evaluate video-to-audio generation quality, we follow the method of [23] which proposed the metries Dellbackk-Libbes Leibber-Diveeeeeeeee (KLD) with PassT [63, Fréchet Audio Distance (FAD) (64] and Align Accuracy (Align Acc) 25 KLD measures how closely the generated audio matches the GT through pairwise comparison, reflecting how well the audio captures the concepts in the video. FAD evaluates the overall distribution, indicating the overall quality of the audio. Align Ace assesses the relevance and temporal alignment of the audio and the video. Additionally, incorporate generation speed (time taken per waveform sample) to measure efficiency. We also compute the CLAP score [65] to evaluate the adherence generated audio to text prompts compare results with text-to-audio generation. Further details of these metrics described in Appendix
Evaluation Metrics:
For video-to-audie captioning, we use two types of metrics, natural language generation (NLG) metrics and audio-text relevance metric. NLG metrics evaluate the generated captions with respect to the ground truth audio captions using rule-based matching in terms of precision and recall. These
metrics include BertScore [66). BLEU-4 [67], ROUGE-L [68] and CIDEr [69]. To assess the
relevance of generated audio captions with the actual audio, compute the CLAP-score (65 as cosine similarity between audio and text embeddings.
Quantitative Evaluation of Audio Generation: We evaluate audio generation of VATT models on the VGGSound test split. For each the 15,446 video samples, we generate 10-second audio waveform. We compare VATT variants against existing video-to-audio generation methods as well as text-to-audio generation methods including AudioLDM-2 32 and AudioGen [37] using different text prompts. The results on the metrics described above are summarized in Table [ and Table 2 VATT models achieve best KLD score and Align Ace against other methods while maintaining competitive FAD (top 2). Notably, when guided by GT audio captions (VATT-LLama- and VATT-Gemma-T, bottom) our models generate sounds that match the GT audio more accurately, as indicated by lowest KLD score of 1.41 and 1.66 for VATT models with two LLM backbones, surpassing both video-to-audio and text-to-audio methods. In comparison to text-to-audio methods, VATT models achieve competitive audio-text alignment in terms of CLAP score, demonstrating strong capability to follow text prompts. Implementation details of these baselines are included in Appendix E
Quantitative Evaluation of Video-to-Audio Captioning: We evaluate video-to-audii captioning by prompting VATT Converter to generate audio captions. We use the prompt "Describe the possible audio for this video:" to generate captions for all VGGSound test videos. For baselines, we prompt CLLAVA-1B--v1 model two zero-shot modes to generate visual and audio descriptions respectively. Since LLAVA can take single image input only, select the middle frame of videos. We

---

Table 2: Quantitative results comparing VATT with text-to-audio generation methods on VGGSound test set. '-T' refers to model with text prompts. CLAP score calculated as the cosine similarity of generated audio with respect to the GT audio caption.
Methods
Text Prompi
KLD
FAD 4
Align Acc
CLAP Score
AudioGen 37
LLAVA visual caption
3.65
6.03
41.66
AudioGen
37
GT audio caption
2.19
3.17
48
0.409
AM-
32
LLAVA visual caption
3.54
3.62
53.49
AudioLDM-22
32
audio caption
2.09
2.46
51.84
0.326
VATT-LLama-T (Ours)
GT audio caption
1,41
2.54
80.16
0.347
VATT-Gemma-T (Ours)
GT audio caption
1.66
2,98
81.48
0.310
Relevance Preference
Fidelity Preference baselines
SpecVQGAN
74.2%
SpecVQCAN
73.4%
71.0%
IMEWAV
IM2WAV
68.9
Dif-Foley
2%
56 2%
FoleyGen
FaleyGan
64.4%
11.6%
V2A-mapper
V2A-mapper
20%
40%
60%
80%
100%
10%
20%
40%
60%
80%
100%
Strong preference VATT
Weak preference VATT
preference
VATT
Waakk prefirance VATT
preference
Weak preference bassling
Strang proference bencinn
Weal preferance beocking
Strong proference basclins
Figure 4: Qualitative evaluation results: Pairwise Comparison of generated audio VATT v.s other methods comparing Fidelity and Relevance aspects.
use "Provide concise, descriptive caption for the following image." as the visual prompt, and "Describe the sounds that this scene could yield in a short sentence without reasoning" as the audio prompt. We also compare against video LLM baseline, Video-LLAMA-77,, to perform zero-shot video-to-audio captioning. Specifically, we directly input VGGSound videos into the VL branch of the Video-LLAMA model, and prompt it to generate audio captions using the instruction "User/ What sounds could match the video?" Since Video-LLAMA has not been pretrained on VGGSound dataset and LTU generated captions, implement similar structure of Video-LLAMA and train on our LTU-generater captioning data. We replaced the original BLIP-2 visual features used by Video-LLAMA with our eva02-CLIB L- visual features due to the expensive pre-procee pre-processing time for all BLIP-2 features from videos in VGGSound and AudioSet. For the Video-QFormer component of Video-LLLAMA, we keep it the same as Video-LLAn LLAMA, and we name this model as VATT-Qformes LLama. Our evaluation is summarized in Table 3 VATT models with LLMs outperform LLAVA- prompted and Video-LLAMA zero-shot results demonstrating stronger capability to infer sounds from videos semantically. In particular, when measuring audio-text relevance, our model with LLama achieves an increase of +5.0% % in accuracy when compared with LLAVA visual caption baselines. For reference, the ground truth audio captions generated by LTU 5 have an average CLAP score of 0.379.
Table 3: Comparison of video-to-audio captions on NLG evaluation metrics and text-audio relevance (CLAP Score).
Methods
BertScore (FI)
BLEU-4
ROUGE-L 1
CIDBr
CLAP Score
I.AAVA w/ Visual Prompt
0.855
0.089
0.137
0.026
0.213
LLAVA w/ Audio Prompt
0.870
0.123
0.155
0.095
0.182
Video-LLAMA w/ Audio Prompt
0.861
0.091
0.117
0.021
0.204
VATT Converter Gemma (nurs)
0,900
0.345
0.337
0.926
0.229
VATT VATT-Qforeer LLama
0.907
0.419
0.375
1.264
0.245
VATT Converter LLarna (ours)
0.909
0.424
0.384
1.354
0.263
In addition to quantitative evaluations, we also conduct a qualitative (subjective) study to evaluate audio generation perceptual quality of VATT. Specifically, we randomly select 100 videos from VGGSound test split with stratified sampling according to video categories. For each method in the baseline, we pair the generated samples against VATT. Two aspects of the generation are evaluated, Fidelity and Relevance. Fidelity focuses solely on audio quality, while Relevance evaluates the semantic relevance and temporal alignment of audio to the vidco. For each
Qualitative Evaluation:

---

Table 4: Architecture Ablation Study.
Methods
KLD 4
FAD 4
Align Acc 1
VATT-V
2.43
2.53
82.43
VATT-Cross-AL
2.76
3.63
76.85
VATT-Gemma (Ours)
2.25
2.35
82.81
pair, raters asked rate their scoring of VATT versus compared baseline on Likert scale from (strongly prefer baseline) to 5 (strongly prefer VATT). We use our best VATT variant, VATT- LLama-T (with GT text guidance), for the comparison. As shown in Table VATT surpasses other methods in Relevance. In terms of Fidelity, VATT is consistently being preferred when compared with most baselines and slightly less preferred when compared with V2A-Mapper. The reason could be that V2A-Mapper is directly optimized with diffusion techniques on AudioLDM, a large-scale pretrained text-to-audio model, such models tend perform better in fidelity aspect in comparison to ooken-based models. Further details of qualitative evaluations are incorporated in Appendix G and qualitative samples are provided in Appendix B
We study the effectiveness of VATT Converter by removing the LLM and directly feed the visual features into the decoder. We denote such model VATT-V. While VATT-V does not handle textual inputs or generate text, still serves as a strong variant of VATT for video-to- audio generation. To study the contribution of audio token decoder, replace the decoder part of VATT with interleaving attention blocks proposed in vanilla transformer [19], and denote this variant as VATT-Cross-AIt As shown in Table 4 VATT-Gemma model outperforms both VATT-V and PATT-Cross-Attu When VATT is conditioned on visual inputs only its performance is lowest across variants.. The VATT Converter enhances the visual features through audio-relevant text, thereby improving the relevance and quality of the generated audio. In addition. we find that the bi-directional transformer design in VATT Audio is critical for learning the associations between audio and conditioning inputs to enhance audio generation performance. Additional ablation studies can be found in Appendix A
Ablation Studies:
5 Conclusion
In this work, we propose a multi-modal generative framework that enables both text-guided video-to- audio generation and video-to-an auud captioning. Experiments show that our method can generate high quality audio through text in both unconditional and conditional modes, as well as to generate reasonable audio captions from videos. One area for improvement is the diversity of the text generated by current audio LLMs. In cases where the user-provided text prompts significantly differ in style there is possibility for conflict of audio quality and adherence to the instructions. Future work could enhance the capability of the model to generalize across different text styles and to further develop capabilities for informative iterative conversation-liks video-to-audio generation.
Broader Impact
VATT could augment existing audio-van video creation tools for content creators by allowing generation of custom audio tracks for given visual content through user provided text prompts. Also, VATT has the ability to suggest potential sounds for given video which can inspire creators by presenting audio options that may not have been considered otherwise. This feature can be useful for brainstorming of content creation, where audio choices can influence the style of the final product.
Further extensions of this work could involve conversational video-to-audio generation such that the audio content is iteratively being refined. By integrating conversational interface, the users can engage in dialogue with the system, making requests and receiving responses. This approach goes beyond static text inputs, offering a more accessible toolset that does nto require significant audio editing expertise. Moreover, the conversational system can seek clarifications or propose alternatives, functioning like an assistant to avoid misunderstandings and enhance audio quality. More broadly, the generative approach proposed here has the potential to adapt to other generative areas not limited to audio, video, but also potentially impact fields such as biochemistry, physics where generative approach is utilized, e.g., generative modeling of high-energy particle events [70].
10

---

While VATT presents potential for content creation, the ability to generate realistic audio from visual inputs could lead to misuse, such as creating deceptive content or deepfake audio and ethical concerns must be addressed before utilization. Furthermore, similarly to audio generation, text generation capability could result in misuse such as offensive language or privacy violations. To mitigate these risks, in further development or potential code release we will establish clear ethical guidelines, evaluate for biases, and implement safeguards to ensure responsible use and fair outputs.
Acknowledgments and Disclosure of Funding
We acknowledge the support of HDR Institute: Accelerated AI Algorithms for covery (A3D3) National Science Foundation grant PHY-21 17997 and the departments of Applied Mathematics and Electrical and Computer Engineering at the University of Washington.
Dis-
References
[1] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[2] Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Mareo Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305..m 096666 2023.
[3] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[4] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, IEEE, 2017.
[5] Yuan Gong, Hongyin Luo. Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint 2.t.::233...1999, 2023,
[6] Kun Su, Xiulong Liu, and Eli Shlizerman. Audeo: Audio generation for a silent performance video. Advances in Neural Information Processing Systems, 33:3325-3337, 2020.
[7] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley Learning to generate music from vidcos. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28 2020, Proceedings, Part XI 16, pages 758-775. Springer, 2020.
music:
[8] Kun Su, Xiulong Liu, and Eli Shlizerman. Multi instrumentalltttt net: Unsupervised generation of music from body movements. arXiv preprint arXhv:2012.03478 2020.
[9] Kun Su, Xiulong Liu, and Eli Shlizerman. How does it sound? Advances in Neural Information Processing Systems, 34:29258 29973,, 2021.
[10] Xiulong Liu, Kun Su, and Eli Shlizerman. Let the beat follow you creating interactive drum sounds from body rhythm. In Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7187-7197, January 2024.
[11] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross- moosl music and image generation. arXiv preprint (tt.:2200600777..1 2022.
[12] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. Visually indicated sounds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[13] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550-3558, 2018.

---

[14] Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:/6/2.078377 2016.
[15] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[16) Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[17] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprini arXiv:2210. 13438, 2022.
[18] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1700.03762. 2017.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[21] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arX1v:2110.08791, 2021.
[22] Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[23] Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually- -uidedd audio generation. arXiv preprint arX1v:2309.10537, 2023.
[24] Kun Su, Judith Yue Li, Qingqing Huang, Dima Kuzmin, Joonseok Lee, Chris Donahue, Fei Sha, Aren Jansen, Yu Wang. Mauro Verzetti, et al. V2meow: Mcowing to the visual beat via music generation. arXiv preprint 250..36350069444 2023.
[25] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to- audio synthesis with latent diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vol- ume 36, pages 48855-48876. Curran Associates, Inc., 2023.
[26] Heng Wang. Jianbo Ma, Santiago Pascual. Richard Cartwright, and Weidong Cai. V2a-mapper: lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 15492-15501, 2024.
[27] Kun Su, Xiulong Liu, and Eli Shlizerman. From vision to audio and beyond: A unified model for audio-visual representation and generation. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 46804-46822. PMLR, 21-27 Jul 2024.
[28] Shentong Mo, Jing Shi, and Yapeng Tian. Diffava: Personalized text-to-audio generation with visual alignment. arXiv preprint arXiv:2305. /2003, 2023.
[29] Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. arXiv preprint 2401v22401.04577, 2024.
12

---

[30] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren. Luping Liu, Mingze Li. Zhenhui Ye. Jinglin Liu. Xiang Yin, and Zhou Zhao. Make-an- audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916-13932. PMLR, 2023.
[31] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the International Conference on Machine Learning, 2023.
[32] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang. and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308. 05734, 2023.
[33] Hao-Wac Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg- Kirg-Kirkpatrick, and Julian McAuley. Clipsonic: Text-to-audio synthesis with unlabeled videos and pretrained language-vision models. In 2023 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pages 1 1-5, IEEE, 2023.
[34] Qingqing Huang, Daniel S Park. Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang. Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint 2.0.:v2302.039111111 2023.
[35] Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf. Mo'," usai: Text-to-musi generation with long-context latent diffusion. arXiv preprint arXiv:2301. /757, 2023.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[37] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprini arXiv:2209. 15352, 2022.
[38] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
[39] Zalán Borsos, Raphaël Marinier, Damien Vincent, Fugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, al. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[40] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint 2rXiv:2301.1133 2023.
[41] Hugo Flores Garcia, Prem Scetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686. 2023.
[42] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495 507,5-5070000 2021.
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 2...Xiv2230..3997... 2023.
[44] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followinn Ilama model. https://gi thtp::////tttb....... com/ tttssttttsuaaaa http://tttttta//tttotorrrdooddd aalpcaaaaaaaaaaaaaaeaaaaaaaaaaaa 2023.
[45] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Erie P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
13

---

[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[47] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-ture audio-visual language model for video understanding. arXiv preprint arXiv:2306.02880 2023.
[48] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 18090-18108, Curran Associates, Inc., 2023.
[49] Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Ausialvisual large language model for grounding space and time. arXiv preprint arXiv:2407.01851, 2024.
[50] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288-5288099 2016.
[51] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[52] Xiulong Liu, Zhikang Dong, and Peng Zhang. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4478 4487, 2024.
[53] Zhikang Dong, Xiulong Liu, Bin Chen, Pawel Polak, and Peng Zhang. Musechat: A conversa- tional music recommendation system for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12775-12783 June 2024.
[54] Xiulong Liu, Sudipta Paul, Moitreya Chatterjee, and Anoop Cherian. Caven: An embodied conversational agent for efficient audio-visual navigation in noisy environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3765-3773, 2024.
[55] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
[56] Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[57] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. arXiv preprint arXiv:2309.05519, 2023.
[58] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0900 2021.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach. R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[60] Wouter Kool, Herke Van Hoof, and Max Welling, Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 3499-3508. PMLR, 2019.
[61] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXh::2003.15389 2023.
[62] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 24X 2024.
14

---

[63] Khaled Koutini, Jan Schlüter, Hamid Eghbal-Zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint arXiv:2110.05069, 2021.
[64] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fry 'echet audio dis- tance: A metric for evaluating music enhancement algorithms. arXiv preprini arXiv: arXiv:1812.00 182..0866, 2018.
[65] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Beng-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-auue audio pretraining with feature fusion and keyword- to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEBE, 2023.
[66] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:/904.090 2019.
[67] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[68] Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- rion Branches Out, pages 81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[69] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[70] Qibin Liu, Chase Shimmin, Xiulong Liu, Eli Shlizerman, Shu Li, and Shih-Chieh Hsu. Calo- vq: Vector-quantized two-stage generative model in calorimeter simulation. arXiv preprint arXiv:2405.0066 06665 2024.
[71] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12500 2022.
arXiv preprint
15

---

Appendix
In this Appendix, we provide:
Additional Ablation Studies and Comparisons, see Appendix A
Qualitative Examples and Analysis, see Appendix B
Details and Examples of our synthetic "V2A Instruction" Dataset, see Appendix C
Additional Implementation Details of VATT, see Appendix D
Additional Implementation Details of baselines, see Appendix
Details of Evaluation Metrics, Appendix
Human Evaluation Details, see Appendix G
A
Additional Ablation Studies and Comparisons
Masking Ratio Distribution Ablation: Designing an appropriate varying masking ratio distribution for training is essential to achieve high audio quality and relevance. We study several commonly used masking ratio distributions, including Uniform distribution, Gaussian distribution and are cosine distribution. For Gaussian distribution, we experiment with distributions with 4 different mean values, 0.55, 0.75, 0.95, and moving mean following a sine schedule with respect to the training epoch (similar curriculum learning), in the range of [0.25, 0.95]. The standard deviation kept fixed 0.25. We use VATT Gemma-2B model for the masking ratio ablation study. As shown in Table 5 the model performs better when the distribution has higher mean in masking ratio, especially cosine distribution and Gaussian mean with 0.75. This is due to the fact that the initial steps during the sampling stage are important for future decoding steps. The initial steps correspond to high masking ratio cases. For later steps, new tokens are unmasked conditioned on more clues such that the masking ratio decreases and the generation becomes less challenging, thus making the learning at lower masking ratio easier during training.
Self- self-prompting Ablation: When additional text prompts are provided as inputs, could use the audio captions generated by the VATT Converter as the text prompt to generate the audio. In this self-prompting mode, the generated audio could be interpreted by the same model in terms of the caption. As shown in Table 6 when our model is fed with corresponding generated captions, the model performs slightly worse than the model without prompt input, showing the space for improvement in the quality of generated captions. Also, generation of audio with the captions from VATT-Converter-II LLama outperforms captions from VATT- Coe---oonn--eeeminnee from the FAD and KLD scores. As the caption quality improves, the text-conditioned tideo-to-audio generation performance also improves. The GT audio captions generated by LTU obtains the highest CLAP score (measured with respect to the GT audio in original video) of 0.379, reflecting the best caption quality. Feeding such GT captions as input to the model also leads to the best audio generation results.
in particular evident
Comparison with Text-to-Audin generation methods on AudioCaps: We use VGGSound as main dataset and benchmark to evaluate since it is a large-scale audio-visual dataset with around 200K videos across many categories, and also the quality of audio-visual alignment is high. To further show the generalization capability of VATT, we experiment with AudioCaps dataset. Due to limited video samples in AudioCaps, we finetuned our VGGSound pretrained VATT model on AudioCaps dataset in two settings, with and without text prompts. To keep the comparison fair, we use the GT audio captions from AudioCaps as the text prompts. We use VATT-LLame and VATT-LLama-T to compare against AudioGen and AudioLDM-2. As shown in Table 7 VATT-LLamas similar level to AudioGen terms of FAD and KLD score, while falling behind AudioLDM-2. is noteworthy that the audio decoder of both AudioGien and AudioL.Ms- are pretrained on much larger data scale (7000 hrs and 30000 hrs audio respectively) than ours (700 hrs audio). Despite this, VATT still performs reasonably well on this datasct.
performs on
16

---

Video Inputs
Text Prompt 1
Text Prompt 2
Text Prompt
Live music along with people shouting splashing sounds,
Music and splashing
splashing
A man speaking, followed shouting the background,
woman is speaking. followed by loud explosion.
music heard while the volcano eruption happens,
explosion
child is crying in the background mixed with catl meowing.
The cat is meowing.
The crow is cawing the cat.
Figure 5: Qualitative samples that showcase text controllability: For same video inputs, VATT is able generate different sounds that align with the additional text prompts, showcasing its capability of performing controllable generation.
B
Qualitative Examples and Analysis
Controllable audio generation through text prompt: A unique advantage of our model lies in its capability to control the generated details of audio through text prompts. We show few samples where different text prompts
applied to the same video to generate different variations of sounds. As shown in Figure 5 our model able to generate reasonable sounds that are distinct in their semantic meaning but fit both the context of the video and aligned with the text description. The text prompts shown are all human-winn written prompts rather than synthetic ones.
VATT without prompt v.s VATT with ground truth audio captions: using our model with GT audio captions as prompt versus generation without prompt to understand why the KLD score with prompt outperforms the generation without prompt by a large margin. Upon inspection of generated samples, as shown in Figure we find that GT audio captions could steer the model generation towards the GT audio in the test set. For example, the first video shows man performing with a rope, so the rope tapping sounds (when hitting the ground) should be heard video. However, it occurs that the model without prompt fails capture this important detail, but instead generates noises from the surrounding crowd. Similar cases apply to the other two examples shown in the figure. KLD measures the pairwise difference between generated sounds and GT sounds in the feature space. Therefore, low score means that the model closely matches the semantic meaning of the GT audio in the original video, which indicates that the model is able to follow the text prompt instruction to generate the desired audio.
We also compare generation
Video-to-Audio Captioning: In addition to controllable iideo-to-addio generation through text, VATT is also able to generate audio captions from videos, providing textual suggestions interpreting what sounds could given video make. As shown in Figure 7 VATT could produce reasonable audio captions for videos across a variety audio-visual categories, showcasing the capability VATT Converter in capturing the audio relevant features from the video.
17

---

Table 5: Ablation Masking Ratio Distribution.
Methods
KIDD 4
FAD 4
Align Acc f
Uniform
2.52
2.75
80.37
Are cosine
2.34
2.26
82,88
Gaussian w./ mean 0.55
2.31
2.34
82.27
Gaussian w// mean 0.75
2.25
2.36
82.81
Gaussian w./ mcan 0.95
2.24
2.49
81.80
Gaussian w./ moving mean
2.42
2.32
81.81
Table 6: Ablation on self- coff-prompiing text-guided generation.
Methods
Text Prompt
KLD | FAD
Align Acc 1
VATT-LLama
x x
2.39
2.38
80.32
VATT-Gemma
2.25
2.35
82.81
VATT-LLama-T
CATT-Converter-LLaame
2.38
2.58
80.41
VATT-LLama-T
VATT-Converes Gemma
2.57
2.67
79.20
VATT-Gemma-T
Gemma
2.40
3.67
80.07
VATT-Gemma-T
VATT-Converter Converter-LLame
2.26
3.20
80.42
-ATT-LLama-T
GT audio caption
1.41
2.54
80.16
VATT-Gemma-T
GT audio caption
1.66
2.98
81.48
Table 7: Quantitative results against text-to text-to-audio generation methods AudioCaps test set.
Methods
KLDD [
FAD
Align Ace
CLAP Score
AudioGen
2.09
3.13
58.26
0.447
Audiol.DMe
1.64
1.86
60.32
0.432
VATT-LI
2.53
3.42
75.76
ama--LLama-T
2.07
3.25
74.89
0.376
Video Inputs
No Prompt
With Ground Truth Text Prompt
"Tapping" patterns spectrogram
Tapping sounds heard with noises from the crowd.
"Loud
caterwauling" patterns in the spectrogram
Two cats are meowing and caterwauling at each other loudly.
"plano music" patterns in the spectrogram
Music and piano sounds are heard.
Figure 6: Steering generation towards ground truth audio: For same video inputs, we compare our generation results without text prompt v.s feeding ground truth audio caption additional prompt.
18

---

horse neighs and clip-clop sounds.
Church bells ringing loudly, reverberating and resounding.
A smoke detector beeps.
snake hisses loudly,
Whales are making sounds underwater.
A chop heard multiple times.
Crushing cracking occur.
subway train is moving clickety- clack sounds and background noise.
thunderstorm heard with the background.
Figure 7: Video-to-Audie Captioning samples by VATT.
C Details and Examples of V2A Instruction Dataset
We describe the synthesis procedure our V2A Instruction dataset. To obtain audio captions for training and evaluating VATT, which is pretrained on a large-scale audio understanding OpenAQA dataset, including audio from VGGSound and AudioSet-2MM LTU demonstrates strong capability in audio captioning zero-shot prompting manner, accurately reflecting what happens in the audio.
existing existing audio large language model, LTU [3],
Specifically, adopt the prompt "Close-ended question: sound." used during LTTU training for audio captioning task, and feed 10-second audio from VG- GSound and AudioSet dataset inputs into LTU-13B model (max length 108 version). In figure 8 we show 15 examples of synthesized captions from videos in VGGSound along with the correspond- ing video IDs and start-to-end time. The generated captions clear natural language and faithfully describe the audio content in details, serving as a reliable dataset for training and evaluation.
Write an audio caption describing the
D Additional Implementation Details of VATT
Data Preprocessing: For visual inputs, extract video frames 5fps rate, resulting in 50 frames for each 1Os video. For audio, we extract tokens from audio waveform using pretrained Encodec-1 16kHz, resulting a 500 token matrix for each 10s audio. To extract visual features, we resize each video frame to 336 x 336 and normalize, and feed into the eva-CLIP-L [61] image encoder. By extracting the mean-pooled vector from the hidden states, we represent each video frame with 768-dim vector. For textual inputs, we use template text prompts input to instruct video-to-audin captioning, including 10 numan-writted prompts. For training the VATT Audio, two cases are considered. In unconditional generation case where no additional audio caption is provided, the video along with one of the template text prompts (shown in Table are fed as inputs to the model. In conditional generation case where the ground truth audio caption is provided, the caption replaces the template prompt as the textual input to the model. For both cases, textual inputs are formatted using "alpaca short" instruction style, "μнн Instruction:
instruction ### Response:".
19

---

YouTube ID
Start end time (s)
Audio caption
r5bG4Zd0Z6M
1129 1139
Paper being crumpled and with mechanisms making noise.
R6pY_ivHxx
70 80
A car is skidding and a man speaks in the background, with tire squealing and speech.
eLxrJINB980
-00
Music is playing and saxophone is being played in the background.
qHgTFW NMRs
41
Insects are chirping, man speaking and roaring lions be heard.
D7DZMOC0ty8
30
Rain falls and thunder booms.
#FTFyawoWiE
67 77
A steam whistle blows and a train moves with hissing sounds.
-CfhAXuBCJQ
118 128
A man is speaking, basketball bounces and people are clapping, cheering, and whistling while crowd heard in the background, with occasional male speech and background noise.
SutoPvInL8
33 43
Bouncing sounds of ping pong game are heard.
XyjsrZegPnl
20 30
People are speaking, shouting, and clapping amidst firecrackers, hubbub, and speech.
dtyLvXtucLs
40
A church bell rings and ticks.
BI FyZ4c__pp
53
A beatboxer is performing, with crowd cheering and speaking sounds.
8ay0Mt Z8GQ
30 40
A man is speaking, chopping food, and breathing with background noise.
N1VHutvx7JU
109 119
A choir sings with yodeling and breathing the background.
JUBdOrrHHss
30 40
accordion is playing music, with reedy and airy quality.
C6xMq9aur5Y
121 - 131
Wind blows and a person skiing on snow with wind in the background.
Figure 8: Examples of V2A Instruction Dataset.
No.
Text Prompt Template
Imagine possible sounds for this video.
Describe possible sounds that could match the video.
What audio could be inferred from this video?
What sounds could match the video?
Infer the sounds that match the video.
What audio could best match this video?
What sound events could the video yield?
Caption possible sound events that describe the video.
What sound events make sense this video?
10
Imagine audio events that match this video.
Table Text prompt templates used in video-to-audio captioning and unconditional video-to-audin generation.
Training: For VATT Converter, we perform V2A cnrruction-Tuningg with our 1.77M audio captions from VGGSound and Audioset altogether. To project visual features to LLM embedding dimensions, we first apply adaptive pooling along the temporal axis, reducing the temporal dimension from to 10 to conserve GPU memory. Then linear layer is applied each time step of the visual features to project 768-dim to the LLM dimension (4096 for LLama and 2048 for Gemma-2B). The training process involves two sub-stages. In the first stage, we turn off the LoRA setting and tune the VATT Projector layer only. After 2 epochs, we start the second stage training by tuning both projection weights and LoRA parameters for another 4 epochs. We use Adam W optimizer with a base learning rate le-4, and limit the maximum length the audio captions to be 108. For both LLama-7B and Gemma-2B, initialize our model with instruction fine-tuned checkpoints available on huggingface.
For VATT Audio, we explore various masking ratio distribution and end up using a truncated Gaussian distribution with mean of 0.75 and standard deviation of 0.25, truncated between 0.5 and 1.0. To enable classifier-free guidance during sampling, we randomly replace the conditioning features with vectors of all zeros with 10% rate. The training also takes two sub-stages: We first train the model in unconditional generation mode without ground truth audio caption text prompts as inputs until convergence, and then ii) train the model conditioned on GT audio captions textual inputs (limiting
20

---

maximum text length to 64). Additionally, we apply two types of data augmentations during the unconditional training phase to facilitate the temporal alignment between audio and video: temporal mixup following [25 with a probability of 0.5, ii) temporal rolling on the video features together with audio tokens by same amount of time. We train our model with a base learning rate of 2e-4 with a warmup step of 40k and linear decay schedule with 600k steps. The number of trainable parameters for VATT Audio is 415M. We use a batch size of 48 for Gemma model and 36 for LLama model.
All our training procedure are conducted on a single A A00 80GB GPU, VATT Converter training takes 16 hours and VATT Audio takes 3 days in total.
Inference: For Video-to- Video-to-Audic Audio Captioning, we adopt the generation configuration with temperature of 0.1, top_ of 0.95, top_ k of 500 and repetition penalty of 1.1 for both LLMs. We use one of the templates "Describe possible sounds that could match the video." to prompt LLMs to generate captions.
For audio generation, we adopt the iterative parallel decoding strategy with total decoding steps of 16, softmax sampling temperate 1.0 with top_k k being 256. For masking sampling with Gumbel top-k strategy, we use initial temperature of 27.5 with linear decay each iteration, where the re-masking ratio follows the cosine schedule, same as 11 Following [23], we also use classifier-free guidance [7] during sampling, and we find a cfg_scale of 5.0 works best for our model.
E Implementation Details of video-to-audio generation baselines
We follow the open source SpecVQGAN's codebase instructions to extract visual frame-level RGB features 21.5 21.5fps rate using ResNet-50 checkpoints for all VGGSound test videos. Following the evaluation script setup for VGGSound, we generate a 10-second audio waveform per each video in the test set.
SpecVQGAN [1]:
Following the open source codebase of IM2WAV, we extract CLIP visual features from videos at 30fps rate for VGGSound test videos. IM2WAV was originally trained to generate 4-second audio. In order to adapt it to generate a 10-second audio, we infer the model with 3 forward passes to obtain three 4-second audio segments without overlap, and concatenate them together and then trim to a 10-second audio.
IM2WAV [22]:
Doff-Foley [25] Diff-Foley uses their pretrained CAVP audio-visur model to extract features at 4fps rate. Following their open source codebase, we extract visual features for test videos in VGGSound, and apply their best generation configuration with double guidance scale CFG scale = 4.5, CG scale 7 = 50, to generate three non-overlapping 4-second audio segments the same way as IM2WAV.
The authors do not open source their implementation. We strictly follow the setup in the paper and uses the open source version of Encodec- 16kHz to replicate their model. For visual features, we follow their implementation to extract the CLIP features at 1fps rate. FoleyGen is a 24-layers transformer architecture with hidden size of 1024 and 16 heads. Using their best visual attention configuration "All-frame" attention along with random visual condition dropout with a probability of 1, we train FoleyGen with specified hyperparameter settings as described in the paper. In the inference stage, we apply classifier-free guidance scale of 3.0 as well as top-k 256 sampling configuration in the paper to generate 10-second video per test video in VGGSound. Upon evaluation, we find that there is a noticeable gap between our implementation results (KLD: 2.89, FAD: 2.59) and reported results in paper (KLD: 2.35, FAD: 1.65). To study where the gap comes from, our extracted Encodec tokens to reconstruct the audio waveform in VGGSound test set, and measure the FAD score of reconstructed audio waveform with respect to the ground truth audio. We find that the open source Encodec- 1occe-166kke 16kHz on huggingface could only achieve a FAD score of 1.86, which still falls behind their reported result of 1.65, indicating that the released Encodec model is sub-optimal version.
FoleyGen (23]:
V2A-Mapper 26] V2A- is not yet open sourced, but the authors publish their generated audio for 15,446 video samples in VGG$ound test set. We therefore download their samples and conduct both objective and subjective experiments based on them.
21

---

F Details of Evaluation Metrics
KLD score: To compute the KL.-Divergence between generated samples and ground truth audio, we adopt the pretrained PaSST 63]. an audio transformer classifier, on AudioSet-2M dataset to extract the classification output probabilities. KLD is evaluate in a pairwise manner and we use the mean value over all 15,446 samples as our KLD score.
Fréchet Audio Distance 66 (FAD) evaluates the difference between the distributions of generated samples and ground truth samples. Specifically, we adopt the pretrained VGG-ish network on AudioSet-2M dataset to extract the features of generated audio and ground truth audio. Using the multivariate gaussian assumption on extracted features, we compute the mean and covariance of the generation sample set and ground truth audio set, and then apply the Fréchet distance formula to obtain the FAD score. Specifically, to ensure the correctness of computation, we use the original implementation of Google Research's 's tensorflow version to perform evaluation.
FAD score:
Align Ace: To evaluate temporal alignment and relevance of audio to video, we also incorporate Align Ace metric proposed by [25] Specifically, Align Ace is computed using a CAVP (contrastive audio-visual pretraining) model by taking video frames along with the audio mel -el-spectrogram as inputs. The model outputs an accuracy score representing the alignment between audio and video. Following their configuration, we use visual frames at 4 fps rate as visual inputs (40 frames for 10-second video), and convert the 16kHz audio waveform into mel-spectrogram with FFT Num 1024, mel basis Num 128 and hop size 250, resulting 640 x 128 mel-spectrogram for 10-second audio waveform.
We benchmark the generation speed on a single A6000 GPU, and measure the average sampling time (in second) for single 10-second sample. For all baselines and our method, we use batch size of to run the test over 15,446 test videos on VGGSound. For V2A-Mapper, since we are unable to obtain their source code for testing, we instead report the AudioLDM-L inference speed on single sample as an approximation of infer Time of V2A-Mapper since the method is a close adaption of AudioLDM.
Infer Time:
CLAP score For comparing adherence of generated audio to the text prompt, we use CLAP model to extract the audio and text embeddings, and then measure the cosine similarity between the generated audio embedding and text prompt embedding. For evaluating the video-to-audie captioning, we again apply CLAP to compute the cosine similarity between the generated audio captions and the ground truth audio for the video.
G Human Subjective Studies Details
For human evaluations results on audio generation shown in Table[4, we used the Amazon Mechanical Turk platform create survey and crowdsource responses. We evaluated 100 video samples randomly selected from VGGSound test set. We used stratified sampling such that each video comes from different audio-visual categories.
To ensure the quality of the survey, we applied constraints on accepted human raters for our survey. In particular, we selected raters that who have historical approval rate of greater than 95% well as possess language proficiency in English. Responses that take less than 20 seconds or longer than 10 minutes excluded from the answers. In addition, no samples could be evaluated twice by the same worker to avoid potential bias.
We set up two types of surveys: audio quality survey and audio-video relevance survey. In audio quality survey, we ask the raters to focus only on the audio quality aspect by providing the question "In Which video the overall audio quality is better?". In audio-vide video relevance survey, question of "Which video whose audio is more relevant to and temporally aligned with the vidco." " asked. In both surveys, we ask the rater to choose their preference at Likert scale from (strong preference of baseline method) to (strong preference of our method), 5 levels in total.
For each video, we request 5 responses from 5 distinct raters. To comply with the NeurIPS code of conduct and rules of the platform, we pay at rate of 0.05 USD per each response, satisfying the lowest wage requirement in any region of the world. Upon running the evaluations on all 10 specific surveys (5 methods and each with two evaluation types), we are able to collect 500 valid responses for each pairwise comparison, from 23 distinct raters on average with 21.7 + 5.6 ratings per participant
22

---

Ours VZA-Mapper
Ours FoleyGen
Ours Diff-Foley
Fideityy
Fidertty Relevance
Fidelity Relevance
Relevence
310314
120
Counts
122
Caunts
Caunts
65
88
14
23
28
Preference
Preference Score
Profarence Score
Ours v.s IM2WAV
Ours v.s SpecVOGAN
Hooelly Relevence
Fidertyy
200
Relevance
204
761
Counts
120
Churts
31.
100
123
20
24
Preference Score
Preference Score
Figure 9: Details of pairwise comparison results between VATT and baseline video-to-audio methods.
We show the details of comparison results between our method and baseline methods bar charts in Figure 9
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We clearly state the contributions and the experiment results match these contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention the limitations and the room for improvements in the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[No]
Justification: Our paper is mainly application and experiment-driver
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include sufficient details of implementations of our models, baseline models, and also evaluation metrics along with human evaluation surveys.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and synthetic data that we generated for training models can be obtained for limited use upon request from the corresponding author. We also aim to release the code and synthetic data that we generated for training models as public repository on the Github upon obtaining involved approvals.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
sublic/guides CoodeSubmissionoolliii hataai/ggiidee/ooeeubbm.mse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
http:/ddeeeeeeeeeeeuuaasiiiioaaa wattamisssiioooooiiooo cilips.cc/publle http://uwlccgguudeeeeeeeeeeeeeee httpa/////uideee/ddeebbbbiibsis
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all details training, inference, dataset and evaluation metrics necessary achieve the results we claim in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our models are evaluated on large-scale test set and the variations should be small and negligible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the details of computing devices, memory and time of execution used to conduct training and evaluation of our method.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
We have reviewed and carefully followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix ??.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss the need for safeguards regarding the negative impact associated with multi-modal generative modeling and propose precautions and guidelines prevent misuse. These details are included as part of the Broader Impact section in Appendix ??.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cited all open-source works that we use including the data, models and codes.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We incorporated section in supplementary text describing synthetic data and placed samples. We aim to release the synthetic data that we generated for training upon obtaining involved approvals.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We included detailed description of human study instructions, and and the setup of the of the study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We submitted IRB approval request to our institution for review and was approved.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "OcZPuPcO6d", "reviewer": "Reviewer_5yrj", "summary": "- The paper proposes a model for video-to-audio generation (main task) and video-to-audio captioning (auxilliary task). \n- The text is optionally used to control audio generation for ambiguous video cases. \n- They use two step approach i.e. video-to-caption stage and video+text-to-audio stage\n- Stage 1: video-to-caption\n  - This stage basically converts videos and text to audio relevant captions\n  - They train a VATT converter that uses LoRA finetuning to get audio captions. \n  - This stage uses AudioSet and VGGSound and captions generated using LTU model.\n\n- Stage 2: video+text-to-audio \n  - This stage generates audio tokens, given video+text as input. The video+text is converted to audio relevant captions using stage-1\n  - They use LM based audio token generation. The VATT Audio decoder learns to predict the masked audio tokens. During inference, iterating parallel decoding is used.\n  - VGGSound is used to train this stage.\n\n- There model shows comparative and better performance on video-to-audio(V2A) generation task and text to audio generation task. Add text to V2A task further improves KLD score.\n- Stage 1 when compared to baselines can perform better on video-to-audio captioning.\n- Qualitative and human evaluation shows preference towards their work.", "questions": "My questions are based on the arguments mentioned in the weakness section:\n\n- Can the correctness of the LTU-generated captions be verified? Since they are used as the ground-truth for several tasks?\n\n- Can the baseline text-to-audio(TTA) generation models be trained on VGGSound+LTU captions? Can VATT be compared on AudioCaps?\n\n- Can VATT be compared on other video captioning datasets? Or the existing video-captioning models be finetuned/trained from scratch on this LTU data?\n\n- Why are some metrics different from the numbers quoted in the original paper? For eg. Diff-foley alignment score.", "limitations": "Yes the authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a new problem of utilizing text for video-to-audio generation.\n- Several novel and interesting technical contributions. i.e. \n - Video+text conditioning to generate audio by converting this condition to joint audio-caption.\n - Using synthetic captions for training and LoRA for MLLM finetuning.\n - Using iterative parallel decoding and bi-directional self-attention. \n- Their approach improves on video-to-audio generation, and text-to-audio generation both quantitatively and qualitatively.\n- The results also show improvement in video-to-caption generation.", "weaknesses": "- Text captions generated by LTU are used as ground truth for video-to-audio generation, text-to-audio generation, and video-to-audio captioning. The authors should verify the correctness of the captions. \n\n- Table 2 results for Text-to-audio generation seem slightly unfair for the baselines. \n      - The LTU-generated caption may have some biases or patterns of caption generation that VATT has seen during training while the \n         baselines have not.\n      - Though AudioGen has seen VGGSound tags, it has not seen the captions generated by LTU (which VATT has). \n      - AudioLDM2 has not even seen VGGSound data for training. \n      - A fair comparison would be finetuning/training from scratch Text-to-audio models on VGGSound with LTU-generated captions. Or VATT should be compared on AudioCaps (with videos downloaded from YouTube). \n\n- Table 3 results are also a little unfair:\n  - Similar to the argument above LLAVA has not seen LTU-generated captions for VGGSOUND.\n  - Also LLAVA takes only 1 frame while VATT takes more frames. \n  - Maybe a fair comparison would be to train some captioning models on LTU-generated data.\n\nMinor comments:\n- The captions for the tables should come above the table. For eg. Table 1,2,3\n- Missing reference: A previous work by Mo. et al [1] that utilizes text+video-to-audio generation. This should be mentioned \n- Align-acc for diff-foley is 82.47 in the paper vs 94.05 in the original paper\n\n[1] DiffAVA: Personalized text-to-audio generation with visual alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review and valuable feedback. We appreciate the reviewer feedback that our proposed methods include novelties and technical contributions. We address the raised concerns below.\n\n**W1 & Q1:** We manually verified the validity of LTU-generated captions prior to using them as synthetic GT. As a result of reviewer feedback, we performed an experiment to further evaluate its correctness. We randomly selected 100 videos from VGGSound test set with stratified sampling according to video categories to conduct a human study. We use the 1-5 point MOS (Mean-Opinion-Score) scale (the higher the better) to measure correctness of the captions. We provide pairs of videos and the corresponding captions to the raters, asking “How accurately the provided caption reflects the sound events happening in the video? **1. Inaccurate and irrelevant.** 2. Relevant but inaccurate with many mistakes. 3. Partially accurate but missing details and with mistakes. 4. Mostly accurate with some minor mistakes. **5. Accurate and complete.**” We used the MTurk platform to perform the evaluation and collected a total of 300 responses. The generated captions have a high MOS of mean 4.72 and std 0.37, providing another indication for the validity of the synthetic ground truth.\n\n\n**W2 & Q2:** We thank the reviewer for asking regarding further comparison against existing text-to-audio methods. Due to the time constraints of the rebuttal, we chose to finetune our VATT model on AudioCaps benchmark instead of training AudioGen and AudioLDM-2 using LTU generated captions from scratch. Specifically, due to the limited data size of AudioCaps, we use our VGGSound pretrained checkpoint to finetune instead of training on AudioCaps from scratch. As shown in the Table 1R1, VATT achieves similar performance against AudioGen on FAD and KLD metrics, while still showing some gaps from AudioLDM-2. One clear reason is that AudioGen and AudioLDM-2 are pretrained on much larger audio-text data (7000 hrs and 30000 hrs audio respectively) than ours (700 hrs audio). In addition, we also find that the audio-visual alignment from AudioCaps dataset (videos from AudioSet) is not as good as VGGSound, often suffering from static frames and weak correspondence of audio and visual. Therefore, the visual condition is not as effective as the VGGSound videos.\n\n\nTable 1R1: Quantitative results against text-to-audio generation methods on AudioCaps test set.\n| Methods       |  KLD ↓ |  FAD ↓ | Align Acc ↑ | CLAP Score ↑ |\n|---------------|------|------|-----------|------------|\n| AudioGen      | 2.09 | 3.13 | 58.26     | 0.447      |\n| AudioLDM-2    | 1.64 | 1.86 | 60.32     | 0.432      |\n| VATT-LLama    | 2.53 | 3.42 | 75.76     | -          |\n| VATT-LLama-T  | 2.07 | 3.25 | 74.89     | 0.376      |\n\n\n**W3 & Q3:** We appreciate the reviewer pointing to the need for further comparison on video-to-audio caption tasks. We have added two additional models/setups in the comparison. \n\n\nTo address the concerns that LLAVA is only trained on single frames rather than the whole video, we instead use an existing video LLM , Video-LLAMA-7B, to perform zero-shot video-to-audio captioning. Specifically, following the Video-LLAMA’s Github repository instructions, we directly input the VGGSound videos into the VL branch of the model, and prompt it to generate audio captions using the instruction “User/ What sounds could match the video?”\n\nSince Video-LLAMA is still not pretrained on VGGSound dataset and LTU generated captions, we implement a similar structure of Video-LLAMA and train on our LTU-generated captioning data. We replaced the original BLIP-2 visual features used by Video-LLAMA with our eva02-CLIP-L visual features due to the expensive pre-processing time for all BLIP-2 features from videos in VGGSound and AudioSet. For the Video-QFormer component of Video-LLAMA, we keep it the same as Video-LLAMA, and we name this model as VATT-Qformer - LLama.\n\nAs shown in Table 2R1, zero-shot performance of Video-LLAMA is similar to LLAVA, while VATT-Qformer - LLama trained from scratch performs very close to VATT Converter - LLama. It is note-worthy that the only difference between VATT Converter - LLama and VATT-Qformer - LLama is the projection method, and we find that the linear projection adopted by VATT Converter  is enough for the task to perform well.\n\nTable 2R1: Comparison of video-to-audio captions on NLG evaluation metrics and text-audio relevance (CLAP Score). Bold denotes the new results.\n\n| Methods                    | BertScore (F1) ↑ | BLEU-4 ↑ | ROUGE-L ↑ | CIDEr ↑ | CLAP Score ↑ |\n|----------------------------|------------------|----------|-----------|--------|--------------|\n| LLAVA w/ Visual Prompt     | 0.855            | 0.089    | 0.137     | 0.026  | 0.213        |\n| LLAVA w/ Audio Prompt      | 0.870            | 0.123    | 0.155     | 0.095  | 0.182        |\n| **Video-LLAMA w/ Audio Prompt** | **0.861**    | **0.091** | **0.117**| **0.021** | **0.204** |\n| VATT Converter - Gemma     | 0.900            | 0.345    | 0.337     | 0.926  | 0.229        |\n| **VATT-Qformer - LLama**   | **0.907**        | **0.419** | **0.375** | **1.264** | **0.245** |\n| VATT Converter - LLama     | 0.909            | 0.424    | 0.384     | 1.354  | 0.263        |\n\n\n**Minor Comments 1:** We will move the table titles and captions above the tables to follow the NeurIPS format.\n\n**Minor Comments 2:** We will make sure to cite the reference mentioned by the reviewer as it is closely related to our work.\n\n**Minor Comments 3 & Q4:** We did find differences between our experiment results and the ones reported in Diff-Foley’s results. However, we strictly follow the guidelines and instructions from their official repository to perform audio generation on the VGGSound test set, and we could not achieve their claimed performance. We provided all the necessary implementation details for the baseline comparisons in Appendix F for reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lyt5FIqOmm", "reviewer": "Reviewer_rzvn", "summary": "In this paper, the authors propose VATT, a multi-modal generative framework for text-guided video-to-audio generation. VATT consists of two key modules VATT Converter and VATT Audio. The former component maps video features to LLM vector space with a projection layer. The latter one generates audio tokens with a bi-directional transformer. Experiments on datasets VGGSound and AudioSet-2M demonstrate that the proposed VATT framework surpasses existing SOTA in terms of KLD score.", "questions": "See weaknesses", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors identified limitations within the existing video-to-audio generative modeling techniques, indicating a clear understanding of the challenges in the field.\n2. The publication of the generated samples is also commendable.", "weaknesses": "1. As stated by the authors, the diversity of text generated by LLM could influence the generated audio. However, it is unclear how the quality of the generated caption affects the results. It is suggested that the authors include an ablation study comparing the results of different captions quality and choose some examples to analyze the reasons.\n2. In the section on quantitative results, the proposed method is only evaluated on VGGSound, which is not enough to prove the generalization and robustness of the proposed method. It would be more convincing to conduct experiments and ablation studies on more datasets. \n3. There seems to be a writing error in part D of Appendix, “we use existing an existing audio large language model…”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful feedback and acknowledging the quality of generated samples from VATT. We address the feedback and concerns below.\n\n**W1:** The quality of generated captions do affect the generation quality of the model in the V+T -> A stage. We conduct an additional ablation study by providing captions generated by different models as inputs to the same VATT model, i.e., VATT-LLama-T or VATT-Gemma-T to evaluate the impact of models’ captioning ability on the audio generation ability. We compare two generated captions from VATT-Converter - Gemma and VATT Converter - LLama. For reference, we also report the result of ground truth audio caption as inputs to the model. As shown in the Table 1R2, when the same model is fed with generated captions with different quality, the model’s audio generation performance are different. This effect is already expected from “Table 6: Ablation on self-prompting text-guided generation” in our paper. Combined with results in Table 3 in our paper, we can see that as the caption quality improves, the text-conditioned video-to-audio generation performance also improves, in particular from the FAD and KLD scores. Notably, the ground-truth audio captions generated by LTU has the highest CLAP score (measured with respect to the ground truth audio in original video) of 0.379 as stated in the paper, reflecting the best caption quality. Feeding such ground truth captions as input to the model also leads to the best audio generation results. \n\nAs for the qualitative analysis on how the captions affect the audio generation quality, we select an example video with YouTube ID “j1kM-hC44Ok_000002.mp4” from VGGSound test set to analyze. The caption generated by VATT-LLama Converter is “A car is driving and crushing.”, while the caption generated by VATT-Gemma is “A car is driving and a man speaks with wind noise in the background.” As can be seen from the video, the car is crushing over some objects and making cracking sounds. The former caption is better aligned with the visual scene, and it generates the sound that matches closely to what is intended. While the latter caption is unrelated to the key event happening in the video, leading to poorer results. We will incorporate qualitative examples as well as demos on our website, and make more detailed analysis in our revised manuscript to help understand the correspondence between the caption quality and the generated audio.\n\nFrom another perspective, our design by enabling the model to take an extra caption as input is a useful feature since the user can provide reasonable text input to steer the generated sound with variations to some extent. In this aspect, our qualitative samples from the provided link in the paper as well as Figure 5 in the Appendix C demonstrate some interesting scenarios where a user provides different plausible captions for the model to generate.\n\nTable 1R2: The impact of caption quality on V + T -> A performance.\n\n| Methods        | Text Prompt             | KLD ↓ | FAD ↓ | Align Acc ↑ |\n|----------------|-------------------------|-------|-------|-------------|\n| VATT-Gemma-T   | VATT-Converter Gemma    | 2.40  | 3.67  | 80.07       |\n| VATT-Gemma-T   | VATT-Converter LLama    | 2.26  | 3.20  | 80.42       |\n| VATT-Gemma-T   | Ground Truth            | 1.66  | 2.98  | 81.48       |\n| VATT-LLama-T   | VATT-Converter Gemma    | 2.57  | 2.67  | 79.20       |\n| VATT-LLama-T   | VATT-Converter LLama    | 2.38  | 2.58  | 80.41       |\n| VATT-LLama-T   | Ground Truth            | 1.41  | 2.54  | 80.16       |\n\n\n**W2:** We use VGGSound as our main dataset and benchmark to evaluate since it is not only a large-scale audio-visual dataset with around 200K videos across many categories, but also the quality of audio-visual alignment is quite good. To further show the generalization capability of VATT, we experiment with AudioCaps dataset. Due to limited video samples in AudioCaps, we finetuned our VGGSound pretrained VATT model on AudioCaps dataset in two settings, with and without text prompts. To keep the comparison fair, we use the ground truth audio captions from AudioCaps as the text prompts. We use VATT-LLama and VATT-LLama-T to compare against AudioGen and AudioLDM-2. As shown in Table 2R2, VATT-LLama-T performs on a similar level to AudioGen in terms of FAD and KLD score, while falling behind AudioLDM-2. It is noteworthy that the audio decoder of both AudioGen and AudioLDM-2 are pretrained on much larger data scale (7000 hrs and 30000 hrs audio respectively) than ours (700 hrs audio). Despite this, our method still performs reasonably well on this dataset. We plan to conduct further  ablation studies by scaling up the training by incorporating more data to train VATT on the video-to-audio generation task.\n\nTable 2R2: Quantitative results against text-to-audio generation methods on AudioCaps test set.\n\n| Methods       |  KLD ↓ |  FAD ↓ | Align Acc ↑ | CLAP Score ↑ |\n|---------------|------|------|-----------|------------|\n| AudioGen      | 2.09 | 3.13 | 58.26     | 0.447      |\n| AudioLDM-2    | 1.64 | 1.86 | 60.32     | 0.432      |\n| VATT-LLama    | 2.53 | 3.42 | 75.76     | -          |\n| VATT-LLama-T  | 2.07 | 3.25 | 74.89     | 0.376      |\n\n**W3:** We will correct the grammar errors and typos like this in our revised manuscript through careful proofreading.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDW7nLeM1J", "reviewer": "Reviewer_Gn3y", "summary": "The paper introduces a two-stage model for video-to-audio generation controlled by text. In the first stage, a large language model (LLM) generates audio captions based on the video content. In the second stage, the model uses video frames and the generated audio captions to predict masked audio tokens, thereby generating the audio.", "questions": "1.\tSee the 3rd item in “Weakness”.\n2.\tIn the quantitative evaluation of video-to-audio captioning, comparing the proposed model with LLAVA, which only accepts single image input, seems unfair. Why not compare it with large language models designed for video understanding, such as VideoChat or Video-LLaMA?", "limitations": "Lack of novelty, too dense content", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tAn anonymous website is provided to showcase the audio generation results, clearly illustrating the effectiveness of the model.\n2.\tThe proposed method can accomplish both video-to-audio generation and captioning tasks, and it demonstrates superior performance compared to the mentioned audio generation models.", "weaknesses": "1.\tThe content of the paper is densely packed, with many details relegated to the appendix, making it somewhat difficult to understand.\n2.\tThere are some grammatical errors in the paper, such as in lines 133-134 where it states \"showed improvement in multi-modal understanding in in tasks such as captioning and question-answering.\"\n3.\tThe paper's novelty is somewhat limited. I noticed that the bidirectional-attention, Iterative Parallel Decoding, and masking design mentioned in the paper were already proposed in reference [1]. How do these differ from what is presented in this paper? If the only contribution is integrating video input into a large language model, I believe this approach alone is insufficient to support the acceptance of the paper.\n\n[1] Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback. We are glad that the reviewer is satisfied with the quality of provided samples and generation capability of our model, and that our model is capable of performing both video-to-audio captioning and video-to-audio generation tasks.  We address the concerns raised by the reviewer below.\n\n**W1:** We aimed to incorporate the most important experiments in the main section of our manuscript. However, we find it difficult to include all ablation studies as well as qualitative samples within the page limit due to the fact that our model works simultaneously on two tasks. We will try our best to adjust some part of the contents so that it becomes easier to read.\n\n**W2:** We will make sure to correct these grammar errors and typos in our manuscript through careful proofreading upon revision.\n\n**W3 & Q1:** In terms of technical aspect, the main differences of our proposed method from Soundstorm lie in the masking design. \n\n1. Soundstorm only tried an arc-cosine masking ratio distribution similar to the one proposed in MaskGIT. However, we explored different masking ratio distributions during training and compared them in Table 5. The study provides critical insights regarding which masking ratio distribution would be suitable for audio generation. In general, the model performs better when the distribution has a higher mean in masking ratio. This is due to the fact that initial steps during the sampling stage are important for future decoding steps. And these initial steps are in the situations of high masking ratio. For later steps, new tokens are unmasked conditioned on more clues as the masking ratio goes lower, the generation becomes less challenging, thus making the learning at lower masking ratio easier during the training.\n\n\n2. Soundstorm designed a complicated masking pattern, where any selected token will be masked together with tokens at higher levels than it. While our approach does not require this pattern dependencies and still performs well.\n\nWe will highlight these technical similarities and differences in our revised manuscript.\n\nAside from directly comparing masking novelty of VATT, we would like to reiterate that additional key novelty in our work is integrating LLM into the audio generation pipeline which enables the capability of suggesting an audio for the video through captioning, as well as generation of audio from video with text feedback. This cannot be achieved by mere combination of existing approaches since the ability to produce text as well as encode text as a condition for the output of another modality like audio is not achievable using these commonly used off-the-shelf text encoders like BERT or T5. Our work is the first to leverage the capability of LLM to achieve video-to-audio generation through text. Such a unique design boosts interactivity via text control as well as provides interpretability of the model by providing captions, which we believe is an important novelty of our method. As mentioned by Reviewer BS9v, “The paper has several technical highlights, especially on the V2A Instruction Tuning part which enables text control on the video-to-audio generation.” \n\n\n**Q2:**  We further compare our method against existing video LLM models. We experiment with Video-LLAMA as suggested by the reviewer in two ways:\n\n1. We conduct zero-shot captioning using Video-LLAMA to generate audio captions for the VGGSound test dataset. We tried different prompts and ended up using “User/ What sounds could match the video?” as it allows the model to follow the instruction closely. \n\n2. Because Video-LLAMA has not been trained on VGGSound dataset and LTU generated captions, we implement a similar structure of Video-LLAMA and then train it on our LTU-generated captioning data. Specifically, we replaced the BLIP-2 visual features with our eva02-CLIP-L visual features due to the expensive pre-processing time for all videos in VGGSound and AudioSet data. For the Video-QFormer component of Video-LLAMA, we keep it the same as Video-LLAMA, and we name this model as VATT-Qformer LLama.\n\nAs shown in Table 1R3, Video-LLAMA’s zero-shot video-to-audio captioning performance stays close to LLAVA albeit utilizing all of the video frames. When we train a similar structure to Video-LLAMA from scratch on LTU captions, we find that its performance is close to VATT-Converter LLama. The linear projection adopted by us and the Qformer projection methods don’t make much difference in our video-to-audio captioning task, and we choose a simpler way of projection.\n\nTable 1R3: Comparison of video-to-audio captions on NLG evaluation metrics and text-audio relevance (CLAP Score). Bold denotes the new results.\n| Methods                    | BertScore (F1) ↑ | BLEU-4 ↑ | ROUGE-L ↑ | CIDEr ↑ | CLAP Score ↑ |\n|----------------------------|------------------|----------|-----------|--------|--------------|\n| LLAVA w/ Visual Prompt     | 0.855            | 0.089    | 0.137     | 0.026  | 0.213        |\n| LLAVA w/ Audio Prompt      | 0.870            | 0.123    | 0.155     | 0.095  | 0.182        |\n| **Video-LLAMA w/ Audio Prompt** | **0.861**    | **0.091** | **0.117**| **0.021** | **0.204** |\n| VATT Converter - Gemma     | 0.900            | 0.345    | 0.337     | 0.926  | 0.229        |\n| **VATT-Qformer - LLama**   | **0.907**        | **0.419** | **0.375** | **1.264** | **0.245** |\n| VATT Converter - LLama     | 0.909            | 0.424    | 0.384     | 1.354  | 0.263        |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjzbsXk2Hw", "reviewer": "Reviewer_BS9v", "summary": "This paper proposes a multimodal video-to-audio generative model called **VATT** that can generate audio and audio captions given input silent videos and optional text prompts. It is capable of performing controllable text-guide video-to-audio generation and video-to-audio captioning. The framework consists of two parts: \ni) **VATT converter**, an instruction fine-tuned LLM that projects video features into language space. To train such a model, the authors caption the existing audio-visual dataset by LTU-13B. \nii) **VATT audio**, a bi-directional transformer audio decoder based on **Maskgit** to generate audio tokens and text tokens with given inputs by iterative parallel decoding and Encodec as audio tokenizer. \n\nThrough the quantitative and qualitative experiments on the VGGSound dataset, they demonstrate that the proposed method achieves state-of-the-art performance in the aspect of audio quality and inference speed.", "questions": "The reviewer has questions about the paper:\n- The reviewer wonders why Maskgit as the decoder framework. From the reviewer's side, the inference speed seems to be the main advantage. \n- For the visual encoder eva-clip, it encode the feature frame-by-frame. The reviewer thinks it might lose temporal information, do authors explore this with a different encoder?\n- The reported inference speed of Diff-Foley is much slower than the number reported in their paper, any clue why it is happening", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The motivation is clear. All details are clearly stated in the paper and appendix.\n- The paper has several technical highlights, especially on the V2A Instruction Tuning part which enables text control on the video-to-audio generation. \n- The authors perform the throughout experiments and ablations to demonstrate its technical advance. The proposed method outperforms baselines on the quantitative evaluation and human study.", "weaknesses": "- The reviewer thinks the major contribution lies in the V2A Instruction Tuning part which enables the text control on the video-to-audio generation. While the contribution of maskgit-based audio decoder seems to be incremental. \n- From the reviewer's perspective, using the VATT converter to encode visual information will lose some temporal information from the video. From the provided qualitative examples, the reviewer thinks the proposed method is not performing quite well on the generated synchronized audio but only roughly correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and valuable feedback. We are glad to see that the reviewer acknowledges the presentation of our work, the technical highlights and thorough experiments. We address the feedback below. \n\n**W1.** Indeed, one major contribution of our work is the V2A tuning part which allows both video-to-audio captioning as well as video-to-audio generation via text control. For the audio decoder part, our methods are inspired from previous works like MaskGIT and Soundstorm. However, we still want to emphasize our key differences from these works. MaskGIT explores only the task of class-label conditioned image generation, and it is not designed to work on cross-modal generation like video-to-audio generation. For Soundstrom, it enables text-conditioned audio generation. However, its masking designs, in particular the masking ratio distribution, are different from ours. We conducted an ablation study (see Table 5) to study how the masking distribution design affects the performance of generation quality.  The experiment could give some insights on how to design the masking ratio, which was not explored in previous works.\n\n**W2 & Q2:** We agree with the reviewer that frame-by-frame eva-CLIP features will lose fine-grained temporal details. At the current stage, we do not find any more powerful video encoder techniques to efficiently extract these fine-grained spatio-temporal features for better synchronizations. We will add this as part of the limitations in our revised manuscript, and motivate for the future research in this direction.\n\n**Q1:** We agree with the reviewer that one of the key motivations to use masking-based decoder is that it enables parallel decoding, which makes the inference speed a lot faster. The other reason we aim to use masking-based decoder is that we aim to unify different modalities using token representation, such that this kind of framework could be easily worked with multi-modal generation purposes within one model. In such cases, the choices are masking-based decoder and autoregressive decoder. For efficiency purposes again, we design our model's framework with masking-based decoder.  \n\n**Q3:** For all inference speed evaluation, we perform generation with batch size = 1, while we found that Diff-Foley performs inference in a batch size of 64 and the inference time is calculated by per batch inference time divided by 64, which is the reason that their reported inference speed is significantly faster. For all other parts, we kept the same setting as theirs and used their Github repo’s notebook to perform inference. We will clarify this in our revised manuscript.\n\n\n**Flag for Ethics Review:** We obtained Institutional Review Board Approval (IRB Approval) from our institution on conducting human subject evaluations for this research project before the submission deadline. Per further request, we could supply the evidence in an anonymous way to the reviewing board committee.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the task of video-to-audio generation task. The authors propose to additionally include text conditioning to improve and stabilize the generation process. The authors achieve this joint audio generation and captioning framework. 
The authors addressed the reviewers comments during the rebuttal phase, while providing additional results and clarifications. I highly encourage the authors to include these in the final manuscript. 
There are several missing references such as: 
[1] Mo, Shentong, Jing Shi, and Yapeng Tian. "DiffAVA: Personalized text-to-audio generation with visual alignment." arXiv preprint arXiv:2305.12903 (2023).
[2] Ziv, Alon, et al. "Masked audio generation using a single non-autoregressive transformer." arXiv preprint arXiv:2401.04577 (2024).
Decision: Accept (poster)