Taming Data and Transformers for Audio Generation

## Abstract

Abstract {NoHyper} * * footnotetext: Work partially done during an internship at Snap Inc. Generating ambient sounds is a challenging task due to data scarcity and often insufficient caption quality, making it difficult to employ large-scale generative models for the task. In this work, we tackle this problem by introducing two new models. First, we propose AutoCap , a high-quality and efficient automatic audio captioning model. By using a compact audio representation and leveraging audio metadata, AutoCap substantially enhances caption quality, reaching a CIDEr score of 83.2 83.2 83.2 83.2 , marking a 3.2 % percent 3.2 3.2\% 3.2 % improvement from the best available captioning model at four times faster inference speed. Second, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. Using AutoCap to generate caption clips from existing audio datasets, we demonstrate the benefits of data scaling with synthetic captions as well as model size scaling. When compared to state-of-the-art audio generators trained at similar size and data scale, GenAu obtains significant improvements of 4.7 % percent 4.7 4.7\% 4.7 % in FAD score, 22.7 % percent 22.7 22.7\% 22.7 % in IS, and 13.5 % percent 13.5 13.5\% 13.5 % in CLAP score, indicating significantly improved quality of generated audio compared to previous works. Moreover, we propose an efficient and scalable pipeline for collecting audio datasets, enabling us to compile 57M ambient audio clips, forming AutoReCap -XL, the largest available audio-text dataset, at 90 times the scale of existing ones. Our code, model checkpoints, and dataset are publicly available.

## 1 Introduction

Text-conditioned generative models have revolutionized the field of content creation, enabling the generation of high-quality natural images (Ramesh et al., ; Rombach et al., ; Podell et al., ; Haji-Ali et al., ), vivid videos  Ho et al. (); Villegas et al. (); Wang et al. (); Qiu et al. (); Menapace et al. (),
and intricate 3D shapes (Cheng et al., ).
The domain of audio synthesis has undergone comparable advancement (Huang et al., ; ; Liu et al., ; Xue et al., ; Guan et al., ; Saito et al., ; Niu et al., ; Yang et al., ; Evans et al., ; Liu et al., ; Wang et al., ; Guo et al., ), with three broad areas of study: speech, music and ambient sounds.
The success in these domains rests on two key pillars: (i) the availability of high-quality large-scale datasets with text annotations, and (ii) the development of scalable generative models (Ho et al., ; Song et al., ).

In the field of audio synthesis, ambient audio generation emerges as a critical domain, which is the main focus of this work.
Unlike speech and music, ambient sound generation suffers from a lack of extensive, well-annotated datasets (Kim et al., ; Drossos et al., ).
Attempts to curate ambient audio from online videos
predominantly failed, primarily due to the dominance of speech and music content in such videos. For instance, AudioSet (Gemmeke et al., ), the largest available audio dataset sourced from online videos, contains $99\%$ speech or music clips. Previous efforts to filter ambient audio from similar datasets involved using expensive classifiers on the video or audio content, making it impractical to compile a large-scale dataset due to the high rejection rate. In this work, we propose a simple, yet scalable filtering approach that leverages existing automatic video transcription to identify segments with ambient sounds. This method is not only more efficient but also more feasible, as it eliminates the need to download the audio or video content. Additionally, by using time-aligned transcripts, we reduce the rejection rate to only $83\%$. Through this approach, we built AutoReCap-XL, a dataset containing 57 million ambient audio clips sourced from existing video datasets, representing a 90-fold increase over the size of previously available datasets.

Another challenge in compiling large-scale text-audio datasets is providing accurate textual descriptions. For visual modalities, such as images and videos (Xue et al., ; Miech et al., ), researchers often relied on the raw description and metadata to train strong visual-text models including reliable captioners (Chen et al., ). Similarly, speech modality benefits from strong automatic transcription models to provide textual annotations. For ambient sounds, however, the task is substantially more challenging as accompanying raw text tends to describe visual information or convey feelings, rather than detailing the audio content. Moreover, human-captioned audio datasets are limited, containing fewer than $51k$ text-audio pairs in total. This significantly impacts the training of current captioning models, making them more susceptible to overfitting and reducing their ability to generalize effectively. In this work, we address this challenge by introducing AutoCap, an efficient and high-quality audio captioning model that leverages visual information to enhance captioning.

AutoCap refines the commonly used encoder-decoder design based on a pretrained BART (Lewis et al., ) model by introducing a Q-Former (Li et al., ) that learns to summarize the encoded audio tokens into four times fewer tokens. By reducing the number of input tokens to the BART model, we speed up inference—an important step towards large-scale audio captioning—and provide better alignment with the original BART token representation due to the Q-Former additional capacity compared to simple projection layers used in previous work (Kim et al., ). Second, we propose to use metadata and captions derived from video content to aid the captioning process and in this way, remedy the data scarcity problem. Critically, we augment the encoder inputs to assume both audio features and a set of descriptive textual metadata including audio title and a caption derived from the visual modality. This dual-input approach not only allows our model to achieve state-of-the-art performance on AudioCaps (Kim et al., ), marking a 3.2% improvement in CIDEr score, but it also helps reduce the domain gap with in-the-wild audios.

Moreover, to adapt audio generative models for larger scale training, we introduce GenAu, a scalable transformer-based architecture that achieves *significant* improvements over state-of-the-art audio generation models. Our approach introduces key architectural modifications over existing audio latent diffusion models (Liu et al., ; Huang et al., ; Ghosal et al., ; Huang et al., ). First, we train an efficient 1D-VAE (Huang et al., ) to transform a Mel-Spectrogram representation to a sequence of tokens and search for the optimal latent space for audio generation. Second, we recognize that audio grows fast temporally, yet contains many silent and redundant segments. Therefore, an efficient architecture that can handle such properties is needed. In particular, we employ a transformer architecture in the denoising backbone where we modify the FIT transformer (Chen & Li, ) to generate audio in the latent space. Lastly, we extend the proposed FIT architecture to incorporate text conditioning through a dual encoder strategy. This involves an instruction-finetuned language model, FLAN-T5 (Chung et al., ), and an audio-centric CLAP encoding (Kim et al., ). This adaptations significantly improves the model’s performance over exiting methods, achieving *$22.7\%$* higher Inception Score, *$4.7\%$* better FAD, and *$13.5\%$* improvement in CLAP score, demonstrating superior audio-text alignment and audio generation quality.

Finally, we explore the scaling behavior of text-to-audio diffusion models in relation to model size and data size. Prior studies in text-to-image generation have established a scaling trend where performance improves with the increases in data volume and model size (Peebles & Xie, ; Li et al., ). Such exploration, however, has not been sufficiently conducted for the audio modality. Initially, we analyze the impact of augmenting the dataset with synthetic captions on model performance. Our findings reveal a clear trend of improvement in FD and IS as we increase the amount of training data. Furthermore, we observe a consistent trend of enhanced performance across all metrics when scaling up the model size, concluding that the audio modality also benefits significantly from increases in both model size and data scale.

In summary, this work introduces: (i) AutoCap, a state-of-the-art audio captioner tailored towards the annotation of data at a large scale, which leverages audio metadata to improve accuracy and robustness, and a Q-Former to improve inference time and reduce overfitting; (ii) GenAu, a novel audio generator based on a scalable transformer architecture specifically adapted to the audio domain. Our model achieves significantly improved quality when compared to the previous state-of-the-art. (iii) AutoReCap-XL, the largest available audio dataset, comprising 57M audio clips paired with synthetic captions derived from the proposed audio captioner.

## 2 Related Work

Automatic Audio Captioning (AAC). The goal of AAC is to produce natural language descriptions for given audio content.
Most recent AAC methods (Deshmukh et al., ; Wu et al., ; Salewski et al., ; Sridhar et al., ; Kadlčík et al., ; Cousin et al., ; Labbé et al., ; Xu et al., ; Zhang et al., ) employ encoder-decoder transformer architectures, where an encoder receiving the audio signal produces a representation that is used by the decoder to produce the output caption. WavCaps (Mei et al., ) employs the CNN14 (Kong et al., ) and HTSAT (Chen et al., ) audio encoders and uses a pretrained BART (Lewis et al., ) language decoder. CoNeTTE (Étienne Labbé et al., ) proposes an audio encoder based on the ConvNeXt architecture and uses a vanilla transformer decoder (Vaswani et al., ) trained from scratch. Recently, EnCLAP (Kim et al., ) proposes the joint use of two audio representations in the form of CLAP (Elizalde et al., ) sequence embeddings and a discrete EnCodec (Défossez et al., ) audio representation and uses a pretrained BART model as the language backbone. Other work explores augmentation strategies to counter data scarcity (Kim et al., ; Étienne Labbé et al., ; Ye et al., ). Liu et al. () recently proposed to leverage the visual information using a pre-trained visual encoder to address sound ambiguities, reporting improvements. BART-Tags (Gontier et al., ) generates captions conditioned on a sequence of predicted AudioSet tags. Our method uses audio metadata and visual information as additional conditioning signals and leverages a lightweight Q-Former (Li et al., ) model that summarizes the audio feature to improve captioning speed and reduce overfitting.

Text-conditioned audio generation. The current state-of-the-art text-to-audio generation methods widely adopt diffusion models (Yang et al., ; Kreuk et al., ; Liu et al., ; ; Huang et al., ; Ghosal et al., ; Evans et al., ; Vyas et al., ; Kreuk et al., ).
AudioLDM (Liu et al., ) makes use of a latent diffusion model conditioned on CLAP embeddings, reducing the need for the textual modality at training time. AudioLDM 2 (Liu et al., ) introduces a general representation of audio unifying the tasks of music, speech, and sound effects generation. Similarly, Audiobox (Vyas et al., ) generates audio across different modalities such as speech and sound effects. StableAudio (Evans et al., ) introduces timing embeddings to allow the generation of long audios up to 95s. Recent work also explored controllable audio generation (Shi et al., ; Xu et al., ; Melechovsky et al., ; Paissan et al., ; Zhang et al., ; Liang et al., ; Liu et al., ), visual-conditioned audio generation (Wang et al., ; Mei et al., ; Wang et al., ), and more recently joint audio-video generation (Tang et al., ; ; Xing et al., ; Hayakawa et al., ; Tian et al., ; Vahdati et al., ; Chen et al., ; Kim et al., ; Wang et al., ; Mao et al., ; Chen et al., ). In this work, we show that improvements to data captioning quality and size, and the adoption of scalable architecture designs lead to state-of-the-art text-to-audio generation performance.

Text-Audio Datasets.
The performance of text-audio models (Zhu et al., ; Li et al., ; Deshmukh et al., ; Mahfuz et al., ; Deshmukh et al., ; Shu et al., ; Elizalde et al., ; Liu et al., ; Tang et al., ; Gong et al., ; Cheng et al., ; Zhang et al., ), including AAC, is currently hindered by the lack of high-quality large-scale paired audio text data of ambient sounds. The two main existing datasets are AudioCaps (Kim et al., ) and Clotho (Drossos et al., ), comprising only $46k$ and $6k$ respectively of human-captioned audio clips. Another challenge is the limited availability of audio clips from sound-only platforms. LAION-Audio (Wu et al., ) relied on numerous sources of audio platforms such as BBC Sound Effects (BBC Sound Effects, ),  (Font et al., ) FreeSounds, and SoundBible (SoundBible, ) to form a dataset consisting of 630k audio samples with highly noisy raw descriptions. WavCaps (Mei et al., ) proposes a filtering procedure based on ChatGPT (Achiam et al., ) to collect $400k$ audio clips and weakly caption them based on the noisy descriptions alone. Several subsequent work (Majumder et al., ; Sun et al., ) adopted similar strategies of using large language models to augment captions. While weak-captioning does improve downstream metrics, it is suboptimal because it fails to incorporate the audio signal itself. A recent work (Huang et al., ) explored a knowledge distillation approach that leverages data labels and a pre-trained audio captioner and retriever to improve caption quality. Chen et al. () attempted to extract audio clips from videos by employing classifiers to detect ambient audio, speech, and music. In this work, we introduce an efficient dataset collection pipeline that relies on video datasets to extract ambient audio clips. We use this approach to collect $57M$ audio clips and use our state-of-the-art captioning method to add audio-aligned text descriptions, compromising the largest available text-audio-video dataset.

Figure: Figure 1: (Left) Overview of AutoCap. We employ frozen CLAP (Kim et al., ) and HTSAT (Chen et al., ) encoders to produce the audio representation. We then compact this representation into 4x fewer tokens using a Q-Former (Li et al., ) module. This representation, along with tokens derived from pertinent metadata, is processed by a pretrained BART encoder-decoder model to generate the final caption. (Right) Overview of GenAu. Following latent diffusion models, we use a frozen 1D-VAE to convert a Mel-Spectrogram into latent sequences, which are then divided into groups and processed using ‘local’ attention layers based on the FIT architecture (Chen & Li, ). ‘Read’ and ‘write’ layers, implemented as cross-attention, facilitate information transfer between input latents and *learnable* latent tokens. Finally, ‘global’ attention layers on *latent tokens* allow for global communication across all groups.
Refer to caption: x1.png

## 3 Method

In this section, we describe our approach to high-quality text-to-audio generation, starting with audio captioning using AutoCap in section 3.1, data collection and processing in section 3.2, and ambient audio generation with GenAu in section 3.3

### 3.1 Automatic Audio Captioning

Audio is an inherently ambiguous modality, as many events can produce similar sound effects—a phenomenon often leveraged in animation, where soundscapes are artificially constructed. AAC attempts to generate textual descriptions for audio clips. Previous AAC methods have generally adopted an encoder-decoder transformer design, where an audio encoder
is responsible for producing a representation that is processed by the decoder to produce a caption. Recent state-of-the-art methods (Étienne Labbé et al., ; Kim et al., ) employ a pretrained audio encoder and finetune a pre-trained language model as the decoder, relying solely on the audio content for captioning. We believe that this approach is suboptimal. By directly finetuning the pre-trained language model on the limited available dataset, these methods often suffer from overfitting and limited expressiveness and accuracy.
Audio files from many sources, however, are still commonly associated with metadata that might be relevant for captioning such as raw user descriptions, or a related modality (i.e. accompanied visual information). Motivated by this observation, we propose AutoCap, an audio captioning model that employs an intermediate audio representation to connect the pretrained encoder and decoder, and uses metadata to aid with the audio captioning. Figure 1 (left) presents an overview of AutoCap.

We consider a dataset of audio signals paired with a corresponding caption $\langle\mathbf{a},\mathbf{y}\rangle$ and metadata represented as a set of token sequences $\{\mathbf{m}_{j}\}_{j=1}^{j=M}$.
Inspired by state-of-the-art AAC methods (Mei et al., ; Étienne Labbé et al., ; Kim et al., ), we employ an encoder-decoder architecture. We start by computing a global feature representation of the audio:

$$ $\mathbf{x}_{\mathrm{clap}}=\mathcal{P}_{\mathrm{clap}}(\mathcal{E}_{\mathrm{ clap}}(\mathbf{a})),$ (1) $$

where $\mathcal{P}_{\mathrm{clap}}$ is a learnable projection layer and $\mathcal{E}_{\mathrm{clap}}$ is the audio encoder of a pretrained CLAP model (Elizalde et al., ). Then we compute a local feature representation of the input audio:

$$ $\mathbf{x}_{\mathrm{audio}}=\mathcal{Q}(\mathcal{E}_{\mathrm{a}}(\mathbf{a})),$ (2) $$

where $\mathcal{Q}$ is a Q-Former (Li et al., ) that outputs a compact sequence audio representation and $\mathcal{E}_{\mathrm{a}}$ is a pretrained HTSAT (Chen et al., ) audio encoder that produces a time-aligned representation. The Q-Former efficiently learns 256 latent tokens, which serve as keys in cross-attention layers with the input features, thereby condensing the audio input features into 256 tokens. Metadata sequences $\mathbf{m}_{i}$ are then embedded using the embedding layer of the pretrained decoder model to obtain corresponding embedding sequences $\mathbf{x}_{\mathrm{meta}_{i}}$. For our experiments, we use video titles and captions as the metadata. We represent the input audio and metadata as the following input sequence:

$$ $\mathbf{x}=\mathbf{x}_{\mathrm{clap}}~{}\texttt{[boa]}~{}\mathbf{x}_{\mathrm{ audio}}~{}\texttt{[eoa]}~{}\texttt{[bom]}_{1}~{}\mathbf{x}_{\mathrm{meta}_{1}} ~{}\texttt{[bom]}_{1}~{}...~{}\texttt{[bom]}_{M}~{}\mathbf{x}_{\mathrm{meta}_{ M}}~{}\texttt{[bom]}_{M}\text{,}$ (3) $$

where [boa][eoa] represent beginning and end of audio sequence embeddings $\mathbf{x}_{\mathrm{audio}}$, and $\texttt{[bom]}_{i},\texttt{[bom]}_{i}$ represent beginning and end of metadata embeddings $\mathbf{x}_{\mathrm{meta}_{i}}$. This input sequence is then used to obtain an output predicted caption $\mathbf{\hat{y}}$ as follows:

$$ $\mathbf{\hat{y}}=\mathcal{D}_{\mathrm{t}}(\mathbf{x})\text{,}$ (4) $$

where $\mathcal{D}_{\mathrm{t}}$ is a pretrained BART transformer model (Lewis et al., ) serving as the decoder.
Finally, we train our model using a standard cross-entropy loss over next token predictions:

$$ $\mathcal{L_{\mathrm{CE}}}=-\frac{1}{T}\sum_{t=1}^{t=T}\mathrm{log}~{}p(\mathbf {y}_{t}|\mathbf{y}_{1:t-1},\mathbf{x})\text{.}$ (5) $$

To avoid degrading the quality of the pretrained BART and audio encoder models, we adopt a two-stage training procedure. In Stage 1, both the audio encoders and BART model are kept frozen, thus allowing the Q-Former, projection layers, and newly introduced delimiter tokens to align to the existing BART input representation. In this stage, we pretrain the model using a larger dataset of weakly-labeled audio clips. In Stage 2, we unfreeze all BART model parameters apart from the embedding layer and finetune the model on the Audiocaps dataset at a lower learning rate to make the captioning style align more closely to the target dataset. This training strategy effectively leverages the larger, weakly-labeled dataset while minimizing the knowledge drift in the pretrained BART. The use of Qformer to learn an intermediate representation is pivotal for such training strategy. Furthermore, the Qformer summarizes the audio representation into four times fewer tokens, significantly reducing the inference time.

Figure: Figure 2: Audio data collection pipeline. We employ online video transcripts to identify audio segments without speech or music. These are processed by AutoCap to generate captions. We retain only ambient clips with captions lacking music and speech keywords.
Refer to caption: x2.png

### 3.2 Data Collection and Re-captioning Pipeline

Generative models in the image and video domains have shown benefits from increased quantities of data and improved quality of captions. In the audio domain, however, the major human-annotated audio-text datasets, namely AudioCaps (Kim et al., ) and Clotho (Drossos et al., ), provide only 51k audio clips combined. Previous methods attempted to extract additional ambient audio clips from existing video datasets using pretrained audio classifiers, but a high rejection rate of $99\%$ marked this method impractical. Instead, we found that automatic transcripts offer reliable information about the segments containing ambient sounds. In particular, we propose to select the parts of the videos that contain no automatic transcription, suggesting the absence of speech and music. Such an approach offers specific advantages over using pretrained classifiers. Automatic transcripts, readily available for most online videos, eliminate the need to download and process video and audio data before filtering. Additionally, as these transcripts provide precise time-aligned information, they facilitate the extraction of more segments, effectively reducing the rejection rate to $83\%$. Subsequently, we leverage our AutoCap model to provide textual descriptions of the extracted audio clips. Despite the effectiveness of this method in collecting ambient sounds, some clips still inadvertently contain music or speech due to transcription errors, particularly with speech in less common languages. We address this by analyzing captions and filtering out clips with keywords related to speech or music. Figure 2 summarizes our data collection pipeline.

We follow this process to extract 466k audio-text pairs from Audioset (Gemmeke et al., ) and VGGSounds (Chen et al., ). Additionally, we recaption audio-only dataset such as Freesound, BBC Sound Effects, and SoundBible. To provide metadata, we employ the captioning model of Chen et al. () to extract a caption whenever a video content is available and pass an empty text otherwise. In total, we form AutoReCap, a large-scale dataset compromising of *761,113* audio-text pairs with precise captions. As an additional contribution, we introduce AutoReCap-XL, in which we scale our approach by analysing four additional large-scale video dataset (Lee et al., ; Xue et al., ; Zellers et al., ; Nagrani et al., ) with a total of *71M* videos and *$715.4k$* hours. In total, we collect and re-caption *57M* ambient audio clips spanning *$123.5k$* hours from *20.3M* different videos, forming by far the largest available dataset of audio with paired captions. More details about the dataset can be found in the *Appendix*.

Figure: Figure 3: Scaling analysis of model size (left) and data volume with synthetic captions (right) reveal consistent improvements in FD and IS.
Refer to caption: x3.png

### 3.3 Scalable Text-2-Audio Generation

We design our audio generation pipeline, GenAu, as a latent diffusion model. Figure 1 (right) shows an overview of our proposed model. In the following section, we describe in detail the structure of our latent variational autoencoder (VAE) and the latent diffusion model.

Latent VAE.
Directly modeling waveform audio data is complex due to the high data dimensionality of audio signals. Instead, we replace the waveform with a Mel-spectrogram representation and use a VAE to further reduce its dimensionality, following prior work (Melechovsky et al., ; Huang et al., ). Once generated, Mel-spectrograms can be decoded back to a waveform through the use of an audio vocoder (Kong et al., ). However, commonly-used 2D autoencoder designs (Liu et al., ; ; Melechovsky et al., ), are not well suited to the Mel-spectrogram representation, as the separation between the Mel channels is non-linear, which is not well suited for 2D convolutions. We instead opt for a 1D-VAE design based on 1D convolutions similar to Huang et al. (). We train our VAE using a combination of reconstruction, adversarial, and KL regularization losses following Esser et al. ().

Latent diffusion model.
Following the latent diffusion paradigm, we generate audio by training a diffusion model in the latent space of the 1D-VAE. Transformer-based diffusion models currently attain state-of-the-art performance in audio generation (Huang et al., ). To improve model scalability, we propose to use an efficient transformer architecture due to its success in handling long-range interactions as in video generation (Chen & Li, ; Menapace et al., ). In particular, we adopt the FIT architecture of Menapace et al. () which was originally proposed to work in the *pixel space* and revise it for the *latent space* of the audio modality.

Given a 1D input $\mathbf{x}$, we first apply a projection operation to produce a sequence of input patch tokens. We then apply a sequence of FIT blocks to the input patches where each block divides patch tokens into contiguous groups of a predefined size. A set of *local* self-attention layers are then applied separately to each group to avoid the quadratic computational complexity of attention computation. Differently from the video domain (Menapace et al., ) where the high input dimensionality makes the *local* layers excessively expensive, we found them to be beneficial for audio generation. To further reduce the amount of computation while maintaining long-range interaction, each block considers a small set of latent tokens. First, a *read* operation implemented as a cross-attention layer transfers information from the patches to the latent tokens. Later, a series of *global* self-attention operations are applied to the latent tokens, allowing information-sharing between different groups. Finally, a *write* operation implemented as a cross-attention layer transfers information from the latent tokens back to the patches. Due to the reduced number of latent tokens when performing the global self-attention, computational requirements of the model are reduced with respect to a vanilla transformer design (Vaswani et al., ). Such a design is also particularly suited for the audio modality, which contains mostly silent or redundant parts. Unlike DiT and UNet-based methods (Ronneberger et al., ; Peebles & Xie, ) which allocate the computation resources uniformly across input tokens, the FiT architecture selectively focuses on the more informative parts.

To condition the generation on an input prompt, we use a pretrained FLAN-T5 model (Chung et al., ) and a CLAP (Elizalde et al., ) text encoder to produce the their respective embeddings $e_{\mathrm{FLAN}}$ and $e_{\mathrm{CLAP}}$, which we concatenate with the diffusion timestep $t$ to form the input conditioning signal $c$. We insert an additional cross-attention operation inside each FIT block immediately before the ‘read’ operation that makes latent tokens attend to the conditioning. Moreover, we use conditioning on dataset ID to adapt the generation style to different types of datasets.

We follow a linear noise scheduler and train the model using the epsilon prediction objective:

$$ $\mathcal{L}=\mathbb{E}_{t,\mathbf{x},\bm{\epsilon}}~{}\big{\lVert}\mathcal{G}( \mathbf{x}_{t},c)-\bm{\epsilon}\big{\rVert}^{2}_{2}\text{,}$ (6) $$

where $\mathcal{G}$ is the FIT generator backbone, $\mathbf{x}_{t}$ is the input with applied noise at diffusion timestep $t$, and $\bm{\epsilon}$ is noise sampled in $N(0,1)$ with the same shape as the input.

**Table 1: AutoCap results on AudioCaps test split for various models. AS: AudioSet, AC: AudioCaps, WC: WavCaps, CL: Clotho, MA: Multi-Annotator Captioned Soundscapes.**
| Model | Pretraining Data |  | BLEU1 | BLEU4 | ROUGEL | METEOR | CIDEr | SPICE | SPIDEr |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACT | AS |  | 64.7 | 25.2 | 46.8 | 22.2 | 67.9 | 16.0 | 42.0 |
| V-ACT | - |  | 69.8 | 28.1 | 49.4 | 23.7 | 71.1 | 17.2 | 44.2 |
| BART-tags | AS |  | 69.9 | 26.6 | 49.3 | 24.1 | 75.3 | 17.6 | 46.5 |
| AL-MixGEN | - |  | 70.0 | 28.9 | 50.2 | 24.2 | 76.9 | 18.1 | 47.5 |
| ENCLAP-Large | - |  | - | - | - | 25.5 | 80.2 | 18.8 | 49.5 |
| HTSAT-BART | - |  | 67.5 | 27.2 | 48.3 | 23.7 | 72.1 | 16.9 | 44.5 |
| HTSAT-BART | AC+CL+WC |  | 70.7 | 28.3 | 50.7 | 25.0 | 78.7 | 18.2 | 48.5 |
| CNext-trans | - |  | - | - | - | - | - | - | 46.6 |
| CNext-trans | AC+CL+MA+WC |  | - | - | - | 25.2 | 80.6 | 18.4 | 49.5 |
| AutoCap (audio) | AC |  | 70.0 | 28.0 | 51.7 | 24.6 | 77.3 | 18.2 | 47.8 |
| AutoCap (audio+text) | AC |  | 72.1 | 28.6 | 51.5 | 25.6 | 80.0 | 18.8 | 49.4 |
| AutoCap (audio) | AC+CL+WC |  | 73.1 | 28.1 | 52.0 | 25.6 | 80.4 | 19.0 | 49.7 |
| AutoCap (audio+text) | AC+CL+WC |  | 72.3 | 29.7 | 51.8 | 25.3 | 83.2 | 18.2 | 50.7 |

## 4 Experiments

We structure the experiments section as follows: section 4.1 evaluates AutoCap by quantitatively comparing it to previous work, section 4.2 demonstrates the capabilities of GenAu quantitatively.
For both, we discuss training details, baselines, metrics, results, and ablations.

### 4.1 Automatic Audio Captioning

Training dataset and details. We train our captioning model in two stages. During stage 1, we pretrain on a large weakly labeled dataset of 634,208 audio clips, constructed from AudioSet, Freesound, BBC Sound Effects, SoundBible, AudioCaps, and Clotho datasets. We use the ground truth captions from AudioCaps and Clotho dataset, WavCaps captions for Freesound, SoundBible, and BBC Sound Effects, and handcrafted captions through a template leveraging the provided ground truth class labels for AudioSet. As metadata, we use the title provided with each clip, and pre-compute video captions using a pretrained Panda70M model (Chen et al., ) when the video modality is available or pass an empty string otherwise. We pretrain the model for 20 epochs with a learning rate of 1e-4, while keeping the audio encoder and pretrained BART frozen. In Stage 2, we fine-tune the model for 20 epochs only on AudioCaps using a learning rate of 1e-5. We randomly sample 10-second clips at 32KHz for all of our captioning experiments.

Baselines. We compare with ACT (Mei et al., ), V-ACT (Liu et al., ), BART-tags (Gontier et al., ), AL-MixGEN (Kim et al., ), ENCLAP (Kim et al., ), HTSAT-BART (Xu et al., ) and CNext-trans (Étienne Labbé et al., ). Among these baselines, ENCLAP and CNext-trans achieve the best performance. ENCLAP benefits from a stronger audio encoder and the use of a CLAP representation for additional guidance. CNext-trans trains a lightweight transformer instead of fine-tuning a pretrained language model to reduce overfitting.

Metrics and evaluation. We report results using the the established BLEU1 (Papineni et al., ), BLEU2 (Papineni et al., ), ROUGE (Lin, ), Meteor (Lavie & Agarwal, ), CIDEr (Vedantam et al., ), and SPIDEr (Liu et al., ) metrics. We evaluate our method on the AudioCaps test split using the last checkpoint of our trained model. We used only 876 clips for evaluation as some videos were deleted since the original data release. We follow the same evaluation pipeline as baselines and include their reported results. Results that were not provided in these publications are excluded from our analysis.

Results. In Tab. 1 we report the quantitative comparison. Our method outperforms previous methods on all metrics, achieving notable improvements in the CIDEr and BLUE1 scores, with values of 83.2 and 73.1, respectively. We found that incorporating metadata significantly enhances the CIDEr scores but slightly reduces the SPICE scores. This trade-off likely results from the enhanced descriptive detail brought by the metadata, which while enriching the content, introduces noise that may compromise the model’s semantic precision. In addition, AudioCaps is labeled based on audio information alone. Thus, the evaluation penalizes the description of information that can not be deduced with certainty from the audio modality only, such as the specific type of object producing a rustling sound. Compared to ENCLAP-Large (Kim et al., ), and CNext-trans (Étienne Labbé et al., ), we find the captions produced by our method to be more descriptive and precise with a better temporal understanding. ENCLAP-Large often misses important details and exhibits lower temporal accuracy. CNext-trans, while accurate, often produces short captions that lack details. We include qualitative comparisons in the project *Website*. Moreover, AutoCap is *four times* faster than ENCALP, producing a caption for a 10-second clip in *0.28* seconds, compared to ENCALP which takes *1.12* seconds. Furthermore, we observe consistent improvements when pretraining on a large scale of weakly-labeled data during the first stage, validating the effectiveness of our training strategy in benefiting from a larger, weakly-labeled dataset.

**Table 2: AutoCap ablation study on AudioCaps.**
| Model | METEOR $\uparrow$ | CIDEr $\uparrow$ | SPICE $\uparrow$ | SPIDEr $\uparrow$ |
| --- | --- | --- | --- | --- |
| Ours | 25.3 | 83.2 | 18.2 | 50.7 |
| - w/o CLAP | 25.3 | 80.7 | 18.4 | 49.6 |
| - w/o Stage 2 | 24.2 | 75.6 | 17.3 | 46.5 |
| - w/o Stage 1 | 22.6 | 59.6 | 15.4 | 37.5 |
| - Unfreeze Word Embed | 22.5 | 82.6 | 18.1 | 50.4 |

Ablations.
In Tab. 2, we ablate model design choices. We observe the use of the CLAP embedding to bring a $2.5$ points increase in the CIDEr score. We also validate that when not performing Stage 2 training, which involves finetuning of the BART (Lewis et al., ) model, performance degrades on all metrics, a finding we attribute to the necessity of adapting BART’s decoder to the sentence structure typical of AudioCaps. A more severe degradation in performance is observed if Stage 1 is not performed, with the misaligned representation between the encoder and the decoder causing catastrophic forgetting in the language model. Finally, if BART word embeddings are finetuned in Stage 2 instead of being kept frozen, we observe a slight performance degradation.

### 4.2 Text-2-Audio Generation

Training dataset and details. We train on similar data settings to baselines. We use our best-performing captioning model to re-caption the WavCaps dataset. In addition, we obtain 339,387 videos from AudioSet and 126,905 videos from VGGSounds, totaling 761,113 clips. For those obtained from sound-only platforms, we input an empty string as the video caption. For full details of the data sources of our training dataset, please refer to the *Appendix*. We additionally use Clotho and AudioCaps training datasets with their ground truth caption. To stay consistent with baselines, we train at 16kHz resolution. We use a patch size of 1 and a group size of 32. We use LAMB optimizer (You et al., ) with a LR of 5e-3. We train for 220k steps and choose the checkpoint with the highest IS, at steps 210k and 207k for the large and small models. We also disable EMA as found it to make the metrics unstable.
Baselines. We compare with TANGO 1 & 2, (Ghosal et al., ), AudioLDM 1 & 2 (Liu et al., ; ), and Make-An-Audio 1 & 2 (Huang et al., ; ). Both AudioLDM and Make-an-Audio train a UNet-based latent diffusion model (Rombach et al., ) on Mel-Spectrogram representation of the audio, by regarding the Mel-Spectrogram as a single channel image, and use a pretrained CLAP encoder to condition the generation on an input prompt. TANGO proposed to use FLAN-T5 (Chung et al., ) as the text encoder and reported significant improvements. AudioLDM-2 and Make-an-Audio-2 proposed to use a dual encoder strategy of a T5 (Raffel et al., ) and CLAP encoder. AudioLDM-2 focused on extending the generation and conditioning to various domains. Specifically, they use the language of audio (LOA) to condition the generation on images, audio, or transcripts and train their model for music and speech generation. Make-an-Audio-2 proposes to use a 1D VAE representation and employ a feed-forward Transformer-based model to replace the UNet.
Recently, Tango-2 proposed to use instruction fine-tuning on a synthetic dataset to enhance the temporal understanding. In our experiments, we focus on text-conditioned natural audio generation and generate 10s clips at a resolution of 16Khz.

**Table 4: GenAu results on AudioCaps test split.**
| Model | Prams | $\#$ Samples | FD $\downarrow$ | IS $\uparrow$ | FAD $\downarrow$ | $\text{CLAP}_{LAION}$ $\uparrow$ | $\text{CLAP}_{MS}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GroundTruth | - | - | - | - | - | 0.251 | 0.671 |
| AudioLDM-L | 739M | 634k | 37.89 | 7.14 | 5.86 | - | 0.429 |
| AudioLDM 2-L | 712M | 760k | 32.50 | 8.54 | 5.11 | 0.212 | 0.621 |
| TANGO | 866M | 45k | 26.13 | 8.23 | 1.87 | 0.185 | 0.597 |
| TANGO 2 | 866M | 60k | 19.77 | 8.45 | 2.74 | 0.264 | 0.590 |
| Make-An-Audio | 453M | 1M | 27.93 | 7.44 | 2.59 | 0.207 | 0.621 |
| Make-An-Audio 2 | 937M | 1M | 15.34 | 9.58 | 1.27 | 0.251 | 0.645 |
| GenAu w/ U-Net | 462M | 811K | 25.57 | 9.54 | 1.98 | - | - |
| GenAu-Large | 1.25B | 811K | 16.51 | 11.75 | 1.21 | 0.285 | 0.668 |

**Table 5: User study between various baselines. % of votes in favor of the baseline to the left.**
| Model | Realism | Quality | Prompt Alignment | Overall Preference |
| --- | --- | --- | --- | --- |
| GenAU-L vs GenAU-S | 61.20% | 58.00% | 61.20% | 60.40% |
| GenAU-L vs GenAU-L (AC) | 60.40% | 54.80% | 60.40% | 59.20% |
| GenAU-L vs MAD-2 | 64.00% | 62.40% | 68.40% | 66.40% |
| GenAU-S w/o Recap. vs MAD-2 | 64.40% | 64.-0% | 63.20% | 64.80% |

Metrics. We compare the performance of our method with baselines using the standard Frechet Distance (FD), Inception score (IS), and CLAP score on the Audioset test dataset, containing 964 samples. There is little consistency between baselines when computing the metrics. Some prior work reported the Fréchet distance results using the VGGish network (Hershey et al., ), denoted as (FAD) (Kilgour et al., ), while other uses PANNs (Kong et al., ). Additionally, to compute the CLAP score, some prior work (Liu et al., ) used CLAP from LAION, which we denote as $\text{CLAP}_{LAION}$ (Wu et al., ), while others (Majumder et al., ; Huang et al., ; ) used CLAP from Microsoft (Elizalde et al., ), which we denote as $\text{CLAP}_{MS}$. Furthermore, some prior (Liu et al., ; ) used CLAP re-ranking with 3 samples for computing the metrics. Due to such inconsistencies in evaluation pipelines and varying results for the same baselines reported in different studies, we recompute all metrics using the official checkpoints to ensure consistent comparisons. We follow the same evaluation protocols of AudioLDM (Liu et al., ) without CLAP re-ranking and use the AudioLDM evaluation package to compute the metrics. Besides, we run our ablations on the Bigsoundbank split from WavText5k (Deshmukh et al., ), which serves as an out-of-distribution evaluation for our models. This is to prevent biasing the evaluation based on the training data. Finally, to further validate our results we run a user preference study. Details about the user study can be found in the *Appendix*.

Results. In Tab. 4, we report evaluation results. Our method achieves superior performance compared to the state-of-the-art methods in terms of IS, FAD, $\text{CLAP}_{MS}$ and $\text{CLAP}_{LAION}$ scores, marking an improvement of $22.7\%$, $4.7\%$, $3.6\%$, and $13.5\%$, respectively. This shows that GenAu can produce high audio quality and achieve better semantic alignment with the conditioning text.

Data scaling. We consider two key aspects: data quality and quantity. First, in Tab. 5 ($2^{nd}$ fow), we show that GenAu-L trained with AutoReCap is generally favoured over training only with AudioCaps (AC). This is confirmed in Tab. 3 ($1^{st}$ vs $2^{nd}$ row), where increasing the dataset size significantly boosts the results across all metrics, improving IS by $56.3\%$. Additionally, we show ($3^{rd}$ vs $4^{th}$ row) that using AutoCap to recaption the dataset significantly enhances the results over all metrics, confirming the importance of data quality. Interestingly, expanding the data size at a lower caption quality does not yield similar gains even at a bigger model ($2^{nd}$ vs $4^{th}$ row), aligning with results reported by Liu et al. (). This highlights that data quality brought by AutoCap is as crucial as the data quantity. Lastly, we examine the effect of scaling the data with synthetic captions. For this, we train for 50k steps by fixing AC and Clotho in the training data and varying the amount of synthetic data. As reported in Fig. 3 (right), increasing the amount of data with synthetic caption has a clear improvement over both IS and FD, with the model trained on the whole AutoReCap achieving the best results.
Model size scaling. In Tab. 5, we report ($1^{st}$ row) that GenAu-L (1.25B params) is constantly favoured over GenAu-S (493M params). This is further confirmed by our automatic evaluation in Tab. 3 ($1^{st}$ vs $3^{rd}$ row), where the larger model shows significant improvements across all metrics. The scaling trend is also evident in Fig. 3, which demonstrates a clear correlation between model size and performance in terms of both IS and FD scores.
Model architecture ablation. Until recently, A UNet (Ronneberger et al., ) has been the most popular choice for the diffusion backbone. Yet, as reported in Tab. 4, replacing the FiT backbone with a UNet drastically reduces performance across all metrics. This supports baseline findings where UNet-based methods lag behind transformer-based approaches (Huang et al., ). Another choice that has recently gained popularity is the DiT architecture (Peebles & Xie, ). Make-an-Audio-2 (MAD-2) employs a DiT at a similar model size and data scale as GenAU-L. However, as we show in Tab. 5, our model is consistently preferred over MAD-2 ($3^{rd}$ row), even without dataset recaptioning ($4^{th}$ row) (i.e. at similar data settings). We infer that the FiT architect, with its read and write operations, allocates compute more efficiently to the key segments of the input, making it more suitable to ambient audio clips which often include silent or redundant parts.

## 5 Conclusion

We take a holistic approach to improve the quality of existing audio generators. Starting by addressing the scarcity of large-scale captioned audio datasets, we build a state-of-the-art audio captioning method, AutoCap, which leverages audio metadata to collect a dataset of 57M annotated audio clips. We then built a latent diffusion model based on a scalable transformer architecture which we trained on our re-captioned dataset to obtain GenAu, a state-of-the-art open-sources model for audio generation. Our approach not only improves ambient audio generation but also opens up possibilities for extending GenAu to other domains, such as speech and music generation. As an additional contribution, we built AutoReCap-XL, a text-audio-video ambient audio dataset with an unprecedented size of $57M$ pairs. AutoReCap-XL can potentially serve as a joint text-audio-video dataset and broadens novel applications such as text-to-audio-video joint generation, a more natural and desired choice of video generation. We leave larger analyses on this dataset for future work.

## Appendix

## Appendix A AutoReCap -XL Details

This section outlines the collection and filtering processes for AutoReCap-XL.

### A.1 Stage 1: Data Selection

We selected existing video datasets primarily from YouTube for the ease of accessing automatic transcriptions. Specifically, we chose 73 million videos from the datasets AudioSet (Gemmeke et al., ), VGGSound (Chen et al., ), ACAV100M (Lee et al., ), VideoCC (Nagrani et al., ), YTTEMP1B (Zellers et al., ), and HDVila-100M (Xue et al., ). We select these datasets for their likelihood of containing videos with strong audio-video correspondence.

### A.2 Stage 2: Speech and Music Filtering

We downloaded English transcripts from YouTube and used automatically generated ones for videos without existing transcripts. However, we discard videos without any transcripts. While some datasets provide only video segments with specific timestamps, we processed the full videos, totaling around 73 million videos. We accepted audio segments longer than one second that lacked any corresponding subtitles, indicating the absence of speech and music. After filtering, we isolated approximately 327.3 million segments from 55.1 million videos. Fig. 5 displays the distribution of the number of segments per video. We denote this dataset as AutoReCap-XL-Raw. Subsequently, we use AutoCap to caption the audio segments. Fig. 7 shows the distribution of caption lengths. Given that AutoReCap was trained for 10-second audio, we limited segments to this duration. Additionally, we concatenate consecutive segments yielding identical captions to form longer audio clips. Fig. 9 illustrates the audio length distribution, and a word cloud of the captions is shown in Fig. 11. Despite filtering, the dataset was still dominated by captions related to speech and music. We attribute this to the limitations of YouTube’s automatic transcription, particularly with certain types of music and less common languages.

Figure: Figure 4: Distribution of number of segments per-video in AutoReCap-XL-Raw
Refer to caption: extracted/5952318/figures/supp/num_segments_pre_filtering.png

Figure: Figure 6: Distribution of caption length of AutoReCap-XL-Raw
Refer to caption: extracted/5952318/figures/supp/caption_length_pre_filtering.png

Figure: Figure 8: Distribution of audio duration of AutoReCap-XL-Raw
Refer to caption: extracted/5952318/figures/supp/audio_legnth_pre_filtering.png

Figure: Figure 10: Word cloud of audio captions in AutoReCap-XL-Raw
Refer to caption: extracted/5952318/figures/supp/word_cloud_pre_filtering.png

### A.3 Stage 3: Post-filtering of Speech and Music.

To further refine the dataset from speech and music, We follow a simple filtering approach. Specifically, we employed a large language model (LLM) to generate keywords associated with speech and music, such as "talking", "speaking", and "singing," and excluded all audio segments whose captions contained such keywords. This process yielded 57 million audio-text pairs from 20.3 million videos. Fig. 5 shows the number of segments per video, Fig. 7 shows the caption length distribution, Fig. 9 shows the audio length distribution, and Fig. 11 presents a word cloud of the final captions. We outline the data sources for constructing this dataset in Tab. 6. Our proposed dataset is not only 90 times larger than the previously largest available dataset, LAION-Audio-630KWu et al. () in terms of the number of audio clips, but also provides more accurate captions compared to existing datasets that rely on raw textual data. A comprehensive comparison with other datasets is detailed in Tab. 7

## Appendix B Limitations.

### B.1 AutoCap

Sounds emitted by various objects can often sound similar, such as a waterfall compared to heavy rain, or a can versus a motorcycle engine. In scenarios where metadata lacks detail, our audio captioning model may struggle to disambiguate these sounds accurately. The model also tends to falter in capturing the temporal relationships between sounds and differentiating foreground from background noises. Additionally, since it is fine-tuned on AudioCaps, which contains a limited vocabulary of 4,892 unique words (excluding common stop words), the model frequently produces repetitive words and captions.

### B.2 GenAu

Although our model is trained to generate natural sound effects, it underperforms in specialized areas like music generation or text-to-speech synthesis, where more targeted models are superior. Moreover, the limited vocabulary of the paired texts, even though extensive, hampers the model’s ability to accurately generate audio for long and detailed prompts.

### B.3 AutoReCap -XL

Our proposed dataset, AutoReCap-XL, is substantial in size but features a constrained vocabulary of only 4,461 unique words, excluding stop words, due to the vocabulary limitations of the AudioCaps-trained captioner. Furthermore, despite its potential as a significant contribution, this dataset has not yet been extensively analyzed for caption accuracy or performance in downstream tasks.

## Appendix C Evaluation Details

### C.1 Audio Captioning

While the established practice in the evaluation of audio captioning methods is to report the results on the test set using the checkpoint that performs best on the validation subset, prior work (Étienne Labbé et al., ; Kim et al., ) reported high instability of the metrics on the validation subset and weak correlation between the validation and test performance, making the model’s results vary significantly for different seeds. To alleviate this, ENCLAP (Kim et al., ) selects around five best-performing validation checkpoints and reports their best results on the test set. CNext-trans (Étienne Labbé et al., ) uses the FENSE score to pick the best validation checkpoint. This method of choosing the best checkpoint may produce misleading results and potentially disadvantage baselines. Our model, thanks to the two-stage training paradigm, significantly reduces this instability and we observe steady performance gains as training progresses. Therefore, we report the results at convergence, specifically after 20 epochs of pre-training and 20 epochs of fine-tuning.

### C.2 Audio Generation

There is a lack of consistency in the metrics used across text-to-audio generation baselines. Some baselines, such as Liu et al. () and Huang et al. (), employ the VGGish network (Hershey et al., ) to compute the Fréchet Distance, while others, like Liu et al. (), utilize the PANNs network (Kong et al., ), and still others rely on OpenL3 embeddings, such as Evans et al. (). Additionally, some baselines use the LAION CLAP network (Wu et al., ) to compute the CLAP score, whereas others use the Microsoft CLAP network (Elizalde et al., ). To further complicate matters, different baselines often report varying results in various publications. To address these discrepancies, we recalculated all metrics for the baselines using their publicly released checkpoints under identical evaluation configurations. Our method significantly outperforms the baselines across all metrics, except for the Fréchet Distance, where it is slightly behind Make-An-Audio 2 (Huang et al., ). Nevertheless, our user study, detailed in the main paper, indicates that GenAu is generally preferred over Make-An-Audio 2.

Figure: Figure 12: A screenshot of the user study interface.
Refer to caption: extracted/5952318/figures/supp/user_study.png

**Table 6: Overview of the employed dataset sources and audio clips counts for each of them.**
| Data Source | $\#$ pairs |
| --- | --- |
| AudioSet | 339.4k |
| VGGSounds | 126.9k |
| Freesounds | 262.3k |
| BBC Sound Effects | 31.2k |
| YouTube Videos | 57.0M |
| ACAV-100M |  |
| VideoCC |  |
| YTTEMP1B |  |
| HDVila-100M |  |
| AutoReCap | 761.1k |
| AutoReCap-XL | 57.0M |

**Table 7: Comparative overview of the main audio-language datasets.**
| Dataset | # Text-Audio Pairs | Duration (h) | Text source |
| --- | --- | --- | --- |
| AudioCaps | 52,904 | 144 | Human |
| Clotho | 5,929 | 37 | Human |
| MACS | 3,537 | 10 | Human |
| WavText5K | 4,072 | 23 | Online raw-data |
| SoundDescs | 32,979 | 1,060 | Online raw-data |
| LAION-Audio-630K | 633,526 | 4,325 | Online raw-data |
| WavCaps | 403,050 | 7,567 | Processed raw-data |
| AutoReCap | 761,113 | 8,763 | Automatic re-captioning |
| AutoReCap-XL | 57M | 123,500 | Automatic re-captioning |
| AutoReCap-XL-Raw | 327.3M | - | Automatic re-captioning |

**Table 8: Audio Evaluation Criteria**
| Criterion | Description |
| --- | --- |
| Realism | Which audio is more realistic? In other words, is more likely to be a result of a real action. |
| Quality | Which audio has better quality, regardless of the realism of the audio. Please note that some audio may have background noise, which should not be confused with low quality. |
| Prompt Alignment | Considering the prompt to generate the audio is "A sewing machine operating as a machine motor hisses loudly in the background", which audio better follows the given prompt? |
| Overall Preference | Considering the realism, quality, and prompt alignment of the audio, which audio do you prefer more overall? The prompt is: "A sewing machine operating as a machine motor hisses loudly in the background." |

### C.3 User Study

Each user study reported in this paper involved 5 different participants, yielding a total of 1000 responses per study. Samples were selected from the AudioCaps test split, specifically choosing the top 200 samples with the longest text prompts and sampling 50 for each study to enhance the likelihood of obtaining more complex audio scenarios. To minimize discrepancies between baselines, we fix the seed and other sampling parameters across all experiments.

During the user study, participants were initially presented with two audio clips from the compared baselines and asked to judge which one sounded more realistic. They were then prompted to choose the audio they believed had better quality. Next, after showing the prompt used to generate the audio, participants were asked to select the clip that most faithfully followed the prompt. Finally, they were asked to choose their overall preferred audio clip. A screenshot of the user study interface is included in Fig. 12, and the questions posed to the annotators are detailed in Tab. 8.

**Table 9: Qualitative comparison of captioning results on the AudioCaps dataset. See the *Website* for qualitative results accompanied by the respective audio.**
| Method | Caption |
| --- | --- |
| Ground Truth | *A man talking as ocean waves trickle and splash while wind blows into a microphone* |
| Ours | *A man speaks as wind blows and water splashes* |
| CoNeTTE | *A man is speaking and wind is blowing* |
| ENCLAP | *A man is speaking and wind is blowing* |
| Ground Truth | *An adult male speaks, birds chirp in the background, and many insects are buzzing* |
| Ours | *Birds chirp in the distance, followed by a man speaking nearby, after which insects buzz nearby* |
| CoNeTTE | *A man speaking with birds chirping in the background.* |
| ENCLAP | *Birds are chirping and a man speaks* |
| Ground Truth | *A telephone dialing tone followed by a plastic switch flipping on and off* |
| Ours | *A telephone dialing followed by a series of plastic clicking then plastic clanking before plastic thumps on a surface* |
| CoNeTTE | *A telephone ringing followed by a beep.* |
| ENCLAP | *A telephone dialing followed by a series of electronic beeps* |
| Ground Truth | *A running train and then a train whistle* |
| Ours | *A train moves getting closer and a horn is triggered* |
| CoNeTTE | *A train horn blows and a steam whistle is blowing* |
| ENCLAP | *A train running on railroad tracks followed by a train horn blowing as wind blows into a microphone* |
| Ground Truth | *A child is speaking followed by a door moving* |
| Ours | *A child speaks followed by a loud crash and a scream* |
| CoNeTTE | *A woman speaking followed by a door opening and closing.* |
| ENCLAP | *A young girl speaks followed by a loud bang* |
| Ground Truth | *Water splashing as a baby is laughing and birds chirp in the background* |
| Ours | *A baby laughs and splashes, and an adult female speaks* |
| CoNeTTE | *A baby is laughing and people are talking.* |
| ENCLAP | *A baby laughs and splashes in water* |
| Ground Truth | *Leaves rustling in the wind with dogs barking and birds chirping* |
| Ours | *Birds chirp in the distance, and then a dog barks nearby* |
| CoNeTTE | *A dog is barking and a person is walking.* |
| ENCLAP | *Birds chirp and a dog barks* |
| Ground Truth | *Tapping followed by water spraying and more tapping* |
| Ours | *Some light rustling followed by a clank then water pouring* |
| CoNeTTE | *A toilet is flushed and water is running.* |
| ENCLAP | *A faucet is turned on and runs* |

**Table 10: Ablation of different FIT architectural variations in terms of patch size number of latent tokens and adopted text encoders on the AudioCaps dataset.**
| Tokens | Patch size | FLAN-T5 | CLAP | FD $\downarrow$ | FAD $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| 256 | 1 | ✓ | ✓ | 16.45 | 1.29 | 10.26 |
| 256 | 1 |  | ✓ | 17.41 | 1.39 | 10.0 |
| 256 | 1 | ✓ |  | 20.47 | 1.86 | 8.89 |
| 384 | 1 |  | ✓ | 17.41 | 1.39 | 10.0 |
| 192 | 1 |  | ✓ | 18.0.1 | 2.01 | 8.91 |
| 128 | 1 |  | ✓ | 25.56 | 1.77 | 7.49 |
| 256 | 2 | ✓ | ✓ | 18.53 | 1.70 | 9.0 |

**Table 11: Ablation of different 1D-VAE designs on audio generation on the AudioCaps dataset.**
| Channels | Recon. loss | FAD $\downarrow$ | FD $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- | --- | --- |
| 64 | 0.159 | 1.29 | 16.45 | 10.26 |
| 128 | 0.107 | 1.43 | 16.78 | 10.11 |
| 256 | 0.064 | 1.80 | 18.63 | 9.43 |

## Appendix D Training and Inference Details

### D.1 AutoCap

We train the audio captioning model using the Adam optimizer, starting with a learning rate of $10^{-4}$ in stage 1, and reducing to $10^{-5}$ in stage 2. The training was completed over 9 hours on eight A100 80GB GPUs. Although our model is training with 10-second audio clips, we observed qualitatively that it generalizes well to short audios, such as 1-2 second audio clips.

### D.2 GenAu

We employ the LAMB optimizer for our audio generation model, setting the learning rate at $0.005$ with a cosine schedule, and incorporating a weight decay of $0.1$ and a dropout rate of $0.1$. The small model variant is trained for 210k steps with a batch size of 2,048, while the large model variant is trained for 220k steps with a batch size of 3,072. The large model is trained over 48 hours on 48 A100 80GB GPUs, and the small model on 32 GPUs. Ablation studies are conducted on eight A100 80GB GPUs using a batch size of 512. We further condition the model on the training dataset with a conditioning dataset ID. For generation, we utilize the AudioCaps dataset ID as it is the most reliable dataset.

## Appendix E Additional Results

In this section, we present additional results which are complemented by our *Website*.

### E.1 Additional Audio Captioning Evaluation

In Tab. 9 we show qualitative results of the captions produced by our method and compare them with state-of-the-art AAC methods. See the *Website* for qualitative results accompanied by the original audio. While ENCLAP (Kim et al., ) and CoNeTTE (Étienne Labbé et al., ) tend to produce short captions, our method produces the most descriptive captions, capturing the most amount of elements from the ground truth audio, an important capability to allow high-quality audio generation (Shi et al., ).

### E.2 Additional Audio Generation Evaluation

In this section, we report additional evaluation results and ablations on the task of audio generation.

In Tab. 10, we evaluate fundamental architectural choices in the design of our scalable FIT model.
When removing either the Flan-T5 or CLAP encodings, we notice a steady reduction in all metrics. When increasing the number of latent tokens we also notice a steady improvement in performance as more compute is allocated to the model. Similarly, increasing the patch size to 2 results in a performance decrease under all metrics due to the reduced amount of allocated computation.

In Tab. 11, we ablate the 1D-VAE bottleneck size in terms of reconstruction loss and performance of a subsequently trained latent audio diffusion model, in terms of FAD, FD, and IS. Similarly to the phenomenon observed in the image and video generation domain (Gupta et al., ; Esser et al., ), we observe that a larger number of channels allocated to the latent space results in lower reconstruction losses, but making the latent space more complex, hindering generation quality. We adopt 64 1D-VAE channels for all our experiments.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see weakness section.
soundness: 2
strengths: - The illustrations in the paper are clear and it is well written.
- No evident flaws.
- AutoReCap-XL will be great asset for the audio research community if open-sourced.
summary: This paper proposes an audio captioning method called AutoCap, an audio generation model called GenAu and an audio dataset called AutoReCap-XL.
weaknesses: - The authors shared the audio samples from Stable-Audio 1.0[7] but the comparison is missing in results table.
- Typo in table 5, last row "Quality" column: "-".
- The performance improvement when compared to increase in number of parameters in GenAu is marginal.
- Recently, Large Audio Language models [1,2,3,4] are being employed for audio captioning task, but the authors don't compare AutoCap to these baselines which in my opinion should be an important comparison.
- Inconsistent use of word "Q-Former" and "Qformer".
- Why do authors not show generation results of GenAu without Recap dataset or baseline + Recap in normal settings? It should be an important ablation to show where the performance gain is coming from.
- The CLAP text conditioner is confusing, the text encoder employed by CLAP [5,6] is BERT. What do authors mean when they say that they have used CLAP text encoder, do they use the text encoder weights from CLAP's checkpoint?
- There should be clear distinction between CLAP and EnCLAP. Authors mention CLAP but cite EnCLAP.
- The authors propose to use video caption and title as metadata, how sensitive is the output to the accuracy and relevance of metadata? Have the authors performed any ablation to show the robustness of this method?
- It would be interesting to analyse the distribution of sounds across various classes of non-ambient sounds in AutoReCap-XL. Have the authors explored any distribution patterns, and is there a noticeable bias toward any particular class as there is a lot of filtering while preparing the dataset?
- In section A.2 authors mention "Given that AutoReCap was trained for 10-second audio", I think there is a typo and it should be "AutoCap" instead of "AutoReCap" as the later one is the dataset.
- I want to understand what is the novelty of GenAu when compared to other models? In my opinion all other baselines and other audio generative models are **almost** capable of generating more or less similar audios with lesser parameters.

References

[1] GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities

[2] Listen, Think, and Understand

[3] Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

[4] Pengi: An Audio Language Model for Audio Tasks

[5] ENCLAP: COMBINING NEURAL AUDIO CODEC AND AUDIO-TEXT JOINT EMBEDDING FOR AUTOMATED AUDIO CAPTIONING

[6] CLAP : LEARNING AUDIO CONCEPTS FROM NATURAL LANGUAGE SUPERVISION

[7] Stable Audio Open

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Didn't train baselines on the new dataset to show the proposed architecture is actually superior. I would have liked to see the baselines trained on the 57M example dataset. This would clearly show if the better performance of the proposed method is due to architecture or just the scaling of the dataset.
* Is there a way to verify the quality of the dataset in terms of captioning? assuming the community adopts the use of this new data how are you to ensure the data is of high quality? maybe some human evaluation would be in place.
* Please move the limitations section into the main body of the paper.
soundness: 3
strengths: * The paper presents a simple pipeline to label audio data in order to generate larger data than ever before.
* The paper shows the new dataset improves the quality of trained models.
* Train a SOTA model using the new dataset.
* The authors say they will release the dataset which could be a good contribution to the community.
summary: Introduce two new models. First, we propose AutoCap, a high-quality and efficient automatic audio captioning model. By using a compact audio representation and leveraging audio metadata, AutoCap substantially enhances caption quality Second, the work proposes GenAu, a scalable transformer-based audio generation architecture that is scaled up to 1.25B parameters. Using AutoCap to generate caption clips from existing audio datasets.
weaknesses: * In scenarios where metadata lacks detail, audio captioning may struggle to disambiguate sounds accurately. The model also tends to falter in
capturing the temporal relationships between sounds and differentiating foreground from background
noises. 
* Fine-tuned on AudioCaps, which contains a limited vocabulary of 4,892 unique words. The limited vocabulary of the paired texts, even though extensive, hampers the model’s ability to accurately generate audio for long and detailed prompts.

* The proposed dataset, AutoReCap-XL, is substantial in size but features a constrained vocabulary of
only 4,461 unique words. Furthermore, despite its potential as a significant contribution, this dataset has not
yet been extensively analyzed for caption accuracy or performance in downstream tasks.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: My questions are broadly mentioned in the Weaknesses section. I am mostly looking for the "Why?" answers to the choices made that are related to my listed Weaknesses.
soundness: 2
strengths: - The paper is well written and well presented. The figures are good and the writing and everything is crisp. It is a nice to read paper.
- The method shows good improvements. Open-sourcing the artifacts in future would help the audio community.
- Th intuitions are good. The fact that good captions can improve audio generation is a good finding and well conveyed. Although I feel some parts are over-claimed which I mention next.
- I don't see many technical flaws with the paper.
summary: The paper proposes a new method to improve audio (sound) generation. The authors first propose AutoCap, a novel method to generate audio captions using auto-regressive models. Next, the authors propose a novel audio generation model which is trained on their generated dataset. The proposed method shows improvement on benchmark datasets.
weaknesses: I have several issues with the paper. I will first point out the technical weaknesses:
- Fig. 1 says CLAP Encoder has one token. CLAP uses HTSAT as the audio encoder  which also has intermediate representations. This only means that the authors used the CLS token (or some pooled representations) which is not specified. The authors should also clearly mention "CLAP audio encoder". 
- The caption says "We then compact this representation into 4x fewer tokens using a Q-Former (Li et al., 2023a) module.". The figure shows only HTSAT encoder representation is passed to QFormer. The authors should rewrite the caption.
- The claim "4x fewer tokens" just because QFormer was employed in one part is not justified. The authors are also using BART embeddings, etc. Do you mean "4x fewer tokens" compared to your own baseline which does not use a QFormer?
- QFormer has been used for audio earlier [3]. Additionally, it does not seem like you are pre-training the QFormer? Does this mean you are training it E2E? An E2E model as big as QFormer trained on such small datasets is not very sound.
- The authors say fewer tokens but do not talk about the increase in parameter count. I would like to see ore discussion on this please.
- Some audio captioning prior art missing from comparison [1,2].
- I am concerned why only audio clips with "No subtitles" were uses. Is the synchronization of time and subtitle correct? Also, human speech is an important sound and heavily present in audiocaps. If the authors believe not, then I think the paper should refocus on "environmental sounds" and not "audio".
- (Not a big weakness) I feel its important that synthetic data generation pipelines have some verification strategy (human or automatic).
- The freezing and unfreezing paradigm has been heavily employed in prior art in the DCASE challenge. The paper misses some citations. Please see DCASE captioning challenges that use BART like architectures.
- The fact that visual modality helps in audio captioning is not new and has been explored. Comparison or discussion with papers like SOUND-VECAPS [4] or [5] is missing.
- For GenAu, a discussion with Stable Audio is missing. I understand where at some places the authors mention DiT and how GenAu mitigates some limitations of DiT based architectures, but the introduction of time embeddings and the use of attention modules makes it very similar to StableAudio overall. A discussion would do justice.
- Table 4 does not compare with Stable Audio. Same training data + Stable Audio comparison is a must for a fair comparison.
- The table has missing comparison of baselines + their re-captioned data. I struggle to understand where the gains are coming from. The only ablation of GenAU w. U-Net is not sufficient. I also understand computational constraints but a couple of baselines with the data would do more justice to the results.

[1] https://ieeexplore.ieee.org/abstract/document/10448030.   
[2] https://arxiv.org/abs/2309.07372.   
[3] https://arxiv.org/abs/2406.11768.  
[4] https://arxiv.org/abs/2407.04416.   
[5] https://arxiv.org/abs/2309.11500.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. What does the rejection rate in line 282 refer to? Is the rejection related to downloads?

2. In Section 3.2, the authors mention that they use the captions of each video from the video-captioning model in Panda-70M. Does this imply that the "metadata" is essentially raw captions generated by another captioning system?

3. Can the authors provide more details about the collected video datasets? Specifically, what is the average length of each video sample?

4. How does GenAu perform if only one text encoder (either FLAN-T5 or CLAP) is used?

5. What is the performance of GenAu when trained on the proposed AudioReCap-XL dataset?

6. Could the authors provide some demo examples of GenAu?
soundness: 4
strengths: 1. The authors employ a Q-Former structure to compress audio representations from the pretrained HTSAT model, which significantly reduces system inference time. Additionally, to develop a more stable system, the paper adopts a two-stage training strategy: in stage one, only the Q-Former and CLAP-projector are trained on weakly labeled samples, while in stage two, the system is fully fine-tuned on the AudioCaps dataset.
2. The paper collects the majority of its data from videos, filtering out ambient clips that lack automatic transcripts. This is a straightforward yet interesting method to exclude speech and music content.
3. According to experimental results, both the captioning and generation systems achieve state-of-the-art (SoTA) performance.

Overall, this paper proposes an interesting approach by leveraging external metadata (e.g., captions or titles from visual information) to enhance audio captioning performance. Moreover, the introduction of the Q-Former module effectively reduces complexity and inference time. Finally, the authors present AutoReCap-XL, a dataset significantly larger than any existing audio-language datasets.
summary: This paper introduces a high-quality and efficient audio captioning model, named AutoCap, which demonstrates improvements in generation quality while being four times faster than current state-of-the-art (SoTA) models. The authors further present AutoReCap-XL, a large-scale audio-language dataset comprising 57 million ambient clips paired with automatically generated captions. Additionally, the paper proposes an audio generation model named GenAu, which also achieves SoTA performance.
weaknesses: 1. The paper lacks a detailed pipeline explaining how captions are analyzed and how speech or music-related content is filtered out. The authors should provide more clarification on this method.
2. The proposed audio captioning system appears to require external "metadata" to generate captions. However, the definition of this "metadata" is ambiguous throughout the paper. The authors should offer more details on what this metadata consists of and how it is used. If this metadata is not readily accessible from raw audio, it limits the applicability of the system to real-world scenarios.
3. According to Table 1, the comparison indicates that the proposed AutoCap does not achieve the best performance on several metrics, especially when only audio is used as input data.
4. For the audio generation system, the proposed model largely follows the architecture from Huang et al. (2023), incorporating the 1D-VAE and LDM modules. This feels more like an engineering effort, where the existing system is applied and scaled to larger datasets. Moreover, the model uses both FLAN-T5 and CLAP for conditional input, whereas previous models generally employ only one text encoder to achieve satisfactory results. The authors should explain why both encoders are necessary, along with experimental comparisons showing if using only one encoder leads to a drop in performance.
5. The paper lacks an evaluation of the effectiveness of the proposed AutoReCap-XL dataset.

Overall, the paper presents an interesting approach to improving the captioning system using external metadata. However, the explanation of certain methods and strategies needs more detail. Additionally, the metadata required by the captioning system may not be easily accessible from raw audio, making it difficult to apply the system to general tasks. The proposed generation model seems more like an engineering effort rather than a novel contribution. Lastly, the large-scale dataset introduced in the paper requires experimental validation to demonstrate its effectiveness in audio-language tasks. I am willing to change the score if the author can fulfil the experiments for the proposed two systems as well as the dataset.

Meta-review: > This paper introduces a high-quality and efficient audio captioning model, named AutoCap, which demonstrates improvements in generation quality while being four times faster than current state-of-the-art (SoTA) models. The authors further present AutoReCap-XL, a large-scale audio-language dataset comprising 57 million ambient clips paired with automatically generated captions. Additionally, the paper proposes an audio generation model named GenAu, which also achieves SoTA performance.

I agree with the authors that the rating of 3 from TUnZ may be a bit harsh post-rebuttal, but even if I were to fully ignore it, there is no champion review. So, weaknesses remain, in particular in disantangling contributions of data and modeling and supporting modeling claims.

This an interesting paper, but it fails a bit short of the scholarship required for ICLR.

Decision: Reject