AudioGen: Textually Guided Audio Generation
Authors: Felix Kreuk
Gabriel Synnaeve
Adam Polyak
Uriel Singer
Alexandre Défossez
Jade Copet
Devi Parikh
Yaniv Taigman
Yossi Adi
Venue: ICLR 2023 poster (2023)
Keywords: text-to-audio
audio generation

Abstract: In this work, we tackle the problem of generating audio samples conditioned on descriptive text captions. We propose AudioGen, an auto-regressive generative model, operating on a learnt discrete audio representation, that generates audio samples conditioned on text inputs. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high fidelity audio requires one to operate over extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. Finally, we apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. We further conduct an ablation study to gauge the effects of pre-trained text and audio components.

Published as a conference paper at ICLR 2023
AUDIOGEN: TION
TEXTUALLY GUIDED AUDIO GENERA-
Felix Kreuk', Gabriel Synnaeve', Adam Polyak', Uriel Singer', Alexandre Défossez! Jade Copet', Devi Parikh', Yaniv Taigman', Yossi Adi
FAIR Team, Meta AI Thhe Hebrew University of Jerusalem fe eelixkreuk@meta... com
ABSTRACT
We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AUDIOGEN, an auto-regressive generative model that generates audio samples conditioned on text inputs. AUDIOGEN op- erates on learnt discrete audio representation. The task of text-to-audio genera- tion poses multiple challenges. Due to the way audio travels through medium, differentiating "objects" can be a difficult task (c.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability scale models. Finally, model- ing high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges pro- pose an augmentation technique that mixes different audio samples, driving the model internally learn separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining similar bi- trate and perceptual quality. We apply classifier-free guidance to improve adher- ence to text. Comparing to the evaluated baselines, AUDIOGEN outperforms over both objective and subjective metrics. Finally, we explore the ability of the pro- posed method to generate audio continuation conditionally and unconditionally. Samples: https: / fel rkuk g thub. i o aud ogge..
INTRODUCTION
Neural generative models have challenged the way we create digital content. From generating high- quality images (Karras et al., 2019; Park et al., 2019) and speech (Ren al., 2021; Oord et al., 2016), through generating long textual spans (Brown et al., 2020; Zhang et al., 2022), to the recently proposed text prompted image generation (Ramesh al., 2022: Rombach et al., 2022), these mod- els have shown impressive results. This begs the question what would be the audio equivalent to textually guided generative models? From generating soundscapes to music or speech, solution to this problem that is high fidelity, controllable, and diverse in its outputs, would be useful addition the modern toolbox of creators of movies. video games, and any virtual environments.
While image generation and audio generation have a lot in common, there are a few key differ- ences. Audio is intrinsically a one dimensional signal and thus has less degrees of freedom to differentiate overlapping "objects"" (Capon, 1969; Frost, 1972). Real-world audio inherently has re- verberations, which makes the task of differentiating objects from the surrounding environment even harder. Moreover, psychoacoustic and psychovisual properties differ, for instance hearing "resolu- tion" (equal-loudness) is U-shaped in frequencies with a dip at 4kHz and bump at 8kHz (Suzuki et al., 2003). Last but not least, the availability of audio data with textual descriptions is orders of magnitude below that of text-image paired data. This makes generating unseen audio compositions a hard task (e.g. generating audio equivalent of image of "an astronaut riding horse in space"").

---

Published as a conference paper at ICLR 2023
In this work, tackle the problem of generating audio samples conditioned on descriptive text captions. We additionally extend the proposed method to conditional and unconditional audio con- tinuation. Here, generate "a dog barks while somebody plays the trumpet in a busy street". In the above prompt, the model must generate three categories of acoustic content, with varying degrees of background/foreground, durations, and relative position in the temporal axis, the composition of which is highly unlikely to be present in the training set. Generating such audio is thus challenge in generalization, acoustic fidelity, production and mastering.
We propose AUDIOGEN, an autoregressive textually guided audio generation model. AUDIO- GEN consists of two main stages. The first encodes raw audio discrete sequence of tokens using a neural audio compression model (c.g. Zeghidour et al. (2021)). This model is trained in an end-to-end fashion to reconstruct the input audio from the compressed representation, with an addition of perceptual loss in the form of set of discriminators. Such an audio representation designed to generate high-fidelity audio samples while still being compact. The second stage, leverages an autoregressive Transformer-decode language-mode that operates on the discrete au- dio tokens obtained from the first stage while also being conditioned on textual inputs. We represent text using& a separate text encoder model pre-trained et al., 2020). The pre-trained text encoder enables the generalization to text concepts that are absent from current text-audio datasets. This is especially important when working with text annotations limited in terms of diversity and descriptiveness.
large corpus of text, namely T5 (Raffel
Compared to the existing text-to-audio work (Yang et al., 2022), AUDIOGEN generates samples that obtain better objective and subjective metrics. In particular, AUDIOGEN creates more natural sounding unseen audio compositions. Lastly, we empirically show how the proposed approach can be extended to audio continuation considering both conditional and unconditional generation.
regressive audio generation model condi- tioned on textual descriptions or audio prompts, as evaluated with objective and subjective (human listeners) scores. Specifically we propose two model variations, one with 285M parameters and another one with 1B parameters; (ii) We improve text-to-audio generation in two axes. We im- prove text adherence by applying classifier free guidance on top of the audio language model. We improve compositionality by performing proposed approach can be extended to audio continuation conditioned on text and unconditionally; (iv) We explore the trade-off between audio-fidelity and sampling time by utilizing residual vector quantization (for acoustic units) and multi- sttti-stream transformers.
Our contributions:
(i) We propose a state-of-the-ar
the fly text and audio mixing; (iii) We show that the
2 RELATED WORK
Speech Representation Learning. Studies on unsupervised speech representation learning can be roughly divided into reconstruction and self-supervised learning methods. Auto-encoding is the common approach for signal reconstruction, where speech is first encoded into a low-dimensional latent representation, and then decoded back to speech. Various constraints can be imposed on the encoded space, such as temporal smoothness (Ebbers et al., 2017), discreteness (van den Oord et al., 2017b), and hierarchy (Hsu et al., 2017). Self-Supervised Learning (SSL) methods for speech have shown remarkable results for automatic speech recognition (Schneider al., 2019; Baevski et al., 2020; Wang et al., 2021), phoneme segmentation (Kreuk et al., 2020), and audio compres- sion (Zeghidour al., 2021; Polyak et al., 2021). van den Oord et al. (2018) and Schneider et al. (2019) suggested training a convolutional neural network to distinguish true future samples from random distractor samples using a Contrastive Predictive Coding (CPC) loss function. Ao et al. (2022) proposed a speech version of the T5 model and showed its efficiency on various speech tasks. Similar to CPC, Baevski al. (2020) use an encoder and a predictor. which is trained con- trastively to distinguish positive and negative samples. Unlike (Schneider et al., 2019), it discretizes and masks segments of the encoder's 's output. Hsu et al. (2021) proposed the HuBERT model which is trained with masked prediction task similar to BERT (Devlin et al., 2019) but with masked con- tinuous audio signals. Chen et al. (2022) proposed a similar version of HuBERT trained on larger and augmented dataset. More recently, Huang et al. (2022) proposed Masked Auto Encoding ap- proach for learning a speech representation and show it efficiency on several audio classification tasks.
Another line of relevant prior work relates to modeling audio discrete representations. Recent studies suggest quantizing SSL representations using k-means and later perform language modeling (Lakho-
2

---

Published as a conference paper at ICLR 2023
tia et al., 2021; Kharitonov al., 2022a; Borsos et al., 2022), multi- processing (Kharitonov et al., 2022b). speech emotion conversion (Kreuk et al., 2022), spoken dialuge (Nguyen et al., 2022), and speech-to-speed translation (Lee et al., 2022a;b; Popuri et al., 2022).
Text-to-Image has seen great advances recently. DALL-E (Reddy et al., 2021) first transforms the patches of an image to discrete codes using a pre-trained VAE, During training, codes rep- resenting image patches were appended to codes representing text. Then, a Transformer-deecodrr model was trained to model these codes in autoregressive fashion, while Gafni al. (2022) sug- gested a similar approach and incorporated segmentation maps for increased controllability. In the Parti model (Yu et al.) the authors suggested modeling the task of lext-to-image as sequence-to- sequence problem using Transformers in an encoder-decodee architecture.
More recently, diffusion model have gained increased popularity (Nichol al., 2022; Ramesh al., 2022: Saharia et al.; Rombach et al., 2022). DALLE-2 (Ramesh et al., 2022) used diffusion model to predict the CLIP visual features given the CLIP text encoding (prior), and another diffusion model to predict the image pixels given the predicted CLIP visual features (decoder). The predicted image is upsampled to a higher resolution using a cascade of super-resolution models. Imagen (Saharia et al.) employed a similar approach but omitted the prior component in favor of using pre-trained text-encoders such as T5 (Raffel et al., 2020).
The most relevant to work is the one proposed by Yang et al. (2022), in which the authors proposed DiffSound, a text-to-audio model based on a diffusion process that operates on audio discrete codes. The audio codes were obtained from VQ-VAE (van den Oord et al., 2017a) based model trained over mel-spectrogram. To further improve model performance, Yang al. (2022) suggested pre-training the diffusion model using labeled tags with a random input masking. They additionally explore the usage of an auto-regressive Transformer decoder model, however found be inferior to the diffusion based model.
Text-to-Audio.
The proposed method differentiate from DiffSound in the following: (i) our audio representation being learned directly from the waw-waveform; (ii) we create new audio compositions using data augmentation allowing the model to generate audio from complex text captions; (iii) we apply and study the effect of classifier free guidance under the auto-regressive setting: (iv) in contrast to Yang et al. (2022), we empirically demonstrate that text- rext-conditioned auto-regressive models can generate high-quality audio samples.
3 METHOD
The proposed method, AUDIOGEN, is based two main steps: (i) learning discrete representation of the raw audio using an auto-encoding method; (ii) training a Transformer language model over the learnt codes obtained from the audio encoder, conditioned on textual features. Then, during inference time, we sample from the language model to generate new set of audio tokens given text features. These tokens can later be decoded into the waveform domain using the decoder component from step (i). A visual description of the proposed method can be seen on Figure 1.
3.1 AUDIO REPRESENTATION
An audio signal of duration d can be represented by a sequence æ E [-1,1]Cux2 with Ca the number of audio channels, T d far the number of audio samples given sample rate far. In this work we set for 16kHz. The audio representation model is composed of three components: (i) an encoder network E which gets as input an audio segment and outputs latent representation 2: (ii) a quantization layer Q produces a compressed representation &g. using Vector Quantization (Vasuki & Vanathi, 2006) layer; (iii) a decoder network G reconstructs the time-domain signal, ă, from the compressed latent representation Zg. The whole system is trained end-to-end to minimize reconstruction loss applied over both time and frequency domain, together with a perceptual loss in the form of several discriminators operating at different temporal resolutions. Using pre-trained model, we can leverage the encoder and quantizer components as a discrete feature extractor (i.e., Q E) and G to decode the representation to the time-domain signal. For Q, use single codebook with 2048 codes, where each code is a 128 dimensional vector. visual description of the proposed method can be seen in Figure (right).
3

---

Published as a conference paper at ICLR 2023
La La
e
Audio Becoder
Terner Decoder
00.00...1
bust Tokana
XX
Tees Deaitions:
Ardso Fesisioral
Teatt Rrooder
Aadia Escedat
A dog is barking in
A general overview of the AUDIOGENSystem. Left: the audio representation model. Right: the audio language model. Both text and audio embeddings concatenated over the time dimension and fed in K causal self-attention and cross-attention blocks with the embedded text.
Figure I:
Architecture. We follow similar auto-encoder model architecture as in Zeghidour et al. (2021); Li et al. (2021). The encoder model E' consists of ID convolution with C' channels followed by B convolutional blocks. Each convolutional block is composed of single residual unit followed by Nlon-ssapling layer consisting of a strided convolution, with kernel size K of twice the stride S. The residual unit contains two convolutions and a skip-connection. The number of channels is doubled whenever down-sampling occurs. The convolutional blocks are followed by two-layer LSTM for sequence modeling and a final ID convolution layer with a kernel size of and D output channels. We use C 32. B 4 and (2, 2, 2, 4) strides. We use ELU as a non non-lineat activation function (Clevert et al., 2015) and LayerNorm (Ba et al., 2016). The decoder mirrors the encoder, using transposed convolutions instead of strided convolutions, and with the strides in reverse order as in the encoder, outputting the final audio.
Training Objective. We optimize GAN based training objective similar to (Kong et al., 2020; Zeghidour et al., 2021) of jointly minimizing a combination of reconstruction losses and adversarial losses. Specifically, we minimize the L1 distance between the target and reconstructed audio over the time domain, i.c. (xx, Ø) = || + 1- For the frequency domain loss, we use a linear combination between the LI and L2 losses over the mel-spectrogram using several time scales (Yamamoto al., 2020; Gritsenko et al., 2020). Formally,
E .))
Σ E,Ea iEe
Σ
[S S,(a)]l -
|s|
where S; is a 64-bins mel-spectrogram using a normalized STFT with window size of 2' and hop length of 2' 2//4, e 5,..., .,. 11 is the set of scales, and a represents the set of scalar coefficients balancing between the LI and L2 terms. Unlike Gritsenko et al. (2020), we set a; = 1.
To further improve the quality of the generated samples, we additionally optimize a multi-scale STFT-based (MS-STFT) discriminator. Multi-scale discriminators are popular for capturing differ- ent structures in audio signals (Kumar al., 2019; Kong et al., 2020; You et al., 2021). The MS- STFT discriminator is based on identically structured networks operating on multi-scaled complex- valued STFT where its real and imaginary parts are concatenated. Each sub-network composed of 2D convolutional layer (using kernel size 3 x 8 with 32 channels), followed by 2D convolutions with increasing dilation rates in the time dimension of 1, 2 and 4, and a stride of 2 over the frequency axis. A final 2D convolution with kernel size 3 x 3 and stride (1, 1) provide the final prediction. We use different scales with STFT window lengths of [2048, 1024, 512, 256 128). The adversarial loss for the generator is constructed as follows, (z( 1 Σg max(0, 1 D& (i))). where K is the number of discriminator networks. Similarly to previous work on neural vocoders (Kumar et al.,
2019; Kong et al., 2020; You al., 2021), we additionally include feature matching loss for the
4

---

Published as a conference paper at ICLR 2023
generator. Formally,
K L
(x,
- 11)
(2)
KL
k=1
where (Dx) are the discriminators, and L is the number of layers in discriminators.
Overall the discriminators are trained to minimize the following: La(x, σ) = Σ max(O, Px(x:)) + max(0, 1 + Px())), where K the number of discriminators, while the generator trained to minimize the following: Lg = l + - .)) +>g l (ā)) f & (2r, it).
3.2 AUDIO LANGUAGE MODELING
Recall, in this work our goal is to generate audio conditioned on text. Specifically, given a textual input the Audio Language Model (ALM) component outputs sequence of audio tokens Eg. which can be later decoded into raw audio using G.
Consider text encoder F which maps raw text input into semantic dense representation, F(c) u. Then, a Look- Look-Up-Taall UplTable (LUT) embeds the audio tokens, εn, into a continuous space, LUT(22) = v. We then concatenate both u and create Z wi..... uT, Ui,..., UT., where Tú and T,
the length of the text representation and audio representation respectively.
Using the above representation, train Transformer-decoder language-model parameterized by 0 using the cross-entropy loss function:
N Tc
L = log (ujlu),.... u4. "t,..,,,
0)-1).
(3)
The text representation is obtained using a pre-trained text-encoder (Raffel et al., 2020). We additionally experimented with learning text embeddings using LUT. Although produces com- parable results to the T5 model, it limits our ability to generalize to unseen words during training, hence we did not pursue this direction. The transformer-decoder language-model is implemented using a GPT2-like architecture (Radford et al., 2019). To achieve a better text adherence we add cross-attention between audio and text each attention block of the transformer. See a visual de- scription on Figure 1 (left). The entire system may be alternativeyy viewed as an encoder-decoder model, where the encoder (T5) is pre-trained and fixed throughout training.
Classifier Free-Guidance. It was recently shown by Ho & Salimans (2021); Nichol et al. (2022) that using the Classifier Free Guidance (CFG) method is an effective mechanism for controlling the trade-off between sample quality and diversity. Although the CFG method was originally proposed for the score function estimates of diffusion models, in this work we apply it auto-regressive models. During training we optimize the Transformer-LM conditionally and unconditionally. In practice, we randomly omit the text conditioning 10% of training samples. At inference time we sample from distribution obtained by linear combination of the conditional and unconditional probabilities. Formally we sample from,
log pa (ujjlu)..................) ur., o1 w)-1) + (1 7) log (o(
where y is the guidance scale.
Multi-stream audio inputs. In order to generate high-quality audio samples we down-sample the raw audio by a factor of 32, which corresponds to operate over extremely long sequences as each second of audio is represented by 500 tokens. Modeling such long sequences is a notoriously difficult problem (Rae et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). To alleviate this problem, propose Multi-Stream representation and modeling paradigm. It was shown by Kharitonov et al. (2022b) that transformers are capable of modeling multiple streams simultaneously.
ms for each audio token. This requires us
Consider a sequence of length Te, we can learn a representation length T./2 using two par- allel streams of approximately the same bit-rate. This approach can be generalized & streams, where each stream is of length T,/ and each codebook is of size 2048/k. Such representation can be obtained by generalizing Q from single code book Vector- Quatto-Qaatizttioon to Residual Vector
5

---

Published as a conference paper at ICLR 2023
Quantization module done by Zeghidour al. (2021). At time the network is fed with & discrete codes, which then embedded using embedding layers. The final embedding at time t is the mean of these & embeddings. We adapt the network to output & codes using & LM prediction heads. The prediction heads operate independently of each other, we explored conditioning stream i on stream i but did not observe any performance gains.
4 EXPERIMENTS
In this section we start by providing a detailed description of the experimental setup. Next, we present the main results for audio generation, and we conclude this section with ablation study.
4.1 EXPERIMENTAL SETUP
Dataset. We use a set of several datasets: AudioSet (Gemmeke et al., 2017), BBC sound effects 1 AudioCaps (Kim et al., 2019), Clotho v2 (Drossos et al., 2020). VGG-Sound (Chen et al., 2020), FSD50K (Fonseca al., 2021), Free To Use Sounds 2. Sonniss Game Effects 3, WeSSondEffectss Paramount Motion Odeon Cinematic Sound Effects All audio files were sampled at 16kHz.
For textual descriptions we use two types of annotations. The first one multi-label annotations, available for the datasets: AudioSet, VGG-Sound, FSD50K, Sinniss Game Effects, WeSoundEf- fects, Paramount Motion - Odeon Cinematic Sound Effects. We form pseudo-sentences by con- catenating lists of tags available per audio samples (e.g.. "dog, bark, park" is transformed to "dog bark park"). The second type of annotation is natural language captions available for the datasets: AudioCaps, Clotho v2, Free To Use Sounds, and BBC Sound Effects. A more elaborate description of the used datasets can be found in Appendix 3. We apply a pre-processing step to better match the class-label annotation distribution. Specifically, lemmatize the remaining words (e.g. "a dog is barking at the park" is transformed to "dog bark park") using the WordNet lemmatizer in NLTK (Bird al., 2009). As speech the dominant class in the data, we filter all samples where the tag or caption contains the word "speech" to generate more balanced dataset. Overall we are left with 44k hours for training data.
remove stop words and numbers, finally
Data Augmentations. One of the most impressive capabilities of recently proposed generative models (Ramesh et al., 2022: Saharia et al.; Gafni al., 2022) is their ability to create unseen object compositions (e.g., "An astronaut riding horse in space"). To achieve similar capabilities with regards to audio generation we propose an augmentation method that fuses pairs of audio samples and their respective text captions, thus creating new concept compositions during training. Formally, given two audio samples æ 22 and their respective text captions C1, C2, we first randomly draw temporal offset merge the two audio samples. Next, we draw random Eegnal-to-Noiss Ratio (SNR) in the interval [-5, 5], and finally we mix the audio samples and concatenate the text captions C1 C2-
Evaluation Methods. We evaluate all models and baselines using both objective and subjective metrics. For the objective functions we compute the Fréchet Audio Distance (FAD) (Kilgour al., 2019) over both real and generated samples. FAD, adapted from the Fréchet Inception Distance (FID) to the audio domain, a reference-free evaluation metric that closely correlates with human perception. Similarly to Yang et al. (2022) we additionally compute the KL-Divergence between the output of state-of-the-ard audio classification model (Koutini et al., 2021) while feeding it both the original samples and the generated audio. The FAD was shown to correlate well with human perception terms of audio quality. On the other hand, the KI. measure computed using the label distribution produced by a pre-trained classifier. Hence, it reflects more on the broader audio concepts occurring in the recording. As a result, the two metrics are complementary.
For subjective methods, we follow the similar setting to (Yang et al., 2022). We ask human raters to evaluate two main aspects of the audio signal (i) overall quality (OVL), and (ii) relevance to the
co...dcccc...... co.uk/
httpss://eecunddfeeetsscccm otrp://sssococooooooooooooeeeeee

---

Published as a conference paper at ICLR 2023
Table 1: Results are reported for DiffSound together with several versions of AUDIOGEN. For DiffSound data augmentation, we follow the authors suggested mask-based text generation (MBTG) strategy. For subjective tests we report overall quality (OVL), and text relevenace (REL.) together with 95% Confidence Interval. For the objective metrics report FAD and KL.
SUBJECTIVE
OBJECTIVE FAD
#params
AUG.
TEXT-COND.
OVL1
REL.1
KLJ
Reference
92.08=1166
92.97=0885
DiftSound
400M
MBTG
CLIP
65.68=1558
53.91
7.39
2.57
AUDIOGEN-hase
285M
T5-base
70.85 1.06
1.62
2.84
2.14
AUDIOGEN-bai base
285M
Mix
TS-base
71.68 1.89
66.01
3.13
2.09
AUDioGEN-large
IB
Mix
T5-Large
71.85=1007
68.73:1161
1.82
1.69
3.5
3.0
Tve
2.5
124
2.0
1.5
1
0.5
2.0
3.0
4.0
0.3
1.0
2.0
3.0
4.0
1.0
0.5
1.01
23
3.0
4.0
Audio Proonpt Duntion
Audio Frompt Duration
Pooupt
(a)
(b)
(c)
Figure 2: Audio continuation results. In (a) we compare model generations against audio corre- sponding SRC and RND. In (b) compare model generations with and without text condi- tioning against audio corresponding to RND text. In (c) we compare model generations with and without text conditioning against audio corresponding to SRC text. In all settings we report KL results. Specific details can be found on Section 4.2.
text input. We follow the MUSHRA protocol (Series, 2014), using both a hidden reference and a low anchor. For the overall quality test raters were asked to rate the perceptual quality of the provided samples in a range between 1 to 100. For the text relevance test, raters were asked rate the match between audio and text on scale between to 100. Raters were recruited using the Amazon Mechanical Turk platform. We evaluate 100 randomly sampled files from the AudioCaps test set, where each sample was evaluated by at least 5 raters. We verified that the majority of the samples (85%) contain at least two audio concepts (e.g.. "a dog barks while a bird chirps"). We use the CrowdMOS package to filter noisy annotations and outliers. We remove annotators who did not listen to the full recordings, annotators who rate the reference recordings less then 85, and the rest of the recommended recipes from the CrowdMOS (Ribeiro al., 201 1). Participants in this study were paid at least the American minimum wage.
Hyper- prrper-prramttrs. We trained two sets of ALMs, one with 285M parameters (base) and the other with IB parameters (large). In the smaller model we use a hidden-size of 768, 24 layers and 16 attention-heads, while for the large variant we use hidden size 1280, 36 layers and 20 attention- heads. We use the Adam optimizer with a batch size of 256, a learning rate of 5e-4 and 3k steps of warm-up followed by inverse-square root decay. The small model was trained on 64 A100 GPUs for 200k steps (~55 days) and the large model was trained on 128 A100 GPUs for 200k steps (~1 week). For the small model we use T5-base and for the large model we use TS-large. For sampling, we employ top-p (Holtzman et al., 2019) sampling with p 0.25. For the CFG we use 7 3.0.
4.2 RESULTS
We start by comparing AUDIOGEN to DiffSSund. Objective metrics are reported on the AudioCaps test set. We use the official pre-trained model provided by DiffSSoud authors ? The model was trained on AudioSet and fine-tuned on AudioCaps. Results are presented in Table
'http:/ // /www. wwwwdmos............o........... org/ downl
Pree-trained model can he found under http://gghhhhoooooooogggh/////ee
7

---

Published as a conference paper at ICLR 2023
AUDIOGEN-base outperforms DiffSound considering all metries while being smaller in terms of parameters count. As expected AUDIOGEN-large significantly outperforms both DiffSound and AUDIOGEN-base. Notice, for AUDIOGEN-base the model trained without mixing obtained supe- rior FAD and comparable OVL to the model trained with mixing augmentations. However, when considering relevance text, i.e., KI. and RAL. the model with mixing augmentations reaches bet- ter performance. This is especially noticeable for the REL. metric as it contains mostly complex compositions (see Table 1). We additionally compare AUDIOGEN-base to the same dataset setup used by DiffSound (i.e., training on AudioSet and AudioCaps). AiDIOGEN-bass reached KL of 2.46 vs. 2.57 for DiffSound and FAD of 4.39 vs. 7.39 for DiffSound. These results suggest that AUDIOGENIS still significantly superior to DiffSound.
Next, we experimented with pre-training the ALM component on only audio tokens without condi- tioning on text (learning an audio prior). We did not observe gain in doing ALM pretraining. We hypothesize that due to the pre-training process taking place on the same labeled data. Samples can be found under the following link: https: / ffellelllleeee www.feeeel.keeeeeeee wwt.wkw.................. 1o/ audiogen.
Audio Continuation. Similarly to (Lakhotia et al., 2021), we first encode an audio prompt into a sequence of audio tokens, feed it into the ALM, and sample a continuation. Unlike previous work, we can additionally steer the generated audio towards textual captions. We evaluate our model given audio prompts of lengths [0.5s, 1s, 2s, 3s, 4s] and different combinations of text prompts: (i) no-text; (ii) text corresponding to the audio prompt (SRC); (iii) text taken from a randomly sampled audio file (RND). For each of the above settings we measure the KL. between the generated audio and either the audio corresponding to the source text or the target text.
In Figure 2(a) we input the model with audio prompt together with RNDaS conditioning text. We evaluate KL between the output of the classification model using the generated audio against either the audio prompt or the audio corresponding to the RND. Results suggest that by using short audio prompts we could better steer the generation towards the conditioning text. In contrast, using long audio prompts leaves less room for textual guidance. Text and audio prompts have roughly the same impact at 1.5s. In Figure 2(b) we input the model with an audio prompt with and without RNDtext conditioning. We evaluate the KI. between the output of the audio classification model using the generated audio against the audio corresponding to the RND. Although using longer audio prompts leaves less room for textual guidance, we still observe a gap between generations with and without text, even when using longer audio prompts (~4s). This suggests that text has significant effect on the generation process. Lastly, In Figure 2(c) we condition the model on an audio prompt with and without SRCTeXt conditioning. We evaluate the KI. between the output of the audio classification model using the generated audio against the input audio. Results suggest that when using short audio prompts, text has major impact on the generated output. However we feed longer sequences, the audio is sufficient to steer the generation towards the target concept classes with a minimal gain when adding text.
Full results together with FAD scores for all settings together with a visual depiction can be found in Table on the supplementary 7.2.
4.3 ABLATION STUDY.
The effect of classifier-free guidance scale. As pointed out by Ho & Salimans (2021), the choice of the CFG scale offers trade-off between sample diversity and quality with respect to the condi- tioning text. To gauge the effect of the 7 parameter in our setting on Figure we report results for 7 E (1.0, 2.0, 3.0). Notice, setting 7 = 1 is equivalent to vanilla sampling procedure (no CFG).
Removing the CFG results in poor performance compared to 7 >> 1.0. The FAD score reaches its' minimum at y 3.0, while the KL monotonically decreases and converges 7 4.0. This implies that using y = 3.0 provides the best trade-off in the evaluated setting between quality and diversity.
Multi-stream processing. To better understand the benefit of using multiple streams over single stream we optimized three different audio discrete representation models using Down Sampling Factors (DSF) of (x32, x64, x128) using {1, 2, 4} different codebooks respectively. For a fair comparison we kept all models at 2048 codes overall (c.g.. 2 codebooks of size 1024 for the x64 model and 4 codebooks of size 512 for the x x228 model).
8

---

Published as a conference paper at ICLR 2023
4.5
2.75
2.50
3.0
2.00
2.5
1.75
2
4
2
Cuidance Scale
Guidance Scale
Figure 3: Results FAD (left) and (right) as function of the guidance scale.
Table 2: Multi-stream results: we report two sets of results encoding and generation. For the encoding metries report Bitrate (kbps), SI-SNR (dBs), and ViSQOL. For the generation scores we report FAD, KL and inference speed-up. We additionally include the number streams and the Down Sampling Factor (DSF). Notice, the encoding metrics the for the large and base model as the same audio representation model was used.
ENCODING SI-SNR
GENERATION
# STREAMS
DSF
Bitrate (kbps)
VISQOLA
FADL
KLJ
SPEED-UP UPE
1
x32
5.37
5.1
3.95
3.13
2.09
x1.0
base
2
x64
4,88
4.5
3,94
10.35
2.17
x2.0
4
x128
4.39
4.2
3.91
9.68
2.36
x5.1
1
x32
5.37
5.1
3.95
1.82
1.69
x1.0
large
2
x64
4.88
4.5
3.94
6.89
1.86
x2.3
x128
4.39
4.2
3.91
2.59
x3.6
To assess the quality of the audio representation we report three encoding metrics namely Scale- Invariant Signal-to-Noise Ration (SI-SNR), ViSQOL (Chinen et al., 2020), and bitrate. These met- rics characterize the audio representation only while ignoring the ALM. Note, both SI-SNR and ViSQOL reference based metrics hence are computed between the audio reconstructed from the learned representation and the original audio sequence. Results are reported on Table 2. While single stream encoder achieves better SI-SNR results, all representations estimated perceptual quality as measured by ViSQOL. While the total amount of codes is the same across all settings, the effective bitrate of the multi-stream models was degraded, leading to lower SI-SNR values.
comparable in terms of
Next, we report the FAD, KL and inference speed-up for the ALM on top of the learned represen- tations. Results suggest that increasing the number of streams degrades the performance in both base and large models when compared to a single-stream. AUDIOGEN-base improves the KI score over DiffSound (2.57) while yielding higher FAD. AUDIOGEN-large with two streams improves over DiffSound both FAD and KL (7.39 and 2.57 respectively). While showing inferior FAD and KL results when compared to the single-stream model, the multi-stream setting offers trade-off' between inference quality and speed.
LIMITATIONS
As we operate on audio tokens using relatively small down-sampling factor, the audio tokens se- quences can be extremely long. This impose two main limitations: (i) modeling long range se- quences; (ii) high inference time. In this work, we propose one possible relaxation to the first limitation, however such an approach comes at the cost of producing less quality audio samples. Such issues will become worse when considering high resolution audio samples, (e.g., sampling rate 48kHz). Another limitation of the proposed approach relates to the audio compositions. Al- though, the mixing augmentation greatly improves the models ability to separate the sources and

---

Published as a conference paper at ICLR 2023
create complex compositions, it is still lacks the understanding of temporal ordering in the scene, e.g., a dog is barking then a birds humming, vs. a dog is barking and a birds humming in the back- ground. Lastly,
omit most of the speech samples in training set, the proposed approach often generates unintelligible speech. This can be mitigated by either using more speech data, better data augmentation recipes for speech, or by providing additional speech features. Another limita- tion of the datasets used is its diversity. The datasets were mainly collected from YouTube, in which specific demographic and geographic locations are more represent than others. This may create bias in the generated sample.
6 CONCLUSION
In this work, we propose a Transformer based generative model, named AUDIOGEN, which operates on learnt discrete audio representation. Unlike previous work, we empirically demonstrate that auto-regressive models can generate high-quality audio samples conditionally or unconditionally. We show that on-the-fly text and audio mixing augmentations can improve model performance and provide an ablation study, analyzing the effect of CFG and multi-stream processing.
As for broader impacts, this work serves as the foundation for building better text-to-audio models. In addition, the proposed research could open up future directions involved with benchmarking, semantic audio editing, audio source separation from discrete units, etc.
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5723-5738, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 100.006400 2016.
Alexei Baevski et al. wav2vec 2.0: A framework for self-supervised learning of speech representa- tions. In ICLR, 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkir. O'Reilly Media, Inc.". 2009.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: lan- guage modeling approach to audio generation. arXiv preprint arXiv:2209.03143; 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Jack Capon. High-resolution frequency-wavenunie spectrum analysis. Proceedings of the IEEF, 1788):1008-1418, 1969.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale supe-supe pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 2022.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp. 1-6, IEEE, 2020.
10

---

Published as a conference paper at ICLR 2023
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arxiv:5111.07289, 2015.
Jacob Devlin et al. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740. IEEE, 2020.
Janek Ebbers et al. Hidden markov model variational autoencoder for acoustic unit discovery. In INTERSPEECH 2017, 2017.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactoons Audio, Speech, and Language Processing, 30:829 852, 2021.
Otis Lamont Frost. An algorithm for linearly constrained adaptive array processing, Proceedings of the IEEE, 60(8))926-935, 1972.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 89-106. Springer, 2022.
Jort F Gemmeke. Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore. Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeles dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snock, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33: 14:13362-13000 13072, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in Neural Information Processing Systems, 2017.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460, 2021.
Po- Yao Huang, Hu Xu, Juncheng B Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Pro- cessing Systems, 2022. URL https: / www.revieeeeeeeeeeeeeeeeee net forum? d= 9bbL..
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, et al. textless-lib: a library for textless spoken language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, pp. 1-9, 2022a.

---

Published as a conference paper at ICLR 2023
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8666-8681, 2022h.
Kevin Kilgour, Mauricio Zuluaga. Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In INTERSPEECH, 2019.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Pupers), pp. 119-132, 2019.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-1 17033, 2020.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint 1cXiv:2//000506: 2021.
Felix Kreuk, Joseph Keshet, and Yossi Adi. Self-supervised contrastive learning for unsupervised phoneme segmentation. In INTERSPEECH 2020, 2020.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei- Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete & decomposed representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11200-11214, 2022.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Ann Lee. Peng-Jen Chen, Changhan Wang, Jiatao Gu. Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. Direct Proceedings of the 60th Annual Meeting of the Association Computational Linguisties (Volume 1: Long Papers), pp. 3327-3339, 2022a.
translation with discrete units. In
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino. Jiatao Gu, and Wei-Ning Hsu. Textless speech- to-speech translation real data. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 860-872, Seattle, United States, July 2022b. Association for Computational Linguis- maa.:m2.maacmmmm....... URL https: / acl ant org//022.222222
tics. doi: 10. naacl-me maimmmain... 63.
Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu. and Dominik Roblek. Real-time speech frequency bandwidth extension. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 691 691-695. IEEF, 2021.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. arXiv preprint arXiv:2203.16502, 2022.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models, In International Conference on Machine Learning, pp- 16784-16804. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2023
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03444 2016.
Taesung Park. Ming-Ya Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2337-2346, 2019.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In INTERSPEECH, 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre- training and Data Augmentation. In Proc. Interspeech 2022, pp. 5195-5199, 2022. doi:
1022443//meerspecch2222-- 11032.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap- Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 06/25, 2022.
Mr Murahari Reddy, Mr Sk Masthan Basha, Mr M Chinnaiahgari Hari, and Mr N Penchalaiah. Dall-e: Creating images from text. UGC Care Group I Journal, 8(14):71-75, 2021.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. International Conference on Learning Representations, 2021.
F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. CROWDMOS: An approach for crowdsourcing mean opinion score studies. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2416-2419, 2011. doi: 10. 10.1109//AASP..211....55 1.5226991.9999111
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Pho- torealistic text-to-inet to-tiiimagee diffusion models with deep language understanding. In Advances in Neural Information Processing Systems.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised Fre-Training for Speech Recognition. In INTERSPEECH, 2019.
Series. Method for the subjective assessment of intermediate quality level audio systems. International Telecommunication Union Radiocommunncation Assembly, 2014.
Yôiti Suzuki, Volker Mellert, Utz Richter, Henrik Møller, Leif Nielsen, Rhona Hellman, Kaoru Ashihara, Kenji Ozawa, and Hisashi Takeshima. Precise and full-range determination of two- dimensional equal loudness contours. Tohoku University, Japan, 2003.
13

---

Published as a conference paper at ICLR 2023
Aaron den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL ht.pps: http//pooceeeiiggg/e neurips cc.papprr2217/fi1ee pdf.
a 98 7 63a (eee00992eeee.d. (eeaaa00000eeeeeee32222e0eeeee00
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017b.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint urXiv: 2018.
Vasuki and PT Vanathi. review of vector quantization techniques. IEEE Potentials, 25(4): 39 47, 2006.
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data. In International Conference on Machine Learning, pp. 10937-10947. PMLR, 2021.
Ryuichi Yamamoto, Funwoo Song, and Jae-Min Kim. Parallel wavegan: fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1199-62033 IEEE, 2020.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Dift- sound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.0998. 2022.
Jaeseong You, Dalhyun Kim, Gyuhyeon Nam, Geumbyeol Hwang, and Gyeongsu Chae. Gan vocoder: Multi-resolution discriminator is all you need. arXiv preprint arXiv:2103.05236, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283 17297, 2020.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 0000000 2022.
14

---

Published as a conference paper at ICLR 2023
7 APPENDIX
7.1 DATASETS
We use a set of several datasets: AudioSet (Gemmeke et al., 2017), BBC sound effects Au- dioCaps (Kim et al., 2019), Clotho v2 (Drossos et al., 2020), VGG-Sound (Chen et al., 2020), FSD50K (Fonseca et al., 2021), 1), Free Use Sounds 9, Sonniss Game Effects 10 WeSoundEffects Paramount Motion Odeon Cinematic Sound Effects 12, All audio files were sampled at 16kHz.
Table 3: Datasets description. Duration is reported in hours for original audio, i.., before pre- processing.
DATASET
TEXT CONDITIONING
DURATION (II)
AudioSet
5.42k
BBC
captions
463
AudioCaps
captions
145
Clotho v2
captions
37
VGG-Sound
560
FSD50K
lags & captions
108
Free Use Sounds
tags & captions
Sonniss Game Effects
tags
85
WeSoundEffects
lags
12
Paramount Motion
lags
20
7.2 ADDITIONAL RESULTS
We present a visual qualitative depiction of conditional and unconditional audio continuation in Fig- ure 4. We use the following text as conditioning, "speech and a goat bleating", and the first second of its corresponding audio as prompt. We mark the start of the generated segment using a dashed white line. In Figure 4 (left) we visualize unconditional audio continuation (i.e., text). In Fig- ure (right) we visualize audio continuation conditioned text. As the audio prompts contains only human speech, the unconditional model generates continuation that contains speech utterances, but does not generate any goat sounds. In contrast, the model that is conditioned on text successfully generates both human speech and goat sounds (left).
4096
4096
2048
2048
Hz
1024
512
512
%
0
1.2
1.8 2.4 Time
3.6 4.2
4.8
1.8
2.4 Time
3.6
4.2
4.8
Figure 4: A visual example of the text guided audio continuation. We plot Mel-Spectrograms of both audio continuation without text conditioning (left), and textually guided audio continuation (right). Input text is: "speech and goat bleating".
In table 4 we report the complete results presented in Figure 2.
www.::ewct....b.bbbb....i..... co. co.uk/
"https:/ Phttess//sommssscc//gnneudddddd
"nttpss/wessunddffeets.com http:////o...etooe//o/oo///n//aa
15

---

Published as a conference paper at ICLR 2023
TEXT
AUDIO PROMPT DURATION (SEC.)
FAD
KL RND
KL SRC
SRC
1.97
1.49
no text
0.5
5,66
5.04
3,39
RND
0.5
3.08
2.38
3.85
SRC
1.96
1.35
text
4.70
4.95
2,86
RND
3.17
2.65
3.25
SRC
1.92
1.21
text
3.15
5.00
2.10
RND
2.92
3.13
2.47
SRC
1.90
1.09
text
2.46
5.04
1.58
RND
2.58
3.54
1.84
SRC
1.97
1.00
text
2.14
5.10
1.18
RND
2.29
4.37
1.34
Table 4: Full results of the FAD and KI. metrics for all text condition and audio prompt settings
Finally, we analyze the effect of model sizes when considering text encoder and ALM, on the generated audio. In Table 5 report KL and FAD scores for four different combinations: (TS-base, T5-large} x {ALM-base, ALM-large). When using a larger T5 encoder we observe big improvement in terms KI., and minor improvement in FAD. On the other hand, when using larger ALM, we see similar improvement in terms or KL with a significantly bigger improvement FAD. Using both Ia-lerge and ALM-large yields the best results overall.
T5
ALM
KI.
FAD
Base
Base
2.09
3.13
Base
Large
1.92
2.27
Large
Base
1.91
3.03
Large
Large
1.69
1.82
Table 5: Ablation study. We report KL and FAD scores for four different text-encoder and ALM setups, considering base and large models.
16


Reviews:
[{"review_id": "hPgpVU8OWe", "reviewer": "Reviewer_J2ki", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a system named AudioGen that can generate audio given text description of sound. The system is trained using a set of many audio datasets. Unlike (image, text) pairs, the size of (audio, text) pairs is not large enough, at least in the public domain. This limitation is tackled by using audio tag labels. Although this means what AudioGen does is rather a tag-based generation than text description, this is probably the best output using the currently available dataset. The loss is also carefully chosen, which seems to help. ", "summary_of_the_review": "I believe the paper is good enough to be accepted overall. I'll only describe some issues here.\n- After the text processing, is it correct to call them \"sentences\"? I find this a bit misleading. Although a description-based system does not need to fully reflect what is written in the description (it is probably not possible), the proposed system seems rather (open vocabulary) tag-based generation than sentence-based one.\n- Very minor issue, but In 4.3, last paragraph: \"AudioGen improves the KL score over.. higher FAD\" → it'd be easier to follow if the scores of DiffSound is presented again in this sentence.\n- In 5. Limitation, I don't think such a system is supposed to generate intelligible speech (it's not a TTS system.)\n- It'll be useful if the text preprocessing code is shared somewhere. Not every reader is knowledgeable on text processing.\n", "strength_and_weaknesses": "Strength:\n- Clever use of available datasets\n- Well designed and executed experiment that shows the system works well.\n- Limitation of the proposed system is given", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MyymJVdu_wK", "reviewer": "Reviewer_Tiv8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The combine existing methods (SoundStream audio codec, Transformer LMs) and datasets in a new way to demonstrate significantly higher quality generation of text-conditional environmental sounds than previous state of the art. Architecture and loss functions are largely the same to previous papers, but the extension to text-to-audio makes this paper a significant contribution to the community and a demonstration of the potential for follow-up research.\n\nThe authors also explore a variety of extensions to the task at hand (augmenting data by mixing audio clips/captions, speeding up inference through predicting multiple independent streams of audio tokens. The experiments are clearly laid out provide both quantitative and qualitative metrics to support the improvement over a baseline method (DiffSound) and the increase of quality with increasing model size. Technical limitations of the approach are appropriately explained.", "summary_of_the_review": "This paper presents an impressive empirical demonstration of text2audio using existing techniques. Datasets are carefully curated and methods selected to enable this demonstration, and it is of high significance to the ML community as it points towards the potential of pushing these techniques further, and the limitations of current datasets and methods. \n\nIn general, the experiments are carefully performed, with good metrics and ablations. Several important questions remain to ensure that the comparisons between models are fair, the value of techniques such as mixing data augmentation are justified, and the relative importance of scaling encoders / decoders are understood. Some discussion of future ethical implications would also help the limitations section of the paper.", "strength_and_weaknesses": "Strengths\n* Strong empirical results. On both quantitative and qualitative metrics, this paper clearly demonstrates a \"0-to-1\" improvement on a new task of text2audio, even given the limitations of labeled data and long sequence lengths.\n* Evaluation. Good choice of metrics for the experiments at hand, both quantitative and qualitative.\n* Ablation studies motivate the choice of CFG scale and trade-offs of the multi-stream generation strategy.\n\nWeaknesses\n* The benefits/need for the mixing data augmentation strategy is not well supported by experiments. The results in Table 1 from including mixing augmentation are, well \"mixed\" at best. More importantly, an important baseline is missing. For a situation with multiple sources, it would be good to compare to the baseline of generating several audio clips from a model trained without augmentation, and then mixing those outputs posthoc. Clearly this approach has the downside of taking longer to generate, but can handle arbitrary combinations of sources, regardless of the training. It also exposes a shortcoming of the current data augmentation strategy, as it ignores the fact that real audio often has many correlations between sound sources, including acoustic environment, background noise, and interactions in the natural scene. \n* The comparisons with model size in Table 1 could also benefit from clarifying what improvements are due to increasing the size of the decoder (ALM) vs. increasing the size of the encoder (T5). Currently they are both scaled together, and an ablation experiment could help to demonstrate where the benefits emerge.\n* It is unclear if DiffSound was retrained on the same datasets as AudioGen for the comparisons in Table 1. If the model is indeed trained on different datasets (using a pretrained checkpoint) then the comparison is less useful as a comparison of model architectures and needs to be highlighted as such. AudioGen is still an impressive empirical feat, but it would then be unclear how much is due to the model architecture vs. the datasets, and the claim couldn't be made that the AudioGen architecture outperforms the DiffSound architecture in this case. \n* The limitations section doesn't sufficiently address limitations of the datasets. Datasets with fine-grained multi-labels would be a better approach at capturing true scene dependencies than the linear augmentation. Further, the model will be significantly biased by the distribution of the training data in ways that cut across demographics and geography. As the quality of these models improve, this can present bias and representation challenges in a similar manner to text2image models currently, and it's important to note that in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "96JiU03aFMK", "reviewer": "Reviewer_pGs2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "AudioGen is a text to audio model, focusing primarily on non-speech, non-music audio events. It supports rendering multiple overlapping events simultaneously depending on the given text description and can generalize to unseen text/audio pairs by using a pre-trained language model encoder (T5).\n\nThe paper does several ablation studies exploring the effects of classifier-free guidance, text/audio training data mixing, and multi-stream decoding.", "summary_of_the_review": "This paper makes significant contributions to the relatively new field of text-to-audio. Overall the paper is clear and well written, but a few sections could be made more clear. I believe the paper will be influential in the field.", "strength_and_weaknesses": "Strengths:\n- The outputs of the model are remarkable and clearly follow the text prompts.\n- There is a clear improvement over previous work, with plenty of both quantitative and qualitative evaluations.\n- Ablation studies help determine the relative benefits of classifier-free guidance, mixing, etc.\n\nWeaknesses:\n- Some sections of the paper were not clear or difficult to follow (see next section).\n- Audio quality is still relatively low resolution (16 KHz).\n- Generated speech is unintelligible.\n- Exact timing of overlapping audio events is not controllable.\n\nThe weaknesses other than the first one are acknowledged in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1pSIQLt5t7", "reviewer": "Reviewer_iK2a", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a solution to generate audio from text prompts. The proposed solution uses a Vector-Quantized VAE, and in the latent space a language model is employed. ", "summary_of_the_review": "There is some unclear aspects I noted above. However I think this is a good quality work, and therefore it is above the acceptance threshold. ", "strength_and_weaknesses": "Strengths: \n-The results sound impressive. \n-Methodology seems clear. \n-The paper is relatively well written. \n\nWeaknesses: \n-The employed model seems to be very large, and the employed computational resources are enormous.\n-I am not sure if the comparison with diffsound is exactly fair. I am sorry if I am missing something, but it doesn't seems to be indicated how did you get the results with Diffsound. Also, a comparison on how much data each model uses. From what I understand you use more data than diffsound to train your model. Ideally it would be nicer to train the architecture of diffsound on the dataset you have curated. This would help to ascertain if your proposed architecture contributes more significantly than the size of the dataset. I am sorry if you have done this experiment, however from the manuscript I do not see it.  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster