Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a system named AudioGen that can generate audio given text description of sound. The system is trained using a set of many audio datasets. Unlike (image, text) pairs, the size of (audio, text) pairs is not large enough, at least in the public domain. This limitation is tackled by using audio tag labels. Although this means what AudioGen does is rather a tag-based generation than text description, this is probably the best output using the currently available dataset. The loss is also carefully chosen, which seems to help. 
strength_and_weaknesses: Strength:
- Clever use of available datasets
- Well designed and executed experiment that shows the system works well.
- Limitation of the proposed system is given
clarity,_quality,_novelty_and_reproducibility: - Clarity is high. Clear writing about what was done.
- Quality is high. The proposed system is well designed with state-of-the-art architectures and sensible, intuitive, and efficient idea of using the available datasets.
- Novelty is medium-high (or high enough)
- Reproducibility is high. In practice though, with the current computational cost, it is not easy to actually reproduce this work. 
summary_of_the_review: I believe the paper is good enough to be accepted overall. I'll only describe some issues here.
- After the text processing, is it correct to call them "sentences"? I find this a bit misleading. Although a description-based system does not need to fully reflect what is written in the description (it is probably not possible), the proposed system seems rather (open vocabulary) tag-based generation than sentence-based one.
- Very minor issue, but In 4.3, last paragraph: "AudioGen improves the KL score over.. higher FAD" → it'd be easier to follow if the scores of DiffSound is presented again in this sentence.
- In 5. Limitation, I don't think such a system is supposed to generate intelligible speech (it's not a TTS system.)
- It'll be useful if the text preprocessing code is shared somewhere. Not every reader is knowledgeable on text processing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a solution to generate audio from text prompts. The proposed solution uses a Vector-Quantized VAE, and in the latent space a language model is employed. 
strength_and_weaknesses: Strengths: 
-The results sound impressive. 
-Methodology seems clear. 
-The paper is relatively well written. 

Weaknesses: 
-The employed model seems to be very large, and the employed computational resources are enormous.
-I am not sure if the comparison with diffsound is exactly fair. I am sorry if I am missing something, but it doesn't seems to be indicated how did you get the results with Diffsound. Also, a comparison on how much data each model uses. From what I understand you use more data than diffsound to train your model. Ideally it would be nicer to train the architecture of diffsound on the dataset you have curated. This would help to ascertain if your proposed architecture contributes more significantly than the size of the dataset. I am sorry if you have done this experiment, however from the manuscript I do not see it.  


clarity,_quality,_novelty_and_reproducibility: -Some aspects regarding the experimental details are not super clear to me. For instance, as I wrote above I am not sure about under which exact conditions did you compare with Diffsound. Also it is not indicated how long does the training takes with the indicated number of GPUs. 

-The proposed architecture is not particularly novel from a technical standpoint, but the curation of several different aspects (dataset, design of the architecture, delivery of the results) make this work good quality. 

-Reproducibility is extremely difficult as the number of GPUs used in this dataset is very large. Also the code is not available.  
summary_of_the_review: There is some unclear aspects I noted above. However I think this is a good quality work, and therefore it is above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The combine existing methods (SoundStream audio codec, Transformer LMs) and datasets in a new way to demonstrate significantly higher quality generation of text-conditional environmental sounds than previous state of the art. Architecture and loss functions are largely the same to previous papers, but the extension to text-to-audio makes this paper a significant contribution to the community and a demonstration of the potential for follow-up research.

The authors also explore a variety of extensions to the task at hand (augmenting data by mixing audio clips/captions, speeding up inference through predicting multiple independent streams of audio tokens. The experiments are clearly laid out provide both quantitative and qualitative metrics to support the improvement over a baseline method (DiffSound) and the increase of quality with increasing model size. Technical limitations of the approach are appropriately explained.
strength_and_weaknesses: Strengths
* Strong empirical results. On both quantitative and qualitative metrics, this paper clearly demonstrates a "0-to-1" improvement on a new task of text2audio, even given the limitations of labeled data and long sequence lengths.
* Evaluation. Good choice of metrics for the experiments at hand, both quantitative and qualitative.
* Ablation studies motivate the choice of CFG scale and trade-offs of the multi-stream generation strategy.

Weaknesses
* The benefits/need for the mixing data augmentation strategy is not well supported by experiments. The results in Table 1 from including mixing augmentation are, well "mixed" at best. More importantly, an important baseline is missing. For a situation with multiple sources, it would be good to compare to the baseline of generating several audio clips from a model trained without augmentation, and then mixing those outputs posthoc. Clearly this approach has the downside of taking longer to generate, but can handle arbitrary combinations of sources, regardless of the training. It also exposes a shortcoming of the current data augmentation strategy, as it ignores the fact that real audio often has many correlations between sound sources, including acoustic environment, background noise, and interactions in the natural scene. 
* The comparisons with model size in Table 1 could also benefit from clarifying what improvements are due to increasing the size of the decoder (ALM) vs. increasing the size of the encoder (T5). Currently they are both scaled together, and an ablation experiment could help to demonstrate where the benefits emerge.
* It is unclear if DiffSound was retrained on the same datasets as AudioGen for the comparisons in Table 1. If the model is indeed trained on different datasets (using a pretrained checkpoint) then the comparison is less useful as a comparison of model architectures and needs to be highlighted as such. AudioGen is still an impressive empirical feat, but it would then be unclear how much is due to the model architecture vs. the datasets, and the claim couldn't be made that the AudioGen architecture outperforms the DiffSound architecture in this case. 
* The limitations section doesn't sufficiently address limitations of the datasets. Datasets with fine-grained multi-labels would be a better approach at capturing true scene dependencies than the linear augmentation. Further, the model will be significantly biased by the distribution of the training data in ways that cut across demographics and geography. As the quality of these models improve, this can present bias and representation challenges in a similar manner to text2image models currently, and it's important to note that in the paper.
clarity,_quality,_novelty_and_reproducibility: In general the paper is clearly written, with appropriate motivation, references, and equations. As mentioned in the summary, the novelty of the paper is primarily empirical, demonstrating the viability of text2audio. 

 Below I point out some things and questions that could be addressed to strengthen the clarity and reproducibility of the paper:

* In the section on page 5, "Multi-stream audio inputs" I inferred from the text that each stream is sampled independently given the logits for a frame of audio, is that correct? If so, it might clarify to make that explicit.
* In describing the datasets, it could help to make a table to clarify the datasets used, their size pre and post filtering, and any other characteristics that vary between datasets such as their primary composition (speech, birds, etc.) and examples of the labels.
* Figure 2 is a bit confusing compared to the rest of the paper. Especially the graph on the right, where "Rnd text" is used for labels for two lines, but I believe in one, it refers to the conditioning and in the other it refers to the reference. frankly, there's not a whole lot in those figures that isn't presented clearer in table 3. I believe the authors are trying to highlight the Delta KL between rnd text and source text in the left, and text cond and no text cond in the right. Just plotting that value itself with a single line would make things clearer I think, or just adding it to table 3 and removing figure 2.
* Table 2 likewise is a bit redundant with the encoder info duplicated for both rows. Perhaps consider splitting into two tables? Or at least noting the redundancy in the caption as they are independent on the size of the decoder.
* The paragraph right before "Limitations" compares AudioGen in Table 2 to DiffSound. It would help to add DiffSound to table 2 to make the comparison easier.


Small typos:
* page 2: "generation in two axis" -> "generation in two axes"
* page 3: "audio tokes" -> "audio tokens"
* page 6: "we remove annotator" -> "we remove annotators"
* page 8: "three encoding mehtods" -> "three encoding metrics"
* page 9: "Limitation" -> "Limitations"
* page 15: Caption says the text is "a crowd applauds followed by a woman and a man speaking", but the text above says it should be "speech and a goat bleating" which looks correct given the spectrogram.
summary_of_the_review: This paper presents an impressive empirical demonstration of text2audio using existing techniques. Datasets are carefully curated and methods selected to enable this demonstration, and it is of high significance to the ML community as it points towards the potential of pushing these techniques further, and the limitations of current datasets and methods. 

In general, the experiments are carefully performed, with good metrics and ablations. Several important questions remain to ensure that the comparisons between models are fair, the value of techniques such as mixing data augmentation are justified, and the relative importance of scaling encoders / decoders are understood. Some discussion of future ethical implications would also help the limitations section of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: AudioGen is a text to audio model, focusing primarily on non-speech, non-music audio events. It supports rendering multiple overlapping events simultaneously depending on the given text description and can generalize to unseen text/audio pairs by using a pre-trained language model encoder (T5).

The paper does several ablation studies exploring the effects of classifier-free guidance, text/audio training data mixing, and multi-stream decoding.
strength_and_weaknesses: Strengths:
- The outputs of the model are remarkable and clearly follow the text prompts.
- There is a clear improvement over previous work, with plenty of both quantitative and qualitative evaluations.
- Ablation studies help determine the relative benefits of classifier-free guidance, mixing, etc.

Weaknesses:
- Some sections of the paper were not clear or difficult to follow (see next section).
- Audio quality is still relatively low resolution (16 KHz).
- Generated speech is unintelligible.
- Exact timing of overlapping audio events is not controllable.

The weaknesses other than the first one are acknowledged in the paper.
clarity,_quality,_novelty_and_reproducibility: The text-to-audio field is still quite new, and this paper is a clear and novel step forward in that domain. For the most part, the paper is clear enough that it could be reproduced, though source code and pretrained models would help.

However, there are a couple sections that I found unclear:

**Section 3.2, Multi-stream audio inputs.**

It's not clear to me how this setup works. Do the multiple tokens predicted at the same time cover the same timestep? Is there any conditioning of the tokens on each other, or are they predicted independently? Is the RVQ implementation providing increasing levels of quality, as in Soundstream, or is it somehow related to decoding multiple timesteps at once?

I think some concrete examples and maybe a figure would help clarify this section.

**Section 4.2 Audio Continuation**

I found Figure 2 and its associated paragraph difficult to follow due to the combination of source/random text and audio and what they were being evaluated against. Also, I think there is a typo in the figure because there are 2 lines labeled "Rnd text". I think just being a little more explicit in the description would make things clear, describing exactly what was used for the audio prompt, the text conditioning, and the target/evaluation audio.

**Other comments**

Section 1, paragraph 2: Did you mean to say "astronaut riding a horse"?

Section 1, paragraph 3: When you that some compositions are unlikely to be in the training dataset, do you have any way of quantifying or testing that?

Section 1, paragraph 4: What size of pre-trained T5 did you use?

Section 3, paragraph 1: tokes -> tokens

Section 3.1: Just to clarify, does this mean that RVQ was not used other than in the multi-stream experiments? So a single token represents a single timestep? Could you also be more clear here about what the stride is? How much time does a token cover?

Section 3.1, Training Objective: I think it's worth discussing why the reconstruction loss is frequency magnitude only but the discriminator can also see phase.

Section 3.2: Instead of saying the architecture is like a decoder-only GPT-2, but with cross-attention added, wouldn't it be simpler to say it's an encoder-decoder Transformer with a frozen (T5) encoder?

Section 4.1 Dataset: Several of the quote marks are backwards.

Section 4.1 Dataset: Do you always train with input text of the form "dog bark park"? How does this work when the model is tested with phrases like "a dog barks while somebody plays the trumpet in a busy street”? Do you just rely on the T5 encoder to encode full sentences to the same representation as the sequence of individual words?

Section 4.1 Hyperparameters: Missing "hidden size" before "768".


summary_of_the_review: This paper makes significant contributions to the relatively new field of text-to-audio. Overall the paper is clear and well written, but a few sections could be made more clear. I believe the paper will be influential in the field.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: the paper combines existing methods (SoundStream, Transformer LM) and datasets in a new way to demonstrate significantly higher quality generation of text-conditioned environmental sounds. 

Strengths: The empirical results are strong. The presentation of the paper is clear and ablation studies are solid. 

Weaknesses: The novelty of the paper is primarily empirical. 
justification_for_why_not_higher_score: The novelty of the paper mainly lies in developing a high quality solution to the environment sound generation.
justification_for_why_not_lower_score:  The study and the presentation are solid. The evaluation, ablation studies and discussions on limitations are sound.
Decision: Accept: poster