Masked Audio Generation using a Single Non-Autoregressive Transformer
Authors: Alon Ziv
Itai Gat
Gael Le Lan
Tal Remez
Felix Kreuk
Jade Copet
Alexandre Défossez
Gabriel Synnaeve
Yossi Adi
Venue: ICLR 2024 poster (2024)
Keywords: Audio modeling
audio generation
music generation
non-autoregressive models
Primary area: generative models
TL;DR: We present MAGNeT, a fully non-autoregressive model for music and audio generation. MAGNeT reaches comparable performance to autoregressive models (e.g., MusicGen) but x7 faster during inference time.

Abstract: We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking scheduler, while during inference we gradually construct the output sequence using several decoding steps. To further enhance the quality of the generated audio, we introduce a novel rescoring method in which, we leverage an external pre-trained model to rescore and rank predictions from MAGNeT, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAGNeT, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNeT for the task of text-to-music and text-to-audio generation and conduct an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach is comparable to the evaluated baselines, while being significantly faster (x$7$ faster than the autoregressive baseline). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNeT, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.

Published as a conference paper at ICLR 2024
MASKED AUDIO GENERATION USING A SINGLE NON- AUTOREGRESSIVE TRANSFORMER
'Alon Ziv!.3, Itai Gat', Gael Le Lan', Tal Remez', Felix Kreuk', Alexandre Défossez? Jade Copet', Gabriel Synnaeve', Yossi Adi 13
FAIR Team, Meta
2 Kyutai
The Hebrew University of Jerusalem al clonzi @cs. wuwi..iaee aac.il ill
ABSTRACT
We introduce MAGNET, masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAG- NET is comprised of single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking sched- uler, while during inference gradually construct the output sequence using several decoding steps. To further enhance the quality the generated audio, we introduce novel rescoring method in which, we leverage external pre- trained model to rescore and rank predictions from MAGNET, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAG- NET, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNET for the task of text-to-music and text-to-audio generation and con- duet an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach comparable the evaluated base- lines, while being significantly faster (x7 faster than the autoregressive base- line). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNET, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality. Samples available cw..ujs.ii......... snrjj.aa ac / adi wosyyyosssssleeseooe ab/MAGNeT
demo
page https: / /pagapaaa..///////////////..ee.. cs. hujii.iii..
INTRODUCTION
Recent developments in self-supervised representation learning (Hsu al., 2021; Défossez al., 2022), sequence modeling (Touvron et al., 2023; Rozière et al., 2023), and audio synthesis (Lee al., 2022; Polyak et al., 2021) allow great leap in performance when considering high quality conditional audio generation. The prominent approach, in recent years, is to represent the audio signals compressed representation, either discrete continuous, and apply generative model on top of it (Lakhotia et al., 2021; Kharitonov et al., 2022; Borsos et al., 2023a; Kreuk et al., 2022a; Copet et al., 2023; Lam et al., 2023; Agostinelli al., 2023; Gat et al., 2023; Sheffer Adi, 2023; Maimon & Adi, 2022: Schneider et al., 2023; Huang et al., 2023b; Liu et al., 2023a; Li et al., 2023; Liu et al., 2023b). Recently, Défossez al. (2022); Zeghidour et al. (2021) proposed to apply a VQ- (Q-V directly the waveform using residual vector quantization discrete representation of the audio signal. Later on, Kreuk et al. (2022a); Wang et al. (2023); Zhang et al. (2023); Copet et al. (2023); Kreuk et al. (2022b) presented conditional language modeling on such audio signals representations. In parallel, Schneider et al. (2023); Huang et al. (2023b); Liu al. (2023a) proposed training a conditional diffusion-based generative model operating learned continuous representations of the audio signal obtained from a pre-trained auto-encoder model.
obtain multi-stream
Work was done part of Alon's 's internship FAIR.

---

Published as a conference paper at ICLR 2024
Overall, the family of generative models explores in prior work can be roughly divided into two: (i) autoregressive (AR) in the form of language models (LMs), usually operating on discrete audio rep- resentations; and (ii) diffusion- based models usually operating continuous latent representations of the audio signal. A Although providing impressive generation results, these approaches have several main drawbacks. Due its autoregressive nature, following the LMM approach yields relatively high inference time which turns into high latency values, hence making it less appalling for interactive applications such as music generation and editing under Digital Audio Workstations (DAW). On the other hand, diffusion models perform parallel decoding, however, to reach high-quality music sam- ples recent studies report using a few hundred diffusion decoding steps (Huang et al., 2023a; Liu et al., 2023b). Moreover, diffusion models struggle with generating long-form sequences. Recent studies present results for either 10-second generations (Liu al., 2023b; Li et al., 2023; Yang et al., 2022) or models that operate in low resolution followed by a cascade of super-resolution models to reach 30-second segments (Huang ct al., 2023a).
In this work, we present MAGNET, a short for Masked Audio Generation using Non-auvoreerressvve Transformers. MAGNET is a novel masked generative sequence modeling operating on a multi- stream representation of an audio signal. The proposed approach comprised of single transformer model, working in a non-autoregressiv fashion. During training, we first sample a masking rate from the masking scheduler, then, we mask and predict spans of input tokens conditioned un- masked ones. During inference, we gradually build the output audio sequence using several de- coding steps. We start from fully masked sequence and each iteration step, we fix the most probable token spans, i.c., the spans that got the top confidence score. To further enhance the quality of the generated audio, we introduce a novel rescoring method. In which, we leverage an external pre-trained model to rescore and rank predictions from MAGNET. Lastly, we explore Hybrid version of MAGNET, in which we fuse autoregressive and con-autoreeressive models. The hybrid- MAGNET generates the beginning of the tokens sequence in autoregressive manner while the rest of the sequence is being decoded in parallel, similarly to the original MAGNET. visual de- scription of the inference of the proposed method can be seen in Fig. 1.
Similar non-autoregressive modeling was previously proposed by Ghazvininejad et al. (2019) for machine translation, Chang et al. (2022) for class-conditional image generation and editing, and Chang et al. (2023) for image generation guided by rich textual description followed by a super- resolution component. Borsos et al. (2023b) recently proposed SoundStorm, non-autoregressive method for the task of text-to-speech and dialogue synthesis. SoundStorm is conditioned on "seman- tic" tokens obtained from an autoregressive model. Unlike, SoundStorm, MAGNET is composed of a single non-autoregressivv model and was evaluated on music and audio generation which, unlike speech, leverages the full frequency spectrum of the signal.
We evaluate the proposed approach considering both mext-to-music and text-to-audio generation. We report objective metries together with human study and show the proposed approach achieves com- parable results to the evaluated baselines while having significantly reduced latency (x7 faster than the autoregressive-basee method). We further present an analysis of the proposed method consid- ering latency, throughput, and generation quality. We present the trade-offs between the two when considering autoregressive and non-autoregressive models. Lastly, we provide an ablation study that sheds light on the contribution of each component of the proposed approach to the performance.
Our contributions: (i) We present a novel aoo--euoorerrsssvve model for the task of audio modeling and generation, denoted as MAGNET. The proposed method is able to generate relatively long sequences (30 seconds long), using a single model. The proposed approach has a significantly faster inference time while reaching comparable results the autoregressive alternative; (i)) We leverage an external pre- preinrrined model during inference to improve generation quality via a rescoring method; and (iii) We show how the proposed method can be combined with autoregressive modeling to reach a single model that performs joint optimization, denoted as Hybrid-M AGNET.
2 BACKGROUND
Audio representation. Modern audio generative models mostly operate on a latent representation of the audio, commonly obtained from compression model (Borsos al., 2023a; Kreuk et al., 2022a; Yang et al., 2022). Compression models such as Zeghidour et al. (2021) employ Residual Vector Quantization (RVQ) which results in several parallel streams. Under this setting, each stream
2

---

Published as a conference paper at ICLR 2024
is comprised of discrete tokens originating from different learned codebooks. Prior work, proposed several modeling strategies to handle this issue (Kharitonov et al., 2022; Wang et al., 2023).
Specifically, Défossez et al. (2022) introduced EnCodee convolutional auto-encoder with latent space quantized using Residual Vector Quantization (RVQ) (Zeghidour et al., 2021), and an adver- sarial reconstruction loss. Given reference audio signal Rk with d the audio duration and f. the sample rate, EnCodec first encodes it into a continuous tensor with a frame rate fr < fs- Then, this representation is quantized into E fI,...,, N "_.x-ff,, with K being the number of codebooks used in RVQ and N being the codebook size. Notice, after quantization we are left with K discrete token sequences, each of length T d fr, representing the audio signal. In RVQ, each quantizer encodes the quantization error left by the previous quantizer, thus quantized values for different codebooks are in general dependent, where the first codebook the most important one.
Audio generative modeling. Given díscrete representation of the audio signal, our goal is to model the conditional joint probability distribution po ( (ly) where y a semantic representation of the condition. Under the autoregressive setup we usually follow the chain rule of probability, thus the joint probability of a sequence can be computed as a product of its conditional probabilities:
2n (1) = II
21,8).
(I)
The above probability chain rule can be thought of as a masking strategy, where, in each time step i, we predict the probability of the i-th token, given its past tokens, while we mask future tokens. For that, we define a masking function m(i), that mask out all tokens larger than which results in:
po(21 ................................ znly)= II po(z:l(1 (i)))) O 2,y),
where each element in mi11 = [m (i)...... ( is defined as (t( = 1 1]. Notice, Eq. (2) does not hold for any masking strategy. One should pick masking strategy that satisfies the proba- bility chain rule.
Extending Eq. (2) to the non-autoregressive setup be done by modifying the masking strategy and the decomposition of the joint probability to predict an arbitrary subset of tokens given the unmasked ones using several decoding steps. Let us formally define the masking strategy as follows,
mi(1)= ~ 1 lj E M,J where MM~ ~ U(A (1-(AMM :44| = r(;;)) TT),
(3)
and 7 is the masking scheduler, with decoding steps, defined as >(i;s) cos( (((ttt)))) and Mo {I...., J. In other words, at each time step we mask a subset of (i; 8) ' tokens sampled from the masked set at the previous time step. Thus the modified version of Eq. (2) is,
Po(21 ... 2n v)
II po (1)) @ 21(1 m()) o 2, v)
(4)
In practice, during training, a decoding time step i E [1, and the tokens to be masked from Mo randomly sampled. The tokens indices .M; then replaced by a special mask token, and the model is trained to predict the target tokens at the masked positions M, given the unmasked
tokens. This modeling paradigm was previously explored by Ghazvininejad et al. (2019); Chang
et al. (2022; 2023); Borsos et al. (2023b).
Recall, the audio representation is composed of multi-stream sequences created by RVQ. which, the first codebook encodes the coarse information of the signal while later codebooks encode the quantization error refine the generation quality. To handle that, Borsos al. (2023b) proposed to predict tokens from codebook i given its preceding codebooks. During training, a codebook level k, is being uniformly sampled from {1, .1,... K). Then, we mask and predict the tokens of the k-th codebook given previous levels via teacher forcing. At inference we sequentially generate the token streams, where each codebook is being generated conditioned on previously generated codebooks.
3 METHOD
Following the approach presented in the previous section solely does not lead to high-quality audio generation. We hypothesize this is due three factors: (i) The masking strategy operates over
3

---

Published as a conference paper at ICLR 2024
Beacoring
Reacoring
Reacoring
EnCodez
RIO Tobend
Zeanafosrme
fsunsformm:
TransforrRE
Transfocmac
with an alectronte
Span
HPQ Tokeso
Kaaking
Maaking
Maakinng
Figure 1: Inference of MAGNET model. During each iteration, we mask a subset of token spans (starting from fully masked sequence). Next, rescore the tokens based on an external pre- trained model. Finally, we select the token spans to be re-masked for the next decoding iteration.
individual tokens that share information with adjacent tokens. Hence, allowing the model to "cheat" during tokens prediction while being trained using teacher forcing: (ii) The temporal context of the codebooks at levels greater than one, is generally local and influenced by small set of neighboring tokens. This affects model optimization; (iii) Sampling from the model at different decoding steps requires different levels of diversity with respect to the condition. Also sampling be combined with external scoring models.
In this section, we present MAGNET in details. MAGNET consists of non-autoregressive audio- based generative masked language model, conditioned on a semantic representation of the condition, operating on several streams of discrete audio tokens obtained from EnCodec. We follow similar modeling strategy as presented in Section while introducing core modeling modifications consist- ing of masking strategy, restricted context, sampling mechanism, and model rescoring.
3.1 MASKING STRATEGY
Adjacent audio tokens often share information due the receptive field of the audio encoder. Hence, we use spans of tokens as the atomic building block of our masking scheme, rather than individual ones done in prior work. We evaluated various span lengths / between 20ms to 200ms and found 60ms span length to give the best overall performance (see Section 5.3 for detailed results). We sample masking rate (i), from the scheduler, and compute the average amount of spans to be masked accordingly. As spans may overlap, this process requires careful design. We select the number of spans u, that satisfies expression is the expected masking rate over all possible placements of u spans of length / over the sequence. Full derivation similar strategy, in which we re-mask the least probable spans instead of individual tokens as done in prior work. We consider the span's probability as the token with the maximal probability. For computational efficiency, we use non-overlapping spans.
("z)/(") s 7()). where / is the span length. The above
be found in the Appendix C. During inference time, we follow
3.2 RESTRICTED CONTEXT
Recall, the used audio tokenizer is based RVQ, where each quantizer encodes the quantization emror left by the previous quantizer. Thus quantized codebooks later than the first one heavily de- pend on previous codebooks rather than surrounding tokens. To leverage that we analyze the used EnCodec and restrict the context of the codebooks accordingly.
Specifically, the audio encoder consists of multi-layer convolutional network and final LSTM block. Analyzing the receptive field for the used EnCodec shows that the receptive field of the con- volutional network is ~ 160ms, while the effective receptive field when including the LSTM block ~ 180ms. We empirically estimate the receptive field the model, using shifted impulse func- tion over time while measuring the magnitude of the encoded vector in the middle of the sequence.
4

---

Published as a conference paper at ICLR 2024
Fig. 3 in Appendix G depicts such process. Notice, although theoretically, the LSTM has an infinite memory, practically we observe it is bounded.
We utilize this observation to improve model optimization, by restricting the self-attention of code- books greater than 1, to attend only on tokens at a temporal distance smaller than 200ms. Similar ideas were proposed in the context of language modeling by Rae & Razavi (2020); Roy al. (2021). We depict the used attention map for the restricted context in Fig. 8.
3.3 MODEL INFERENCE
Sampling described in Eq. (3) uses uniform sampling to choose spans from the previously set of masked spans. In practice, we use the model confidence at the i-th iteration as a scoring function to rank all possible spans and choose the least probable spans be masked accordingly. However, the scoring function does not have to be part of the generative model.
A common practice in Automatic Speech Recognition (ASR) decoding, is generate of dif- ferent hypotheses from one model and rescore them using another model (Benesty et al., 2008; Likhomanenko et al., 2020). Inspired by the ASR rescoring method, we propose novel strategy in which at iteration i we generate a candidate token sequence using MAGNET, Then, we feed it to an external model and get a new set of probabilities for each of the token spans. Lastly, we use convex combination of both probabilities (the one emitted by MAGNET and the one obtained from the rescorer model). sample from:
p(=ly) = w (=( + (1 - w) Pressorer(2ly)-
(5)
In this work, we use MUSICGEN and AudioGen as our rescorering models (in non autoorerrreeevvvve manner). The proposed rescoring method is generic and is not tied any specific rescoring model. Following the proposed approach improves the generated audio quality and stabilizes inference. A pseudo-code of our entire decoding algorithm is described Fig. Appendix D.
Classifier-free guidance anncaling, Token prediction is done using a Classifier-Free Guidance (CFG) (Ho & Salimans, 2022). During training, we optimize the model both conditionally and unconditionally, while at inference time we sample from a distribution obtained by linear combi- nation of the conditional and unconditional probabilities.
While prior work (Copet et al., 2023; Kreuk et al., 2022a) used fixed guidance coefficient, À 1, instead use CFG annealing mechanism controlled by the masking schedule 7. As the masking rate (i) decreases, the guidance coefficient is annealed during the iterative decoding process. The motivation behind this approach to gradually reduce text adherence and guide the generation process toward the already fixed tokens. Intuitively, this transforms the sampling process from textually guided to contextual infilling. Formally, we use a CFG coefficient of
A(t) = 71) Àu + (1 (i)) A1
(6)
where Ao and A are the initial and final guidance coefficients respectively. This approach was also found to be beneficial in 3D shape generation (Sanghi et al., 2023).
4 EXPERIMENTAL SETUP
Implementation details. We evaluate MAGNET the task of text-to-muut generation and text- to-audio generation. We use the exact same training data as using by Copet et al. (2023) for music generation and by Kreuk et al. (2022a) for audio generation. dataset can be found on Appendix A.2. We additionally provide a detailed description about the datasets used train the evaluated basclines in Table 4.
detailed description of the used
Under all setups, we use the official EnCodec model as was published by Copet et al. (2023); Kreuk al. (2022a)*. The model gets as input audio segment and outputs 50 Hz discrete represen- tation. We use four codebooks, where each has codebook size of 2048. We perform the same text preprocessing as proposed by Copet al. (2023); Kreuk al. (2022a). We use pre- trained T5 Raffel et al. (2020) model to extract semantic representation from the text description and use it as model conditioning.
hhttps://gii wttph/cccomecccccccoeeeeeerrreee
ocra
5

---

Published as a conference paper at ICLR 2024
Table Comparison to Text-to- Music Baselines. The Mousai and MusicGen models were retrained on the same dataset, while for MusicLM use the public API for human studies. We report the original FAD for Au2, and MusicLM. For human studies, we report mean and C195.
MODEL
FADup
KL
CLAPss
OVL.
REL.
# STEPS
LATENCY (S)
Reference
92. 02.691.0.89
93,971082
Mousai
1.59
0.23
13.97
74.12:1443
200
44.0
MusicLM
4.0
84.03:1:.28
85.57
AudioLDM
3.1
1.20
0.31
77.6911.93
82,41
208
18.1
MUsiCGEN-small
3.1
1.29
0.31
84.68+1.45
83.89
1500
17.6
MUsIcGEN-large
3.4
1.23
0.32
85.65+1.51
84.12+1.12
1500
41.3
MAGNET-small
3.3
1.12
0.31
81.674:.72
83,21
180
4.0
MAGNET-large
4.0
1.15
0.29
84.26+144
84.21
180
12.6
We train non-autoregressive transformer models using 300M (MAGNET-smal)) and 1.5B (MAGNET-large) parameters. We train models using 30-second audio crops sampled at random from the full track. We train the models for 1M steps with the AdamW optimizer (Loshchilov & Hutter, 2017), a batch size of 192 examples, 31 = 0.9, 82 = 0.95, a decoupled weight decay of 0.1 and gradient clipping of 1.0. We further rely on D-Adaptation-biee based automatic step-sizes (Defazio & Mishchenko, 2023). We use cosine learning rate schedule with a warmup of 4K steps. Addi- tionally, we use an exponential moving average with decay of 0.99. We train the models using respectively 32 GPUs for small and 64 GPUs for large models, with float16 precision. For compu- tational efficiency, we train 10-second generation models with a batch size of 576 examples for all ablation studies. Finally, for inference, we employ nucleus sampling (Holtzman et al., 2020) with top-p 0.9, and a temperature 3.0 that linearly annealed to zero during decoding iterations. We CFG with condition dropout of 0.3 at training, and guidance coefficient 10.0 annealed to 1.0.
Evaluation metries. We evaluate the proposed method using the same setup as proposed by Copet et al. (2023); Kreuk et al. (2022a), which consists of both objective and subjective metrics. For the objective metrics, we use: the Fréchet Audio Distance (FAD), the Kullback-Leibe Divergence (KL), and the CLAP score (CLAP). We report the FAD (Kilgour al., 2018) using the official implemen- tation in Tensorflow with the VGGish model Following Kreuk et al. (2022a), we use a state-of- the-art audio classifier (Koutini et al., 2021) compute the KL-divergence over the probabilities of the labels between the original and the generated audio. We also report the CLAP score (Wu al., 2023; Huang et al., 2023b) between the track description and the generated audio quantify audio-text alignment, using the official CLAP model
For the human studies, we follow the same setup as in Kreuk al. (2022a). We ask human raters to evaluate two aspects of the audio samples (i) overall quality (OVL), and (ii) relevance to the text input (REL). For the overall quality test, raters were asked to rate the perceptual quality of the pro- vided samples in range of to 100. For the text relevance test, raters were asked to rate the match between audio and text on a scale of 1 to 100. Raters were recruited using the Amazon Mechanical Turk platform. We evaluate randomly sampled files, where each sample was evaluated by at least raters. We use the CrowdMOS package* to filter noisy annotations and outliers. We remove annotators who did not listen to the full recordings, annotators who rate the reference recordings less than 85, and the rest of the recommended recipes from CrowdMOS (Ribeiro et al., 2011).
RESULTS
5.1 TEXT-TO-MUSIC GENERATION
We compare MAGNET to Mousai (Schneider et al., 2023), MusicGen Copet et al. (2023). Audi- oLDM2 Liu al. (2023b) 5, and MusicLM (Agostinelli et al., 2023). We train Mousai using our dataset using the open source implementation provided by the authors
seteeccceeeooeeeeeeooeeeeeeeeeee .iittace
http://www.cowwlmmssoo/dddwwoll/
wwtpp//cwwcaaiiiaaaaaaaaaaaa/eee (September 2023)
Immlemennattion from hww..nhhwi..occciiiiiitaaaaaaaia (March 2023)

---

Published as a conference paper at ICLR 2024
1.00
60
Autoregressive sutoregressive
Somouaa sooe comminn
Autoing ressvv
Non-sutoregnessive
AR prompt
46.1
40
24.92
0.75
20
10
16
64
128
256
32 128
256
0.60
12 Latency (5)
16
Batch size
Batch size
(a) Latency
(b) Throughput
(c) Latency/FAD trade-off
Latency and throughput analysis: MAGNET is particularly suited to small batch sizes (up to 10 times lower latency than MUSICGEN), while MUSICGEN benefits from higher throughput for bigger batch sizes. MAGNET offers flexibility regarding the latency/quality trade off by allowing a customizable decoding schedule or following the Syyrid-MAGNER variant.
Figure 2:
Table presents the results of MAGNET on the task of text-to-music generation compared to var- ious baselines. Results are reported on the MusicCaps benchmark. As can be seen, MAGNET reaches comparable performance to MusicGen, which performs autoregressive modeling, while be- ing significantly faster both in terms of latency and decoding steps. When comparing to Audi- oLDM2, which is based on latent diffusion, MAGNET gets worse FAD and CLAP scores, while reaching better KL subjective scores. Notice, AudioLDM2 was trained on 10-seconds generation 16kHz while MAGNET was trained on 30-seconds generation at 32kHz. When we reduce the sequence length to 10-second generations our FAD reaches to 2.9 and CLAP score of 0.31. We additionally evaluate MAGNET on the task of text-to-audio generation (environmental sound gen- eration). Results and details regarding the baselines can be found in Appendix G. Results show similar trends as on text-to-music of MAGNET providing comparable performance to the autore- gressive baseline while being significantly faster.
5.2 ANALYSIS
Latency vs. Throughput. We analyze the trade-offs between latency and throughput as a function of the batch size, as illustrated in Fig. 2a and Fig. 2b. Latency and throughput are reported for generated samples of 10-second duration on an A100 GPU with 40 GB of RAM. Due to CFG, the batch size value is typically twice the generated sample count. Indeed the model outputs two distributions in parallel: one conditioned on the text prompt and the other unconditioned.
Compared with the baseline autoregressive model (red curve), the non- autorrrrrreees model (dashed blue curve) especially excels on small batch sizes due to parallel decoding, with a latency as low 600 ms for single generated sample (batch size of two in the 2a), more than 10 times faster than the autoregressive baseline. This is especially interesting in interactive applications that require low-latency. The non-autoregressive model is faster than the baseline up batch size of 64.
However, in scenarios where throughput is a priority (e.g. generate as many samples as possible, irrespective of the latency), we show that the autoregressive model is favorable. While the autoregressive model throughput is bounded to ~ 2.8 samples/second for batch sizes bigger than 64, the autoregressive model throughput is linear in batch size, only limited by the GPU memory.
Jybrid-MAGNE' Next, we demonstrate how hybrid version can also be combined. We boot- strap the non-autoregressivv generation with an autoregressive-genrrti single model that incorporates both decoding strategies. During training, we sample a time step t E {1...., ,T) and compute the autoregressive training loss for all positions that precede The rest of the sequence is being optimized using the MAGNET objective. This is done by designing custom attention mask that simulates the inference behavior during training (causal attention before t, parallel attention after t). During inference, the model can be used autoregressively to generate short audio prompt and switch to non-autoregressive decoding to complete the generation faster. A detailed deseription of the hybrid training can be found on Appendix E.
audio prompt. We train
We analyze the effect of the chosen in Fig. 2c using a 30-second generations without rescor- ing. Starting from fully non-autoregressive generation decoding, ablate the autoregressive- generated prompt duration. The results indicate that the longer the prompt, the lower the FAD. The Hybrid- Mindd-MAGI MAGNET is even able to outperform the full autoregressive baseline (when consid-

---

Published as a conference paper at ICLR 2024
Table Span length and restricted context ablation. We report FAD scores for MAGNET using an In-domain test set considering different span lengths, with and without temporally restricted context.
Span-length
2
Restricted context
x
X
X
FAD
3.07
2.13 |0.74 0.66 0.82 0.61 0.97 0.63
1.13
0.84
1.66
1.05
Table 3: We evaluate the effect of the rescorer model performance. We report mean and C195.
MODEL
RE-SCORE
FgDegg + Kt.
CLAPser
OVI. f
REL. t
LATENCY (s)
MAGNET-small
X
3.7
1.18
0.30
80.65
81.06:1.199
2.5
-AGNET-small
V
3.3
1.12
0.31
83.31
84.53
4.0
MAGNET-
X
4.2
1.19
0.30
82,19=1.23
82.96:00.1
8.2
MAGNETTlarge
V
4.0
1.15
0.29
1..4
85.57
12.6
ering FAD), starting from a I-second prompt while still being significantly faster (3.2s of latency down from 17.6s). This Hybrid strategy is another way control quality/latency trade-offs when performance of the non-autoregressive model does not quite match its autoregressive counterpart.
5.3 ABLATION
The effect of modeling choices. To validate our findings regarding the necessity of span masking for audio modeling, well the necessity of temporal context restriction for efficient optimization, we train different model configurations and report the resulting FAD in Table 2. Results suggest that using restricted context consistently improves model performance across all settings. Moreover, using a span-length of 3, which corresponds to spans of 60ms yields the best performance.
The effect of CFG annealing. Table 6 in the Appendix presents results computed over in-domain samples using several CFG coefficient configurations. We evaluate both constant CFG schedules, e.g. by setting Äg X 3, and annealing CFG. Results suggest that using Åg 10, A 1 yields the best FAD score over all evaluated setups. This finding aligns with our hypothesis that during the first decoding steps stronger adherence required, while later decoding steps we would like the model to focus on previously decoded tokens.
The effect model rescorer. Next, we evaluate the effect of model rescorering on the overall perfor- mance. Results are presented in Table 3. Results suggest that applying model rescoring improves performance for almost all metries. However, this comes the expense of slower inference time.
The effect of decoding steps. We explore the effect of less decoding steps on the overall latency and performance, see Fig. 7. seems that reducing the decoding steps for higher levels does not impact quality as much as for the first level. For scenarios where minimizing the latency is the top priority, one should consider only 1 step per higher codebook level: in such case, latency drops to 370 ms, the expense of 8% increase of FAD compared to 10 steps per higher levels.
Decoding visualization. We visualize the masking dynamics of MAGNET's iterative decoding process. In specific, we plot the mask mii) chosen by MAGNET during the generation of 10-second audio sample, for each itcration i E [l...., ,20.. As can be seen, MAGNET decodes the audio sequence in non-causal manner, choosing first sparse set of token spans at various disconnected temporal locations, and gradually "inpaint" the gaps until convergence to a full token sequence. Visualization and full details can be found in Appendix F.
RELATED WORK
Autoregressive audio generation. Recent studies considering text-to-audio generation can be roughly divided into two: (i) environmental sounds generation; and (ii) music generation. As for environmental sound generation, Kreuk et al. (2022a) proposed applying a transformer language model over discrete audio representation, obtained by quantizing directly time-domain signals using EnCodec Défossez et al. (2022). Sheffer & Adi (2023) followed a similar approach to Kreuk et al. (2022a) for image-to-audio generation. Dhariwal et al. (2020) proposed representing music samples
8

---

Published as a conference paper at ICLR 2024
in multiple streams of discrete representations using hierarchical VQ-VAE. Next, two sparse trans- formers applied over the sequences to generate music. Gan et al. (2020) proposed generating music for a given video, while predicting its midi notes. Recently, Agostinelli et al. (2023) proposed similar approach to M (Borsos et al., 2023a), which represents music using multiple streams of "semantic tokens" and "acoustic tokens". Then, they applied a cascade of transformer decoders conditioned on a textual-music joint representation (Huang et al., 2022). Donahue et al. (2023) fol- generation. Copet
lowed a similar modeling approach, but for the task of et al. (2023) proposed a single stage Transformer-basee autoregressive model for music generation, conditioned on either text or melodic features, based on EnCodec.
Non-autoregressive audio generation. The most common approach for non -on-autoregressive gener- ation is diffusion models. These models naturally apply over continuous representations however can also operate over discrete representations. Yang et al. (2022) proposed representing audio spec- trograms using a VQ-VAE, then applying a discrete diffusion model conditioned on textual CLIP embeddings for the generation part Radford et al. (2021). Huang et al. (2023b); Liu et al. (2023a;b) proposed using latent diffusion models for the task of text-to-audio, while extending to various other tasks such as inpainting, image-to-audie etc. Schneider et al. (2023); Huang et al. (2023a); Maina (2023); Forsgren & Martiros (2022); Liu et al. (2023b) proposed using a latent diffusion model for the task of text-to-music. Schneider al. (2023) proposed using diffusion models for both audio encoder-decoder and latent generation. Huang et al. (2023a) proposed a cascade of diffu- sion model to generate audio and gradually increase its sampling rate. Forsgren & Martiros (2022) proposed fine-tuning Stable Diffusion (Rombach et al., 2022) using spectrograms to generate five- second segments, then, using image-to-image mapping and latent interpolation to generate long sequences. Li et al. (2023) present impressive generation results using a latent diffusion model with a multi- task training objective, however for 10-second generation only.
The most relevant prior work to ours involves masked generative modeling. Ghazvininejad et al. (2019) first proposed the Mask-Predict method, a masked language modeling with parallel decoding for the task of machine translation. Later on, Chang et al. (2022) followed similar modeling strat- egy, denoted as MaskGIT, for the task of class-conditioned image synthesis and image editing, while Chang et al. (2023) extended this approach to high-quality textually guided image generation over low-resolution images followed by super-resolution module. Lezama et al. (2022) further proposed the TokenCritic approach, which improves the sampling from the joint distribution of visual tokens over MaskGIT. Recently, Borsos et al. (2023b) proposed the SoundStorm model, which has a similar modeling strategy as MaskGIT but for text-to-speech and dialogue synthesis. Unlike MaskGIT, the SoundStorm model is conditioned on semantic tokens obtained from an autoregressive model. The proposed work differs from this model as we propose a single non-autoregressive model, with a novel audio-tokens modeling approach for the task of text-to-audio. Another concurrent work, is VampNet (Garcia et al., 2023), a non-autoregressive music generation model. Unlike MAGNET, VampNet is based on two different models (one to model the "coarse" tokens and one to model the "fine" tokens), and do not explore text-to-music generation without audio-prompting.
DISCUSSION
Limitations. As discussed in section 5.2, the proposed non-autoregresssvve architecture targets low- latency scenarios. By design, the model re-encodes the whole sequence at each decoding step, even for time steps that have not changed between two consecutive decoding steps. This is a fundamental difference with autoregressive architectures that can benefit from caching past keys and values and only encode one-time step per decoding step, which efficiently scales when the batch size increases. Such caching strategy could also be adopted for non- -oooertoreerrssvvee architectures, for time steps that do not change between consecutive decoding steps, however, this requires further research.
Conclusion. In this work, we presented MAGNET which, to the best of our knowledge. is the first pure non-autoregressive method for text-conditioned audio generation. By using single-stage en- coder during training and a rescorer model, we achieve competitive performance with autoregressive methods while being approximately 7 times faster. We also explore hybrid approach that combines autoregressive and non-autoregressive models. Our extensive evaluation, including objective metrics and human studies, highlights MAGNET's promise for real-time audio generation with comparable or minor quality degradation. For future work, intend to extend the research work on the model rescoring and advanced inference methods. We believe this research direction holds great potential in incorporating external scoring models which will allow better non-left-to-righ model decoding.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS.
The authors would like to thank Or Tal, Michael Hassid and Nitay Arcusin for the useful theoret- ical discussions. The authors would additionally like to thank Kamila Benzina and Nisha Deo for supporting this project. This research work was supported in part by ISF grant 2049/22.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11322 2023.
J Benesty, J Chen, and Huang. Automatic speech recognition: deep learning approach, 2008.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.094 096666 2023b.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- cration via masked generative transformers. arXiv preprint htt.:2300.000004 2023.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. arXiv preprini arXiv:2306.05284 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems, 2022.
Aaron Defazio and Konstantin Mishchenko. Leamning-rate-iree learning by d-adaptation. arXiv preprint orXiv:2301.0770 00773,, 2023.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 20000:0000500331,, 2020.
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Gen- erating musical accompaniments from singing. arXiv preprint arXiv:2301. 2023.
Alexandre Défosse7, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. 13438, 2022.
S Forsgren and Martiros. Riffusion-stable diffusion for pall-time music generation. 2022. URL https://riffusion. com/about, 2022.
Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley mu- sie: Learning generate music from videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. Springer, 2020.
Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music gener- ation via masked acoustic token modeling. arXiv preprint arXiv:2307.0466 2023.
Itai Gat, Felix Kreuk, Tu Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi Augmentation invariant discrete representation for generative spoken language modeling. In /WSL.7. 2023.
10

---

Published as a conference paper at ICLR 2024
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel
decoding of conditional masked language models. arXiv preprint arxiv:1904.09324, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
arXiv preprini
arXhv:2207.12598, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mu- lan: A joint embedding music audio and natural language. arXiv preprint arXiv:2208, 12415, 2022.
Qingqing Huang, Daniel $ Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917 2023a.
Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. 12661, 2023b.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Pupers), 2019.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint arXiv:21 10.05069, 2021.
Felix Kreuk, Gabriel Synnaeve. Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniy Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv: /5352, 2022a.
Felix Kreuk, Yaniv Taigman, Adam Polyak, Jade Copet, Gabriel Synnaeve, Alexandre Défossez, and Yossi Adi. Audio language modeling using perceptually- guiided discrete representations. arXiv preprint arXiv:221 1.02221.112.2... 2022b.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 2021.
Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. arXiv preprint arXiv:2305. 15719, 2023.
Sang-gil gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint 24..482206.04688 2022.
11

---

Published as a conference paper at ICLR 2024
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. pettps:/ggithe com/ facebook / x rme rs, 2022.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa Improved masked image generation with token-critic. In European Conference on Computer Vision. Springer. 2022.
Peike Li, Boyu Chen, Yao Yao, Yikai Wang, Allen Wang, and Alex Wang. Jen-l: Text-guided uni- versal music generation with omnidirectional diffusion models. arXiv preprint arXiv:2308.0472) 04729, 2023.
Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010.11745, 2020.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023a.
Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with sepf-supe pretraining. arXiv preprint 21iv:2300.007344 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
2017.
Gallil Maimon and Yossi Adi. Speaking style conversion with discrete self-supervised units. arXiv preprint arXiv:2212.09730, 2022.
Kinyugo Maina. Msanii: High fidelity music synthesis on a shoestring budget. arXiv preprint wrXiv:2301.064668 2023.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.000355 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
Jack W Rae and Ali Razavi. Do transformers need deep long-range memory. arXiv preprint arXiv:2007.03356, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. Crowdmos: An approach for crowdsourcing mean opinion score studies. In IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2011.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/VVF Con- ference Computer Vision and Pattern Recognition, 2022.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguis- tics, 2021.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint 2308 /2950, 2023.
12

---

Published as a conference paper at ICLR 2024
Aditya Sanghi, Rao Fu, Vivian Liu, Karl DD Willis, Hooman Shayani, Amir H Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation feh--fidlitty and diverse shapes from natural language. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, 2023.
Flavio Schneider, Zhijing Jin, and Berhard Schölkopf. Mo usai: Text-to-music generation with long-context latent diffusion. arXiv preprint arXiv:2301.11757, 2023.
Roy Sheffer and Yossi Adi. hear your true colors: Image guided audio generation. In ICASSP 2023- 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint mrXiv:2301.02111 2023.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui Taylor Berg-Kirkpaune Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword augmentation. In IEEE International Conference Acoustics, Speech and Signal Processing, 2023.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diff- sound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.0990 09983, 2022.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint 2xXi:23033.09226 2023.
13

---

Published as a conference paper at ICLR 2024
Table 4: Details about the training sets used train the proposed method and the evaluated baselines.
MEtHOD
No. OF HOURS
SAMPLING RATE
DETAILS
MusicGen
20,000
32KHz
ShutterStock, Pond5, and proprietary data
MusicLM
280,000
24 kHz
Proprietary data
Mousai
2,500
48kHz
ShutterStock, Pond5, and proprictary data
AudioSet, WavCaps, AudioCaps, VGGSound, Free Music Archive, Million Song Dataset, LJSpeech, and GigaSpeech
DM2
29,510
16kHz
MAGNET
20,000
32kHz
ShutterStock, Pond5, and proprietary data
A EXPERIMENTAL SETUP
A.1 IMPLEMENTATION DETAILS
Under all setups, we use the official EnCodec model as was published by Copet al. (2023)². The model gets as input an audio segment and outputs a 50 Hz discrete representation. We use four codebooks where each has codebook size of 2048. We perform the same text preprocessing as proposed by Copet et al. (2023); Kreuk et al. (2022a).
We train non- autorrrrereeev transformer models using 300M (MAGNET-small) and 1.5B (MAGNET-large) parameters. We use a memory efficient Flash attention (Dao al., 2022) from the xFormers package (Lefaudeux al., 2022) to improve both speed and memory usage. We train models using 30-second audio crops sampled random from the full track. We train the models for 1M steps with the Adam W optimizer (Loshchilov & Hutter, 2017), a batch size of 192 examples, 31 0.9, 82 0.95, decoupled weight decay of 0.1 and gradient clipping of 1.0. We further rely on D-Adaptation based automatic step-sizes (Defazio & Mishchenko, 2023). We use a cosine learning rate schedule with warmup of 4K steps. Additionally, use exponential moving average with decay of 0.99. We train the models using respectively 32 GPUs for the small model and 64 GPUs for the large ones with f precision.
Finally, for inference, we employ nucleus sampling (Holtzman et al., 2020) with top-p 0.9, and a temperature of 3.0 that linearly annealed zero during decoding iterations. We use CFG with condition dropout rate of 0.3 during training, and a guidance coefficient 10.0 that is annealed to 1.0 during iterative decoding.
A.2 DATASETS
We follow the same setup as in Copet et al. (2023) and use 20K hours of licensed music to train MAGNET. Specifically, we rely on the same 10K high-quality music tracks, the ShutterStock, and Pond5 music data collections instrument-only music tracks. All datasets consist of full-length music sampled at 32 kHz with metadata composed of textual description and additional information such as the genre, BPM, and tags.
used in Copet et al. (2023) with respectively 25K and 365K
For the main results and comparison with prior work, we evaluate the proposed method the MusieCaps benchmark (Agostinelli et al., 2023). MusicCaps is composed of 5.5K samples (ten- second long) prepared by expert musicians and 1K subset balanced across genres. We report objective metrics on the unbalanced set, while we sample examples from the senre-balanced set for qualitative evaluations. We additionally evaluate the proposed method using the same in-domain test set as proposed by Copet et al. (2023). All ablation studies were conducted on the in-domain test set.
httpps://gin thub. hwtphhwww..o..ocoooecooc..hh.eeh facebook hww.orcccreeeecccceecaaa ocra ft www..hutterstckk.cmmm/// and www.pond5.com
14

---

Published as a conference paper at ICLR 2024
A.3 EVALUATION
Baselines. For music generation we compare MAGNET Mousai (Schneider al., 2023), Music- Gen Copet et al. (2023), AudioLDM2 Liu et al. (2023b), and MusicLM (Agostinelli et al., 2023). For Mousai, we train a model using our dataset for fair comparison, using the open source imple- mentation provided by the authors.. ?
Evaluation metrics. We evaluate the proposed method using the same setup as proposed in Copet al. (2023); Kreuk et (2022b), which consists of both objective and subjective metrics. For the objective methods, we use three metrics: the Fréchet Audio Distance (FAD), the Kullback-Leiber Divergence (KI.) and the CLAP score (CLAP). We report the FAD (Kilgour et al., 2018) using the official implementation in Tensorflow with the VGGish model 10 A low FAD score indicates the generated audio is plausible. Following Kreuk et al. (2022a), we use a state-of-the-e audio classifier trained for classification on AudioSet (Koutini al., 2021) to compute the KL-divergence over the probabilities of the labels between the original and the generated audio. For the music generation experiments only additionally report the CLAP score (Wu et al., 2023; Huang et al., 2023b) between the track description and the generated audio to quantify audio-text alignment, using the official pretrained CLAP model !!
For the human studies, we follow the same setup as in Kreuk et al. (2022a). We ask human raters evaluate aspects of the audio samples (i) overall quality (OVI.), and (ii) relevance to the text input (REL). For the overall quality test, raters were asked to rate the perceptual quality of the provided samples in range of 1 to 100. For the text relevance test, raters were asked rate the match between audio and text on a scale of to 100. Raters were recruited using the Amazon Mechanical Turk platform. We evaluate randomly sampled files, where each sample was evaluated by at least raters. We use the CrowdMOS package 12 to filter noisy annotations and outliers. We remove annotators who did not listen to the full recordings, annotators who rate the reference recordings less than 85, and the rest of the recommended recipes from CrowdMOS (Ribeiro et al., 1). For fairness, we include the same normalization scheme as proposed in Copet et al. (2023) of normalizing all samples at - 14dB LUFS.
B RECEPTIVE FIELD ANALYSIS
We present the receptive field analysis of the EnCodec model in Fig. 3. We slide an impulse function, in the form of a one-hot input vector, and measure the norm of the encoded latent vector in the middle of the sequence,
function of the temporal distance from the impulse. We perform the process twice: (i) For the full encoder (left) and (ii) while omitting the LSTM block and remaining only with the convolutional network (right). Fig. shows that the effective receptive field of EnCodec is upper bounded by 100ms in each direction, supporting our choice to design MAGNET's restricted transformer s.t. codebooks greater than one attend only tokens in a neighborhood of 100ms in each direction.
C SPAN MASKING
Sampling placement of token spans can be implemented by first sampling subset of indices from {1,... .1....,, ....., TT, serving as the span starts, and then extending each index to a span. Formally, we sample (u) ~ (HAA C [I,...,?}: |A| = w)), and then extend each index E [() to the span of
indices t,... t,..., 1. The total set of masked indices would be
M (1 A U Utt. ++
(7)
tE7(w)
'Implementation from hetlluacwwuuattiiiiiaaaiiillilip (March 2023) http://wwww.o.eoeooooooooeeeeeee
audio distance
"htpss//gituu..com//// wttp://hhhaa.aamaaaaooAaAAAAAAA
12 Pttt://www.cooddmss..gdddoobbbl
15

---

Published as a conference paper at ICLR 2024
70-
60
Zmd
Zmd
40-
30-
-200
-150
-100
-50
50
150
-200
-150
-100
-50
50
100
150
200
Impulse dt (ms)
Impulse dt (ms)
(a) EnCodec's middle latent vector's impulse
(b) The impulse response of the vector when omitting the LSTM block from the encoder.
sponse.
Figure 3: A visualization of the receptive field analysis.
Proposition C.1. Given random placement of u spans of size over sequence of length T, the expected masking rate is as follows,
M spans
(1W)1)
(LA (HAAACSIT...TAAALLL THA-=uH)
Derivation: First, note that for given token the probability that would remain unmasked, is the probability to choose u span starts only from the indices:
A (1...., 73 Xe 1
The total number of placements is (.), i.e., the number of possibilities to choose u span starts out of set of T indices without replacement. Similarly, the total amount of placements for which all span starts are in A., is (TT)). Thus,
p P|te MITUN (1())
(10)
Consequently, the masking probability for each token is (/T)) Finally, we define the indicator random variable t MNoe for each f E (1...T). and conclude the derivation by
T'
Ey00
M spans
[x();:)]
E
(EM
(11)
T
=
Ey(w)
(12)
(T-I)
1
T.
(13)
D MODEL INFERENCE
Fig, 4 presents the inference process of MAGNET. For clarity, omit CFG and nucleus sampling, and assume T is multiple of the span length To further ease the reading, we present the inference algorithm for single codebook, while in practice, Fig. 4 for every codebook k E {1...K) K).
16

---

Published as a conference paper at ICLR 2024
int, int, text: List, int, model: nn. mn.dodele, resocrers nn, Module, mask_id: int, temprr float, ws float)
Start from fully masked sequence
gen_seq torch. full ((B, 2), mask_ie, stypentorch.long!
n_apens // span_len
apans_ shape 18, s_apons)
apan_acores terch. seros (spans_ahape,, stype-torsh.tlottee
Run MAGNeT iterative decoding for 's' iterations for in sange(s):
mask_p torch.oos (math..pl ) r_masked_spans maxlint (mack_p n_spans), 11
Nasking
basked_spant sponessoress topk (n maammaaeeepaans, dim--1) indices
mask get_mask ispans_shaps, masked_spans) gon_seq(maask] maak_id
Forward pass
logits, probs compute e_treeettttttteeee Tona (gen_sequenne, text, ofg-True, temperature-tempr)
classifier free guidence with anneeling
logitogits (fackkk legits, annesling-
Sampling
= sampl cample_tone e_t_ee_ppppppppppppe p-top_P)
Place the sampied tokens in the masked positions mask = gen_seg mo mask_id
gen_seg placo_aampled aane__aaplled__tken imask, sampled_tokena(.... 0], gen_aeg)
sampled
proos = probs (prebs, ssmeled_tokens]
if Rescoring
peacorer_logite
compute ceciiteeppeeeeeee ct (gen_seg, text) procs preas (rescorer_ Lokens]
tescorer__ampled__pae
Final probs are the convex combination of probs and percorer__prbb sempled_probe rescocer_.empled
(1 sampled
Spon scoring max
span_scores get_apans__sone scores tsang probs)
Prevent remasking by placing -inf scores apan_scores
fill spans_mask, -1c5]
return gen
Figure 4: MAGNET's text-to-audio inference. MAGNET performs iterative decoding of steps. In each step. the least probable non-overlapping spans are being masked, where the probability is convex combination of the restricted traanstooffmrrr confidence and the probability obtained by the pre- trained rescorer. Finally, the span probabilities are re-updated, while assigning oc: to the unmasked spans, prevent its re-masking and fix it anchors for future iterations.
HYBRID-MAGNET TRAINING
The aim of Hybrid-MAGNET is to switch from autoregressive generation to non-autoregressive during inference, as generate an audio prompt with the same quality as MUSICGEN that can be completed fast using MAGNET inference. The goal is to find compromise between MUSICGEN quality and MAGNET speed, To give Hybrid-MAGNETT the ability to switch between decoding strategies, it requires few adaptations from MAGNET training recipe. One of them is to train jointly on two different objectives as illustrated by Figure 5. Similarly to Borsos et al. (2023b) a time step is uniformly sampled from (1...., T} that simulates audio prompt for MAGNET to complete from. For all positions that precede and for all codebook levels we propose to compute the autoregressive training objective. using causal attention masking. For all succeeding positions we keep the MAGNET training objective: the model can attend tokens from the audio prompt. Moreover the codebook pattern is adapted for the autoregressive generation work well, to that
MAGNET
17

---

Published as a conference paper at ICLR 2024
222
loss
76
892
37
1009
432
439
20
249
591
992
32
55
53
23
31
921
821
NAR loss
993
129
323
422
83
13
993
2
612
429
83
AR-generated prompt
NAR completion
Figure 5: Training of Hybrid-MAGNET.. During training random timestep é is sampled. For timesteps preceding a causal attention mask applied and cross entropy loss computed for all levels (blue highlighted squares). For timesteps succeeding the standard MAGNET training strategy is applied. Codebook levels are shifted following the delay pattern from Copet et al. (2023).
Decoding Timestep
Decoding Timestep
10
10
11
11
13
50
100
150
200
250
300
350
400
450
100
150
200
250
300
350
400
450
Temporal Timestep
Temporal Timestep
Decoding Timestep
Decoding Timestep
50
100
150
200
25.0
300
350
4000
100
150
200
250
300
350
400
450
Temporal Timestep
Temporal Timestep
Figure 6: Decoding visualization of the chosen anchor tokens function of decoding steps, for an iterative decoding process with = 20. We plot the mask m(i) chosen by MAGNET, for each i E (1...., s). during the generation of a 10-second audio sample for the text prompt "A dynamic blend of hip and orchestral elements, with sweeping strings and brass', The x-axis represents time while the y-axis represents the decoding steps.
end we use the delay pattern from Copet et al. (2023). Thus the temporally restricted context from MAGNET is adapted take into account the codebook level-dependent shifts.
18

---

Published as a conference paper at ICLR 2024
Table Text-to-Audio generation results. We report FAD and KL scores for all methods.
PARAMETERS
TEXT CONDITIONING
FADL
KL.}
DiffSound
400M
CLIP
7.39
2.57
AudioGen-base
285M
T5-base
3.13
2.09
AudioGen-large
1500M
TS-large
1.77
1.58
AudioL.DM2-small
346M
Ta-large, CLAP, ImageBind.
1.67
1.01
AudioLDM2-large
712M
T5-large, CLAP, ImageBind, PE
1.42
0.98
Make-an- anduioo
332M
CLAP
4.61
2.79
MAGNET
300M
T5- 15-large
3.22
1.42
M MAGNET-large
1500M
T5-large
2.36
1.64
1.2
steps
First codebook
1.1
Codebook >1
1.0
FAD
0.9
0.8
0.7
$10
ATRON
0.6
100
0.4
0.6
0.8
1.0
1.2
1.4
Latency (s)
Figure 7: Effect of the decoding schedules on the quality/latency trade off. We vary the number of decoding steps for first codebook level (dashed red curve) and higher codebook levels (dotted blue curve) around (20, 10, 10, 10) decoding schedule.
F ITERATIVE DECODING DYNAMICS
Fig. 6 presents the masking dynamics of MAGNET's iterative decoding process with 20. In specific, we plot the mask m²() chosen by MAGNET, for each i E {1...s), s), during the genera- tion of 10-second audio sample for the text prompt 'A dynamic blend of hip-hop and orchestral elements, with sweeping strings and brass'. To demonstrate MAGNET's stochasticity, we repeat the process several times. As can seen, MAGNET decodes the audio sequence in non-causal manner, choosing first a sparse set of token-spans at various disconnected temporal locations, and gradually "inpaint" the gaps until convergence a full token sequence.
G ADDITIONAL RESULTS
Text-to-audio generation We follow Kreuk al. (2022a) and use the exact same training sets to optimize MAGNET. We train MAGNET in two model scales, of 300M and 1.5B parameters respectively, and compare it AudioGen (Kreuk et al., 2022a), DiffSound (Yang al., 2022), AudioLDM2 (Liu et al., 2023b), and Make-an-Audio Huang et al. (2023b). Results are reported in Table 5. Results are reported on the AudioCaps testset (Kim al., 2019). All audio files were sampled at 16kHz. As can be see MAGNET results are comparable autoregressive alternative (AudioGenn while having significantly lower latency (the latency values the same in Table for MAGNeT, while AudioGen has the same latency MusicGen). For inference, different than the MAGNET models trained for music generation, we use top-p 0.8, an initial temperature of 3.5, and initial CFG guidance coefficient of 20.0.
slightly worse than the
The effect of decoding steps. The latency of the aon-autoregressive model can controlled by configuring the appropriate decoding steps, at the expense of quality degradation. In Fig. 7, report the in-domain FAD as a function of latency for different decoding steps. We ablate on the first codebook level step count (dashed red curve) and the higher codebook levels step count (dotted
19

---

Published as a conference paper at ICLR 2024
Figure 8: We restrict the attention maps to focus on local context for codebooks levels greater than
1. In this figure consider 2 time-steps restrictions for each side, in practice use time-steps for each side, resulting in 11 tokens.
Table 6: CFG annealing ablation. We report FAD scores for different λo A1 configurations, well as KL and CLAP scores.
A1
1
10
+
20 +
20
FAD +
3.95
0.99
0.63
0.61
0.80
0.68
K1.
0.79
0.60
0.56
0.56
0.57
0.56
CLAPar f
0.13
0.28
0.28
0.31
0.31
0.31
blue curve), starting from the (20, 10, 10, 10) decoding schedule. The red curve illustrates that good compromise between latency and quality can obtained around 20 steps for the first level, after which decoding further will only marginally lower the FAD, while adding latency. Regarding the higher codebook levels, we can observe an inflection point happening around 5 steps after which FAD remains stable. It is interesting note that reducing the decoding steps for higher levels does not impact quality as much as for the first level. For example the (10, 10, 10, 10) decoding schedule achieves 0.65 FAD
latency close that of the (20, 5, 5, schedule, which achieves lower FAD of 0.61 despite a smaller total step count.
The effect of CFG guidance annealing. We evaluate both constant CFG schedules. e.g. by setting Ao \ = 3, and annealing CFG. Results are presented in Table 6. Results suggest that using An = 10, A₁ = yields the best FAD score over all evaluated setups. This finding aligns with our hypothesis that during the first decoding steps stronger text adherence is required, while at later decoding steps we would like the model to put more focus on previously decoded tokens.
20


Reviews:
[{"review_id": "SKI5Sxln1a", "reviewer": "Reviewer_Vzfc", "summary": "In this paper, the author proposed MAGNET, a masked discrete generative model on top of RQV representations (EnCodec), able to perform audio generation at a faster inference speed than other baselines while reaching the same generation quality. To improve their model, the authors propose a list of techniques: 1. a masking strategy where instead of masking tokens, spans of tokens are masked, given that neighboring tokens are inter-dependent temporally paired with a restricted context on the self-attention for codes greater than one (given the convolutional nature of EnCodec and the limited receptive field of its LSTM in practice) 2. a re-scoring strategy at inference time, where the probability to mask a span is given by an external audio model (e.g., MUSICGEN and AudioGen) 3. an annealer for the classifier free guidance embedding strength. Additionally, the authors experiment with a hybrid model trained both with autoregression and masking. The resulting model results 10 times faster than the autoregressive baseline when using low batch sizes.", "questions": "- The scheduler $\\gamma(i; s) = \\cos( \\pi(i−1)/ 2s )$ is badly defined because computed on $i=1$ returns still $1$.\n- I do not understand why the new CFG anneal performs better.\n- Can the authors explain why the small 300M model performs better than the 1.5B model? I find this counterintuitive.\n- Additionally, can the authors explain better why the restricted context improves the metrics? Intuitively, the only advantage should be training/inference time improvement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed masking strategy, which uses token spans instead of individual tokens, is an important methodological contribution of the paper since it greatly impacts the performance of the proposed model.\n- The 10x speedup in latency is remarkable, especially in a setting like music, where it is required to sample many tokens, given the long context.", "weaknesses": "- A problem with the work is related to the title and the overall tone, in which the authors claim proposing a new type of audio/music model. In a paper titled \"Masked Audio Generative Modeling,\" it is supposed that the authors proposed for the first time a masked model for audio. Nonetheless, the papers introducing such an idea in audio are SoundStorm (which the authors cite correctly) and VampNet https://arxiv.org/abs/2307.04686 (which the authors should cite as concurrent work). The authors could change the title of the article to better delineate the paper as a set of improvements of such models (providing at least one of the two as baselines).\n- Except for the masking strategy, which I find important, the other introduced techniques seem relatively not critical or novel, especially the rescorer model, given that it improves the FAD only marginally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vi5Yd7AMiJ", "reviewer": "Reviewer_4Cc9", "summary": "MAGNeT constitutes a mask generative modeling approach for conditional audio synthesis, leveraging the neural audio codec (EnCodec) to transform audio waveforms into compressed and discrete sequences. The main contribution lies in transfering the masked generative modeling, as exemplified by MaskGIT in the image domain, to the audio domain. In my opinion, the fact that the methods are shown to be effective in other domains (image, for example) does not necessarily undermine the findings in this work. My assessment therefore is based on completeness and rigor in justification of design choices and experimental details.", "questions": "There have been several concurrent works using a similar approach, including SoundStorm, which is adequately mentioned in the manuscript. I would like to mention another concurrent work, VampNet[1], which applied MaskGIT-like training and a neural audio codec for music generation. While the comparison is not strictly necessary, it would be beneficial for the audience to have a summary of the differences between the models, such as highlighting the spanned mask prediction training and rescoring method of MAGNeT.\n\n[1] Garcia, Hugo Flores, et al. \"Vampnet: Music generation via masked acoustic token modeling.\" arXiv preprint arXiv:2307.04686 (2023).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The choice of spanned mask prediction training with restricted context for RVQ is well justified both intuitively and through ablation studies. The use of a shifted impulse function for analyzing EnCodec's latent vector is a welcomed addition, especially to better understand the dependency between multi-level RVQ tokens, which will inspire the community to improve the methods to train the codec model and/or the generative model alike.\n\nThe accuracy-latency tradeoff analysis for the first-level RVQ token versus the rest is also helpful in designing a better decoding strategy for the RVQ tokens. It is worth mentioning that such dynamics can potentially change when an improved codec model is developed. The analysis methods in MAGNet are generalizable to assess future-generation codec models as well, and will likely become useful tools to design an optimal generative model for the target codec model.", "weaknesses": "The ablation study is satisfactory overall, but I would also like to see quantitative analysis on the proposed annealed classifier-free guidance scale, which seems to be missing in the current manuscript. While intuitive, documenting the performance gain obtained by the method would make the paper more convincing and complete.\n\nAlthough the manuscript states that the training dataset is the same for several baseline models (Mousai and MusicGen), others (MusicLM and AudioLDM2) are trained on different data. Understanding the challenge in matching the music dataset (and open sourcing the data due to copyright concerns), further clarification on the training dataset for all baseline models via a complete paragraph and/or the table will be beneficial to gain a complete picture in the field of audio/music generative models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HqJsUeh1eY", "reviewer": "Reviewer_f8w3", "summary": "The proposed model (MAGNET) is a generative sequence modeling method for discrete audio tokens. It uses a single-stage, non-autoregressive transformer encoder during training to predict masked token spans and gradually constructs the output sequence during inference. The authors further introduce a model rescorer that leverages another pretrained model to rank predictions. A hybrid version combines autoregressive and non-autoregressive models, generating the initial part autoregressively and the rest in parallel. MAGNET operates on a multi-stream audio representation (EnCodec) and uses a single transformer model trained to predict spans of masked input tokens. During inference, it constructs the output audio sequence through multiple decoding steps. The method achieves competitive results in text-to-music and text-to-audio generation, with seven times faster latency compared to autoregressive methods, and offers insights through ablation studies.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors of this research paper address the challenging problem of generating long audio and music sequences, a task that has garnered significant attention due to its relevance in various applications, such as text-to-music and text-to-audio generation. Their proposed solution, known as MAGNET (Masked Audio Generation using Non-autoregressive Transformer), introduces several innovative techniques to tackle this problem effectively.\n\nOne key feature of MAGNET is its use of a training via masking strategy, a popular approach in contemporary self-supervised learning. This approach is particularly well-suited for the task of audio generation as it helps the model to learn dependencies and patterns in the data by predicting masked segments. In the case of MAGNET, the authors employ span masking, a technique rooted in speech and audio processing, which proves to be more effective than traditional token based masking. Span masking mitigates the issue of information leakage and results in more coherent and high-quality audio generation.\n\nMAGNET's underlying representation is based on EnCodec, which combines full self-attention with smaller attention spans derived from the impulse response characteristics of LSTM blocks. This hybrid approach allows the model to efficiently capture long-range dependencies while preserving local details, contributing to the generation of realistic audio sequences.\n\nA notable innovation in this work is the introduction of model rescoring through MusicGen and AudioGen. This technique enhances the robustness of the model during the sampling process, ensuring that the generated audio maintains high quality and coherence. Additionally, the authors incorporate annealing in the classifier-free guidance parameter, a strategy previously demonstrated to be useful in 3-D shape generation. This further contributes to the model's performance.\n\nThe authors conduct a comprehensive evaluation of MAGNET on both text-to-music and text-to-audio generation tasks. They use objective metrics such as FAD, KL Divergence, and CLAP scores to quantitatively assess the model's performance. Subjective metrics, derived from crowd-sourced evaluations and including Mean Opinion Score (MOS) and audio-text relevance assessments, provide a more human-centered perspective on the quality of the generated audio.\n\nTo further validate their approach and elucidate the rationale behind their design choices, the authors conduct ablation studies. These studies help in understanding the individual contributions of different components to the overall performance of MAGNET.\n\nFinally, the authors perform a comparison of MAGNET with existing methods, including MusicGen, Mousai, and AudioLDM2, providing a clear understanding of its strengths and weaknesses in comparison to state-of-the-art approaches.", "weaknesses": "The main weakness of this paper is that there is a heavy amount of engineering involved in the development of this model. While, it is definitely commendable, it makes reproducing the result extremely difficult for the rest of research community. Further, authors have not referred to this work titled \"Masked Autoencoders that Listen\" which has a very similar idea of span masking. It  would have been interesting to see the contrast and comparison against something that is designed with similar idea. Another issue I have with the paper is that the authors do not provide any details about the dataset used for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)