From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Authors: Robin San Roman
Yossi Adi
Antoine Deleforge
Romain Serizel
Gabriel Synnaeve
Alexandre Défossez
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion
audio
compression
TL;DR: A diffusion based approach to decoding compressed audio tokens

Abstract: Deep generative models can generate high-fidelity audio conditioned on various
types of representations (e.g., mel-spectrograms, Mel-frequency Cepstral Coefficients
(MFCC)). Recently, such models have been used to synthesize audio
waveforms conditioned on highly compressed representations. Although such
methods produce impressive results, they are prone to generate audible artifacts
when the conditioning is flawed or imperfect. An alternative modeling approach is
to use diffusion models. However, these have mainly been used as speech vocoders
(i.e., conditioned on mel-spectrograms) or generating relatively low sampling
rate signals. In this work, we propose a high-fidelity multi-band diffusion-based
framework that generates any type of audio modality (e.g., speech, music, environmental
sounds) from low-bitrate discrete representations. At equal bit rate,
the proposed approach outperforms state-of-the-art generative techniques in terms
of perceptual quality. Training and evaluation code are available on the facebookresearch/
audiocraft github project. Samples are available on the following
link (https://ai.honu.io/papers/mbd/).

From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Robin San Roman
Yossi Adi
Antoine Deleforge
Romain Serizel
Gabriel Synnaeve
Alexandre Defossez*
D: FAIR Team, Meta Universite de Lorraine, CNRS, Inria, LORIA, Nancy, France The Hebrew University of Jerusalem
Abstract
Deep generative models can generate high-fidelity audio conditioned on various types of representations (e.g., mel-spectrograms, Mel-frequency Cepstral Coef- ficients (MFCC)). Recently, such models have been used to synthesize audio waveforms conditioned on highly compressed representations. Although such methods produce impressive results, they are prone to generate audible artifacts when the conditioning is flawed imperfect. An alternative modeling approach is to use diffiusion models. However, these have mainly been used as speech vocoders generating relatively low sampling
(i.e., conditioned rate signals. In this work, we propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality (e.g., speech, music, envi- ronmental sounds) from low-bitrate discrete representations. At equal bit rate, the proposed approach outperforms state-of-the-art generative techniques in terms of perceptual quality. Training and evaluation code are available on the face- bookresearch/andiorraff github project. Samples available the following link
mel-spectrograms)
Introduction
Neural-based vocoders have become the dominant approach for speech synthesis due to their ability to
produce high-quality samples [Tan et al.] [02T]. These models are built upon recent advancements in neural network architectures such WaveNet (Oord et al.) 2016) and MelGAN [Kumar et al., 2019].
and have shown impressive results in generating speech with natural-sounding intonation and timbre.
In parallel, Self- SuperSupe Learning (SSL) applied to speech and audio data [Hsu et al. 2021 van den
Oord et al. 2019 have led to rich contextual representations that contain more than lexical content,
e.g., emotion and prosody information [Kharitonov et al., 2022 Kreuk et al. 2022a Generating
waveform audio from such representations hence new topic interest [Liu al. 2019 Polyak
al. 202T |Huang et al., 2022). This is often performed in two stage training pipeline. First, learn
audio representations using SSL objectives, then, decode the speech using Generative Adversarial
Networks (GAN) approach such as the HiFi GAN model [Kong et al. 2020a]. Even though these methods perform well, they are known to be unstable, difficult to train and prone to add audible artifacts to the output waveform.
Compression models [Zeghidour et al. 202 Défossez et al.) 2022 also be considered as SSL models that use the reconstruction loss as a way to learn meaningful representations of the data. Unlike models described before, compression models are trained in an end-to-end fashion, while learning
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Invertible EQ Processor
diffusion process
Split Bands
~ ~N(0.1)
reverse process
Figure 1: MULTI-BAND DIFFUSION process (resp. reverse process). The first step consists of reversible operation (EQ Processor) that normalizes the energy within frequency bands resemble that of a standard Gaussian noise. The audio is then filtered into non non-overlapping bands. Each band has its own diffusion process using a specifically tuned version of the proposed power noise schedule.
both audio representation (often discrete one) and synthesis, and model large variety of audio domains. They are optimized using complex combinations of specifically engineered objectives including spectrogram and feature matching, well multiple adversarial losses [Détossez et al.] 2022]. Even though they have impressive performance compared to standard audio codecs, c.g.. Opus Valin et al. 2012]. they tend to add noticeable artefacts when used at very low bit rates (c.g. metallic voices, distortion) that are often blatantly out of distribution.
After model optimization the learned representation also be used for different audio modeling tasks. Kreuk et al.] (2022b) presented a textually guided general audio generation. [Wang et al.] [2023] proposed zero shot text speech approach. Agostinelli al. [0233 demonstrated how such representation can be used for text-to-music generation, while Hsu et al. 2022 followed a similar modeling approach for silent video speech generation.
In this work, we present MUI.TI BAND DIFFUSION (MBD), a novel diffusion- hased method. The proposed approach generate high-fidelit, samples in the waveform domain of general audio, may it be speech, music, environmental sounds, etc. from discrete compressed representations. We evaluate the proposed approach considering both objective metrics and human studies. As we demonstrate empirically, such an approach can be applied to a wide variety of tasks and audio domains to replace the traditional GAN based decoders. The results indicate that the proposed method outperforms the evaluated baselines by significant margin.
We present novel diffusion based model for general audio synthesis. The proposed method based on: (i) a band-specific diffusion model that independently processes different frequency bands, allowing for less accumulative entangled errors; (ii) frequency equalizer (EQ) processor that reduces the discrepancy between the prior Gaussian distribution and the data distribution in different frequency bands; and (iii) A novel power noise scheduler designed for audio data with rich harmonic content. We conduct extensive evaluations considering both objective metrics and human study, demonstrating the efficiency of the proposed approach over state-of-the of-the-art methods, considering both GAN and diffusion based approaches.
Our Contributions:
Related work
Neural audio synthesis has been originally performed with sample level autoregressive models such as Wavenet [Oord et al. 2016). This type of architecture is notoriously slow and difficult to train. Speech synthesis is one of the dominant area of audio synthesis research. Vocoders are models designed to generate waveform speech from low level phonetic or acoustic features. Different approaches, often

---

conditioned on mel-spectrograms, have been explored for this task, including GAN-based methods such as HiFi-GAN [Kong et al. 2020a Kumar et al. 2019]. Polyak et al. [202T] used HiFi-GAN on other types of conditioning. This method generate speech conditioned on low bit rate representations learned from self-supervised methods such as HuBERT (Hsu et 2021] VQ-VAI vvn Niekerk et al., 2020] or CPC [van den Oord et al., 2019] together with the fundamental frequency Jo and a speaker embedding. By using only a few centroids for clustering, the speech content representation becomes largely disentangled from the speaker and the fundamental frequency (f0), enabling controllable speech generation.
Diffusion based vocoders are inspired by the recent success of diffusion for image generation (Ho al., 2020, Saharia al., 2022, Dhariwal and Nichol 2021 Ramesh al.. 2022 Kong al. [2020b] introduced Difewave, a irrtusion-basee vocoders, that applies the vanilla diffusion equations
to waveform audio. Compared with the adversarial approach, diffusion offers a simpler 1.2 Loss objective, and stable training. PriorGrad [Leee et al. 2021 is an extension of Diffwave that uses non standard Gaussian noise the diffusion process. The authors extract the energy of the conditioning mel-spectrogram and use it to adapt the prior noise distribution to the target speech. Wavegrad [Chen et al. 2020] is similar but uses conditioning on continuous noise levels instead of discrete ones. This allows the model to perform the sampling using any noise schedule with a single training. Takahashi et al. [2023] look at singing voices, which is a more complex distribution than standard read speech due to wider spectrum, and increased diversity. Inspired by super-resolutic
cascaded techniques from image diffusion (Ho et al. 2022], they used hierarchical models. The first diffusion model synthesises at a low sampling rate while later ones, conditioned on the output of their predecessor, perform upsampling. This process can yield high-quality, high-resolution audio samples. Recent work [Pascual al. 2022 applies diffusion generating full band audio at high sampling rate, although the proposed methods allows for unconditional generation, and flexible style transfer, remains limited narrow range of audio modalities.
Most diffusion models that sample data from complex high dimensional distributions use upsampling frameworks (Huang et al. 2023 Takahashi et al. 2023 This type of cascaded models are achieving good performance but they are based on series of diffusion processes conditioned on the output of the previous and thus can not be performed in parallel. In vision, some efforts have been invested in simplifying diffusion pipelines. SimpleDiffusion [Hoogeboom et al., 2023] presents a framework that matches results of cascading diffusion models using single model. The model architecture and training objective are adapted to focus on ow-frequency content while keeping high quality textures. To the best of our knowledge, this type of idea has not been ported to audio processing as of yet.
Finally, our work offers an alternative to the decoder of adversarial neural audio codecs such as SoundStream [Zeghidour et al., 2021] and EnCodec [Défossez et al., [022], which consist in an encoder, a quantizer, and a decoder, and are trained with combination losses including discriminators. spectrogram matching, feature matching. and waveform distances. Our diffusion based decoder is compatible, but offers higher quality generation as measured by subjective evaluations.
3 Method
3.1 Background
Following Ho et al. 2020], we consider a diffusion process given by a Markov chain where Gaussian noise is gradually added to corrupt clean data point zo until a random variable zy close to the standard Gaussian noise obtained. The probability of the full process is given by
(2002120) II (
where (( ~ WWI--WIVV--- B, Grl) and (8:)osesT is usually referred to as the noise sched- ule. One can efficiently sample any step of the Markov chain with
VaLzo + VI CGE,
(2)
where ã. = -I'' (1) is called the noise level and & ~ N (O, 7). Denoising Diffusion Proba- bilistic Models (DDPM) aims going from prior noise ør to the clean data point zo through the
3

---

reverse process
T p(æT:o) p(TT) II pa (o(
where (ay 2++1) ) is a learned distribution that reverses the diffusion chain 4 (z,) and p(ær) is the so-called prior distribution that not learned. Under the ideal noise schedule, one can see from eq. (2 that the prior distribution can be approximated by (0,, T).
Ho al. [2020) show that the distribution (1(22=1(2z) can be expressed N (N (Zre f), ozl) where 14g can be reparameterized as follow:
1
Bg VT
po(x,,
Ft
Eo (2z, c0( a4
VT-
This reparametrization allows to train a neural network Eg to predict the noise in the corrupted data point z2. To train this neural network, consists in sampling æt using eq. (2) and optimizing the following 1.2 loss:
use the simple objective given by Ho al. 2020 that
L =
(Ile co(vo + v1 âte, 1)1
(5)
With such a model, one can reverse the diffusion process iteratively using the following equation:
1
Be V1 - ār co(x)))) + Vote,
St
St
(6)
v1
3.
where a is a parameter that should be chosen between Br = 30= (l--0-1)/(1 at) and 3 Ho et al. 2020 In our experiments we always use
Bt.
3.2 Multi-Band Diffussio
The MULTI-BAND DIFFUSION method is based on three main components: (i) Frequency EQ processor; (ii) Scheduler tuning; and (iii) Band-specific training, which we now describe.
Frequency Eq. Processor The mathematical theory of diffuusin processes allows them to sample from any kind of distribution, regardless of its nature. However, in practice, training a diffusion network for multiple audio modalities in the waveform domain is an open challenge. We make the sumption that the balance of energy levels across different frequency bands in both the prior Gaussian distribution and the target distribution is important obtain an efficient sampling mechanism.
white Gaussian noise signal has equal energy over all frequencies. However natural sounds such as speech and music do not follow the same distribution SSchuup et al. [201T), i.e. music signals tend to have similar energy level among frequency bands that are exponentially larger. For signals of the same scale, white noise has overwhelmingly more energy in the high frequencies than common audio signals, especially at higher sample rate (see Fig. 2 Thus during the diffusion process, high frequency content will disappear sooner than the low frequency counterpart. Similarly, during the reverse process, the addition of noise given by (6) will have more impact over the high frequencies.
To resolve this issue, we normalize the energy of the clean signal, denoted as Ty, across multiple frequency bands. We split 2o into components b, with cascade of band pass filters equally spaced in mel-scale. Given the filtered band b, of an audio signal, we normalize as follow,
0 = b;
where of and of denote the energies in the band for standard Gaussian noise and for the signals the dataset, respectively. The parameter P controls to what extend we align the energy levels. For p 0 the processor does not do any rebalancing and p 1 corresponds to matching exactly the target energy. Given that speech signals often have no content in the high frequency bands, we compute the parameters of over the music domain to avoid instabilities in

---

Original
-5
Processed
Noise
-15
Energy (d6)
-20-
-25
-30
-35
-40
2
6
Mell Bands
Figure 2: Standard deviation in 8 mel scale frequency bands (from lows to highs). For data from our dataset (Original), Equalized data (Processed) and for standard Gaussian Noise (Noise).
13
54-
209 600
600
2009
Figure 3: Left curves depict a comparison of the noise level ā along the diffusion process for cosine schedule and our power schedule. Right figure presents spectrograms along diffusion process. The top row is our power schedule and the bottom row follows cosine schedule.
Scheduler Tuning. The noise schedule is known to entail an important set of hyperparameters in diffusion models, and to play a critical role in the final quality the generation [Karras et al. 2022)
A common approach when generating raw waveform is using either linear or cosine schedules Nichol and Dhariwal 2021). Such schedulers performs good for read speech where the frequency spectrum not wide or for www-samliing-ratee generation followed by cascaded models that iteratively upsample the signal. In preliminary experiments, we found such schedules performs poorly when generating signals at high sampling rate. Hence, we argue that one should prefer more drastic schedule. We propose to use p-power schedules, defined as:
Be (V BBB [(VB--
(8)
where the variance of the injected noise at the first and last step (99 and 3r) are hyperparameters. One could assume that, since the noise schedule used at generation time can be chosen after training, is unnecessary to tune the training noise schedule and only focus on the choice of subset S. As evoked by Chen et al. 22020]. in practice, the training noise schedule is crucial element of diffusion models. Since the train-time steps are sampled uniformly, the training noise schedule parameterizes the sampling distribution of the noise level Vā. As in Fig. 3 using the proposed power schedule will results in sampling most of the training examples very small amount of noise (i.e. very high ā).
We noticed that for time step / close to T, i.e. at the end of the diffusion process, the model estimate <0(x)) of the noise is often worse than simply using x, itself. We hypothesize this is due to the limited precision when training the model. In that regime, the model can advantageously be replaced by the identity function, which is equivalent to skipping those timesteps entirely. We thus choose the 3 values such that /1 ag is large enough avoid this phenomenon.
-and-SSeccfiic Training. Similarly to the findings in image diffusion models [Song et al. 2020 )
audio diffusion models first generate low frequencies and then address high frequencies during the final stages of the reverse process. Unlike images where high frequencies are only locally connected, audio data contains complex entanglements of spectrogram content across both time and frequency
5

---

[Schnupp et al., 2011]. As result, training diffusion model on full-band audio data would always provide the ground truth low frequencies when generating high frequencies. ends up amplifying the errors committed at the beginning of the generation when unrolling the reverse process.
Following that, proposed training each frequency band independently, denoted MUL.TI-BAN BAND DIFFUSION, Through preliminary experiments we found such an approach resulted in significant improvements in the perceptual quality of the samples. Interestingly, dividing the frequency band along model channels did not yield the same quality improvements. This observation supports our intuition that, by not providing the model with previously generated content (lower frequencies) during training, the model can avoid accumulating errors during sampling.
4 Experimental Setup
4.1 Model & Hyperparameters
Overview. Our approach serves replacement for EnCodec's decoder. This approach offers the advantage of flexibility and compatibility. allows one to switch between the original decoder and the proposed diffusion decoder depending on the required trade-off between quality and generation
time.
Architecture. Similarly to Chen et al. 2020 Kong et al. [2020b] Lee et al. [2021] we use fully convolutional symmetrical U-net network Ronneberger et al., 2015] with an alternation of two residual blocks (Détossez et al., |021] and downsampling (resp. upsampling in the decoder) convolutional blocks of stride 4. The input audio conditioning is incorporated in the bottleneck of the network whereas the timestep is embedded through learned lookup table and added every layer. According to the recommendation of [Hooebooo et al.] (2023), it is advisable to allocate additional computational resources close to the bottleneck of the network when applying diffusion to high-dimensional data. Hence, we opted for a growth rate of 4. The weight of one model is GB. A visual description of the model architecture can be seen in Fig. A.4 in the Appendix.
the publicly available EnCodec models at 24kHz [Défossez et al. [022] which are frozen during training. The embedding sequence is upsampled using linear interpolation to match the dimension of the UNet bottleneck. In the experiments we include reconstructions using I, 2 and for EnCodec code books which correspond to bit rates of respectively 1.5kbps, 3kbps and 6kbps, when using multiple code books the embedding used is simply the average of the different code books.
Input Conditioning. We use the latent representation
Schedule. We trained our diffusion models using our proposed power schedule with power p = 7.5, 800 = 1.0e-5 and B7 = 2.9e-2. Although we use very few diffusion steps (20) at generation time, we observed that it is beneficial to use many steps at training time (1000). First, increases the versatility of the model since one can sample using any subset of steps S C (1,.. ...... 1000]. Second, it allows the model to be trained on a more diverse range of noise levels / Vāz. In the experiment section we always use the simplest time steps sub sampling i.e. S {i 1000 i {0, 1, NJ} where N is the number of sampling steps (20 if not precised).
Frequency EQ processor. In the experiments we use a band processor that uses 8 mel scale frequency bands with p = 0.4. We compute the values of the bands of on an internal music dataset.
Band Splitting. As described in 3 we use separate diffusion processes. In this work we always use a split of 4 frequency bands equally space in mel-scale using julius I Those bands are not related to the processor bands. The 4 models share the same hyperparameters and schedule. All models take the same ENCODEC tokens as conditioning input.
Training. We train our models using Adam optimizer with batch size 128 and a learning rate of le-4. It takes around 2 days on 4 Nvidia V100 with 16 GB to train one of the 4 models.
Computational cost and model size. Diffusion model sampling has an intrinsic cost that due to the number of model passes that are required for generation. We provide in A.ble the details for time consumption and number of parameters of MULTI-BAND DIFFUSION.

---

4.2 Datasets
We train on diverse set of domains and data. We use speech from the train set of Common Voice 7.0 (9096 hours) AAdila et al. 2019] together with the DNS challenge 4 (2425 hours) [Dubyy et al, 2022] For music, we use the MTG--ammeno dataset (919h) [Bogdanov et al., 2019]. For the environmental sound we use FSDSOK (108 hours) [Fonecca et al., 2021] and AudioSet (4989 hours) [Gemmeke et al. 2017]. We used AudioSet only for the research that is described in the publication and for the benefit of replicability. For evaluation, we also use samples from an internal music dataset.
4.3 Evaluation Metries
Human evaluation. For the human study we follow the MUSHRA protocol [Series 2014], using hidden reference and a low anchor. Annotators were recruited using a wwowd-soureing platform. in which they were asked to rate the perceptual quality of the provided samples in range between to 100. We randomly select 50 samples of 5 seconds from each category of the the test set and force at least 10 annotations per samples. To filter noisy annotations and outliers remove annotators who rate the reference recordings less then 90 in at least 20% of the cases, or rate the low-anchor recording above 80 more than 50% of the time.
Objective metrics. We use two automatic evaluation functions. The first one is the standard ViSQOL [Chinen et al. 2020) metric. B The second one is novel metric we introduce to measure the fidelity of the mel- meelsppetrogram of the reconstructed signal compared with the ground truth across multiple frequency bands. Let take reference waveform signal E RT. We normalize x to have unit variance, and use the same scaling factor for à. We take the mel-spectrogram of both, computed over the power spectrum with M mels, and hop-length H, e.g..
Rº and reconstructed signal
mel
and
8 = mel
>
6
with 2 FXT/H We compute the mel-spectrogram distortion 8 2. Finally for each time step and frequency bin f. we can compute a Signal-To-Noise ratio. In order to avoid numerical instabilities, and also not let the metric be overly impacted by near zero values in the ground truth mel-spectrogram, we clamp the SNR value between B5d +2ddB, considering that any distortion lower than -25 dB would have a limited impact on perception, and that beyond +25 dB, all distortions would be equally bad. Indeed, due to the limited precision used in the computation and training of neural network, it is virtually impossible to output a perfectly null level of energy in any given frequency band, although such empty bands could happen in real signals. Finally we get,
clamp [10- (logıa (2) Iogio (8)) 25dd +25dB]
(10)
We then average over the time steps, and split the mel-scale bands into 3 equally spaced in mel-scale. We report each band as Mel-SNR-L (low frequencies), Mel-SNR-M (mid frequencies), and Mel- SNR-H (high frequencies). Finally we also report the average over all 3 bands as Mel-SNR-A. At 24 kHz, we use STFT over frames of 512 samples, with hop length H 128 and N 80 mel bands.
5 Results
5.1
Multi modalities model
We first evaluate the performance of our diffusion method compared with ENCODEC on a compression task. Specifically we extract audio tokens from audio samples using the ENCODEC encoder and decode them using MULTI-BAND DIFFUSION and the original decoder.
We perform subjective evaluation on four subsets: 50 samples of clean speech from DNS, 50 samples of corrupted speech using DNS blended with samples from FSD50K, 50 music samples from Jamendo and 50 music samples from an internal music dataset. All speech samples are reverberated with probability 0.2 using room impulse responses provided in the DNS challenge. 3 subjective studies with different bit rate levels: 6kbps, 3kbps, and 1.5kbps. Note that scores should not be compared across the studies since ratings are done relatively to the other samples of the study. We include Opus [Valin et al. 2012] at 6kbps as a low anchor and the ground truth samples. Even
Table we present
We compute visgol with: https: / ht////gtthuttt com./ooogee/visqo] using the recommended recipes.
7

---

Table 1: Human evaluations (MUSHRA) scores for 24kHz audio. The mean and CI95 results are reported. The Opus low anchor and ground truth samples are consistent across all three studies, delimited by horizontal lines. The other methods used a bit rate of 6kbps for the top study on top, 3kbps for the middle one, and 1.5kbps for the bottom one. Higher scores indicate superior quality.
Method
Speech
Music
Average
Reference
93.86=0.014
92.93=00021
93.40
Opus
0.004
3..24
47.69
ENCODEC
79.03
84.67 = 0.062
81.85
MBD (ours)
8.068
83.61
84.15
Reference
93.17=0.015
94.45
93.81
Opus
62.83
36.17
49.5
ENCODEC
18.51=0.070
85.55
82.03
MBD (ours)
0.042
87.3 0.041
85.87
Reference
94.65=0.012
94.7 100022
94.78
Opus
44.65=0.057
38.33
41.49
ENCODEC
49.51 0.072
75.98 0000
62.75
MBD (ours)
6..83_0.056
75.29
70.56
Table 2: Human evaluations (MUSHRA) scores for 24kHz audio. The mean and C195 results reported. The first part of the table reports different methods of ENCODEC tokens at 6kbps decoding while the second part adds other independent compression baselines at 6 kbps.
Method
score
Ground Truth
90.32 1.39
MBD
85.16 1:093
Encodec
82.73:11.11
PriorGrad
65.16::2.2
HifiGan
82.5
DAC
84.44
OPUS
65.12.33
though the comparison with ENCODEC is done with different model sizes of Table A.7 original paper Défossez et al. [2022) makes it clear that the number of parameters of the model is not limiting factor of their method.
Multi-Band Diffusion outperform EnCodec on speech compression by significant margin, up to 30% better, while being on part on music data. Averaging across modalities, our method outperforms EnCodec for all bit rates. Qualitatively, to introduce very sharp and straight harmonics that can lead to metallic artifacts. the other hand, our diffusion method produces more blurred high-frequency content. We provide number of spectrogram in the Supplementary Material, Section A.2
observed that GAN-based methods have a tendency
In [ we compare our approach with other decoders baseline trained using the same condition and data our model. Specifically we compare to HifiGAN Kong et al. [2020a] and PriorGrad Lee et al. [2021 using the hyper parameters proposed on their original papers.
The second part of table 2 adds comparisons other end to end audio codecs that do not rely ENCODEC. Specifically adds the pretrained model of DAC Kumar et al. [2023] at 6kpbs which is a different audio codec 24khz. We show ENCODEC MULTI-BAND DIFFUSION on part with DAC that uses a different quantized space. is likely that training our MUL.T-BAND DIFFUSION on the audio tokens of DAC would results in even higher audio quality.

---

Table Objective and subjective metrics comparing the reconstruction performances of our model and ENCODEC across bit rates.
Setting
ViSQOL (1)
McI-SNR-LL (1)
Mel-SNR-M (1)
McI-SNR-H (f)
Mcl-SNR-A (1)
MBD @ 1.5kbps
3,20 10.02
10.09
8.03
8.26
8.79
ENCODEC 1.5kbps
3.33+am
9.61
10.8
13.37
11.36
MBD 3.0 kbps
3,47+0m
11.65
8.91
8.69
9.75
ENCODEC @ 3.0kbps
3.64+0m
11.42
11.97
14.34
12.55
MBD 6 6.0 kbps
3.67=0m2
13.33
9.85
9.26
10.81
ENCODEC @ 6600kpps
3.92#0.0
13.19
12.91
15.21
13.75
Table 4: Comparing the reconstruction performances of our model at 6kbps.
Setting
ViSQOL (f)
Mel-SNR-L (1)
Mel-SNR-M (t)
McI-SNR-H (t)
Mel-SNR-A (t)
MBD @6.0 kbps
3.67
13.33
9.85
9.26
10,81
W-o Processor
3.38
13.16
9.68
8.46
10.43
Linear Schedule
2.93
10.65
7.10
7.73
8.49
Cosine Schedule
3.29=a.0
12.88
9.60
9.59
10.69
Single Band
3.32+0002
12.76
9.82
8,58
10.39
5.2 Ablations
In this section we use objective metries to compare the reconstruction performances. We compute for every experiments the ViQOL score and Mel-SNR on 3 mel spec bands. Objective metrics are computed on the same 4 types of modalities as in section 5.T usingg 150 samples per category. Even though those metrics seem to not correlate well with human evaluation across different model families (c.f. Tables Tables[andd]3 in our testing was accurately measuring the improvements quality resulting from small design changes. In Table 3 we compare the reconstruction performances of Multi-Ban Band Diffusion and Encodec at different bit rates. It is notable that overall Encodec achieves better objective reconstruction metrics while being outperformed in subjective evaluations. We argue that such models are better spectral distance metrics due to their specific training for content reconstruction. On other hand, diffusion based methods do not feature spectrogram matching and tend create samples that are more "in distribution" resulting in more natural audio. Diffusion based methods have more freedom to generate something that will be different from the original audio. They optimized to keep maximize likelihood of their output with respect to the train dataset. The optimal method might be different depending the purpose. However claim that our MUL.TI-BAND DIFFUSION preferable for most generative tasks based on generation in the codec space.
To evaluate the impact of our individual contributions performed an ablation study that evaluates models in the exact same setting when removing one element introduced in this article.
According to the findings of our study, increasing the number of steps to 20 results in improved output quality. However, further increasing the number of steps shows diminishing returns (results available in the Appendix Table 4 In comparison to our approach utilizing four models, a single model performs less effectively. Despite employing a similar number of neural function estimations has worse audio quality and worse scores in every metrics. By leveraging our processor to rehalance the frequency bands, we achieved notable enhancement of 0.2 in ViSQOL scores. Additionally, our proposed schedule demonstrates a performance increase of 0.4 and 0.2 when compared to standard linear and cosine schedules Nichol and Dhariwal [021]) Moreover, our proposed data processing technique also leads to 0.2 increase in VISQOL scores. The figures displayed in table 3 indicate that the high frequencies (Mel-SNR-HH are primarily affected by this processing technique.
5.3 Text to audio
Although our model alone cannot generate audio without conditioning, show that when combined with generative language model on the audio tokens, it provides substantial quality enhancements.
Text to Speech. Using language models audio codecs has recently gained interest for Text to Speech. Methods such as VALL-EE Wang et al. (2023) or SPEAR-TSS Kharitonov et al. 2023) T
achieved convincing results this task. We claim that one improve the quality of the final

---

Table 5: Human evaluations (MUSHRA) decoding token sequences from various methods.
Bark Singing Voices
MusicGen Music
Method
Speech
Average
ENCODEC
64.3443.6
61.85 73.67
63.10
70.99
MBD 76.04+2.9
73.86
74.97
audio by just switching to our MULTI-BANI DIFFUSION token decoder To test that claim we use the implementation and pretrained models from Bark are publicly available. Bark is composed of three transformer models. The initial model converts text input into high-level self-supervised audio tokens, while the second and third models sequentially process these tokens to produce Encodec tokens with two and eight codebooks, respectively. We used our trained diffusion models to decode the final token sequences. We generated 50 text prompts from Bark in all supported languages. We also include 50 prompts using the music note emoji as suggested the official Github page to generate some singing voices. We removed from the subjective tests the samples for which the language model failed to generate any voice, in our experiments using pretrained bark this append for less than 5% of speech prompts and around30% singing voice prompts. Table5 presents the results, and we include the Encodec generation used in the original code base as a bascline.
Text to Music. There has been significant advancement in the field of music generation us- ing language modeling of audio tokens. Recently, this progress has been exemplified by Musi- cLM [Agostinelli et al., [023]] and MusicGen [Copet et al. 2023), which have greatly improved text-to-music generation. order to demonstrate the versatility of our decoding approach, we utilized the open source version of MusicGen and trained a diffusion model conditioned with the tokens produced by its compression model. Our model is trained on the same dataset as the EnCodee model used by MusicGen, with a sampling rate of 32kHz. Additionally, match the standard deviation of 16 mel scaled bands with the compression model output.
Notably, our method achieved MUSHRA score improvement of +4 compared to standard MusicGen (see 5). Overall, the artifacts generated by the diffusion decoder are less pronounced. We find that in music containing complex elements, such as fast drum playing, the outputs from MULTI-BAND DIFFUSION are much clearer than the original ones.
Discussion
In summary, proposed diffusion-based method for decoding the latent space of compression models offers significant improvements audio quality compared to standard decoders. While does require more compute and is slower, our results demonstrate that the trade-off is well worth it. Our approach generates audio that is more natural and distribution, with fewer artefacts compared to existing methods. However, it is worth noting that method may not be suitable for all use cases. For instance, if real-time performance is a critical factor, our approach may not be ideal.
Ethical concerns. Our approach, although not categorized as generative AI, can seamlessly integrate with techniques like Wang et al. 20233 to enhance the authenticity of generated voices. This advancement opens up potential missuses such as creating remarkably realistic deep fakes and voice phishing. Similar to all deep learning algorithms, our method depends on the quality and quantity of training data. We meticulously train our model substantial dataset to optimize its performance across a wide range of scenarios. Nevertheless, we acknowledge that imbalances in the dataset can potentially introduce biases that may impact minority groups.
References
Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A survey on neural speech synthesis. arXiv preprint 155.61,,6.. 2021.
"htpss//gthubb.oomsunaaaaai
10

---

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: (600:0:3009 2016.
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in nearal information processing systems, 32, 2019.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8666-8681, 2022.
Felix Kreuk, Adam Polyak. Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete & decomposed representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11200-11214, 2022a.
Andy T Liu, Po-chun Hsu, and Hung-yi Lee. Unsupervised end-to-end learning of discrete linguistic units for voice conversion. arXiv preprini arXiv: /563, 2019.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprini wrXiv:2104.00355 2021.
Wen-Chin Huang, Shu Yang, Tomoki Hayashi, Hung- Yi Lee, Shinji Watanabe, and Tomoki Toda. S3prl-vc: Open-source voice conversion framework with self-supervised speech representations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6552-6556. IEEE, 2022.
Jungil Kong, Jachycon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022 17033, 2020a.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 00:495-507, 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022.
Jean-Mare Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. Technical report, 2012.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint 15355,
15352, 2022b.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech 2023.
synthesizers. arXiv preprint arXiv:2301.02/1
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arxiv preprint arXiv:2301.11325, 2023.

---

Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement, arXiv preprint arXiv:2212. 2022.
Benjamin van Niekerk, Leanne Nortje, and Herman Kamper. Vector Decttr-quantized neural networks for acoustic unit discovery in the zerospeech 2020 challenge. arXiv preprint wrXiv22055009400 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 2020b.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior arXiv preprint arXiv:2106.066 06406, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007/3, 2020.
Naoya Takahashi, Mayank Kumar, Yuki Mitsufuji, al. Hierarchical diffusion models for singing voice neural vocoder. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Jonathan Ho., Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
Santiago Pascual, Gautam Bhattacharya, Chunghsin Ych, Jordi Pons, and Joan Serrà. Full-band general audio synthesis with coore-based diffusion, 2022.
Qingqing Huang. Daniel S Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint ar arXiv:2302.08: 00317.39....... 2023.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXlv:2301.11099. 2023.
Jan Schnupp, Israel Nelken, and Andrew King. Auditory neuroscience: Making sense of sound. MIT press, 2011.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models.
In International Conference on Machine Learning, pages 8162 8171. PMLR, 2021.
Yang Song. Jascha
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. URL https: :/app/////////v///////ea////am .rriiiioorrg/ee wttp:oo///abb..oooo/o 1455e/99955000 04597
12

---

Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain, 2021.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henrett, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019.
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, et al. Icassp 2022 deep noise suppression challenge. In /CASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9271-9275. IEEE, 2022.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mig-jamendo dataset for automatic music tagging. ICML, 2019.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 30:829-888 852, 2021.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 20/7 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommanication Assembly, 2014.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 !welfth international conference on quality of multimedia experience (QoMEX), pages 1-6. IEEE, 2020.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan, 2023.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.035400 2023.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation, 2023.
13


Reviews:
[{"review_id": "fyiv96jP28", "reviewer": "Reviewer_kZTC", "summary": "This paper presents a novel approach to processing audio data by developing and implementing a band-specific diffusion model, a frequency equalizer processor, and a power noise scheduler. The band-specific diffusion model processes various frequency bands independently, thereby reducing the accumulation of entangled errors. The frequency equalizer (Eq.) processor helps to lessen the discrepancy between the Gaussian prior distribution and the actual data distribution across different frequency bands by balancing energy-level between gaussian noise and each frequency band. The power noise scheduler, specifically designed for audio data with high sampling rate, is another contribution. The authors conduct extensive evaluations to gauge the efficiency of their approach, using both objective metrics and human studies. The results show that the proposed approach surpasses current state-of-the-art methods, encompassing both GAN and diffusion-based methods.", "questions": "1. Regarding the power noise schedule, what were the rules of thumbs to select hyperparameters?\n2. Did the authors set the power noise schedule differently for each band?", "limitations": "The authors have addressed the limitations of this work in the conclusions such as slow speed of the current model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Generating high-frequency bands using diffusion model is something that has not been tackled very well. I personally have also suffered from this in various experiments.\n\nAll three contributions, training of band-specific models, frequency energy balancer, and power noise scheduler, are all valid approaches and were validated by metrics in experiments.", "weaknesses": "It would have been nicer if there were more GAN-based / Diffusion-based vocoder baseline models other than just encodec.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n**Baselines:** We conducted a new subjective studies (MUSHRA) including new baselines such as a discrete version of HifiGan used in https://arxiv.org/abs/2104.00355  and Priorgrad (https://arxiv.org/abs/2106.06406) both conditioned on EnCodec units at 6 kbps. We also include DAC@6kbps(https://arxiv.org/abs/2306.06546), a concurrent SOTA neural compression model from July 2023. We underline that DAC is not 1-1 comparison since it isn't based on EnCodec codebooks but on its own learned latent space, it is likely that using MultiBand Diffusion on top of those codebooks would improve quality even more.\n\n|                               | Mean              | CI95       |\n|--------------------------|--------------------|-------------|\n| Ground Truth       | 90.32              | 1.39        |\n| MBD                      | *85.16*           |  0.93       |\n| Encodec               | 82.73              | 1.11        |\n| PriorGrad             | 65.16              | 2.2          |\n| HifiGan                 | 82.5                | 1.25        |\n|-|\n| DAC                      | 84.44              | 1.14        |\n| OPUS                    | 65                   | 2.43       |\n\n\n**Choice of Noise Schedule:** To determine our noise schedule function, we tested various functions, listened to diffusion process states, and empirically validated our choice and parameters through a grid search. Throughout this process, our primary objective was to have more steps in the less noisy region as described in the paper. We also adopted a schedule that wouldn’t include states where the model doesn’t manage to predict the noise more accurately than the Identity function.\n\n**Details about hyperparameters:** In order not to add unnecessary complexity, we used the identical noise schedule for every band, however we found that tuning the EQ processor differently for every band was beneficial. Comprehensive details will be added in the Appendix of the final version of the paper with all hyperparameters.\n"}, {"review_id": "3O7B9Nytkj", "reviewer": "Reviewer_s9xT", "summary": "This is an interesting submission proposing the use of a diffusion model in a band-by-band manner to generate high-fidelity audio from (potentially) a variety of low bit rate inputs. The literature so far has only applied diffusion models for audio generation given spectrogram inputs. A specific diffusion noise schedule is proposed; and a diffusion-oriented frequency equalization method is proposed. Significant improvements in audio quality are described from a number of experimental evaluations.", "questions": "What is the computational cost of the proposed MBD compared to Opus and EnCodec?", "limitations": "The authors cite reasonable limitations to their work, including the possibility of their method being used to generate deep fakes, as well as the usual potential issue with training set bias, in spite of their efforts to avoid that.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n- Clear presentation (in spite of persistent minor problems with grammar & spelling);\n- Original multi-band diffusion model;\n- Good literature review;\n- Original frequency equalization model;\n- Good empirical results, i.e. nice improvements in generated audio quality.\n\n", "weaknesses": "The main weakness is a persistent pattern of small mistakes in grammar and spelling. I mention a number below but this is just a small sample. Overall the work is very readable, but the problems with the writing should be corrected.\n\nI note that there is mention of the proposed method being significantly slower than the alternatives examined, but I don't see a clear description of the compute cost of the proposed method compared to existing methods.\n\nSpecific comments:\n\n> 27 artefacts [here and below]\n--> artifacts\n\n> have led to rich contextual representations that contains more\n\ncontains --> contain\n\n> 32 They are optimized using complex combination\ncombination --> combinations\n\n> 47 Results suggest the proposed method achieves significantly\n> 48 superior performance than the evaluated baselines.\n\n\"significantly superior\" is an odd phrasing, rewrite?\n\n> 67 Clustering using a few centroids leads to the speech content representation being\n> 68 mostly disentangled from the speaker and the f0 and thus controllable speech generation.\n\nEnd of sentence is not grammatical; missing words before/around \"thus controllable speech generation\"?\n\n\"Eq. Processor\" in Figure 1 caption vs. \"EQ Processor\" in the figure itself: choose one or the other, and keep consistent.\n\n> 165 As a result, training a diffusion model on full-band audio data would always\n> 166 provide the ground truth low frequencies when generating high frequencies\n\nI don't quite understand this statement, rephrase?\n\n>  170 Interestingly, dividing the frequency band\n> 171 along model channels\n\nI don't understand this statement. What are \"model channels\"?\n\n> 238 5.1 Multi modalities model\nReading this section, i'm not sure what are the \"modalities\" referred to.\n\nTable 1: \"MBD\", though it is clear that this is the proposed method, I don't see the acronym introduced. Introduce it early on, e.g. state \"... Multi Band Diffusion (MBD) ...\"?\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First thank you very much for the review and taking the time to point at typos and grammar issues. We have addressed all of your comments and will conduct a thorough proofreading of the paper to ensure that no error appears in the camera-ready version. \n\n**Regarding the computational cost:** We will include a table with the number of parameters and the compute time of our method vs Encodec, we also include in the total cost of a complete pipeline LM + Decoder such as the one described in the paper. This table will be included to the Appendix of the paper.\n\n|                                                | Compute time (30s) | #parameters |\n|----------------------------------------|------------------------------|-------------------|\n| Encodec                                | 0.1s                            | 56M               |\n| MBD                                       | 21.2s                          | 411M             |\n| MusicGen-large + Encodec | 102s                           | 3.3B              |\n| MusicGen-large + MBD        | 123s                           | 3.7B              |\n"}, {"review_id": "xzJVUzYEWk", "reviewer": "Reviewer_vmZN", "summary": "This paper proposes a novel multi-band diffusion (MBD) model that generates high-fidelity audio of multiple modalities, e.g., speech, music, and environmental sounds, from low-bitrate discrete representations. The authors show that MBD outperforms EnCodec and Opus in terms of perceptual quality.", "questions": "Some questions have been stated above. In summary:\n\n(i) What are the considerations when choosing the baselines? Are these baselines strong enough?\n\n(ii) Is the frozen latent representations of a pre-trained EnCodec necessary for training MBD? Can MBD be end-to-end trained from scratch?\n\n(iii) What are the speed and model sizes of MBD in comparison to EnCodec or other baselines (if any)?", "limitations": "The limitations of this work have been stated in the weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The band-specific diffusion model is novel and appears to be a general technique for universal audio synthesis. The authors design a frequency equalizer to reduce the discrepancy between the prior Gaussian distribution and the data distribution in different frequency bands, which is sensible for improving the consistency and stability of audio generation, especially for general audios. The authors also propose a novel power noise scheduler, which empirically surpasses other commonly used schedulers, e.g., linear and cosine schedulers. I appreciate the high variety of audio generation experiments conducted for evaluating MBD. The generated samples of MBD on the demo page seem promising and are apparently better than EnCodec and Opus at the same bit rates.", "weaknesses": "1. Baselines are not strong enough. The authors only compare MBD to Encodec and Opus, where Opus is an old method proposed in 2012. I am therefore skeptical of the choice of the baselines in this paper. Are these baselines strong enough? The authors mention SoundStream (Zeghidour et al., 2021) but do not take it as a baseline. Considering the wide and successful application of SoundStream in many recent audio generation works, e.g., in AudioLM, Vall-E, and Natural Speech 2, it would be more convincing if MBD can be compared to SoundStream and evidently demonstrate superiority. I consider increasing my rating if such an experiment can be supplemented.\n\n2. MBD depends on the frozen latent representations of a pre-trained EnCodec. The dependency of MBD on EnCodec complicates the analysis of the significance of this work. It remains unclear whether the high quality comes from a well-trained EnCodec or the proposed MDB model. The training of MBD also becomes more unstable. Please explain why the encoder of EnCodec cannot be jointly trained with MBD. \n\n3. It is unfair that the generation speed and model size of MBD are not compared to those of Encodec. While the authors only present the strength of their proposed MBD model in terms of different quality measures, they do not mention the drawback of the generative diffusion model. As an iterative sampling method, it is foreseeable that MBD could be slow than Encodec up to orders of magnitude. It is then unfair to only focus on the quality when it is more reasonable to evaluate the audio codecs based on their practical values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n**Comparison to SoundStream**: Despite our willingness to perform a direct comparison with SoundStream, it's important to note that the authors have not released a public repository, and there is no public reimplementation on GitHub that is reproducing SoundStream performances. \n\n**Baselines** EnCodec is a very strong Neural audio codec that builds on SoundStream. In their paper they provide a comparison showing that it outperforms SoundStream (cf. https://arxiv.org/pdf/2210.13438.pdf Appendix Table A2). Many models such as VALL-E or MusicGen are based on EnCodec. Natural speech 2 uses its own compression model that also builds on SoundStream. To add stronger baseline we add recent SOTA Compression Model DAC(https://arxiv.org/abs/2306.06546) and two other decoders (HifiGan and PriorGrad). We are comparable to DAC while building on different Encoders. It is likely that using MultiBand Diffusion as a decoder replacement for DAC improves performances over their GAN based decoder.\n\n|                               | Mean              | CI95       |\n|--------------------------|--------------------|-------------|\n| Ground Truth       | 90.32              | 1.39        |\n| MBD                      | **85.16**           |  0.93       |\n| Encodec               | 82.73              | 1.11        |\n| PriorGrad             | 65.16              | 2.2          |\n| HifiGan                 | 82.5                | 1.25        |\n|--------------------------|--------------------|-------------|\n| DAC                      | 84.44              | 1.14        |\n| OPUS                    | 65                   | 2.43       |\n\n\n\n**Aim of the paper:** Our approach serves as a replacement for EnCodec's decoder. We will highlight this aspect more prominently in the paper, as it has proven unclear to several reviewers. This approach offers the advantage of flexibility and compatibility across various applications. In the context of Text-to-Audio generation, it provides a means to swiftly preview audio using the fast and lightweight default decoder, and the option to switch to MultiBand diffusion when a desirable sample is identified and higher quality is needed.\nIn all tables, the encoder and RVQ are shared between EnCodec and our method, thereby rendering the performance disparity solely attributable to our proposed diffusion decoder. Everything remains identical except for this aspect. \n\n\n**Training end-to-end compression with MBD:** This is an excellent question. We conducted preliminary experiments in this direction. However, we observed that utilising solely the L2 loss on the waveform does not yield satisfactory latent representations when compared to loss specifically designed for compression such as features and spectrogram matching, as is the case in a standard neural audio codec. How to combine the diffusion objective with perceptual losses is a complex and open problem that is beyond the scope of this paper.\n\n**Compute time:** We will include this table to the appendix of the paper including compute time and number of parameters for the different methods. We also add the comparison of complete pipelines using MusicGen to put the increase in compute time and model size in perspective.\n\n|                                                | Compute time (30s) | #parameters |\n|----------------------------------------|------------------------------|-------------------|\n| Encodec                                | 0.1s                            | 56M               |\n| MBD                                       | 21.2s                          | 411M             |\n| MusicGen-large + Encodec | 102s                           | 3.3B              |\n| MusicGen-large + MBD        | 123s                           | 3.7B              |\n"}, {"review_id": "Vf1I0YITC2", "reviewer": "Reviewer_SCw1", "summary": "This paper presents Multi-Band Diffusion, a diffusion-based decoder for neural audio codecs. The proposed method demonstrates superior generation quality across various audio domains compared to a publicly available state-of-the-art neural audio codec method. Based on the analysis of time-frequency representation of audio, the model provides a novel diffusion-based model for audio synthesis.", "questions": "From the selection of low bit-rate latent representation as the model's conditional input and the use of a small number of diffusion steps during inference, it seems that the authors have considered efficiency alongside the model's performance while empirically validating the proposed method. However, it would be beneficial to extend it and demonstrate the peak performance of the proposed model with larger bit-rate representation and high diffusion steps, as diffusion-based generative models have shown remarkable performance in various fields.", "limitations": "The authors have adequately addressed both the limitations of their research and its possible societal impacts.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The authors provide extensive experimental results, along with a detailed explanation of their methodology.\n* They present novel methods such as the band-specific diffusion model, frequency equalizer processor, and power noise scheduler, demonstrating their necessity and assessing their effectiveness through an ablation study.\n* The model not only outperforms benchmarks in terms of the generation quality of reconstructed samples across different audio domains, but also proves its applicability in tasks such as text-to-audio and text-to-speech through empirical evidence.", "weaknesses": "* There is no comparison of generation quality when producing from high bit-rate latent representations. While the generation quality remains inferior compared to the ground truth audio, the authors only showcase generation quality from low bit-rate representations.\n* There is a lack of a comparison of computational cost or model size. Although the authors mention that the proposed method requires more computation compared to standard decoders, a comparison of generation speed or parameter size would have provided a better understanding of the trade-off between generation quality and synthesis speed of this model. Furthermore, given that the model was trained using 1000 diffusion steps and only 20 steps were \bemployed during inference, an evaluation of quality at different diffusion steps during inference could also be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**High bit rate:** The Neural audio codec literature has mainly focussed on low bit rates. Moreover, these low bitrate setups align with the configurations employed by language modelling approaches that our method seeks to enhance, such as Musicgen, or Vall-E.. While our method is applicable to higher bit rates, the potential improvements are more marginal and might not justify the computational trade off. Finally, there are fewer use cases of high bit rate representations in the realm of text-to-audio (to the best of our knowledge, none).\n\n**Number of denoising steps:** We provide in the appendix of the paper a comparison of different number of denoising steps showing that there are very few improvement going more that 20 steps (see Table A5)\n\n**Computational time & Model size:** We will add this table to the appendix of the paper. Showing generation time and model size of EnCodec vs MBD also put the cost of a full pipeline LM + Decoder to put it in perspective.\n\n|                                                | Compute time (30s) | #parameters |\n|----------------------------------------|------------------------------|-------------------|\n| Encodec                                | 0.1s                            | 56M               |\n| MBD                                       | 21.2s                          | 411M             |\n| MusicGen-large + Encodec | 102s                           | 3.3B              |\n| MusicGen-large + MBD        | 123s                           | 3.7B              |\n"}, {"review_id": "WHOtqsfuFP", "reviewer": "Reviewer_tTLs", "summary": "The paper proposes to use diffusion model to recover high-fidelity audio from compressed audio tokens. To improve the performance of diffusion model, the paper proposes multi-band diffusion with 1) Frequency Eq. Processor; 2) Scheduler Tuning; and 3) Band-Specific Training. The experiments show that the proposed method surpasses baseline method (EnCodec) by a margin. ", "questions": "My main concern is the proposed method should be verified on vocoder task and compared with GAN based and diffusion based vocoder baselines.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a diffusion-based model as decoder for audio compression model (EnCodec), which surpasses the default decoder joint trained with encoder.\n\n2. The paper proposes multi-band diffusion model to improve the generation quality.", "weaknesses": "1. The motivation of multiband is not clear. According to the description in paper, the multi-band is conducted on the hidden of encodec, which seems not have frequency information. If there is no intuition or explanation about the multi-band, it seems that using multi-band is a kind of improving the model parameters (using multiple model, one per band).\n\n2. The experiments are only on EnCodec tokens, which are insufficient. The design in paper should be verified on vocoder tasks (recovering audio from mel-spectrum) and compared with more baselines (e.g., bigvgan, hifigan, wavegrad and multi-band MelGAN). Since mel-spectrum has frequency information, using multi-band on mel-spectrum is more convincing.\n\n3. In ablation study, the comparison with single band seems to be unfair in terms of total model parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and suggestions.\n\n**Clarification about the setting:** We will clarify the model's input-output relationship in the paper, as it currently appears to be unclear. All models take the full Encoder compressed representation as input. The diffusion process occurs in the waveform domain, specifically on waveforms that have undergone filtering using a passband filter. \n\n**Multi Band motivation:** We use a multi band approach is to address the issue of entangled errors that arise when conducting diffusion on full-band audio. In our experiments, this phenomenon manifested regardless of the model sizes. Our intuition suggests that the model overly relies on lower frequencies already present in the noisy waveform during training steps with small t (close to the clean x_0). Training independently for different frequency bands resolves this problem, preventing the model from inferring high-frequency content from the low frequency of noisy waveforms and requiring it to extract such information from the conditioning.\n\n**Discrete Units Focus:** Our paper is focused on generating from low bitrate discrete representation. Everything has been designed with this task in mind to seamlessly plug in prior text to audio work. Performing mel spectrogram to audio is not a task we considered yet. However we’ll add some discrete version of HifiGan and PriorGrad as additional baselines to our work. We’ll also add DAC(https://arxiv.org/abs/2306.06546), a new SOTA Compression model from July 2023. At 6kbps using their public implementation we have comparable performances. However we want to stress that this is not a decoder based on EnCodec but a completely different compression model. It is possible that replacing their decoder with diffusion would improve performances, which is something we might investigate in the future.\n\n\n\n|                               | Mean              | CI95       |\n|--------------------------|--------------------|-------------|\n| Ground Truth       | 90.32              | 1.39        |\n| MBD                      | *85.16*           |  0.93       |\n| Encodec               | 82.73              | 1.11        |\n| PriorGrad             | 65.16              | 2.2          |\n| HifiGan                 | 82.5                | 1.25        |\n|--------------------------|--------------------|-------------|\n| DAC                      | 84.44              | 1.14        |\n| OPUS                    | 65                   | 2.43       |\n\n\n**Comparison to Single Band:** We trained multi band models with 4 times less parameters to compare with the single band model from the ablation table (Table 3). We will add the line \"Multi Band small\" to the paper in table 3:\n\n| Setup | ViSQOL |  Mel SNR-L| Mel SNR-M | Mel SNR-H | Mel SNR-A |\n|--------|-----------|--------------|---------------|--------------|---------------|\n|Multi Band | 3.67±0.02 | 13.33 | 9.85 | 9.26 | 10.81| \n|Multi Band small | 3.56±0.03 |12.93| 9.81 | 9.11 |10.61 |\n|Single Band |3.32±0.02 |12.76| 9.82| 8.58| 10.39|\n"}]

Author rebuttal:
We want to make an ethical statement, the AudioGen and MusicGen team recently released the new public implementations with pre-trained models. We trained MultiBand Diffusion with those new versions of the compression models. We found improvement on MusicGen using MultiBand Diffusion as a decoder. However, in the first experiment that we did with pre-trained AudioGen, MBD was not improving over the baseline decoder. We want to come clean and say that we will remove the results on AudioGen as it is currently in the paper. However this doesn't change any other experiment conducted in the rest of the paper.

Results (MUSHRA) on MusicGen:

|                    | Mean  | CI95 |
|--------------------|-------|------|
| MusicGen + Encodec | 70.99 | 1.19 |
| MusicGen + MBD     | **74.97** | 1.94 |


Meta-review: This paper proposes a so-called multi-band diffusion model that can produce high-fidelity signals of various audio modalities from low-bitrate discrete representations.  In this model, the diffusion is carried out in multiple independent bands. On top of that,  a frequency equalizer and a power noise scheduler are introduced to stabilize the training and improve the quality and efficiency of diffusion.  The authors compare the proposed multi-band diffusion model with existing high-performance models and show its superior quality  across various applications such as speech and music.   The work is considered novel and interesting.  The experimental results are solid.  The authors have cleared most of the concerns raised by the reviewers in their rebuttal and following discussion.  All reviewers are supportive to accept.
Decision: Accept (poster)