SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
z = \operatorname {m e l} \left[ \frac {x}{\epsilon + \sqrt {\langle x ^ {2} \rangle}} \right], \quad \text {a n d} \quad \hat {z} = \operatorname {m e l} \left[ \frac {\hat {x}}{\epsilon + \sqrt {\langle x ^ {2} \rangle}} \right], \tag {9}
$$
with  $z, \hat{z} \in \mathbb{R}^{F \times T / H}$ . We compute the mel-spectrogram distortion  $\delta = z - \hat{z}$ . Finally for each time step  $t$  and frequency bin  $f$ , we can compute a Signal-To-Noise ratio. In order to avoid numerical instabilities, and also not let the metric be overly impacted by near zero values in the ground truth mel-spectrogram, we clamp the SNR value between  $-25dB$ ,  $+25dB$ , considering that any distortion lower than  $-25$  dB would have a limited impact on perception, and that beyond  $+25$  dB, all distortions would be equally bad. Indeed, due to the limited precision used in the computation and training of a neural network, it is virtually impossible to output a perfectly null level of energy in any given frequency band, although such empty bands could happen in real signals. Finally we get,
$$
s = \operatorname {c l a m p} [ 1 0 \cdot (\log_ {1 0} (z) - \log_ {1 0} (\delta))., - 2 5 \mathrm {d B}, + 2 5 \mathrm {d B} ]. \tag {10}
$$
We then average over the time steps, and split the mel-scale bands into 3 equally spaced in mel-scale. We report each band as Mel-SNR-L (low frequencies), Mel-SNR-M (mid frequencies), and Mel-SNR-H (high frequencies). Finally we also report the average over all 3 bands as Mel-SNR-A. At 24 kHz, we use a STFT over frames of 512 samples, with a hop length  $H = 128$  and  $N = 80$  mel bands.
5.1 Multi modalities model
We first evaluate the performance of our diffusion method compared with ENCODEC on a compression task. Specifically we extract audio tokens from audio samples using the ENCODEC encoder and decode them using MULTI-BAND DIFFUSION and the original decoder.
We perform subjective evaluation on four subsets: 50 samples of clean speech from DNS, 50 samples of corrupted speech using DNS blended with samples from FSD50K, 50 music samples from Jamendo and 50 music samples from an internal music dataset. All speech samples are reverberated with probability 0.2 using room impulse responses provided in the DNS challenge. In Table[1], we present 3 subjective studies with different bit rate levels: 6kbps, 3kbps, and 1.5kbps. Note that scores should not be compared across the studies since ratings are done relatively to the other samples of the study. We include Opus [Valin et al., 2012] at 6kbps as a low anchor and the ground truth samples. Even
2We compute visqol with: https://github.com/google/visqol using the recommended recipes.
Table 1: Human evaluations (MUSHRA) scores for  $24\mathrm{kHz}$  audio. The mean and CI95 results are reported. The Opus low anchor and ground truth samples are consistent across all three studies, delimited by horizontal lines. The other methods used a bit rate of 6kbps for the top study on top, 3kbps for the middle one, and 1.5kbps for the bottom one. Higher scores indicate superior quality.
Method	Speech	Music	Average
Reference	93.86±0.014	92.93±0.021	93.40
Opus	61.14±0.094	34.24±0.147	47.69
ENCODEC	79.03±0.053	84.67±0.062	81.85
MBD (ours)	84.68±0.036	83.61±0.072	84.15
Reference	93.17±0.015	94.45±0.014	93.81
Opus	62.83±0.14	36.17±0.12	49.5
ENCODEC	78.51±0.078	85.55±0.045	82.03
MBD (ours)	84.42±0.042	87.31±0.041	85.87
Reference	94.65±0.012	94.71±0.012	94.78
Opus	44.65±0.057	38.33±0.081	41.49
ENCODEC	49.51±0.072	75.98±0.077	62.75
MBD (ours)	65.83±0.056	75.29±0.076	70.56
Table 2: Human evaluations (MUSHRA) scores for  ${24}\mathrm{{kHz}}$  audio. The mean and CI95 results are reported. The first part of the table reports different methods of ENCODEC tokens at 6kbps decoding while the second part adds other independent compression baselines at 6 kbps.
Method	score
Ground Truth	90.32 ±1.39
MBD	85.16 ±0.93
Encodec	82.73±1.11
PriorGrad	65.16±2.2
HifiGan	82.5±1.25
DAC	84.44±1.14
OPUS	65±2.43
though the comparison with ENCODEC is done with different model sizes cf Table A.7, original paper Defossez et al. 2022 makes it clear that the number of parameters of the model is not a limiting factor of their method.
Multi-Band Diffusion outperform EnCodec on speech compression by a significant margin, up to  $30\%$  better, while being on part on music data. Averaging across modalities, our method outperforms EnCodec for all bit rates. Qualitatively, we observed that GAN-based methods have a tendency to introduce very sharp and straight harmonics that can lead to metallic artifacts. On the other hand, our diffusion method produces more blurred high-frequency content. We provide a number of spectrogram in the Supplementary Material, Section A.2
In table 2, we compare our approach with other decoders baseline trained using the same condition and data as our model. Specifically, we compare to HifiGAN [Kong et al. (2020a)] and PriorGrad [Lee et al. (2021)] using the hyper parameters proposed on their original papers.
The second part of table 2 adds comparisons to other end to end audio codecs that do not rely on ENCODEC. Specifically it adds the pretrained model of DAC Kumar et al. 2023 at 6kpbs which is a different audio codec at 24khz. We show ENCODEC + MULTI-BAND DIFFUSION is on part with DAC that uses a different quantized space. It is likely that training our MULTI-BAND DIFFUSION on the audio tokens of DAC would results in even higher audio quality.
Table 3: Objective and subjective metrics comparing the reconstruction performances of our model and ENCODEC across bit rates.
Setting	ViSQOL (↑)	Mel-SNR-L (↑)	Mel-SNR-M (↑)	Mel-SNR-H (↑)	Mel-SNR-A (↑)
MBD @1.5kbps	3.20 ±0.02	10.09	8.03	8.26	8.79
ENCODEC@1.5kbps	3.33±0.02	9.61	10.8	13.37	11.36
MBD 3.0 kbps	3.47±0.02	11.65	8.91	8.69	9.75
ENCODEC@3.0kbps	3.64±0.02	11.42	11.97	14.34	12.55
MBD @6.0 kbps	3.67±0.02	13.33	9.85	9.26	10.81
ENCODEC@6.0kbps	3.92±0.02	13.19	12.91	15.21	13.75
Table 4: Comparing the reconstruction performances of our model at 6kbps.
Setting	ViSQOL (↑)	Mel-SNR-L (↑)	Mel-SNR-M (↑)	Mel-SNR-H (↑)	Mel-SNR-A (↑)
MBD @6.0 kbps	3.67±0.02	13.33	9.85	9.26	10.81
w-o Processor	3.38±0.02	13.16	9.68	8.46	10.43
Linear Schedule	2.93±0.03	10.65	7.10	7.73	8.49
Cosine Schedule	3.29±0.03	12.88	9.60	9.59	10.69
Single Band	3.32±0.02	12.76	9.82	8.58	10.39
5.2 Ablations
In this section we use objective metrics to compare the reconstruction performances. We compute for every experiments the ViQOL score and Mel-SNR on 3 mel spec bands. Objective metrics are computed on the same 4 types of modalities as in section 5.1 using 150 samples per category. Even though those metrics seem to not correlate well with human evaluation across different model families (c.f. Tables 1 and 3) in our testing it was accurately measuring the improvements in quality resulting from small design changes. In Table 3 we compare the reconstruction performances of Multi-Band Diffusion and Encodec at different bit rates. It is notable that overall Encodec achieves better objective reconstruction metrics while being outperformed in subjective evaluations. We argue that such models are better in spectral distance metrics due to their specific training for content reconstruction. On the other hand, diffusion based methods do not use feature or spectrogram matching and tend to create samples that are more "in distribution" resulting in more natural audio. Diffusion based methods have more freedom to generate something that will be different from the original audio. They are optimized to keep maximize likelihood of their output with respect to the train dataset. The optimal method might be different depending on the purpose. However we claim that our MULTI-BAND DIFFUSION is preferable for most generative tasks based on generation in the codec space.
To evaluate the impact of our individual contributions we performed an ablation study that evaluates models in the exact same setting when removing one element introduced in this article.
According to the findings of our study, increasing the number of steps to 20 results in improved output quality. However, further increasing the number of steps shows diminishing returns (results available in the Appendix Table4). In comparison to our approach utilizing four models, a single model performs less effectively. Despite employing a similar number of neural function estimations it has worse audio quality and worse scores in every metrics. By leveraging our processor to rebalance the frequency bands, we achieved a notable enhancement of 0.2 in ViSQOL scores. Additionally, our proposed schedule demonstrates a performance increase of 0.4 and 0.2 when compared to standard linear and cosine schedules [Nichol and Dhariwal (2021)]. Moreover, our proposed data processing technique also leads to a 0.2 increase in ViSQOL scores. The figures displayed in table3 indicate that the high frequencies (Mel-SNR-H) are primarily affected by this processing technique.
5.3 Text to audio
Although our model alone cannot generate audio without conditioning, we show that when combined with a generative language model on the audio tokens, it provides substantial quality enhancements.
Text to Speech. Using language models on audio CODECs has recently gained interest for Text to Speech. Methods such as VALL-E Wang et al. [2023] or SPEAR-TSS Kharitonov et al. [2023] achieved convincing results on this task. We claim that one can improve the quality of the final
Table 5: Human evaluations (MUSHRA) decoding token sequences from various methods.
Method	Bark	MusicGen
Speech	Singing Voices	Average	Music
ENCODEC	64.34±3.6	61.85±4.2	63.10	70.99±1.19
MBD	76.04±2.9	73.67±3.4	73.86	74.97±1.94
audio by just switching to our MULTI-BAND DIFFUSION token decoder To test that claim we use the implementation and pretrained models from Bark<sup>3</sup> that are publicly available. Bark is composed of three transformer models. The initial model converts text input into high-level self-supervised audio tokens, while the second and third models sequentially process these tokens to produce Encodec tokens with two and eight codebooks, respectively. We used our trained diffusion models to decode the final token sequences. We generated 50 text prompts from Bark in all supported languages. We also include 50 prompts using the music note emoji as suggested in the official Github page to generate some singing voices. We removed from the subjective tests the samples for which the language model failed to generate any voice, in our experiments using pretrained bark this append for less than  $5\%$  of speech prompts and around  $30\%$  singing voice prompts. Table<sup>5</sup> presents the results, and we include the Encodec generation used in the original code base as a baseline.
Text to Music. There has been a significant advancement in the field of music generation using language modeling of audio tokens. Recently, this progress has been exemplified by MusicLM [Agostinelli et al., 2023] and MusicGen [Copet et al., 2023], which have greatly improved text-to-music generation. In order to demonstrate the versatility of our decoding approach, we utilized the open source version of MusicGen and trained a diffusion model conditioned with the tokens produced by its compression model. Our model is trained on the same dataset as the EnCodec model used by MusicGen, with a sampling rate of  $32\mathrm{kHz}$ . Additionally, we match the standard deviation of 16 mel scaled bands with the compression model output.
Notably, our method achieved a MUSHRA score improvement of  $+4$  compared to standard MusicGen (see Table5). Overall, the artifacts generated by the diffusion decoder are less pronounced. We find that in music containing complex elements, such as fast drum playing, the outputs from MULTI-BAND DIFFUSION are much clearer than the original ones.
In summary, our proposed diffusion-based method for decoding the latent space of compression models offers significant improvements in audio quality compared to standard decoders. While it does require more compute and is slower, our results demonstrate that the trade-off is well worth it. Our approach generates audio that is more natural and in distribution, with fewer artefacts compared to existing methods. However, it is worth noting that our method may not be suitable for all use cases. For instance, if real-time performance is a critical factor, our approach may not be ideal.
Ethical concerns. Our approach, although not categorized as generative AI, can seamlessly integrate with techniques like Wang et al. [2023] to enhance the authenticity of generated voices. This advancement opens up potential missuses such as creating remarkably realistic deep fakes and voice phishing. Similar to all deep learning algorithms, our method depends on the quality and quantity of training data. We meticulously train our model on a substantial dataset to optimize its performance across a wide range of scenarios. Nevertheless, we acknowledge that imbalances in the dataset can potentially introduce biases that may impact minority groups.
Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2106.15561, 2021.
<sup>3</sup>https://github.com/suno-ai/bark
Alexandre Defossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain, 2021.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henrtty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019.
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matushevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, et al. Icassp 2022 deep noise suppression challenge. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9271-9275. IEEE, 2022.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. ICML, 2019.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-852, 2021.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
B Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunication Assembly, 2014.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pages 1-6. IEEE, 2020.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan, 2023.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.03540, 2023.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation, 2023.