SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. ViSQOL [7]: an intrusive perceptual quality metric that uses spectral similarity to the ground truth to estimate a mean opinion score.
2. Mel distance: distance between log mel spectrograms of the reconstructed and ground truth waveforms. The configuration of this loss is the same as described in 3.5.
Ablation on	Decoder dim.	Activation	Multi-period	Single-scale # of STFT bands	Multi-scale mel.	Latent dim	Quant. method	Quant. dropout	Bitrate (kbps)	Balanced samp.	Mel distance↓	STFT distance↓	ViSQOL ↑	SI-SDR ↑	Bitrate efficiency ↑
1536	snake	✓	X	5	✓	8	Proj.	1.0	8	✓	1.09	1.82	3.96	9.12	99%
Architecture	512	snake	✓	X	5	✓	8	Proj.	1.0	8	✓	1.11	1.83	3.91	8.72	99%
1024	snake	✓	X	5	✓	8	Proj.	1.0	8	✓	1.07	1.82	3.96	9.07	99%
1536	relu	✓	X	5	✓	8	Proj.	1.0	8	✓	1.17	1.81	3.83	6.92	99%
Discriminator	1536	snake	X	X	X	✓	8	Proj.	1.0	8	✓	1.13	1.92	4.12	1.07	62%
1536	snake	✓	X	1	✓	8	Proj.	1.0	8	✓	1.07	1.80	3.98	9.07	99%
1536	snake	X	X	5	✓	8	Proj.	1.0	8	✓	1.07	1.81	3.97	9.04	99%
1536	snake	X	✓	5	✓	8	Proj.	1.0	8	✓	1.08	1.82	3.95	8.51	99%
Reconstruction loss	1536	snake	✓	X	5	X	8	Proj.	1.0	8	✓	1.10	1.87	4.01	7.68	99%
Latent dim	1536	snake	✓	X	5	✓	2	Proj.	1.0	8	✓	1.44	2.08	3.65	2.22	84%
1536	snake	✓	X	5	✓	4	Proj.	1.0	8	✓	1.20	1.89	3.86	7.15	97%
1536	snake	✓	X	5	✓	32	Proj.	1.0	8	✓	1.10	1.84	3.95	9.05	98%
1536	snake	✓	X	5	✓	256	Proj.	1.0	8	✓	1.31	1.97	3.79	5.09	59%
Quantization setup	1536	snake	✓	X	5	✓	8	EMA	1.0	8	✓	1.11	1.84	3.94	8.33	97%
1536	snake	✓	X	5	✓	8	Proj.	0.0	8	✓	0.98	1.70	4.09	10.14	99%
1536	snake	✓	X	5	✓	8	Proj.	0.25	8	✓	0.99	1.69	4.04	10.00	99%
1536	snake	✓	X	5	✓	8	Proj.	0.5	8	✓	1.01	1.75	4.03	9.74	99%
1536	snake	✓	X	5	✓	8	Proj.	1.0	24	✓	0.73	1.62	4.16	13.83	99%
Data	1536	snake	✓	X	5	✓	8	Proj.	1.0	8	X	1.09	1.94	3.89	8.89	99%
Table 2: Results of the ablation study on our proposed codec. The final model is trained with the same configuration as the baseline (top row), but with a quantization dropout of 0.5.
3. STFT distance: distance between log magnitude spectrograms of the reconstructed and ground truth waveforms. We use window lengths [2048, 512]. This metric captures the fidelity in higher frequencies better than the mel distance.
4. Scale-invariant source-to-distortion ratio (SI-SDR) [20]: distance between waveforms, similar to signal-to-noise ratio, with modifications so that it is invariant to scale differences. When considered alongside spectral metrics, SI-SDR indicates the quality of the phase reconstruction of the audio.
5. Bitrate efficiency: We calculate bitrate efficiency as the sum of the entropy (in bits) of each codebook when applied on a large test set divided by the number of bits across all codebooks. For efficient bitrate utilization this should tend to  $100\%$  and lower percentages indicate that the bitrate is being underutilized.
We also conduct a MUSHRA-inspired listening test, with a hidden reference, but no low-passed anchor. In it each one of ten expert listeners rated 12 randomly selected 10-second samples from our evaluation set, 4 of each domain; speech, music and environmental sounds. We compare our proposed system at 2.67kbps, 5.33kbps and 8kbps to EnCodec at 3kbps, 6kbps and 12kbps.
We conduct a thorough ablation study of our model, varying components of our training recipe and model configuration one-by-one. To compare models, we use the four objective metrics described in Section 4.4. The results of our ablation study can be seen in Table 2.
Architecture: We find that varying the decoder dimension has some effect on performance, with smaller models having consistently worse metrics. However, the model with decoder dimension 1024 has similar performance to the baseline, indicating that smaller models can still be competitive. The change with the biggest impact was switching out the relu activation for the snake activation. This change resulted in much better SI-SDR and other metrics. Similar to the results in BigVGAN [21], we find that the periodic inductive bias of the snake activation is helpful for waveform generation. For our final model, we use the largest decoder dimension (1536), and the snake activation.
**Discriminator:** Next, we removed or changed the discriminators one-by-one, to see their impact on the final result. First, we find that the multi-band STFT discriminator does not result in significantly better metrics, except for SI-SDR, where it is slightly better. However, when inspecting spectrograms of generated waveforms, we find that the multi-band discriminator alleviates aliasing of high frequencies. The upsampling layers of the decoder introduce significant aliasing artifacts [29]. The multi-band discriminator is more easily able to detect these aliasing artifacts and give feedback to the generator to remove them. Since aliasing artifacts are very small in terms of magnitude, their effect on our objective metrics is minimal. Thus, we keep the multi-band discriminator.
We find that adversarial losses are critical to both the quality of the output audio, as well as the bitrate efficiency. When training with only reconstruction loss, the bitrate efficiency drops from  $99\%$  to  $62\%$ , and the SI-SDR drops from 9.12 to 1.07. The other metrics capture spectral distance, and are relatively unaffected. However, the audio from this model has many artifacts, including buzzing, as it has not learned to reconstruct phase. Finally, we found that swapping the multi-period discriminator for the single-scale waveform discriminator proposed in MelGAN [19] resulted in worse SI-SDR. We retain the multi-period discriminator.
Impact of low-hop reconstruction loss: We find that low-hop reconstruction is critical to both the waveform loss and the modeling of fast transients and high frequencies. When replaced with a single-scale high-hop mel reconstruction (80 mels, with a window length of 512), we find significantly lower SI-SDR (7.68 from 9.12). Subjectively, we find that this model does much better at capturing certain sorts of sounds, such as cymbal crashes, beeping and alarms, and singing vocals. We retain the multi-scale mel reconstruction loss in our final recipe.
Latent dimension of codebook: the latent dimension of the codebook has a significant impact on bitrate efficiency, and consequently the reconstruction quality. If set too low or too high (e.g. 2, 256), quantitative metrics are significantly worse with drastically lowered bitrate efficiency. Lower bitrate efficiency results in effectively lowered bandwidth, which harms the modeling capability of the generator. As the generator is weakened, the discriminator tends to "win", and thus the generator does not learn to generate audio with high audio quality. We find 8 to be optimal for the latent dimension.
Quantization setup: we find that using exponential moving average as the codebook learning method, as in EnCodec[8], results in worse metrics especially for SI-SDR. It also results in poorer codebook utilization across all codebooks (Figure 1). When taken with its increased implementation complexity (requiring K-Means initialization and random restarts), we retain our simpler projected lookup method for learning codebooks, along with a commitment loss. Next, we note that the quantization dropout rate has a significant effect on the quantitative metrics. However, as seen in Figure 2, a dropout of 0.0 results in poor reconstruction with fewer codebooks. As this makes usage of the codec challenging for downstream generative modeling tasks, we instead use a dropout rate of 0.5 in our final model. This achieves a good trade-off between audio quality at full bitrate as well as lower bitrates. Finally, we show that we can increase the max bitrate of our model from 8kbps to 24kbps and achieve excellent audio quality, surpassing all other model configurations. However, for our final model, we train at the lower bitrates, in order to push the compression rate as much as possible.
Balanced data sampling: When removed, this results in worse metrics across the board. Empirically, we find that without balanced data sampling, the model produces waveforms that have a max frequency of around  $18\mathrm{kHz}$ . This corresponds to the max frequency preserved by various audio compression algorithms like MPEG, which make up the vast majority of our datasets. With balanced data sampling, we sample full-band audio from high-quality datasets (e.g. DAPS) just as much as possibly band-limited audio from datasets of unknown quality (e.g. Common Voice). This alleviates the issue, allowing our codec to reconstruct full-band audio, as well as band-limited audio.
4.6 Comparison to other methods
We now compare the performance of our final model with competitive baselines: EnCodec [8], Lyra [46], and Opus [37], a popular open-source audio codec. For EnCodec, Lyra, and Opus, we use publicly available open-source implementations provided by the authors. We compare using both objective and subjective evaluations, at varying bitrates. The results are shown in Table 3. We find that the proposed codec out-performs all competing codecs at all bitrates in terms of both objective and subjective metrics, while modeling a much wider bandwidth of  $22\mathrm{kHz}$ .
Figure 3: Listening tests at 44 KHz: MUSHRA scores, with  $95\%$  confidence intervals vs bitrate for EnCodec, our proposed approach, and the reference.
Codec	Bitrate (kbits)	Bandwidth (kHz)	Mel distance↓	STFT distance↓	ViSQOL ↑	SI-SDR↑
Proposed	1.78	22.05	1.39	1.95	3.76	2.16
2.67	22.05	1.28	1.85	3.90	4.41
5.33	22.05	1.07	1.69	4.09	8.13
8	22.05	0.93	1.60	4.18	10.75
EnCodec	1.5	12	2.11	4.30	2.82	-0.02
3	12	1.97	4.19	2.94	2.94
6	12	1.83	4.10	3.05	5.99
12	12	1.70	4.02	3.13	8.36
24	12	1.61	3.97	3.16	9.59
Lyra	9.2	8	2.71	4.86	2.19	-14.52
Opus	8	4	3.60	5.72	2.06	5.68
14	16	1.23	2.14	4.02	8.02
24	16	0.88	1.90	4.15	11.65
In Figure 3, we show the result of our MUSHRA study, which compares EnCodec to our proposed codec at various bitrates. We find that our codec achieves much higher MUSHRA scores than EnCodec at all bitrates. However, even at the highest bitrate, it still falls short of the reference MUSHRA score, indicating that there is room for improvement. We note that the metrics of our final model are still lower than the 24kbps model trained in our ablation study, as can be seen in Table 2. This indicates that the remaining performance gap may be closed by increasing the maximum bitrate.
In Figure 4 and Table 4, we compare our proposed model trained with the same exact configuration as EnCodec (24 KHz sampling rate, 24 kbps bitrate, 320 stride, 32 codebooks of 10 bits each) to existing baselines, in both quantitative and qualitative metrics. In Figure 5, we show qualitative results by sound category.
Figure 4: Listening tests at  $24\mathrm{KHz}$ : MUSHRA scores with  $95\%$  confidence intervals vs bitrate for EnCodec, our proposed approach with the same configuration, and the reference. Here all samples under comparison are resampled to  $24\mathrm{KHz}$ .
Table 3: Objective evaluation of the proposed codec at varying bitrates, along with results from competing approaches.
Codec	Bitrate (kbits)	Bandwidth (kHz)	Mel distance↓	STFT distance↓	ViSQOL ↑	SI-SDR ↑
Proposed@24kHz	1.5	12	1.48	2.24	4.04	0.32
3	12	1.24	2.01	4.23	4.44
6	12	1.00	1.78	4.38	8.44
12	12	0.74	1.54	4.51	12.51
24	12	0.49	1.33	4.61	16.40
EnCodec	1.5	12	1.63	2.69	3.98	0.02
3	12	1.46	2.54	4.16	2.99
6	12	1.30	2.39	4.30	6.06
12	12	1.15	2.28	4.39	8.44
24	12	1.05	2.21	4.42	9.69
Table 4: Encodec Configuration: Objective evaluation of the proposed model trained with the same configuration as EnCodec at varying bitrates, along with results from EnCodec.
Figure 5: MUSHRA by category: MUSHRA scores with  $95\%$  confidence intervals vs bitrate for our proposed model, EnCodec and reference.
We have presented a high-fidelity universal neural audio compression algorithm that achieves remarkable compression rates while maintaining audio quality across various types of audio data. Our method combines the latest advancements in audio generation, vector quantization techniques, and improved adversarial and reconstruction losses. Our extensive evaluation against existing audio compression algorithms demonstrates the superiority of our approach, providing a promising foundation for future high-fidelity audio modeling. With thorough ablations, open-source code, and trained model weights, we aim to contribute a useful centerpiece to the generative audio modeling community.
Broader impact and limitations: our model has the capability to make generative modeling of full-band audio much easier to do. While this unlocks many useful applications, such as media editing, text-to-speech synthesis, music synthesis, and more, it can also lead to harmful applications like deepfakes. Care should be taken to avoid these applications. One possibility is to add watermarking and/or train a classifier that can detect whether or not the codec is applied, in order to enable the detection of synthetic media generated based on our codec. Also, our model is not perfect, and still has difficulty reconstructing some challenging audio. By slicing the results by domain we find that, even though the proposed codec outperforms competing approaches across all of the domains, it performs best for speech and has more issues with environmental sounds. Finally, we notice that it does not model some musical instruments perfectly, such as glockenspiel, or synthesizer sounds.
[35] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779-4783. IEEE, 2018.
[36] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.04421, 2022.
[37] Jean-Marc Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. Technical report, 2012.
[38] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[40] Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017.
[41] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023.
[42] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[43] Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band melgan: Faster waveform generation for high-quality text-to-speech. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 492-498. IEEE, 2021.
[44] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021.
[45] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022.
[46] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[47] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33:1583-1594, 2020.
Modified codebook learning algorithm In our work, we use a modified quantization operation, given by:
$$
z _ {q} (x) = W _ {\text {o u t}} e _ {k}, \quad \text {w h e r e} \quad k = \arg \min  _ {j} | | \ell_ {2} \left(W _ {\text {i n}} z _ {e} (x)\right) - \ell_ {2} (e _ {j}) | | _ {2}
$$
Here,  $W_{\mathrm{in}}$  and  $W_{\mathrm{out}}$  are projection matrices, with  $W_{\mathrm{in}}$  mapping the encoder's output to an intermediate representation, and  $W_{\mathrm{out}}$  mapping this intermediate representation to the quantized representation  $z_{q}(x)$ . Specifically,
$$
W _ {\text {i n}} \in \mathbb {R} ^ {D \times M} \quad \text {a n d} \quad W _ {\text {o u t}} \in \mathbb {R} ^ {M \times D}
$$
where  $D$  is the output dimension of the encoder, and  $M$  is the codebook dimension with  $M \ll D$ .
The vector quantizer loss function is then defined to measure the reconstruction error and is given by:
$$
z _ {\text {p r o j}} (x) = W _ {\text {i n}} z _ {e} (x)
$$
$$
\mathcal {L} _ {\mathrm {V Q}} = \left| \left| \operatorname {s g} \left[ \ell_ {2} \left(z _ {\text {p r o j}} (x)\right) \right] - \ell_ {2} \left(e _ {k}\right) \right| \right| _ {2} ^ {2} + \beta \left| \left| \ell_ {2} \left(z _ {\text {p r o j}} (x)\right) - \operatorname {s g} \left[ \ell_ {2} \left(e _ {k}\right) \right] \right| \right| _ {2} ^ {2}
$$
where  $\mathrm{sg}$  is the stop gradient operator, preventing the back-propagation of gradients through  $e_k$ , and  $\beta$  is a hyperparameter controlling the balance between the two terms in the loss function.