High-Fidelity Audio Compression with Improved RVQGAN
Authors: Rithesh Kumar
Prem Seetharaman
Alejandro Luebs
Ishaan Kumar
Kundan Kumar
Venue: NeurIPS 2023 spotlight (2023)
Keywords: audio generation
audio compression
GAN
audio
speech
TL;DR: A method for neural audio compression that outperforms competing approaches in terms of audio quality, at much higher compression rates.

Abstract: Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.

High-Fidelity Audio Compression with Improved RVQGAN
Rithesh Kumar Descript, Inc.
Prem Seetharaman" Descript, Inc.
Alejandro Luebs Descript, Inc.
Ishaan Kumar Descript, Inc
Kundan Kumar Descript, Inc.
Abstract
Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, introduce high- High-fidelity universal neural audio compression algorithm that achieves 90x compression 44.1 KHz audio into tokens just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization tech- niques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with single universal model, making it widely applicable generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.
1 Introduction
Generative modeling of nigh-resolution audio difficult due to high dimensionality (-44,100 100 samples per second of audio) 19]. and presence of structure at different time-scales with both short and long-term dependencies. To mitigate this problem, audio generation typically divided into two stages: 1) predicting audio conditioned some intermediate representation such mel-spectrograms 24
19 30 and 2) predicting the intermediate representation given some conditioning information, hierarchical generative model, with observed intermediate variables. Naturally, an alternate formulation is to learn the intermediate variables using the variational auto-encoder (VAB) framework, with learned conditional prior predict the latent variables given some conditioning. This formulation, with continuous latent variables and training an expressive prior using normalizing flows has been quite successful for speech synthesis 17 36
28
such text 35 34 This be interpreted
closely related idea is to train the same varitional-autoencode with discrete latent variables using VQ-VAE [38] Arguably, discrete latent variables
better choice since expressive priors can be trained using powerful autoregressive models that have been developed for modeling distributions over discrete variables (27]. Specifically, transformer language models (39] have already exhibited the capacity to scale with data and model capacity to learn arbitrarily complex distributions such as
text[6], images [2 44, audio [5] 4I. music [1] etc. While modeling the prior is straightforward, modeling the discrete latent codes using quantized auto-encoder remains a challenge.
"Equal contribution to this work Address correspondence papers@descript.commm or raise an issue at http:///ggthubcccc////ecrctptttp
audio codec
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Learning these discrete codes can be interpreted as lossy compression task, where the audio signal compressed into discrete latent space by vector-quantzing the representations of an autoencoder using fixed length codebook. This audio compression model needs to satisfy the following properties: 1) Reconstruct audio with high fidelity and free of artifacts 2) Achieve high level of compression along with temporal downscaling to learn a compact representation that discards low-level imperceptible details while preserving high-level structure [38 [33 3) Handle all types of audio such as speech, music, environmental sounds, different audio encodings (such as mp3) as well as different sampling rates using a single universal model.
While the recent neural audio compression algorithms such as SoundStream [46 and EnCodec g partially satisfy these properties, they often suffer from the same issues that plague GAN-based generation models. Specifically, such models exhibit audio artifacts such as tonal artifacts (29), pitch and periodicity artifacts (25) and imperfectly model high- figh-frequencies leading to audio that are clearly distinguishable from originals. These models are often tailored to a specific type of audio signal such as speech or music and struggle to model generic sounds. We make the following contributions:
We introduce Improved RVQGAN a high fidelity universal audio compression model, that can compress 44.1 KHz audio into discrete codes at 8 kbps bitrate (-90x compression) with minimal loss in quality and fewer artifacts. Our model outperforms state-of-the-art methods by a large margin even at lower bitrates (higher compression). when evaluated with both quantitative metrics and qualitative listening tests.
We identify a critical issue in existing models which don't 't utilize the full bandwidth due to codebook collapse (where fraction of the codes are unused) and fix it using improved codebook learning techniques.
We identify a side-effect of quantizer dropout- a technique designed to allow a single model to support variable bitrates, actually hurts the full-bandwidth audio quality and propose a solution to mitigate it.
We make impactful design changes to existing neural audio codecs by adding periodic inductive biases, multi-scale STFT discriminator, multi-scale mel loss and provide thorough ablations and intuitions to motivate them.
Our proposed method is a universal audio compression model, capable of handling speech, music, environmental sounds, different sampling rates and audio encoding formats.
We provide code |
models, and audio samples that we encourage the reader listen to.
2
Related Work
High fidelity neural audio synthesis: Recently, generative adversarial networks (GANs) have emerged as a solution to generate high-quality audio with fast inference speeds, due to the feed- forward (parallel) generator. MelGAN [19] successfully trains a GAN-based spectrogram inversion (neural vocoding) model. introduces a multi-scale waveform discriminator (MSD) to penalize structure at different audio resolutions and a feature matching loss that minimizes L1 distance between discriminator feature maps of real and synthetic audio. HifiGAN [88 refines this recipe by introducing multi-period waveform discriminator (MPD) for high fidelity synthesis, and adding an auxiliary mel-reccnstrruction loss for fast training. UnivNet [66 introduces a multi-resolution spectrogram discriminator (MRSD) to generate audio with sharp spectrograms. BigVGAN (21) extends the HifGGAN recipe by introducing a periodic inductive bias using the Snake activation function 47.. also replaces the MSD in HifiGAN with the MRSD to improve audio quality and reduce pitch, periodicity artifacts (25). While these the GAN-based learning techniques are used for vocoding, these recipes are readily applicable to neural audio compression. Our Improved RVQGAN model closely follows the BGAN training recipe, with a few key changes. Our model uses a new mulli--aane multi-scale STFT discriminator that alleviates aliasing artifacts, and multi-scale mel-reconstrruction loss that better model quick transients.
Neural audio compression models: VQ-VAES 38 have been the dominant paradigm to train neural audio codecs. The first VQ-VAF based speech codec was proposed in [3] operating at 1.6 kbps. This model used the original architecture from 38 with a convolutional encoder and autoregressive

---

wavenet [27] decoder. SoundStream [46] is one of the first universal compression models capable of handling diverse audio types, while supporting varying bitrates using single model. They use a fully causal convolutional encoder and decoder network, and perform residual vector quantization (RVQ). The model is trained using the VQ-GAN [22 formulation, by adding adversarial and feature- matching losses along with the multi-scale spectral reconstruction loss. EnCodec [8] closely follows the SoundStream recipe, with a few modifications that lead to improved quality. EnCodec uses a multi- scale STFT discriminator with a multi-scale spectral reconstruction loss. They use a loss balancer which adjusts loss weights based on the varying scale gradients coming from the discriminator.
Our proposed method also uses convolutional encoder-decoder architecture, residual vector quan- tization and adversarial, perceptual losses. However, our recipe has the following key differences: 1) We introduce periodic inductive bias using Snake activations [47 2) We improve codebook learning by projecting the encodings into a low-dimensional space 44 3) We obtain a stable training recipe using best practices for adversarial and perceptual loss design, with fixed loss weights and without requiring a sophisticated loss balancer. We find that our changes lead to a near-optimal effective bandwidth usage. This allows our model to outperform EnCodec even with 3x lower bitrate.
Language modeling of natural signals: Neural language models have demonstrated great success in diverse tasks such as open-ended text generation [6] with in-context learning capabilities. A rey-component of these models self-attentie [39. which is capable of modeling complex and long-range dependencies but suffers from a quadratic computational cost with the length of the sequence. This cost is unacceptable for natural signals such as images and audio with very high dimensionality, requiring compact mapping into a discrete representation space. This mapping is typically learnt using VQ-GANs [2 44. followed by training an autoregressive Transformer on the discrete tokens. This approach has shown success across image [5 44 (2), audio [5] 41, video and
music I domains. Codecs like SoundStream and EnCodec have already been used in generative audio models, like AudioLM [5), MusicLM (., and VALL-E [4]. Our proposed model can serve as a drop-in replacement for the audio tokenization model used in these methods, allowing for highly superior audio fidelity, and more efficient learning due to our maximum entropy code representation.
3 The Improved RVQGAN Model
Our model is built on the framework of VQ-GANs. following the same pattern as SoundStream [46] and EnCodec (8) Our model uses the fully convolutional encoder-decoder network from SoundStream, that performs temporal downscaling with chosen strid- ing factor. Following recent literature, we quantize the encodings using Residual Vector Quantization (RVQ), a method that recursively quantizes residuals following an initial quantization step with distinct codebook. Quantizer dropout is applied during train- ing to enable a single model that can operate at several target bitrates. Our model is similarly trained using frequency domain reconstruction loss along with adversarial and perceptual losses.
of 10-bit codebookk
Camplint rute (HHz)
Target bitrate (kbps)
Compression Factor
Frume male (lle)
Striding factor
Codec
Proposed
44.1
8
512
86
9.
91.16
24
320
75
32
16
EnCodee
24
320
150
16
32
SoundStreum
24
6
320
75
8
64
Table 1: Comparison of compression approaches.
An audio signal with sampling rate fa (Hz), encoder striding factor M, and N₂ layers of RVQ produce a discrete code matrix of shape S x Ng. where S is the frame rate defined as f./M. Table compares our proposed model against baselines to contrast the compression factors and the frame rate of latent codes. Note that the target bitrate mentioned is an upper bound, since all models support variable bitrates. Our model achieves higher compression factor compared all baseline methods while outperforming them in audio quality, as we show later. Finally, lower frame rate is desirable when training a language model on the discrete codes, it results in shorter sequences.
3.1 Periodic activation function
Audio waveforms are known to exhibit high periodicity (especially in voiced components, music, etc.) While current non-autoregressive audio generation architectures are capable of generating high fidelity audio, they often exhibit jarring pitch and periodicity artifacts (25). Moreover, common
3

---

neural network activations (such as Leaky ReLUs) are known to struggle with extrapolating periodic signals, and exhibit poor aut-of-distribution generalization for audio synthesis [2].
To add a periodic inductive bias to the generator, we adopt the Snake activation function proposed by Liu et al. [47] and introduced to the audio domain in the VGAN neural vocoding model [2T]. It is defined as (nake(r) = x + sinz(az)) where a controls the frequency of periodic component of the signal. In our experiments, we find replacing Leaky ReLU activations with Snake function be an influential change that significantly improves audio fidelity (Table
3.2
Improved residual vector quantization
While vector quantization (VQ) is a popular method train discrete auto-encoder, there are many practi- cal struggles when training them. Vanilla VQ- VAEs struggle from low codebook usage due to poor initial- ization, leading to a significant portion the code- book being unused. This reduction in effective code- book size leads to an implicit reduction in target bi- trate, which translates poor reconstruction quality.
10.0
9.8
Entropy (bits)
9.4
9.2
Proposed (24kbps)
Proposed
To mitigate this, recent audio codec methods use means clustering initialize the codebook vectors, and manually employ randomized restarts [ when certain codebooks are unused for several batches. However, we find that the EnCodec model trained at 24kbps target bitrate, well our proposed model with the same codebook learning method (Proposed
EnCadec
8.6
Proposed (w/
30
Codebook Index
Figure 1: Entropy for each codebook, computed using code usage statistics across large test set. w/ EMA) still suffers from codebook under-utilization (Figure [
To address this issue, we use two key techniques introduced the Improved VQGAN image model 44] to improve codebook usage: factorized codes and L2-normalized codes. Factorization decouples code lookup and code embedding, by performing code lookup in a low-dimensional space (8d or 32d) whereas the code embedding resides in a high dimensional space (1024d). Intuitively, this can be interpreted as a code lookup using only the principal components of the input vector that maximally explain the variance in the data. The I22normalization of the encoded and codebook vectors converts euclidean distance to cosine similarity, which is helpful for stability and quality 44
These two tricks along with the overall model recipe significantly improve codebook usage, and therefore bitrate efficiency (Figure and reconstruction quality (Table while being simpler to implement. Our model can be trained using the original VQ- IQ-VAB codebook and commitment losses [38). without k-means initialization or random restarts. The equations for the modified codebook learning procedure are written in Appendix A
3.3 Quantizer dropout rate
Quantizer dropout was introduced in SoundStream [6 to train a single compression model with vari- able bitrate. The number of quantizers N, determine the bitrate, so for each input example we randomly
Dropout prob.
- 0.0
Metreconstructon 1o5
sample {1, 2, ..., N₂) and only use the first ny quantizers while training. However, we noticed
. 1.0
2.5
that applying quantizer dropout degrades the audio reconstruction quality at full bandwidth (Figure 2)
To address this problem, we instead apply quantizer dropout to each input example with some probabil- ity p. Interestingly, we find that dropout probability p = 0.5 closely matches the reconstruction quality of baseline at lower bitrates, while closing the gap to full-bandwidth quality of model trained without quantizer dropout (p 0.0).
1.5
2
3
6
7
Bitrate (Kops)
Figure 2: Effect of quantizer dropout audio quality vs bitrate.

---

Moreover, we provide additional insight into the practical behavior of quantizer dropout and it's interaction with RVQ. Firstly, we find that these techniques put together lead the quantized codes to learn most-significant to least significant bits of information with each additional quantizer. When the codes are reconstructed with 1 .... N₂ codebooks, we can see each codebook adds increasing amounts of fine-scale detail. We believe this interaction is beneficial when training hierarchical generative models on top of these codes [5] 41 M. for example to partition the codes into "coarse" tokens (denoting the most significant codes) and "fine" tokens.
3.4 Discriminator design
Like prior work, we use multi-scale (MSD) and multi-period waveform discriminators (MPD) which lead to improved audio fidelity. However, spectrograms of generated audio can still appear blurry, exhibiting ovor-smoothing artifacts in high frequencies[I6] The multi-resolution spectrogram discriminator (MRSD) was proposed in UnivNet fix these artifacts and Big VGAN [2I] found that it also helps to reduce pitch and periodicity artifacts. However, using magnitude spectrograms discards phase information which could' ve been otherwise utilized by the discriminator penalize phase modeling errors. Moreover, we find that high- Sigh-frreuency modeling is still challenging for these models especially high sampling rates.
To address these issues, we use complex STFT discriminator [46] at multiple time-scales [8] and find that it works better in practice and leads to improved phase modeling. Additionally find that splitting the STFT into sub-bands slightly improves high frequency prediction and mitigates aliasing artifacts, since the discriminator
learn discriminative features about specific sub-band and provide a stronger gradient signal to the generator. Multi-band processing was earlier proposed to predict audio in sub-bands which are subsequently summed to produce the full-band audio.
in 43
3.5 Loss functions
Frequency domain reconstruction loss: while the mel-reccosttructin loss 18 is known to improve stability, fidelity and convergence speed, the multi-scale spectral losses 42
15 encourage model-
ing of frequencies in multiple time-scales. In our model, we combine both methods by using L1 loss on mel-spectrograms computed with window lengths of [32, 64, 128, 256, 512, 1024, 2048) and hop length set to window_Ke /eooowhhhh//////e / 4. We especially find that using the lowest hop size of 8 improves modeling of very quick transients that are especially common in the music domain.
EnCodec [8] uses similar loss formulation, but with both LI and L2 loss terms, and a fixed mel bin size of 64. We find that fixing mel bin size leads to holes in the spectrogram especially low filter lengths. Therefore, we use mel bin sizes [5, 10, 20, 40, 80, 160, 320] corresponding to the above filter lengths which were verified to be correct by manual inspection.
our model uses the multi multi-period discriminator [8] for waveform discrimination, as well the proposed multi-band multi-scale STFT discriminator for the frequency domain. We use the HingeGAN (22 adversarial loss formulation, and apply the L1 feature matching loss [19
Adversarial loss:
Codebook learning: we use the simple codebook and commitment losses with nop-gradients from the original VQ- formulation [38), and backpropagate gradients through the codebook lookup
using the straight-throug! estimator
Loss weighting: we use the loss weightings of 15.0 for the multi-scal mel loss, 2.0 for the feature matching loss, 1.0 for the adversarial loss and 1.0, 0.25 for the codebook and commitment losses respectively. These loss weightings are in line with recent works [8] 21 (which use 45.0 weighting for the mel loss), but simply rescaled to account for the multiple scales and login base we used for computing the mel loss. We don't use loss balancer as proposed in EnCodec [8]
Experiments
4.1 Data sources
We train our model on large dataset compiled of speech, music, and environmental sounds. For speech, we use the DAPS dataset (26). the clean speech segments from DNS Challenge 4 [10,, the Common Voice dataset (2), and the VCTK dataset [40]. For music, the MUSDB dataset
5

---

B1 and the Jamendo dataset 4 Finally, for environmental sound, we use both the balanced and unbalanced train segments from AudioSet [4]. All audio resampled to 44kHz.
During training, we extract short excerpts from each audio file, and normalize them to -24 dB LUFS. The only data augmentation we apply randomly shift the phase of the excerpt, uniformly. For evaluation, we use the evaluation segments from AudioSet [14, two speakers that are held out from DAPS (26] (F10, M10) for speech, and the test split of MUSDB [3]. We extract 3000 10-second segments (1000 from each domain), as our test set.
4.2 Balanced data sampling
We take special care how we sample from our dataset. Though our dataset resampled to 44kHz, the data within it may be band-limited in some way. That is, some audio may have had an original sampling rate much lower than 44kHz. This is especially prevalent in speech data, where the true sampling rates of the underlying data can vary greatly (e.g. the Common Voice data is commonly 8-16kH2). When we trained models on varying sampling rates, we found that the resultant model often would not reconstruct data above certain frequency. When investigating, we found that this threshold frequency corresponded to the average true sampling rate of our dataset. To fix this, we introduce a balanced data sampling technique.
We first split our dataset into data sources that we know to be full-band they are confirmed to contain energy in frequencies up the desired Nyquist frequency (22.05kHz) of the codec- and data sources where we have no assurances of the max frequency. When sampling batches, we make sure that a full-band item is sampled. Finally, ensure that in each batch, there are an equal number of items from each domain: speech, music, and environmental sound. In our ablation study, we examine how this balanced sampling technique affects model performance.
4.3 Model and training recipe
Our model consists of a convolutional encoder, a residual vector quantizer, and a convolutional decoder. The basic building block of our network is a convolutional layer which either upsamples or downsamples with some stride, followed by a residual layer consisting of convolutional layers interleaved with non-linear Snake activations. Our encoder has of these layers, each of which downsamples the input audio waveform at rates [2, 4, 8, 8). Our decoder has 4 corresponding layers, which upsample at rates [8, 8, 4, 2]. We set the decoder dimension to 1536. In total, our model has 76M parameters, with 22M in the encoder, and 54M in the decoder. We also examine decoder dimensions of 512 (31M parameters) and 1024 (49M parameters).
We use the multi-period discriminator [18). and a complex multi-scale STFT discriminator. For the first, we use periods of [2, 3, 5, 7, 11], and for the second, we use window lengths [2048, 1024, 512), with a hop-length that is 1/4 the window length. For band-splitting of the STFT, we use the band- limits [0.0, 0.1, 0.25, 0.5, 0.75, 1.0]. For the reconstruction loss, we use distance between log-mel spectrograms with window lengths [32, 64, 128, 256, 512, 1024, 20448, with corresponding number of mels for each of [5, 10, 20, 40, 80, 160, 320). The hop length is 1/4 of the window length. We use feature matching and codebook losses, as described in Section 3.5
For our ablation study, we train each model with batch size of 12 for 250k iterations. In practice, this takes about 30 hours to train on a single GPU. For our final model, we train with a batch size of 72 for 400k iterations. We train with excerpts of duration 0.38s. We use the AdamW optimizer 23 with a learning rate of le - 4, 31 = 0.8, and 82 = 0.9, for both the generator and the discriminator. We decay the learning rate every step, with 7 0.999996.
4.4 Objective and subjective metrics
To evaluate our models, we use the following objective metrics:
ViSQOL 7 an intrusive perceptual quality metric that uses spectral similarity to the ground truth to estimate mean opinion score.
2.
Mel distance: distance between log mel spectrograms of the reconstructed and ground truth waveforms. The configuration of this loss is the same as described in 3.5

---

Bitrate efficiency ?
# of STFT bunds
Multi-scale mel.
Quant method
Quant dropout
Balanced sump.
Mel distance J
STFT distance J.
Decoder dim.
Bitrate (kbps)
Multi-period
Single sesael
Activation
Latent dim
ViSQOL
SI-SDR 1
Ablation on
1536
snake
V
V
8
Proj.
1.0
8
1.09
1.82
3.96
9.12
99%
512
snake
/
5
/
8
Proj.
1.0
8
1.11
1.83
3,91
8.72
99%
Architecture
1024
snake
/
X
5
<
8
Proj.
1.0
8
1.07
1.82
3.96
9.07
99%
1536
relu
/
X
5
Proj.
1.0
8
1.17
1.81
3.83
6.92
99%
1536
snake
x
X
Proj. 1.0
1.13
1.92
4.12
1.07
62%
Discriminator
1536
snake
/
Proj.
1.0
1.07
1.80
3.98
9.07
99%
1536
snake
x
-
Proj.
1.0
1.07
1.81
3.97
9.04
99%
1536
snake
x
5
Proj.
1.0
8
1.08
1.82
3.95
8.51
99%
Reconstruction loss
1536
snake
V
5
X
8
Proj. 1.0
8
1.10
1.87
4.01
7.68
99%
1536
snake
V
2
Proj. 1.0
8
1.44
2.08
3.65
2.22
84%
1536
snake
/
5
-
4
Proj.
1.0
8
1.20
1.89
3.86
7.15
97%
Latent dim
1536
snake
/
5
32
Proj.
1.0
8
1.10
1.84
3.95
9.05
98%
1536
snake
/
5
256
Proj.
1.0
8
1.31
1.97
3.79
5.09
59%
1536
snake
/
EMA
1.0
1.11
1.84
3.94
8.33
97%
1536
snake
/
Proj.
0.0
0.98
1.70
4.09
14
99%
Quantization setup
1536
snake
/
<
Proj.
0.99
1.69
4.04
10.00
99%
1536
snake
<
Proj. 0.5
1.01
1.75
4.03
9.74
99%
1536
snake
/
5
V
8
Proj. 1.0
0.73
1.62
4.16
13.83
99%
Data
1536
snake
V
5 8
Proj. 1.0
8
X
1.09
1.94
3.89
8.89
99%
Table 2: Results of the ablation study our proposed codec. The final model is trained with the same configuration as the baseline (top row), but with a quantization dropout of 0.5.
3. STFT distance: distance between log magnitude spectrograms of the reconstructed and ground truth waveforms. We use window lengths [2048, 512]. This metric captures the fidelity in higher frequencies better than the mel distance.
4. Scale-invariant source-to-distortion ratio (SI-SDR) 20 distance between waveforms, ratio, with modifications so that it is invariant to scale differences. When considered alongside spectral metrics, SI-SDR indicates the quality of the phase
similar to signal-to
reconstruction of the audio.
5. Bitrate efficiency: We calculate bitrate efficiency as the sum of the entropy (in bits) of each codebook when applied on a large test set divided by the number of bits across all codebooks. For efficient bitrate utilization this should tend 100% and lower percentages indicate that the bitrate is being underutilized.
We also conduct a MUSHRA -iinpireed listening test, with hidden reference. but no low-passed anchor. In it each one of ten expert listeners rated 12 randomly selected 10-second samples from evaluation set, 4 of each domain; speech, music and environmental sounds. We compare proposed system at 2.67kbps, 5.33kbps and 8kbps to EnCodec at 3kbps, 6kbps and 12kbps.
4.5
Ablation study
We conduct a thorough ablation study our model, varying components of our training recipe and model configuration one-by-one. To compare models, we use the four objective metrics described in Section The results of our ablation study can be seen in Table
Architecture: We find that varying the decoder dimension has some effect on performance, with smaller models having consistently worse metrics. However, the model with decoder dimension 1024 has similar performance to the baseline, indicating that smaller models can still be competitive. The change with the biggest impact was switching out the relu activation for the snake activation. This change resulted in much better SI-SDR and other metrics. Similar to the results in Big (21). we find that the periodie inductive bias of the snake activation helpful for waveform generation. For our final model, we use the largest decoder dimension (1536), and the snake activation.
7

---

Discriminator: Next, we removed or changed the discriminators one-by-one, to see their impact on the final result. First, we find that the multi-band STFT discriminator does not result in significantly better metrics, except for SI-SDR, where it is slightly better. However, when inspecting spectrograms of generated waveforms, we find that the multi-band discriminator alleviates aliasing of high fre- quencies. The upsampling layers of the decoder introduce significant aliasing artifacts [29] The multi-band discriminator is more easily able to detect these aliasing artifacts and give feedback to the generator to remove them. Since aliasing artifacts are very small in terms of magnitude, their effect on our objective metrics is minimal. Thus, we keep the multi-band discriminator.
We find that adversarial losses are critical to both the quality of the output audio, as well as the bitrate efficiency. When training with only reconstruction loss, the bitrate efficiency drops from 99% to 62%, and the SI-SDR drops from 9.12 to 1.07. The other metrics capture spectral distance, and are relatively unaffected. However, the audio from this model has many artifacts, including buzzing, as has not learned to reconstruct phase. Finally, we found that swapping the multi multi-period discriminator for the single-scale waveform discriminator proposed in MelGAN 19] resulted in worse SI-SDR. We retain the multi-period discriminator.
Impact of low-hop reconstruction loss: We find that low-hop reconstruction is critical to both the waveform loss and the modeling of fast transients and high frequencies. When replaced with a single-scale high-hop mel reconstruction (80 mels, with window length of 512), we find significantly lower SI-SDR (7.68 from 9.12). Subjectively, we find that this model does much better at capturing certain sorts of sounds, such as cymbal crashes, beeping and alarms, and singing vocals. We retain the multi-scale mel reconstruction loss in our final recipe.
Latent dimension of codebook: the latent dimension of the codebook has significant impact on bitrate efficiency, and consequently the reconstruction quality. If set too low or too high (e.g. 2. 256), quantitative metries are significantly worse with drastically lowered bitrate efficiency. Lower bitrate efficiency results in effectively lowered bandwidth, which harms the modeling capability of the generator. As the generator weakened, the discriminator tends to "win", and thus the generator does not learn to generate audio with high audio quality. We find 8 to be optimal for the latent dimension.
Quantization setup: we find that using exponential moving average as the codebook learning method, as in EnCodec[8], results in worse metrics especially for SI-SDR. It also results in poorer codebook utilization across all codebooks (Figure D. When taken with its increased implementation complexity (requiring K- K-Means initialization and random restarts), we retain our simpler projected lookup method for learning codebooks, along with a commitment loss. Next, we note that the quantization dropout rate has a significant effect on the quantitative metrics. However, as seen in Figure 2 a dropout of 0.0 results in poor reconstruction with fewer codebooks. As this makes usage of the codec challenging for downstream generative modeling tasks, we instead use a dropout rate of 0.5 in our final model. This achieves a good trade-off between audio quality at full bitrate as well as lower bitrates. Finally, we show that we can increase the max bitrate of our model from 8kbps to 24kbps and achieve excellent audio quality, surpassing all other model configurations. However, for our final model, we train the lower bitrates, in order to push the compression rate as much as possible.
Balanced data sampling: When removed, this results in worse metrics across the board. Empirically, we find that without balanced data sampling, the model produces waveforms that have a max frequency of around 18kHz. This corresponds the max frequency preserved by various audio compression algorithms like MPEG, which make up the vast majority of our datasets. With balanced data sampling, we sample full-band audio from high-quality datasets (e.g. DAPS) just as much as possibly band-limited audio from datasets of unknown quality (e.g. Common Voice). This alleviates the issue, allowing our codec to reconstruct full-band audio, as well band-limited audio.
4.6 Comparison to other methods
We now compare the performance of our final model with competitive baselines: EnCodec g Lyra [6]. and Opus 377. popular open-source audio codec. For EnCodec, Lyra, and Opus, we use publicly available open-source implementations provided by the authors. We compare using both objective and subjective evaluations, at varying bitrates. The results are shown in Table 3 We find that the proposed codec out-performs all competing codees at all bitrates terms of both objective and subjective metrics, while modeling much wider bandwidth of 22kHz.
8

---

Bundwidth (kHz)
STFT distance
Bitrate (kbps)
Mel distance
ViSQOL
SI-SDR
Codec
1.78 22.05
1.39
1.95
3.76
2.16
70
Proposed
2.67
22.05
1.85
3.90
4.41
MUSHIA score
5.33
22.05
1.07
1.69
4.09
8.13
60
8
22.05
0.93
1.60
4.18
10.75
50
1.5
2.11
4.30
2.82
-0.02
3
12
1.97
4.19
2.94
2.94
EnCodec
6
12
1.83
4.10
3.05
5,99
30
Reference
12
12
1.70
4.02
3.13
8.36
20
Proposed
24
12
1.61
3.97
3.16
9.59
EnCodec
10
Lyra
9.2
8
2.71
4.86
2.19
-14.52
2
Bitrate (kbps)
8
10
12
8
3.60
5.72
2.06
5.68
Opus
14
16
1.23
2. 2.14
4.02
8.02
24
16
0.88
1.90
4.15
11.65
Figure 3: Listening tests at 44 KHz: MUSHRA scores, with 95% confidence intervals bitrate for EnCodee, our proposed approach, and the erence.
Objective evaluation of the proposed codec varying bitrates, along with results from competing approaches.
Table
In Figure 3, we show the result of our MUSHRA study, which compares EnCodec to our proposed codec various bitrates. We find that our codec achieves much higher MUSHRA scores than EnCodec at all bitrates. However, even at the highest bitrate, it still falls short of the reference MUSHRA score, indicating that there is room for improvement. We note that the metrics of our final model are still lower than the 24kbps model trained in our ablation study, as can be seen in Table 2 This indicates that the remaining performance gap may be closed by increasing the maximum bitrate.
In Figure and Table 4, we compare our proposed model trained with the same exact configuration as EnCodec (24 KHz sampling rate, 24 khps bitrate, 320 stride, 32 codebooks of 10 bits each) existing baselines, in both quantitative and qualitative metrics. In Figure 5, we show qualitative results by sound category.
Bundwidth (kHz)
STFT distance 4
Mel distance 4
Bitrate (khps)
80
ViSQOL
SI-SDR
70
MUSHRA 5c0re
60
Codec
1.5
12
1.48
2.24
4.04
0.32
12
1.24
2.01
4.23
4.44
40
Proposed @ 24kHz
12
1.00
1.78
4.38
8.44
30
12
0.74
1.54
4.51
12.51
Reference
0.49
1.33
16.40
20
Proposed
EnCodec
1.5
12
1.63
2.69
3.98
0.02
1.46
2.54
4.16
2,99
2
8
10
12
Bitrate (kbps)
EnCodec
1.30
2.39
4.30
6.06
12
12
1.15
2.28
4.39
8.44
24
12
1.05
2.21
4.42
9.69
Figure 4: Listening tests at 24 KHz: MUSHRA scores with 95% confidence intervals vs hitrate for EnCodee, our proposed approach with the same configuration, and the reference. Here samples under comparison are resampled to 24 KHz.
Table 4: Encodee Configuration: Objective evalu- ation the proposed model trained with the configuration EnCodec varying bitrates, along with results from EnCodec.

---

8
70
Seference (qpeec))
20
Eefrrenza (uuec)
Sabrennc jend Prgceses (ore) Encocee
Prsessed Lepocni
Prppord mruad
EnCasee trpeach)
(mudc)
Ditrate (copa)
Brtaate (ctpal
B trate (cips)
Figure 5: MUSHRA by category: MUSHRA scores with 95% confidence intervals vs bitrate for our proposed model, EnCodec and reference.
Conclusion
We have presented high-fidelii universal neural audio compression algorithm that achieves markable compression rates while maintaining audio quality across various types of audio data. Our method combines the latest advancements in audio generation, vector quantization techniques, and improved adversarial and reconstruction losses. Our extensive evaluation against existing au- dio compression algorithms demonstrates the superiority of our approach, providing promising foundation for future high-fidelity audio modeling. With thorough ablations, open-source code, and trained model weights, we aim contribute useful centerpiece the generative audio modeling community.
Broader impact and limitations: model has the capability to make generative modeling of full-band audio much easier to do. While this unlocks many useful applications, such as media editing, text-to-speech synthesis, music synthesis, and more, it can also lead to harmful applications like deepfakes. Care should be taken to avoid these applications. One possibility is to add watermarking and/or train classifier that detect whether not the codec is applied, in order enable the detection of synthetic media generated based on our codec. Also, our model is not perfect, and still has difficulty reconstructing some challenging audio. By slicing the results by domain even though the proposed codec outperforms competing approaches across all of the domains, performs best for speech and has issues with environmental sounds. Finally, notice that does not model some musical instruments perfectly, such as glockenspeil, or synthesizer sounds.
find that,
10

---

References
[1] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.1133 2023.
[2] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multiiingge speech corpus. arXiv preprint arXiv: 2019.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprini 1308v1300..444 2013.
[4] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg- jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019, URL http://hdl. www.ha..ee..e.........m.e.e...aa wwtande..ee/t0eee net/12300/42015
[5] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint urXiv:2209003/1 2022.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 187771911 2020.
[7] Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pages 1-6. IEEE, 2020.
[8] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 2022.
[9] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXiv preprint (xXiv:2005.00341 2020.
[10] Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sergiy Matusevych, Sebastian Braun, Emre Sefik Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner. Icassp 2022 deep noise suppression challenge. In ICASSP, 2022.
[11] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint arXiv:2001. .04643,,0004 2020.
[12] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for regouuesolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873- 2021.
[13] Cristina Gârbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and a wavenet decoder. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 735-739. IEEE, 2019.
[14] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human haben-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017
[15] Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33:13062- 2020.
[16] Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation. arXiv preprint arXiv:2106. 07889, 2021.
11

---

[17] Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning. pages 5530-5540. PMLR, 2021.
[18] Jungil Kong. Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-111 17033, 2020.
[19] Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
[20] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 626-630. IEEE, 2019.
[21] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigygan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658 2022.
[22] Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: arXiv:1005.02894 2017.
[23] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[24] Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837,, 2016.
[25] Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive gan for conditional waveform synthesis. arXiv preprint arXiv:21 I0. 10139, 2021.
[26] Gautham J Mysore. Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech? a dataset, insights, and challenges. IEEE Signal Processing Letters, 22(8):1006-100 1010, 2014.
[27] Aaron van den Oord. Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint orXiv:/6099 03499, 2016.
[28] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice Scaling text-to-speech with convolutional sequence learning. arXiv preprint arXiv:1710.07654. 2017.
[29] Jordi Pons, Santiago Pascual, Giulio Cengarle, and Joan Serrà. Upsampling artifacts in neural audio synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3005-3009, IEEE, 2021.
[30] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEEE, 2019.
[31] Zafar Rafii Antoine Liutkus, Fabian-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The musdb18 corpus for music separation, 2017.
[32] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[33] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2.-vae-2. Advances in neural information processing systems, 32, 2019.
[34] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 2005002000004558, 2020.
12

---

[35] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj -yam,, al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing ( (ICASSP), pages 4779-4783, 4783. IEEE, 2018.
[36] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng. Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.044 04421, 2022.
[37] Jean-Mare Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. Technical report, 2012.
[38] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[40] Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017.
[41] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint arXiv:2301 0001010211, 2023.
[42] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[43] Geng Yang, Shan Yang. Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band melgan: Faster waveform generation for high-quality text-to-speech. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 492 498. IEEE, 2021.
[44] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint
10.006277 2021.
[45] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich eex-to-imaae generation. arXiv preprint arXiv:2206. 10789, 2022.
[46] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[47] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how fix it. Advances in Neural Information Processing Systems, 33:1583-1599 2020.
13

---

A Appendix
Modified codebook learning algorithm our work, we use modified quantization operation, given by:
(x) W Ck where A aag min ( E2(e))]l2
Here, Wnn and Wout are projection matrices, with Wn mapping the encoder's output to an intermediate representation, and W out mapping this intermediate representation the quantized representation (z). Specifically,
Win E R DxM and W RRMxD
where the output dimension of the encoder, and M is the codebook dimension with M << D. The vector quantizer loss function is then defined to measure the reconstruction error and is given by:
Epro ()) = Win Ze (x:
Lvo ||sg[22( Sproj (z)) & (cx)) + B11 (2 (x)))(=)) - sg(E2(e=)]1B
where sg is the stop gradient operator, preventing the back-pre procc-pappgttion of gradients through Cj. and is hyperparameter controlling the balance between the terms in the loss function.
14


Reviews:
[{"review_id": "VRIhZMXFoC", "reviewer": "Reviewer_d8D3", "summary": "The authors propose a neural audio codec model that demonstrates superior performance compared to previous works, and present experimental results.", "questions": "Based on the MUSHRA scores curves, it appears that higher bitrates yield better scores, and the highest quality that this method can achieve remains unconfirmed. Is there a specific reason for this?", "limitations": "Limitations have been well described.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors appropriately explain the problem they aim to address.\n- Their method is adequately described.\n- The authors provide a specific implementation, ensuring reproducibility.\n- The claims made are reasonable, and the experiments and results support them.\n- The authors' various ablation studies can be helpful for future work.\n", "weaknesses": "- For a neural audio codec to be utilized like traditional audio codecs, it should not fail in any patterns. Data-driven neural audio codecs have not been proven to be sufficiently stable from this perspective. Although the authors divided the original dataset into a training set and evaluation set, it is necessary to validate whether the proposed audio codec works well on more diverse and completely different audio data. Additionally, finding failure cases of previous works and comparing them can serve as strong evidence supporting the superiority of the authors' proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your  time and feedback. Please find our answers below to clarify some details about our model.\n\n> …*it is necessary to validate whether the proposed audio codec works well on more diverse and completely different audio data. Additionally, finding failure cases of previous works and comparing them can serve as strong evidence supporting the superiority of the authors' proposed method.*\n> \n\nTo the best of our best knowledge, our model doesn’t have clear failure cases on any audio domain. However, we noticed that some specific sounds like cymbal crashes in music or glockenspiel type sounds aren’t modeled accurately. We leave this to future work to identify why there’s certain limitations on the specific sounds that can be modeled.\n\nOtherwise, we found that our proposed model clearly fixes failure cases in previous works (such as EnCodec). Specifically, EnCodec poorly models background noise (such as room tone), reverb in speech data, but our proposed model almost perfectly reconstructs such cases. This is illustrated in our samples page.\n\n> *Based on the MUSHRA scores curves, it appears that higher bitrates yield better scores, and the highest quality that this method can achieve remains unconfirmed. Is there a specific reason for this?*\n> \n\nWe did not test our model at significantly higher bitrates since our goal was to achieve the highest rate of compression possible. However, we additionally trained our model at 16 kbps max bitrate with 44.1 KHz audio and found our metrics to significantly improve over the 8 kbps model (Table 2 in the attached pdf)."}, {"review_id": "3GDoZw7Tv9", "reviewer": "Reviewer_B66v", "summary": "This paper introduces a RVQGAN-based neural audio codec method, demonstrating superior audio reconstruction quality, a high compression rate, and generalization across diverse audio domains. The authors substantiate the significant performance superiority of their model over alternatives through extensive and thorough qualitative and quantitative experiments. They present and validate their technique to fully utilize residual vector quantization, alongside model, discriminator, and loss design choices for enhanced performance.", "questions": "* Could the model be applied to downstream applications such as training text-to-speech (TTS) models? Previous works like EnCodec and SoundStream utilized causal architectures to make them suitable for in-context learning or prompting in TTS tasks.", "limitations": "The authors have adequately addressed both the limitations of their research and its possible societal impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper addresses some of the key challenges in the neural audio codec domain.\n* The authors conducted strong and extensive experiments, providing comprehensive results.\n* The reference list appears to be thorough and comprehensive.\n* The authors support their findings by sharing the developed model, which is beneficial for the research community.\n", "weaknesses": "* The authors derived the proposed methods from existing studies and experimentally validate them in the neural audio codec domain. This approach seems to compromise the scientific novelty of the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and feedback. Please find below our comments and clarifications.\n\n> *The authors derived the proposed methods from existing studies and experimentally validate them in the neural audio codec domain. This approach seems to compromise the scientific novelty of the research.*\n> \n\nWe addressed the novelty concern in the global rebuttal. We sincerely request you to review them for additional context.\n\n> *Could the model be applied to downstream applications such as training text-to-speech (TTS) models? Previous works like EnCodec and SoundStream utilized causal architectures to make them suitable for in-context learning or prompting in TTS tasks.*\n> \n\nWe find that causal architectures for the codec aren’t related to downstream applications such as TTS or music modeling. Causal architectures were traditionally required to support streaming applications, which was the primary reason to work on codecs earlier.\n\nWe trained a music generative model trained using our codec, which is capable of creating high quality variations of music with different styles of prompting (publication details are withheld for anonymity). Additionally, we internally trained AudioLM-style text-to-speech models on top of the learned tokens and found it to be capable of generating very high quality speech with minimal artifacts."}, {"review_id": "YjQqdH8Z78", "reviewer": "Reviewer_ZhX4", "summary": "This paper introduces a novel high-fidelity neural audio compression algorithm that achieves impressive compression ratios while maintaining audio quality. The authors combine advancements in high-fidelity audio generation with improved vector quantization techniques from the image domain, along with enhanced adversarial and reconstruction losses. Their approach achieves a remarkable 90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. One of the notable strengths of this work is its universal applicability, as it can compress various audio domains (speech, environment, music) using a single model.\n\nThe authors conduct a thorough comparison with competing audio compression algorithms and demonstrate the superior performance of their method. Furthermore, they provide detailed ablations for each design choice, allowing readers to gain insights into the effectiveness of different components. Additionally, the paper offers open-source code and trained model weights, which contribute to the reproducibility of the results.", "questions": "- Have you attempted to apply a similar architecture to the vocoder in TTS? \n- Which components do you believe can be applied and generalized to other domains or tasks?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Impressive compression performance**: The proposed algorithm achieves a 90x compression ratio for 44.1 KHz audio at just 8kbps bandwidth, demonstrating its effectiveness in reducing data size while preserving audio quality.\n- **Novel Method**: The proposed \"codebook collapse\" and \"quantizer dropout\" effectively address the issues in lossy audio compression. \n- **Universal applicability**: The single model's ability to compress various audio domains makes it highly versatile and applicable to generative modeling of different audio types.\n- **Comprehensive evaluation**: The authors compare their method against existing audio compression algorithms, demonstrating its superiority in terms of performance.\n- **Thorough ablations**: The paper provides detailed insights into the impact of design choices, allowing readers to understand the effectiveness of different components and their contributions to the overall results.\n- **Reproducibility**: The availability of open-source code and trained model weights enhances the reproducibility of the research, enabling other researchers to build upon and validate the findings.", "weaknesses": "- The novelty of the proposed model structure is a combination of existing models: \n  - factorized codes and L2-normalized codes are from Improved VQGAN image model;\n  - Snake activation function from BigVGAN\n- This paper presents a strong audio compression technique. However, since the proposed novel points are specifically tailored for a narrow domain, their impact may be limited to the machine learning community and other domains like computer vision/NLP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and feedback. Please find below answers and clarifications to your questions.\n\n> *The novelty of the proposed model structure is a combination of existing models:*\n> \n> - *factorized codes and L2-normalized codes are from Improved VQGAN image model;*\n> - *Snake activation function from BigVGAN*\n\nWe have addressed the novelty concern in the global rebuttal. We sincerely request you to review them for additional context.\n\n> *Have you attempted to apply a similar architecture to the vocoder in TTS?*\n> \n\n> *Which components do you believe can be applied and generalized to other domains or tasks?*\n> \n\nWhile this specific work is focused on an improved technique / algorithm for training neural audio codecs, much of the new recipe is widely applicable to other audio generation tasks such as speech enhancement, source separation, neural vocoding, etc. Internally, we have trained it for the tasks of neural vocoding (trained by removing the encoder and quantization step) and speech enhancement (trained by removing quantization step, while retaining the adversarial losses) and found it to perform as good or better than state of the art models for the respective tasks.\n\nWhile some of our recipe is borrowed from the image domain, it would also be influential to apply some audio generation techniques to other domains like image/video . For example: periodic activations, residual vector quantization and spectral reconstruction losses haven’t been deeply explored in the image domain and there is interesting scope for future work on this angle."}, {"review_id": "2GorIEtexk", "reviewer": "Reviewer_5w24", "summary": "In the present paper, the authors introduce RVQGAN, a neural audio codec that uses a convolutional encoder / decoder along with Residual Vector Quantization as a bottleneck, with a multi scale mel reconstruction loss and different adversarial losses.\nThey show state of the art performance from 3 to 8kbps, compared with the EnCodec model [8].\n\nThe key novelties are:\n- in each VQ layer, the authors perform the retrieval of the nearest codebook entry into a lower dimension space, and use cosine similarity instead of L2 distance to boost the utilization of the codebooks.\n- the authors drop the exponential moving average rule for learning the codebooks.\n- the author notice that the original technique from Soundstream [45] to select a varying number of quantizers can hurt the full bandwidth performance, and thus select 50% of the time all the quantizers in RVQ.\n- refinement of the losses and adversaries from previous work (in particular using different weights for different frequency bands).\n- balancing of the dataset to sample more often fullband audio.\n\nThe authors provide extensive ablation studies with objective metrics, and one subjective comparison with EnCodec with various bitrates.", "questions": "As mentioned before, I would need more clarification on the exact algorithm used for VQ. Is a PCA computed ? if so how often is it updated (as the encoder output distribution might change). How are the codebooks updated ?\n\nIn Section 3.4, the architecture for the multi scale discriminator is missing. Is it the same one as [8] or [45]?\n\nParagraph starting 193: this insight has been noted and motivated before in [45], [5] and [40], it doesn't seem like the authors bring any new material evidence here?\n\nIt would be interesting to see the breakdown of Figure 3 by the category of audio.\n\nA last question would be over learnt snake activation parameters. Do the authors have any insight over the distribution of the learnt $\\alpha$? does this vary with the layer? I'm trying to get a better sense of how exactly the model is utilizing this feature.\n\n\n", "limitations": "authors properly address societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- great execution and illustration of the various issues tackled here and the proposed solutions.\n- quality of the final model clearly surpasses the existing state of the art.\n- detailed ablation study with objective metrics.\n- single model for fullband audio over multiple audio domains.", "weaknesses": "- incremental improvement over previous work: overall method is coming from [45], adversarial losses are a combination of the one from [45] and [8]. Minor changes to the objective loss compared with [15, 45]. The authors however claim novelty: l.59, \"we make impactful design changes [...]: multi scale stft discriminator, multi scale losses\".\n- some details are unclear to me, in particular, the authors mention they do not use the EMA rule from [9]. How are the codebooks updated then? The authors also mention a low dimension projection, but do not mention when and how it is computed and updated. See questions.\n- no ablation with subjective evaluations: could have been interesting to clearly identify where most of the subjective gains is coming from e.g. is it from the quantizer, adversarial losses or dataset balancing?\n- seems like the authors do a comparison of a 24kHz baseline model with a 44.1kHz, keeping the ground truth as 44.1kHz, which can have a high impact on subjective and objective metrics independently of the design choices made by the authors. In particular the visqol for Encodec in Table 3 is much lower than reported in [8].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for you review and constructive criticism of our model. We appreciate the effort in getting to understand the details of our model, and we’ve made our best efforts to answer each of your concerns and questions.\n\n> *incremental improvement over previous work: overall method is coming from [45], adversarial losses are a combination of the one from [45] and [8]. Minor changes to the objective loss compared with [15, 45]. The authors however claim novelty: l.59, \"we make impactful design changes [...]: multi scale stft discriminator, multi scale losses\"*\n> \n\nWe discussed novelty concerns at length in the global rebuttal. We sincerely request the reviewer to review that for additional context.\n\n> *some details are unclear to me, in particular, the authors mention they do not use the EMA rule from [9]. How are the codebooks updated then? The authors also mention a low dimension projection, but do not mention when and how it is computed and updated. See questions*\n> \n\nThe residual vector quantization algorithm is an improved version of the one in SoundStream [45], where the codebook learning is modified, inspired by the techniques proposed in Improved VQGAN[43], to encourage uniform codebook usage thereby preventing a “collapse” where many codebook entries are unused.\n\nSpecifically, while the standard quantization operation is implemented as follows:    \n\n$$\n\\displaylines{\nz_q(x) = e_k,\\quad \\text{where}\\quad k = \\text{argmin}_j ||z_e(x) - e_j ||_2 \\\\\\ \\text{e is the codebook, $z_e$ is the encoder, $z_q$ is the quantizer}\n}\n$$\n\nThe modified quantization operation used in our work applies the following equation:\n\n$$\n\\displaylines{\nz_q(x) = W_\\text{out} e_k,\\quad \\text{where}\\quad k = \\text{argmin}_j ||\\ell_2(W_\\text{in}z_e(x)) - \\ell_2(e_j) ||_2  \\\\\\ \\text{where $W_\\text{in} \\in R^{D \\times M}$ and $W_\\text{out} \\in R^{M \\times D}$ are projection matrices,} \\\\\\ \\text{$D$ is the output dimension of encoder, $M$ is the codebook dimension, $M \\ll D$}\n}\n$$\n\nAs illustrated in the paper as well as [43], performing codebook lookup in a low-dimensional space leads to improved codebook utilization. In our work, D = 1024 and M = 8.\n\nAlso, we noticed that directly updating the codebook using the loss functions in the original VQ-VAE paper instead of EMA is sufficient, simpler to implement and leads to slight performance improvements. Specifically, the loss function is defined as follows:\n\n$$\n\\displaylines{\nz_\\text{proj}(x) = W_\\text{in}\\ z_e(x) \\\\\\ \\mathcal{L}_\\text{VQ} = ||\\text{sg}[\\ell_2(z_\\text{proj}(x))] - \\ell_2(e_k) ||_2^2  + \\beta \\  ||\\ell_2(z_\\text{proj}(x)) - \\text{sg}[\\ell_2(e_k)] ||_2^2 \\\\\\ \\text{where sg is the stop gradient operator}\n}\n$$\n\n> *seems like the authors do a comparison of a 24kHz baseline model with a 44.1kHz, keeping the ground truth as 44.1kHz, which can have a high impact on subjective and objective metrics independently of the design choices made by the authors. In particular the visqol for Encodec in Table 3 is much lower than reported in [8]*\n> \n\nThanks for the important feedback. We revisited comparisons against relevant work and discussed it at length in the global rebuttal. Please review that for additional clarifications.\n\n> *In Section 3.4, the architecture for the multi scale discriminator is missing. Is it the same one as [8] or [45]?*\n> \n\nThe architecture for the Discriminators is as follows. We use the same Multi-period Discriminator architecture from HifiGAN (cite), and the multi-scale STFT discriminator architecture from UnivNet except with complex STFT as input rather than magnitude spectrograms. Additionally, we do multi-band processing by splitting the spectrogram into sub-bands and using separate discriminator weights for each sub-band, as motivated in our paper (Section 3.4 and Section 4.5).\n\nAlso, the code attached to the submission has further exact details on the implementation. \n\n> *Paragraph starting 193: this insight has been noted and motivated before in [45], [5] and [40], it doesn't seem like the authors bring any new material evidence here?*\n> \n\nWhile this may seem obvious, earlier work such as [45], [5] and [40] haven’t explicitly stated this intuition or shared audio samples with reconstructions from different quantizers. We found this may be interesting to some readers, especially with a practical demo.\n\nMoreover, this intuition is important to understand since prior work such as EnCodec use quantizer dropout only in groups, rather than at each level, and this could impact downstream performance when learning language models on top of the tokens. For instance, we have internally experienced that training language models on top of a codec trained without any quantizer dropout leads to poor audio quality.\n\n> *It would be interesting to see the breakdown of Figure 3 by the category of audio.*\n> \n\nWe have attached a pdf to the global rebuttal with the requested breakdown.\n\n> *A last question would be over learnt snake activation parameters. Do the authors have any insight over the distribution of the learnt? does this vary with the layer? I'm trying to get a better sense of how exactly the model is utilizing this feature.*\n> \n\nThanks for the insightful question. We were just as curious about the distribution of the snake activation parameters, and expected it to correlate with the frequencies being learnt or introduced across the generator. However, we did not find any clear patterns in the parameters in each layer (across different dimensions) or across layers."}]

Author rebuttal:
The authors of this paper would like to sincerely thank all the reviewers for their time and feedback. Specifically, we thank the reviewers for positively acknowledging:

1) the key challenges in the neural audio codec domain successfully addressed in this work
2) the strong results significantly improving over state of the art
3) comprehensive evaluation and ablation studies that provide insight into all design choices
4) reproducibility of the research through open-sourced code and model weights

We also appreciate and welcome constructive criticism provided for our work and hope to resolve some common concerns noted by the reviewers.

### 1. Addressing novelty concerns

The authors would like to clarify and reiterate that we don’t claim novelty in terms of inventing new techniques, in this paper. However, there is a strong novel contribution of understanding fundamental limitations in existing codec algorithms, and mitigating them with techniques available at our disposal. Each change was strongly motivated by limitations in existing models that have not been addressed till date. Specifically:

1. We found a fundamental limitation in existing audio codecs (such as SoundStream, EnCodec, etc.) that under-utilize their bandwidth due to poor codebook learning (or codebook collapse). This sub-optimal bandwidth usage prevented these models from achieving better quality and higher compression rates which is desirable.
2. We studied the quantizer dropout technique in greater detail and found that it could hurt full bandwidth reconstruction of the model. Addressing this leads to better perceptual reconstruction quality, while still maintaining the benefits of quantizer dropout.
3. While techniques such as periodic (Snake) activations, multi-scale spectral loss, multi-scale STFT discriminator already exist in prior work, there is little consensus in the audio community on the impact of these changes or the motivation behind them. For instance, Snake activation introduced in BigVGAN was not adopted by EnCodec even though it existed earlier. Our work uncovers the theoretical limitations that motivates each of these design changes and also provides thorough ablations for each change. As an additional example, we note in our work that using multi-scale mel reconstruction loss with a lowest hop size of 8 (and filter length 32) leads to a much better modeling of quick transients that exist in audio domains such as music.

We find that scientifically understanding each limitation, and addressing each of them with well-studied and motivated techniques leads to a significant improvement over the state of the art, even leading to a 2x improvement in key metrics such as mel distance (see Table 1 in attached PDF). We request the reviewers to consider the scientific novelty in our work in identifying key limitations of existing models, as well as the disciplined usage of known techniques to mitigate them.

### 2. Revisiting comparisons against relevant work

The reviewers noted that some of our comparisons against baseline models (like EnCodec) are at different sampling rates, which could affect objective and subjective metrics. While this is a valid concern, this was a difficult choice we originally made since downsampling the proposed model would be an unfair comparison. While EnCodec runs natively at 24kHz, by downsampling the output of the proposed model from 44.1kHz to 24kHz we discard all the capacity and bitrate that was allocated to these higher frequencies.

**Listening tests at 24 KHz**: To remove the bandwidth concerns, we downsampled all examples to 24 KHz and redid the subjective listening tests. While this puts our proposed work at a further disadvantage, since it was trained to compress higher bandwidth audio with higher compression factors, we find that our model still significantly outperforms baseline methods (Figure 1 in attached pdf).

**Apples-to-apples comparison with EnCodec**: Moreover, we re-trained our proposed model with the same exact configuration as EnCodec (24 kHz sampling rate, 24 kbps bitrate, 320 stride, 32 codebooks of 10 bits each) to make a thorough apples-to-apples comparison. We have attached quantitative evaluations for the same (Table 1 in attached PDF). In summary, our proposed model significantly improved over EnCodec across all metrics, achieving a mel distance (lower is better) of 0.49 compared to 1.05 of EnCodec.

We believe these updated results should further strengthen the significance of our proposed model. We will add these updated results to the table in the paper to make the comparisons straightforward.

Meta-review: This paper presents an improvement in the discriminator of existing Encodec and Soundstream neural audio codecs. It is particularly effective for HiFi audio with low bitrates (3-8kbps). The reviews all agree that the paper is a solid incremental contribution to a real benchmark, with a paper that describes correctly the method and its advantages. The rebuttal and discussion was productive and the state of the paper is suitable for publication for NeurIPS.
Decision: Accept (spotlight)