High Fidelity Neural Audio Compression
Authors: Alexandre Défossez
Jade Copet
Gabriel Synnaeve
Yossi Adi
Venue: Accepted by TMLR ()

Abstract: We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and samples are available under github.com/facebookresearch/encodec.

Published in Transactions on Machine Learning Research (09/2023)
High Fidelity Neural Audio Compression
Alexandre Défossez"
defossezūmella.c com
Meta AI, FAIR Team, Paris, France
Jade Copet*
jadecopet@melta. com
Meta AI, FAIR Team, Paris, France
Gabriel Synnacvet
gab@emela. com
Meta AI, FAIR Team, Poris, France
Yossi Adit
adlyooss@melee com
Meta AI, FAIR Team, Aei,Avi Isruel
Reviewed on OpenReview: /ttps://openaenn opwnrevee..........oeee net/ forum? id= toCd828eR2
Abstract
We introduce a stale-of-the-arl real-time, high-fidelity, audio codec leveraging neural nelworks. consists in a streaming encoder-decoder architecture with quantized latent space trained an end-to-end fashion. We simplify and speod-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce novel loss balancer mechanism to stabilize training: the weight of loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how light Transformer models can be used to hurt compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architcctural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with ablation study for range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and samples are available under
1 Introduction
Recent studies suggest that streaming audio and video have accounted for the majority of the internet traffic in 2021 (82% according to [Cisco| 2021b). )). With the internet traffic expected to grow, audio compression is an increasingly important problem. In lossy signal compression we aim at minimizing the bitrate of a sample while also minimizing the amount of distortion according to a given metric, ideally correlated with human perception. Audio codecs typically employ a carefully engineered pipeline combining an encoder and a decoder to remove redundancies in the audio content and yield a compact bitstream. Traditionally, this is achieved by decomposing the input with a signal processing transform and trading off the quality of the components that are less likely to influence perception. Leveraging neural networks trained transforms via encoder-decoder mechanism has been explored by Morishima et al. (1990); Rippel et al. 2019); Zeghidour et al. 2021). Our research work is in the continuity of this line of work, with focus on audio signals.
The problems arising in lossy neural compression models are twofold: first, the model has to represent a wide range of signals, such as not to overfit the training set or produce artifact laden audio outside its
Equal contribution.

---

Published in Transactions on Machine Learning Research (09/202))
bu
Discriminator
Ld
Corw2D blocks
Oony Mook
W000 1100
Done Brogk
La
Eg
Btck
mill-crass
tarelomer tarslorer
Encoder
taraformer
Decoder
Lr
Quantizer
ls
la
Figure ENCODEC an encoder decoder codee architecture which trained with reconstruction (C, and 4) well adversarial losses (Eg for the generator and ls for the discriminator). The residual vector quantization commitment loss (fw) applies only to the encoder. Optionally, train small Transformer language model for entropy coding over the quantized units with &,, which reduces bandwidth even further.
comfort zone. We solve this by having a large and diverse training set (described in Section 4.1 as well as discriminator networks (see Section 3.4) that serve as perceptual losses, which we study extensively Section 4.5.1] Table 2 The other problem is that of compressing efficiently, both compute time and size. For the former, we limit ourselves to models that run in real-time on a single CPU core. For the latter, we residual vector quantization of the neural encoder floating-point output, for which various approaches
have been proposed (Van Den Oord et al. 2017 Zeghidour et al. 2021
Accompanying those technical contributions, we posit that designing end-to-end neural compression models is a set of intertwined choices, among which at least the encoder-decoder architecture, the quantization method, and the perceptual loss play key parts. Objective evaluations exist and we report scores on them in our ablations (Section [.5.1]. But the evaluation of lossy audio codecs necessarily relies on human perception, we ran extensive human evaluation for multiple points in this design space, both for speech and music. Those evaluations (MUSHRA) consist in having humans listen to, compare, and rate excerpts of speech music compressed with competitive codecs and variants of our method, and the uncompressed ground truth. This allows to compare variants of the whole pipeline in isolation, as well as their combined effect, in Section 4.5.1 (Figure 3 and Table M.. Finally, our best model, ENCODEC, reaches state-of-the-art scores for speech and for music at 1.5, 3, 6, 12 kbps at kHz, and at 6, 12, and kbps for 48 kHz with stereo channels.
2 Related Work
Speech and Audio Synthesis. Recent advancements neural audio generation enabled computers to efficiently generate natural sounding audio. The first convincing results were achieved by autoregressive models such as WaveNet (Oord et al. 2016), the cost of slow inference. While many other approaches were explored (Yamamotto al. 2020a. Kalchbrenner et al. 2018 Goel et al. 2022), the most relevant ones here are those based Generative Adversarial Networks (GAN) (Kumar et al. 2019 Yamamoto al. 2020a Kong et al. 2020 Andreev et al. 2022) were able to match the quality of autoregressive by combining various adversarial networks operate at different multi-scale and multi-period resolutions. Our work uses and extends similar adversarial losses to limit. artifacts during audio generation.
Audio Codee. Low bitrate paramotric speech and audio codees have long been studied (Atal & Hanauer Juang Gray 1982). but their quality has been severely limited. Despite some advances (Griffin
1971
& Lim 1985 McCree al. 1996). modeling the excitation signal has remained challenging task. The
2

---

Published in Transactions on Machine Learning Research (09/2023)
current state-of-the-arr traditional audio codecs are Opus (Valin et al. 2012) and Enhanced Voice Service (EVS) (Dett et al., 2015). These methods produce high coding efficiency for general audio while supporting various bitrates, sampling rates, and real-time compression.
Neural based audio codecs have been recently proposed and demonstrated promising results (Kleijn et al. 2018
Valin & Skoglund 2019b] Lim et al.
2020; Kleijn et al. 2021 Zeghidour et al., 2021] Omran et al.
2022
Lin
et al. 2022 Jayashankar et al., 2022 Li et al. Jiang et al., 2022), where most methods are based on quant izing
the latent space before feeding it to the decoder. In Valin & Skoglund (001bb), an LPCNet (Valin & Skoglund
2019a vocoder was conditioned on hand-crafted features and uniform quantizer. Gårbacea et al.
(2019)
conditioned a WaveNet based model on discrete units obtained from a VQ-VAE (Van Den Oord et al.
2017 7
Razavi et al., 2019) model, while Skoglund & Valin (2019) tried feeding the Opus codec (Valin et al.
2012
to a WaveNet to further improve its perceptual quality. Jayashankar et al.] 2022); Jiang et al. (2022) propose
an auto-encoder with a vector quantization layer applied over the latent representation and minimizing the
reconstruction loss, while Li et al.) suggested using Gumbel-Softmax (GS) (Jang et al., 2017 for representation quantization. The most relevant related work to ours is the SoundStream model (Zeghidour et al.) 2021 in
which the authors propose fully convolutional encoder decoder architecture with Residual Vector Quantiza- tion (RVQ) GGray 1984 Vasuki & Vanathi 2006) layers. The model was optimized using both reconstruction loss and adversarial perceptual losses. Caillon & Esling (2021) studied compression as part of VAE- audio modeling, but they did not report any objective or subjective evaluations for this particular application.
Audio Discretization. Representing audio and speech using discrete values was proposed to various tasks recently. Dieleman et al. (2018); Dhariwal et al. (2020) proposed a hierarchical VQ-VAE based model for learning discrete representation of raw audio, next combined with auto-regressive model, demonstrating
the ability to generate high quality music. Similarly, Lakhotia et al. (2021:; Kharitonov et a.l. (2021)
demonstrated that self-supervised learning methods for speech (e.g., HuBERT Hsu et al.) 2021)), can be quantized and used for conditional and unconditional speech generation. Similar methods were applied to spoech resynthesis (Polyak ct al.) 2021), speech cmotion conversion (Kreuk ct al. 2021), spoken dialog
system (Nguyen el al. 2022), and speech-Lo-speed translation (Lee el al. 2021a
Popuri el al. 2022)
3 Model
An audio signal of duration d can be represented by a sequence æ E []1, with C. the number of audio channels, T = d for the number of audio samples at a given sample rate sar- The ENCODEC model is composed of three main components: (i) First, an encoder network E' is input an audio extract and outputs a latent representation z; (ii) Next, a quantization layer Q produces compressed representation Eyp using vector quantization; (iii) Lastly, decoder network G reconstructs the time-domain signal, à, from the compressed latent representation Zy. The whole system is trained end-to-end to minimize a reconstruction loss applied over both time and frequency domain, together with a perceptual loss in the form of discriminators operating at different resolutions. A visual description of the proposed method can be seen in Figure [
3.1 Encoder & Decoder Architecture
The ENCODEC model is a simple streaming, invoouttionllbased encoder-decoder architecture with sequential modeling component applied over the latent representation, both on the encoder and on the decoder side. Such modeling framework was shown to provide great results various audio-related tasks, e-g., source scparation and enhancement (Défossoz et al. [2019] Defossc2 et al., 2020). neural vocoders (Kumar ct al. 2019 Kong et al. 2020 audio codec (Zeghidour et al. 2021), and artificial bandwidth extension (Tagliasachii et al. 2020 Li et al. 2021). We use the same architecture for 24 kHz and 48 kHz audio.
Encoder-Decodor. The encoder model E consists in a 1D convolution with C' channels and kernel size of 7 followed by B convolution blocks. Each convolution block is composed of a single residual unit followed by a down-sampling layer consisting in a strided convolution, with a kernel size K of twice the stride S. The residual unit contains two convolutions with kernel size 3 and a skip-connection. The number of channels is doubled whenever down-sampling occurred. The convolution blocks are followed by a two-layer LSTM for sequence modeling and a final 1D convolutoo layer with kernel size of 7 and D 128 out channels.
3

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm Residual Vector Quantization (RVQ) algorithm
procedure RVQ(=, Q,Nx)::: 2q f- Empty List, z4 < 0.0, r
for i = to Ng do
P, ridx (r)
D. Get both codebook dense representation and index
= Z1: +
r r i
2g = 2g + ridx
Append the codebook index
end for return (2h, zn)) end procedure
Following Zeghidour et al. (2021); ) Li et al. (2021), we use C= 32, B = 4 and (2, 4, 5, 8) as strides. We use ELU as a non lineir activation function (Clevert et al. 2015 either layer normalization (Ba et al. 2016) or weight normalization (Salimans & Kingma 2016). We use two variants of the model, depending on whether we target the low-latency streamable setup, or a high fidelity non-streamable usage. With this setup, the encoder outputs 75 latent steps per second of audio at 24 kHz, and 150 at 48 kHz. The decoder mirrors the encoder, using transposed convolutions instead of strided convolutions, and with the strides in reverse order as in the encoder, outputting the final mono or sterco audio.
Non-streamable. In the non-streamable setup, we use for each convolution total padding of K - S, split equally before the first time step and after the last one (with one more before if K S is odd). We further split the imput into chunks of 1 seconds, with an overlap of 10 ms to avoid clicks, and normalize each chunk before feeding it to the model, applying the inverse operation on the output of the decoder, adding a negligible bandwidth overhead to transmit the scale. We use layer normalization (Ba al. 2016), computing the statistics including also the time dimension in order to keep the relative scale information.
Streamable. For the streamable setup, all padding is put before the first time step. For transposed convoluton with stride s, we output the first time steps, and keep the remaining s steps in memory for completion when the next frame is available, or discarding it at the end of a stream. Thanks to this padding scheme, the model can output 320 samples (13 ms) as soon as the first 320 samples (13 ms) are received. We replace the layer normalization with statistics computed over the time dimension with weight normalization (Salimans Kingma, 2016), as the former is ill-suited for streaming setup. We notice a small gain over the objective metrics by keeping a form of normalization, as demonstrated in Table A.3
3.2 Residual Vector Quantization
We use Residual Vector Quantization (RVQ) to quantize the output of the encoder as introduced by Zeghidour let al. (2021). Vector quantization consists in projecting an input vector onto the closest entry in a codebook of given size. RVQ refines this process by computing the residual after quantization, and further quantizing it using a second codebook, and so forth. A pseudo-code describing the algorithm can be found in Algorithm 1
We follow the same training procedure as described by Dhariwal et al. (2020) and Zeghidour et al. [2021 The codebook entry selected for each input is updated using an exponential moving average with a decay of 0.99. Following Dhariwal et al. (2020), we track the moving average of the cluster utilization, and clusters whose average size fall below 2 are replaced by a candidate sampled from the current batch. We use a Atraight-hhroug--esttie
compute the gradient of the encoder, e.g. as if the quantization step was the identity function during the backward phase. Finally, commitment loss, consisting of the MSE between the input of the quantizer and its output, with gradient only computed with respect to its input, is added to the overall training loss.
(Bengio et al., 20033)
By selecting a variable number of residual steps at train time, a single model can be used to support multiple bandwidth target (Zeghidour et al. 2021). For all of our models, we use at most 32 codcbooks (16 for the 48 kHz models) with 1024 entries cach, e.g. 10 bits per codebook. When doing variable bandwidth training, we select randomly number of codebooks multiple of 2, i.e. corresponding to a bandwidth 1.5, 3, 6, 12
4

---

Published in Transactions on Machine Learning Research (09/2023)
MS-STFTD
Dreuftt Pat buhd
STFT Discriminator (w, how/4)
Figure 2: MS-STFT Discriminator archilecture. The input to the network is complex-valued STFT with the real and imaginary parts concatenated. Each discriminator is composed of a 2D convolutional layer, followed by 2D convolutions with increasing dilation rates. Then a final 2D convolution applied.
or 24 kbps at 24 kHz. Given a continuous latent represention with shape [B. D, D) that comes out of the encoder, this procedure turns it into a discrete set of indexes N,, T"] with Ng the number of codebooks selected. This discrete representation can changed again to a vector by summing the corresponding codebook entries, which is done just before going into the decoder.
3.3 Language Modeling and Entropy Coding
Naive coding of the codebook indexes only optimal if the true underline distribution is uniform over the codebooks. Alternatively, one can better estimate the probability distribution over the codebooks while leveraging past decoded information improve compression rates. To do so, we train small Transformer based language model (Vaswani et al. 2017) with the objective of keeping faster than real time end-to-end single CPU core. The model consists of 5 layers, 8 heads, 200 channels,
decompression pttrresiioondecomppessioo dimension of 800 for the leed-forward blocks, and no dropout. At train time, select bandwidth and the corresponding number of codebooks N₂- For a time step t, the discrete representation obtained at time - is transformed into a continuous representation using learnt embedding tables, one for each codebook, and which are summed. For = O, a special token is used instead. The output the Transformer fed into Ng linear layers with as many output channels as the cardinality of each codebook (e.g. 1024), giving the logits of the estimated distribution over each codebook for time We thus neglect potential mutual information between the codebooks at single time step. This allows specdup inference (as opposed to having one time step per codebook, or multi-stage prediction) with limited impact over the final cross entropy. Each attention layer has causal receptive field of 3.5 seconds, and we offset by a random amount the initial position of the sinusoidal position embedding to emulate being in a longer sequence. We train the model on sequences of 5 seconds.
compression/
Entropy Encoding. We use a range based arithmetic coder (Pasco) 1976 Rissanen & Langdon 1981 order to leverage the estimated probabilities given by the language model. As noted by Ballé al. (2018), evaluation of the same model might lead to different results different architectures, evaluation procedures due to floating point approximations. This can lead to decoding errors as the encoder and decoder will not use the exact same code. We observe in particular that the difference between batch evaluation (e.g. all lime steps at once), and the real-life streaming ivaluation that occurs in the decoder can lead to difference larger than 10-5, We first round the estimated probabilities with a precision 10 -6 although cvaluations in more contexts would be ncoded for practical deployment. We use a total range width of 224 and assign a minimum range width of We discuss the impact on the processing time in Section 4.6
with different
3.4 Training objective
We detail the training objective that combines a reconstruction loss term, a perceptual loss term (via discriminators). and the RVQ commitment loss.
Reconstruction Loss The reconstruction loss term is comprised of a time and frequency domain loss term. We minimize the L1 distance between the target and compressed audio over the time domain, i.e.
5

---

Published in Transactions on Machine Learning Research (09/2023)
(x) -- For the frequency domain, use linear combination between the L1 and L2 losses over the mel-spectrogram using several time scales (Yamamoto et al. 2020b Gritsenko et al. 2020 Formally,
(x(x
1
ΣΣ
[[)((()) S,())) + 0z S (,))22,,
(1)
ax
ogEn sEE
where S, is 64-bins mel-spect using normalized STFT with window size of 2 and hop length of 2 /4, e = 5,..., 11 is the set of scales, and a represents the set of scalar coefficients balancing between the L1 and L2 terms. Unlike Gritsenko et al. 22000), we take a; 1.
Discriminative Loss. To further improve the quality of the generated samples, we introduce a perceptual loss term based on a multi-scale STFT-based (MS-STFT) discriminator, illustrated in Figure 2 Multi-scale discriminators are popular for capturing different structures in audio signals (Kunar et al. 2019 Kong et al. 2020 You et al., 2021]. The basic idea of this family of models is to construct set of discriminators operating at different scales, which will act perceptual loss so they can capture different artifacts produced by the generator. Multi-scaling can take place in various shapes. Kumar et al. (2019) proposed Multi-Scale Discriminator (MSD) which consists of multiple sub-discrimiaalors operating at different scales on the raw waveform (i.e., different window sizes). Kong et al. (2020) proposed a Pelti- Discriminator (MPD) which consists of multiple sub-discriminators operating on equally spaced samples from the waveform.
Inspired by this line of work, we proposed The MS-STFT discriminator which consists in identically structured networks operating multi-scaled complex-valued STFT with the real and imaginary parts concatenated. Each sub-network is composed of 2D convolutional layer (using kernel size 3 x 8 with 32 channels), followed by 2D convolutions with increasing dilation rates in the time dimension of 1, 2 and 4, and a stride of 2 over the frequency axis. A final 2D convolution with kernel size 3 x 3 and stride (1, 1) provide the final prediction. We use 5 different scales with STFT window lengths of [2048, 1024, 512, 256, 128). For 48 kHz audio, we double the size of each STFT window and train the discriminator every two batches, and for stereophonic audio, we process separately the left and right channels. We use LeakyReLU as a non-linear activation function and apply weight normalization (Salimans & Kingma 2016) to discriminator network. The MS-STFT discriminator model architecture is visually depicted in Figure 2
The adversarial loss for the generator is constructed as follows, (z) = Σn max (0, 1 D2 )), where K is
the number of discriminators. Similarly to previous work on neural vocoders Kumar et al. 2019 Kong et al.]
2020
You el al. 2021 we additionally include a relative feature matching loss for the generalor. Formally,
- (1)11 mean (c)li)))))
ffeat x) KL
where the mcan is computed over all dimensions, (D&) are the discriminators, and L is the number of layers in discriminators. The discriminators are trained to minimize the following hinge-loss adversarial loss function: La(x, b)- = t Ek max(0, 1- - De(=))) + max(0, 1 + Dz(t)), where K is the number of discriminators. Given that the discriminator tend to overpower easily the decoder, we update its weight with a probability of 2/3 at 24 kHz, and 0.5 at 48 kHz.
Multi-bandwidth training. At 24 kHz, we train the model to support the bandwidths 1.5, 3, 6, 12, and 24 kbps by selecting the appropriate number of codebooks to keep in the RVQ step, as explained in Section 3.2 At 48 kHz, we train to support 3, 6, 12 and 24 kbps. We also noticed that using a dedicated discriminator per-bandwidth is beneficial to the audio quality. Thus, we select a given bandwidth for the entire batch, and evaluate and update only the corresponding discriminator.
VQ commitment loss. As mentioned in Section 3.2 we add a commitment loss Ly between the output of the encoder, and its quantized value, with no gradient being computed for the quantized value. For each rosidual step E (1....C) C. (with C' depeding on the bandwidth target for the current batch), noting 2, the current residual and q. (z.) the nearest entry in the corresponding codebook, we define L, as
hp =
Σ c=l
2 - ge (z(2))11
(3)
6

---

Published in Transactions on Machine Learning Research (09/2023)
100
80
MUSHRA
60
ENCODEC
40
ENCODEC cutropy coded
Lyra-v2
20
EVS
Opus
0
1.5
3
6
9.6
12
Bitrate (kbps)
Figure 3: Human evaluations (MUSHRA: comparative scoring of samples) across bandwidths of standard codecs and neural codecs. For ENCODEC we report the initial bandwidth without entropy coding (in plain) and with entropy coding (hollow). Lyra-v2 is a neural audio codec, while EVS and Opus are competitive standard codecs. The audio samples are from speech and music. The ground truth is 16bits 24kHz wave.
Overall, the generator is trained to optimize the following loss, summed over the batch,
LG = de le 2) + A la (x, +) + X (0) + Afeat ffeat +) + Au (w),,,
(4)
where de, Aos Ago Afeat: and Aw the scalar coefficients balance between the terms.
Balancer. We introduce loss balancer in order to stabilize training, in particular the varying scale of the gradients coming from the discriminators. We also find that the balancer makes it easier to reason about the different loss weights, independently of their scale. Let us take number of losses (4)); that depends only on Be. the output of the model &. We define 9: = de and (Ilg:ll2)s the exponential moving average of gi over the last training batches. Given a set of weights (A:) and a reference norm R, we define
D. d Σ,
9
5
We then backpropagate into the network E, 9., instead of the original Σ, ^gg,- This changes the optimization problem but allows to make the A; interpretable irrespectively of the natural scale of each loss. If E, A 1, be interpreted
the fraction of the model gradient that come from the corresponding loss. We take R = 1 and 8 0.999. All the generator losses from Eq. (4) fit into the balancer, except for the commitment loss, as it is not defined with respect to the output of the model.
then each weight
4 Experiments and Results
4.1 Dataset
We train ENCODEC on 24 kHz monophonic across diverse domains, namely: speech, noisy speech, music and general audio while we train the fullband stereo ENCODEC on only 48 kHz music. For speech, we use the clean spoech segments from DNS Challenge 4 (Dubey et al. 2022) and the Common Voice datasct (Ardila et al. For general audio, we use on AudioSet (Gemmnke et. al. 2017) together with FSD50K (Fonseca al. 2021
2019
For music, we rely the Jamendo dataset (Bogdanov et al, 2019) for training and evaluation and further evaluate our models on music using proprictary music dataset. Data splits are detailed in Appendix
For training and validation, we define a mixing strategy which consists in either sampling a single source from a dataset or performing on the fly mixing of two or three sources. Specifically, we have four strategies: (s1) we sample a single source from Jamendo with probability 0.32; (s2) we sample a single source from the other datasets with the same probability; (s3) we mix two sources from all datasets with probability of 0.24; (s4) we mix three sources from all datasets except music with probability of 0.12.
7

---

Published in Transactions on Machine Learning Research (09/2023)
The audio is normalized by file and we apply random gain between -10 and 6 dB. We reject any sample that has been clipped. Finally we add reverberation using room impulse responses provided by the DNS challenge with probability 0.2, and RT60 in the range [0.3, 1.3] except for the single-source music samples. Such imput normalization was found to be beneficial in improving model performance {Defossez et al. 2020 For lesting, we use four categories: clean speech from DNS alone, clean speech mixed with FSDK50K sample, Jamendo sample alone, proprictary music sample alone.
4.2 Baselines
Opus (Valin et al. 2012) is a versatile speech and audio codec standardized by the LETF in 2012. It scales from 6 kbps narrowbant monophonic audio to 510 kbps fullband stereophonic audio. EVS (Dictz et al.) 2015) is a codec standardized in 2014 by 3GPP and developed for Voice over LTE (VoLTE). It. supports range of bitrates from 5.9 kbps to 128 kbps, and audio bandwidths from 4 kHz to 20 kHz. It is the successor of AMR-WA WB (Bhagat et al. 2012). We use both codecs to serve as traditional digital signal processing basclines. We also utilize MP3 compression at 64 kbps as an additional baseline for the stereophonic signal compression case. MP3 uses lossy data compression by approximating the accuracy of certain components of sound that are considered to be beyond hearing capabilities of most humans. Finally, we compare ENCODEC to the SoundStream model from the official implementation available in Lyra 2 at 3.2 kbps and kbps on audio upsampled to 32 kHz. We also reproduced a version of SoundStream (Zeghidour et al. 2021) with minor improvements. Namely, we use the relative feature loss introduce in Section 3.4 and layer normalization (applied separately for each time step) in the discriminalors, except for the first and last layer, which improved the audio quality during our preliminary studies. Results a reported in Table A.2 in the Appendix A.3
4.3 Evaluation Methods
We consider subjective and objective metrics. For the subjective tests we follow the MUSHRA protocol (Sries 2014), using a hidden reference and a low anchor. Annotators were recruited using a crowd-sourcing platform, in which they were asked to rate the perceptual quality of the provided samples in a range between 1 to 100. We randomly select 50 samples of seconds from each category of the the test set and force at least 10 annotations per samples. To filter noisy annotations and outliers we remove annotators who rate the reference recordings less then 90 in least 20% of the cases, or rate the low-anchor recording above 80 more than 50% of the time. For objective metries, we use ViSQOL (Hines et al.) 2012 Chinen et al. 2020) together with the Invarian Signal-to-Noise Ration (SI-SNR) (Luo & Mesgarani 2019 Nachmani et al.
2020
Chazan et al. 2021).
4.4
Training
We train all models for 300 epochs, with one epoch being 2,000 updates with the Adam optimizer with a batch size of 64 examples of 1 second each, a learning rate of 3- 10 B1 = 0.5, and B2 = 0.9. All the models are traind using 8 A100 GPUs. We use the balancer introduced in Section 3.4 with weights de = 0.1, 'y = 1,
>g = 3, Ateat 3 for the 24 kliz models. For the 48 kHz model, we use instead Xg = 4, Areat = 4.
4.5 Results
We start with the results for ENCODEC with a bandwidth in (1.,,3, 6, 12} kbps and compare them to the baselines. Results for the streamable setup are reported in Figure 3 and a breakdown per category in Table I We additionally explored other quantizers such as Gumbel-Softmax and DiffQ (see details Appendix A.2) however, we found in preliminary results that they provide similar or worse results, hence we do not report them.
When considering the same bandwidth, ENCODEC is superior to all evaluated baselines considering the MUSHRA score. Notice, ENCODEC at 3kbps reaches better performance on average than Lyra-v2 using 6kbps and Opus at 12kbps. When considering the additional language model over the codes, we can reduce the bandwidth by ~ 25- 40%. For instance, we can reduce the bandwidth of the 3 kpbs model to 1.9 kbps.
htttps:/ http:/gttbbboommoo. /gogl///ogll/
We compute visqol with: htsps://ggthubb.c.. com/google/viiqqo using the recommended recipes.
8

---

Published in Transactions on Machine Learning Research (09/2023)
Table 1: MUSHRA scores for Opus, EVS, Lyra-v2, and ENCODEC for various bandwidths under the strcamable setting. Results are reported across different audio categories (clean speech, noisy speech, and music), sampled at 24 kHz. We report mean scores and 95% confidence intervals. For ENCODEC, we also report the average bandwidth after using the entropy coding described in Section 3.3
Model
Bandwidth
Entropy Coded
Clean Speech
Noisy Speech
Music Set-1
Music Sct-2
Reference
1..5+1.6
93.9
93.2+2.5
97.1
Opus
6.0 kbps
30.1+2.8
19.1=5.y
20.6+5.8
17.9
Opus
12.0 kbps
76.5+2.3
61.9#2.1
77.8+3.2
65.4
EVS
9.6 kbps
84.4+22.5
80.0
89.9
87.71:2.3
Lyra-v2
3.0 kbps
53.1+1.9
52.0#4.7
69.3+3.3
42.3=3.5
Lyra-v2
6.0 kbps
66.2+2.9
59.9
75.7+2.6
48.6=2.1
EN ENCODEC
1.5 kbps
0.9 kbps
49.2::2.4
41.3
68.2
66.5
ENCODEC
3.0 kbps
1.9 kbps
67.0+1.5
62.5+223
89.613.1
87,8=2.9
ENCODEC
6.0 kbps
4.1 kbps
83.1+2.7
69.4
92.9+1.8
91.3=2.1
ENCODEC 12.0 kbps
8.9 kbps
90.6::2.6
80.1 1255
91.8+22.5
92.9
Table 2: Comparing discriminators using objective (ViSQOL, SI-SNR) and subjective metrics (MUSHRA).
Discriminator setup
SI-SNR
ViSQOL
MUSHRA
MoD+-Mon-STFF
5.99
4.22
62.91+2.62
MPD MS-STFT+MPD
7.35
4.24
60.7
6.55
4.34
79.0+1.9
MS-STFT
6.67
4.35
77.541.8
We observe that for higher bandwidth, the compression ratio lower, which could be explained by the small size of the Transformer model used, making hard to model all codebooks together.
4.5.1 Ablation study
Next, we perform an ablation study to better evaluate the effect of the discriminator setup, streaming, multi- larget bandwidth, and balancer. We provide more detailed ablation studies the Appendix, Section A.3
The effect of discriminators setup. Various discriminators were proposed in prior work to improve the perceptual quality of the generated audio. The Multi-Scale Discriminator (MSD) model proposed by Kumar
et al. (2019) and adopted in (Kong et al.] 2020 Andreev et al. 2022 Zeghidour et. al. 2021 operates on the raw waveform at different resolutions. We adopt the same MSD configuration as described in Zeghidour et al.) (2021). Kong et al. (2020) additionally propose the Multi-Period Discriminator (MPD) model, which reshapes the waveform to a 2D input with multiple periods. Next, the STET Discriminator (Mono-STFTD) model was introduced in Zeghidour et al. (2021). where a single network operates over the complex-valued STFT.
We evaluate our MS-STFTD discriminator against three other discriminator configurations: (i) MSD+Mono- STFTD (as in Zeghidour et al. (2021)); (ii) MPD only; (iii) MS-STFTD only; (vi) MS-STFTD-MMP. Results are reported in Table 2 Results suggest that using only multi-scale STFT-based discriminator such as MS- STFTD, is enough to generate high quality audio. Additionally, it simplifies the model training and reduces training time. Including the MPD discriminator, adds a small gain when considering the MUSHRA score.
The effect of the streamable modeling. We also investigate strcamable vs. non-streamable setups and report results in Table 3 Unsurprisingl, we notice a small degradation switching from non-streamable streamable but the performance remains strong while this setting enables streaming inference.
9

---

Published in Transactions on Machine Learning Research (09/2023)
Table Streamable vs. Non-streamable evaluations at 6 kbps on an equal mix of speech and music.
Model
Streamable
SI-SNR
ViSQOL
Opus
2.45
2.60
EVS
1.89
2.74
EN ENCODE
6.67
4.35
ENCODEC
X
7.46
4.39
Table 4: Stereophonic extreme music compression versus MP3 and Opus for music sampled at 48 kHz.
Model
Bandwidth
Entropy Coded
Compression
MUSHRA
Reference
1x
95.1+1.8
MP3
64 kbps
24x
82.713.2
Opus
kbps
256x
17.7#5.9
Opus
kbps
64x
82.943.7
ENCODEC
6 kbps
4.2 kbps
256x
82.9424
ENCODEC
12 kbps
8.9 kbps
128x
88.012.7
ENCODEC
24 kbps
19.4 kbps
64x
87.5+2.6
The effect of the balancer. Lastly, we present results evaluating the impact of the balancer. We train the ENCODEC model considering various values λ Ày, >g, and Afeat with and without the balancer. Results are reported in Table A.44 in the Appendix. As expected, results suggest the balancer significantly stabilizes the training process. See Appendix A.3 for more details.
4.5.2 Stereo Evaluation
All previously reported results considered only the monophonic setup. Although it makes sense when considering speech data, however for music data, stereo compression is highly important. We adjust our current setup to stereo by only modifying our discriminator setup as described in Section 3.4
Results for ENCODEC working 6 kbps, ENCODEC with Residual Vector Quantization (RVQ) at 6 kbps, and Opus at kbps, and MP3 64 kbps are reported in Table ENCODEC is significantly outperforms Opus at 6kbps and is comparable to MP3 at 64kbps, while ENCODEC at 12kpbs achieve comparable performance to ENCODEC at 24kbps. Using language model and entropy coding gives variable gain between 20% to 30%.
4.6 Latency and computation time
We report the initial latency and real time factor on Table 5 The real-time factor here defined the ratio between the duration of the audio and the processing time, so that is greater than one when the method is faster than real time. We profiled all models single thread of MacBook Pro 2019 CPU at 6 kbps.
Table 5: Initial latency and real time factor (RTF) for Lyra v2, ENCODEC at 24 kHz and 48 kHz. A RTF greater than 1 indicates faster than real time processing. We report the RTF for both the encoding (Enc.) and decoding (Dec.), without and with entropy coding (EC). All models are evaluated at kbps.
Real Time Factor
Model
Latency
Enc.
Dec.
Enc. + EC Dec. + F.C
Lyra v2 (32 kHz)
27.4
67.2
ENCODEC 24 kHz
13 ms
9.8
10.4
1.6
1.6
ENCODEC 48 kHz
1
6.8
5.1
0.68
0.66
10

---

Published in Transactions on Machine Learning Research (09/2023)
Initial latency. The 24 kHz streaming ENCODEC model has an initial latency (i.e., without the computation time) of 13.3 ms. The 48 kHz non-streaming version has an initial latency of 1 second, due to the normalizations used. Note that using entropy coding increases the imitial latency, because the stream cannot be "flushed" with each frame, in order to keep the overhead small. Thus decoding the frame at time 1 requires for the frame ++1 to be partially received, increasing the latency by 13ms.
Real time factor. While our model is worse than Lyra v2 in term of processing speed, it processes the audio 10 times faster than real time, making it good candidate for real life applications. The gain from the entropy coding comes at cost, although the processing is still faster than real time and could be used for applications where latency is not essential (e.g. strcaming). At 48 kHz, the increased number of step size lead to a slower than real time processing, although a more efficient implementation, or using accelerated hardware would improve the RTF. could also be used for archiving where real time processing is not required.
5 Conclusion
We presented ENCODEC: : Seate-of-the-ars real-time neural audio compression model, producing high-fidelity audio samples across range of sample rates and bandwidth. We showed subjective and objective results from 24kHz monophonic at 1.5 kbps (Figure g to 48kHz stereophonic (Table + We improved sample quality by developing a simple but potent spectrogram-onny adversarial loss which efficiently reduces artifacts and produce high-quality samples. Besides, we stabilized training and improved the interpretability of the weights for losses through novel gradient balancer. Finally, we also demonstrated that small Transformer model can be used to further reduce the bandwidth by up to 40% without further degradation of quality, in particular for applications where low latency is not essential (e.g. music streaming).
11

---

Published in Transactions on Machine Learning Research (09/2023)
References
Pavel Andreev, Aibek Alanov, Oleg Ivanov, and Dmitry Vetrov. Hifi++:: a unified framework for neural vocoding, bandwidth extension and speech enhancement. arXiv preprint arXiv:2203 13086, 2022.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-mulillngual speech corpus. arXiv preprint arXi::1912.066: 2019.
Bishnu S Atal and Suzanne L Hanauer. Speech analysis and synthesis by linear prediction of the speech wave. The journal of the acoustical society of America, 50(2B):637 655, 1971.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint orXix:607.004450 2016.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In ICLR, 2017.
Johannes Ballé, Nick Johnston, and David Minnen. Integer networks for data compression with latent-variable models. In International Conference on Learning Representations, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXic:1308.3422 2013.
Dipesh Bhagat, Ninad Bhatt, and Yogeshwar Kosta. Adaptive multi-rate wideband speech codec based on celp algorithm: architectural study, implementation & performance analysis. In 2012 International Conference on Communication Systems and Network Technologies, pp. 547-551. IEEE, 2012.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML. 2019), Long Beach, CA, United States, 2019. URL http: 7/hdl hww.ll.anndee..eee net./le3/n///22//2000100
Antoine Caillon and Philippe Esling. Rave: A variational autoencoder for fast and high-quality neural audio synthesis. arxiv preprind arXiv:2111.0501. 2021.
Shlomo E Chazan, Lior Wolf, Eliya Nachmani, and Yossi Adi. Single channel voice separation for unknown number of speakers under reverberant and noisy settings. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3730-30 37344. TEEE, 2021.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference quality of multimedia experience (QoMEX), pp. 1 6. IEEE, 2020.
Cisco. Global 2021 forecast highlights - cisco. https://www/ wwww:wwwwwcccooooo cow wtt.c/////////ee//eeeeeeeee/e http////nnnussousitinnnnoo
Lal__aoooeeetttiiiaaaaa Hww.hlighhhhhhhh.....g pdf 2021.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint arXiv:1511.07288. 2015.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254, 2019.
Alexandre Defossez, Gabricl Synnacve, and Yossi Adi. Real time speech cnhancement in the waveform domain. arXiv preprint arXiv:2006.128, 12847, 2020.
Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo quantiza- tion noise. arXiv preprinl or Xrii:22004.0995 2021.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXio preprint arXim:20050003/ 2020.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Sander Dieleman, Aaron den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. Advances in Newral Information Processing Systems, 31, 2018.
Martin Dietz, Markus Multrus, Vaclav Eksler, Vladimir Malenovsky, Erik Norvell, Harald Pobloth, Lei Miao, Zhe Wang, Lasse Laaksonen, Adriana Vasilache, et al. Overview of the codec architecture. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5698 5702. IEEE, 2015.
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sergiy Matusevych, Sebastian Braun, Emre Sefik Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner. Icassp 2022 deep noise suppression challenge. In ICASSP, 2022.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30: 829 852, 2021.
Cristina Gârbacca, Aäron van den Oord, Yazhc Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vae and wavenet decoder. In ICASSP 2019-2011 TEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 735-739. IEFE, 2019.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and hunan-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. TEFE, 2017.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. arXiv preprint or arXim:2202.0091 09729, 2022.
Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4-20 1984.
D Griffin and Jae Lim. A new model-based speech analysis/aynthheis system. In ICASSP, 1985.
Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33: 13062 11602, 2020.
Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Visqol: The virtual speech quality objective listener. In IWAENC 2012; International Workshop on Acoustic Signal Enhancement, pp. 4. VDE, 2012.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. TEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Technical Report 1502.03167, arXiv, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
Tejas Jayashankar, Thilo Koehler, Kaustubh Kalgaonkar, Zhiping Xiu, Jilong Wu, Ju Lin, Prabhav Agrawal, and Qing He. Architecture for variable bitrate neural speech codec with configurable computation complexity. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 861-865. IEEE, 2022.
Xue Jiang, Xiulian Peng, Chengyu Zheng, Huaying Xue, Yuan Zhang, and Yan Lu. End-to-end neural speech coding for real-time communications. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (TCASSP), pp. 866 870. IEEE, 2022.
Brang-Hwang Juang and Gray. Multiple stage vector quantization for speech coding. In ICASSP'82 IEEE Internalional Conference on Acoustics, Speech, and Signal Processing, volume 7, pp. 597-697766 6000 TEFE, 1982.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Nal Kalchbrenner et al. Efficient Neural Audio Synthesis. In JCML, 2018.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, el al. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.0822 03264, 2021.
W Bastiaan Kleijn, Felicia SC Lim, Alejandro Luebs, Jan Skoglun, Florian Stimberg, Quan Wang, and Thomas C Walters. Wavenet based low rate speech coding. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 676-680. IEEE, 2018.
W Bastiaan Kleijn, Andrew Storus, Michael Chinen, Tom Denton, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, and Hengchin Yeh. Generative speech coding with predictive variance regularivation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6478 6482. IEEE, 2021.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-1700 2020.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint wrXiv:2111.07//1 2021.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to-speee translation with discrete units. arXiv preprint arXi::2107.05604, 2021a.
Ann Lee, Hongyu Gong, Aaul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speoch-to-speech translation on real data. arXiv preprinl Xrx:::2112 22. 08352, 2021b.
Shuyang Li, Huanru Henry Mao, and Julian McAuley. Variable bitrate discrete neural represenlations via causal self-attention. In 2nd Pre-registration workshop (NeurIPS 2021), Remote.
Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu, and Dominik Roblek. Real-time speech frequency bandwidth extension. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 691-695. IEEE, 2021.
Felicia SC Lim, W Bastiaan Kleijn, Michael Chinon, and Jan Skoglund. Robust low rate speech coding basod on cloned networks and wavenet. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASPP), PD. 6769 6773. TEEE, 2020.
Ju Lin, Kaustubh Kalgaonkar, Qing He, and Xin Lei. Speech enhancement for low bit rate speech codec. Tn ICASSP 2022-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7777-7781. IEEE, 2022.
Yi Luo and Nima Mesgarani. Conv-tasnct: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256-12 1266, 2019.
Alan McCree, Kwan Truong, E Bryan George, Thomas P Barnwell, and Vishu Viswanathan. A 2.4 kbit/s melp coder candidate for the us federal standard. In ICASSP, 1996.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Shigeo Morishima, H Harashima, and Y Katayama. Speech coding based on multi-layer neural network. In IEEE International Conference on Communications, Including Supercomm Technicol Sessions, pp. 429-433. LEEE 1990.
Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. International Conference on Machine Learning, pp. 7164 7175. PMLR, 2020.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. aivi preprint arXiv:2203. 16502, 2022.
Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slancy, and Marco Tagliasacchi. Disentangling speech from surroundings in a neural audio codec. arXiv preprinl ar 22v: 15577857888 2022.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Scnior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprinl ar 1rx:::1609.03. 03499, 2016.
Richard Clark Pasco. Source coding algorithms for fast data compression. PhD thesis, Stanford University CA, 1976.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrah- man Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2044000555 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lec. Enhanced direct speoch-to-speecl translation using self-supervieed pro-training and data augmentation. arviv preprint arXiv:2204. 02967, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bourdev. Learned video compression. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3454-3463, 2019.
Jorma Rissanen and Glen Langdon. Universal modeling and coding. IEFE Transactions on Information Theory, 27(1):12 23, 1981.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparamcterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
B Series. Mothod for the subjective assessment of intermediate quality level andio systems. International Telecommunseation Union Radiocommumication Assembly, 2014.
Jan Skoglund and Jean-Mare Valin. Improving opus low bit rate quality with neural speech synthesis. anXiv preprint Xix::11905.0 04628, 2019.
Marco Taglissacchi, Yumpeng Li, Karolis Misiunas, and Dominik Roblek. Seanet: A multi-modal speech enhancement network. arXiv preprinl arXiv:2009.02005 2020.
Jean-Marc Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear prediction. ICASSP 2019-2019 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891 5895. IEEE, 2019a.
Jean-Mare Valin and Jan Skoglund. A real-time wideband neural vocoder at 1.6 kb/s using lpenet. arXiv preprint ar Xix:::1900 1 12087, 2019b.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Jean-Mare Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. IETF, September, 2, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
A Vasuki and PT Vanathi. A review of vector quantization techniques. JEEE Potentials, 25(4):39 47, 2006.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of Neural Information Processing Systems, 2017.
Bernard Widrow, Tstvan Kollar, and Bhan-Chang Liu. Statistical theory of quantization. JEEE Transactions on instrumentation and measurement, 45(2):353 361, 1996.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199 6203. IEEE, 2020a.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast. waveform generation model based on generative adversarial networks with melti-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199 6203. IEEE, 2020b.
Jaeseong You, Dalhyun Kim, Gyuhyeon Nam, Geumbyeol Hwang, and Gyeongsu Chae. Gan vocoder: Multi-resolution discriminator all you need. orXiv preprint arXiv:2103. 05236, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Table A.1: Datasets description. License with asterisk annotation * imply that the specific license varies across the dataset and specific to each sample.
Datasct
Audio domain
Sampling rate
Channels
Duration
License
Common Voice 7.0
Speech
48 kHz
1
9,096 h
CC-0
DNS Challenge (speech)
Speech
48 kHz
1
2,425 h
Multiples*
AudioSet
General andio
48 kHz
2
4,989 h
CC BY 4.0*
FSD50K
General audio
44.1 kHz
1
108 h
CC*
Jamendo
Music
multiples
2
919 h
CC*
A Appendix
A.1
Experimental details
Datasets details. We present additional details and statistics over the datasets used for training in Table A.1 Notice that for some datasets, each sub-source or sample has its own license and we then refer the reader to the respective dataset details for more information. We create our datasets splits as followed. For Common Voice, we randomly sample 99.5% of the dataset for train, 0.25% for valid and the rest for test splits. Similarly, we sample 98% of the clean segments from DNS Challenge for train, 1% for valid and 1% for test. For AudioSet, we use the unbalanced train segments as training data and randomly selected half of the eval segments validation set and the other half as test set. We follow the same procedure for FSD50K using the dev set for training and splitting the eval set between validation and test. Finally for the Jamendo dataset, we randomly take 96% of the artists and their corresponding tracks for train, 2% for valid and 2% for test, hence there is no artists overlap in the different sets.
SoundStream model. We additionally re-implemented SoundStream. We follow the implementation details in [Zoghidour et al. (2021) to develop our own SoundStream version as the original implementation is not open sourced. We implement Residual Vector Quantization with k-means initialization, exponential moving average updates and random restart as pointed by the original implementation. For the wave-based discriminator, we follow the details provided in Zeghidour et al. (2021) and refer to the original Multi-Scale Discriminator implementation proposed in Kumar et al. (2019) for additional information. We use LeakyReLU as non-linear activation function and following Kong et al. (2020), we use spectral normalization for the original resolution and weight normalization for other resolutions. For the STFT-based discriminator, we experimented with multiple normalization methods and found that using LayerNorm (Ba et al., 2016) was the only one that prevented the discriminator from diverging. We used LeakyReLU non-linear activation function. Finally, training hyper parameters are not shared either we use the same parameters for our ENCODEC model.
A.2
Alternative quantizers
A.2.1 DiffQ Quantizer
Pseudo quantization noise. We perform scalar quantization of the latent representation. As quantization is non differentiable, use properly scaled additive independent noise, a.k.a pseudo quantization noise, at train time to simulate it. This approach was first use for taalog-to-digital converters design (Widrow let al. 1996), then for image compression (Baléé et al. (017), and finally by the DifIQ model compression method (Défossez et al. 202I) with a differentiable bandwidth estimate. We extend the DiffQ approach for latent space quantization, adding support for streamable rescaling, proper sparsity, and improved prior coding. Formally, we introduce a learnt parameter B E IRD" (with D the dimension of the latent space) such that B(i) represents the number of bits to use of the i-th dimension. In practice B is parameterized B = Bmax sigmoid(aw)), with Bmax = 15, and the 5 factor is used for boosting the learning rate of v, the learnt parameter. We then define the pseudo quantized representation zq.triin used at train time,
11
Zg.train = clamp(z, m L. L.o, o, + L. o)+ + L
(6)
2
17

---

Published in Transactions on Machine Learning Research (09/2023)
with m (resp. o) the mean (resp. standard deviation) of 2 along the time and batch axis, and u[-1, uniform i.i.d noise. The limit L is chosen so that when B goes to O, the noise covers most of the range of values accessible to a Gaussian variable of variance a. In order to prevent outlier values, we clamp the imput 2 to this expected range of values. If L is too large, the dynamic range of the quantization will be poorly used. If L is too low, many values will get clamped and lose gradients. In practice we choose L, = 3, which verifies that values are not clamped 99.5% of the time (against 99.8% if the input were gaussian). In order to learn B, we approximate at train time the bandwidth used by the model by Waiff, = T" E B(*) /d with T" the number of latent time steps, the duration of the sample, and add to the training loss a penalty term of the form Awddftg as long as Waiffq is over a given target, as used in Section 3.4
Test time quantization. At validation and test time, we replace the batch-level statistic with an exponential moving average with decay of 0.9 computed over the train set, similar to batch norm (Loffe & Szegedy 2015). We first normalize and clamp to the segment [o, 1] as clamp((L + o= m))/2L, and define the number of quantized levels NB = (28) and the quantized index as
i = min [floor(N (Ng u), NB - 1] E [O..NB - 1],
(7)
with the minimum taken to avoid the edge case a = 1. We know we can code each entry in á on at most loga loga(N)) (NB) bits. The quantized latent 2q is finally defined as 2q = m + Lo (2 Ng
200.5
Sparsity. We want to allow B to go to O, however additive noise fails to remove all information contained in the latent in that case, Indeed, if x < m for instance, then even after adding the largest possible amount of noise, Eg,train is still biased towards values smaller than m. In order to remove all information about z from Zg,train, the scale of the noise relative to the scale of the signal must go to infinity. This is achieved by scaling down z by a factor min(B: 1) in Eq. (6), while scaling down the additive noise only by a factor /min(B. 1). Thus, the decoder cannot invert the downscaling of without blowing up the noise. In the limit of B - 0, we recover a sparse representation.
A.2.2 Gumbel softmax quantizer
We introduce second fully differentiable vector quantizer composed of No codebooks each with Q entries. The i-th codebook is composed of a set of centroids C, E RoxD, logit bias dy R and learnt prior logit 4 E RR Assuning for simplicity latent vector for the j-th time step zj E RD, define a probability distribution over each codebook entries as ((2) softmax C2z b.). We then sample from the corresponding gumbel-softtax (Jang et al. 2017) with a temperature T == 0.5. This gives us a differentiable approximately 1-hot vector over the codebooks, i.c., noting GS the gumbel-softmax,
Nc
(8) At test time, we replace the gumbel-softnax with a sampling from the distribution q. We define for all i, Ps = softmax(44) the prior distribution over the codebooks entries which is used for coding the quantized value by with an arithmetic coder. We can both train the prior and minimize the bandwidth with single loss term given by the cross entropy between the prior Ps and the posterior (=)),, i.e. the differentiable bandwidth estimate for this method is given by Wgs No D kk=1 log().)
(S((og(a())),)))CC
A.3 Additional Results
Comparing to SoundStream. For fair evaluation, we also compare ENCODEC to our reimplementation of SoundStream (Zeghidour et al.) 2021). For which, the quantizer corresponds to RVQ in and the discriminator corresponds to STFTD+MSD. Results are reported in Table A.2 The results of our SoundStream imple- mentation are slightly worse than ENCODEC using DifiQ quantizer. However, when considering the RVQ as the latent quantizer, ENCODEC is superior to the SoundStream model. Both ENCODEC and SoundStream methods are significantly better than Opus and EVS at 6.0kbps.
The effect of the model architecture. We investigate the impact of different architectural decisions of our ENCODEC model and present our results with objective metrics and real-time factor in Table A.3
18

---

Published in Transactions on Machine Learning Research (09/2023)
Table A.2: comparison between ENCODEC using either DiffQ or RVQ as latent quantizers against our implementation of the SoundStream model 3kbps. We additionally include the results of Opus and EVS 6. 6.0kbps for reference.
Model
Bandwidth
MUSHRA
Reference
96.1+1.11
Opus
6.0
21.1+2.02
EVS
6.0
62.9+2.18
SoundStream
3.0
71.8
ENCODEC (DiffQ)
3.0
72.3+1.18
ENCODEC (RVQ)
3.0
Table A.3: Model architecture Analysis. We explore variations of our architecture including the impact of the number of Residual Units (ResUnits), the sequence modeling with LSTM, the number of channels (C) on objective metrics and real-time factor (RTF greater than is faster than real time).
Model
RTF Enc
RTF Dec
SI-SNR
VISQOL
ENCODEC base
9.8
10.4
6.67
4.35
Channels 16
26.0
25.7
6.40
4.32
Channels 64
1.3
3.1
6.70
4.38
norm None
10.1
10.4
6.45
4.29
LSTM 0
15.0
14.6
6.40
4.35
Residual Layer = 3, LSTM = 0
6.0
7.3
6.32
4.35
For all models, we consider the streamable setup and our reference ENCODEC base model has the number of channels C set 32 and single residual unit. The real-time factor reported here is defined as the ratio between the duration of the input audio and the processing time needed for encoding (respectively decoding) it. We profiled streamable multi-target 24 kHz models during inference at 6 kbps MacBook Pro 2019 CPU. First, we validate that the selected number of channels provide good trade offs in terms of perceived quality and inference speed. We observe that increasing the capacity of the model only marginally affects the scores on objective metrics while has a high impact on the real-time factor. The results also demonstrate that presence of LSTM improves the SI-SNR and the final reconstruction quality, at the detriment of the real-time factor. We also experiment with increasing the number of residual units LSTM in architecture. To do so,
single thread of a
instead of relying dilation used in the first convolutional layer of the residual unit for each subsequent unit. We observe that using Residual Unils has more impact on the real-time factor and nole small degradation of the SI-SNR compared to the LSTM-based version.
3 Residual Units and we double the
The effect of the balancer. Lastly, we presont results evaluating the impact of the balancer. In which, we train the ENCODEC model considering various levels of balancing the loss. For this set of experiments we use the DifQQ quantizer rather than RVQ. All models were trained on Jamando music dataset (see Table A.4). Results suggest the balancer significantly stabilizes the training process. This is especially useful while considering the different terms in the objective together where each term operates at different scale. Following the balancer approach significantly reduce the effort needed for tuning the objective coefficients (i.e., As, A5, Ag, )feat). We demonstrate that the use of the balancer shows no degradation compared to an identified combination of coefficients.
A.4 Societal impact
The majority of the internet traffic represented by audio and video streams (82% in 2021 according to Cisco 2.21)). This share of content is boosted user-generated content, phone and video calls and by the
19

---

Published in Transactions on Machine Learning Research (09/2023)
Table A.4: ViSQOL and SI-SNR results for ENCODEC using DiffQ considering various coefficients to balance the overall objective. All models were trained using Jamando music dataset.
A
A
Xg
feet
Balancer
SI-SNR
ViSQOL
1
1
1
1
/
10.32
4.16
1
1
1
1
X
6.16
3.89
2
1
10.08
4.12
1
2
1
X
5.01
3.77
1
2
2
1
V
10.06
4.17
2
2
1
X
3.84
3.67
2
4
1
V
9.93
4.17
2
4
1
X
1.72
3.52
2
100
1
8.41
4.05
2
100
1
X
-35.83
2.82
2
1
1
4
V
10.53
4.06
2
4
X
7.66
4.03
2
2
2
4
10.19
4.13
2
2
2
4
X
7.16
3.98
2
10
4
9.82
4.11
2
10
4
X
3.98
3.66
2
2
100
4
V
8.52
4.03
2
2
100
4
X
-34.31
2.91
10
1
1
1
V
10.53
3.65
10
1
1
1
X
8.16
4.00
10.72
3.62
X
5.99
3.74
10
1
4
2
/
10.71
3.73
10
1
4
2
X
7.03
3.88
10
2
100
2
/
9.23
4.07
10
2
100
2
X
-33.78
2.85
10
2
100
4
9.22
4.09
2
100
4
X
-16.39
2.95
development of HD music streaming and video streaming services. Compression methods used to reduce the storage requirements and network bandwidth used to serve this content. Furthermore, the growing adoption of wearable devices contributes to making efficient compression an increasingly important problem.
Different audio codecs, including Opus and EVS, have been developed and widely adopted over the past years. Those codecs support audio coding at low latency with high audio quality low to medium bitrates (in the range of 12 24 kbps) but the audio quality deteriorates at very low bitrates (eg. 3 kbps) on non-speech audio. Addressing very low bitrate compression with high fidelity remains essential challenge solve very low bitrate codecs enable communication and improve experiences such videoconferenccng strcaming content even with poor internet connection and therefore allows the internet services to become more inclusive. While further work needs to be done, we hope that sharing the result of this work to the broader community can further contribute to this direction.
20


Reviews:
[{"review_id": "xXbjbhFoKj", "reviewer": "Reviewer_6m85", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1)\tThe simplification of adversarial loss with extensive experimental demonstration is convincing and effective.\n2)\tThe loss balancer is effective to stabilize the codec training. It can be extended to other audio compression works too.\n3)\tThe lightweight LM with entropy encoding is novel. With the help of such a design, the RTF can be reduced.\n4)\tThe paper is well written, and the code is released.\n5)\tThe evaluation and ablation study conducted for a range of audio domains demonstrate the superiority of the proposed approach over baseline methods across all evaluated settings.\n\nWeakness:\n\n1)\tThe insight and methodology of the proposed language model are not clear and straightforward. In this paper, the LM seems to predict latent representation at time t given 1 to t-1 times discrete representations. Then it will predict N_q discrete tokens in parallel at time t without using any audio information of time t. But without time t’s audio, will the results degrade? Will the predicted time t’s discrete representation be used to predict time t+1’s representation?\n2)\tThe entropy encoding is also confusing. The paper uses entropy encoding to compress the probability given by LM. But the codec only uses the discrete tokens. Why the probability of LM instead of the discrete tokens should be compressed?\n3)\tIn Sec. 3.1 Non-streamable paragraph, EnCodec will first normalize the chunk before feeding it to the model and apply an inverse operation on the output, which is not a familiar operation in other codec works. Why is it important? More explanation should be included.\n4)\tIn Sec. 3.2, when training Encodec, the entries in the codebook that are not used are replaced with a candidate sampled from the current batch. Will this replacement happen in every iteration? It seems too frequent to update the code in every batch iteration.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced a novel audio codec to achieve real-time, high-fidelity audio compression. The technical contribution of this paper lies in: 1) They simplify the training of GAN by a single multiscale spectrogram adversary (MS-STFT). 2) They introduce a novel loss balancer mechanism to stabilize the training. 3) They propose a lightweight transformer-based LM and entropy encoding mechanism to further compress the obtained representation. Overall, this paper presents a remarkable advancement in audio compression, showcasing innovative techniques that simplify GAN training, introduce a loss balancer mechanism, and leverage lightweight transformer-based LM and entropy encoding. The extensive evaluation further solidifies the effectiveness of the proposed codec, making it a significant contribution to the field of audio compression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1)\tPlease refer to weaknesses 1 and 2, the authors should provide more explanation of LM and entropy encoding.\n2)\tPlease refer to weakness 3. The authors should include more explanation and verification of the normalization operation.\n3)\tPlease refer to weakness 4. The codebook replacement should be more detailed to avoid confusion.\n", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uy0EjXc4Ph", "reviewer": "Reviewer_gWec", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n-The experimental results seem to suggest that the proposed method is able to outperform the other methods that the authors compare against. I am not an expert in the audio compression domain, so I am not sure if the comparisons are exhaustive. \n\nWeaknesses:\n\n-The motivation for the different components of the proposed method seems a bit arbitrary. (For instance the loss components) However this is okay as the experimental results seem to be good.. \n\n-It's a bit difficult to read the paper as the authors seem to assume prior knowledge of several previous papers. This includes residual vector quantization, and multiscale discriminators. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an autoencoder for audio compression. The method uses discretization in the latent space, and a discriminator loss with the motivation of improving the reconstructed audio quality. The experimental results indicate that with respect to subjective scores and objective scores the method outperforms the other methods that the authors compare with. The method has streamable and non-streamable versions, and it seems the streamable version does not lose much performance over the non-streamable one.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-As I described in section above, it would be nicer to make the reader's life a bit easier by making the paper a bit more self-contained rather than assuming knowledge on previous papers. Specifically, adding a bit more background on residual vector quantization and multiscale discriminators might help. \n\n-It would also help to create a companion website where you provide examples from the baselines and your method. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GW3Fp1H39R", "reviewer": "Reviewer_k3bm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an extremely useful and valuable piece of work!\nSystem and architecture-wise, it's not surprising to someone who are familiar with audio SOTA, but it is exciting to see the idea materialize!\nThe loss-balancer and all these datasets' combination is a great engineering success!\nI played with the code, and it works like a charm! \nI am convinced!\nOne thing I suggest for future is to compare with Transformer-based VAE architecture such as AudioMAE (Huang et al. 2022), it would greatly benefit the community to understand which method would be preferred.\nMy guess is that MAE based method would need a few layers of Conv1d to be equally efficient, but might be better at retaining the global audio structure?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a real-time, high-fidelity audio codec that utilizes neural networks in a streaming encoder-decoder architecture with a quantized latent space. \nThe training process is simplified and accelerated by employing a single multiscale spectrogram adversary, resulting in reduced artifacts and high-quality samples. \nA novel loss balancer mechanism is introduced to stabilize training. The study also explores using lightweight Transformer models to compress the representation by up to 40% while maintaining real-time processing. \nThe paper delves into key design choices such as training objectives, architectural changes, and perceptual loss functions.\n An extensive subjective evaluation is conducted for various bandwidths and audio domains. The approach outperforms baseline methods across all settings, for both 24 kHz monophonic and 48 kHz stereophonic audio. Code and samples are provided in the supplementary material.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have several question for the author:\nThe encodec Model according to my test, is 32 times slower than torchaudio's FFT (on GPU) on CPU is 23 slower.\nDo you think more extreme quantization, and more aggressive distillation can make it faster? Otherwise, this will be specialized for low-bandwidth applications only.\n\nDiffQ is weight-quantization only, without activation quatization, have your experiemented with quantizer's with activation quantization? \nFrom the appendix, it is not clear what bit you quantized to.\n\nAppendix typo near: For this set of experiments we use the DiffQ quantizer **other** RVQ.\n\nIssue: 1% artifact if computed from cuda VS. CPU: using 24kHz model (8, 504) latent:\ndiff:<class 'torch.Tensor'>,tensor([[    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0, -1305,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,  -457,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,   651,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,  -233,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,  2105,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0, -1495,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n             0,     0,     0,     0]])\nthe difference is huge in code book.", "broader_impact_concerns": "The code is open source, and already gaining traction. \nThis is an important milestone for neural codec.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: See the comments above.
Decision: Accept (poster)