Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
Authors: Hubert Siuzdak
Venue: ICLR 2024 poster (2024)
Keywords: audio synthesis
vocoder
GAN
Primary area: generative models

Abstract: Recent advancements in neural vocoding are predominantly driven by Generative Adversarial Networks (GANs) operating in the time-domain. While effective, this approach neglects the inductive bias offered by time-frequency representations, resulting in reduntant and computionally-intensive upsampling operations. Fourier-based time-frequency representation is an appealing alternative, aligning more accurately with human auditory perception, and benefitting from well-established fast algorithms for its computation. Nevertheless, direct reconstruction of complex-valued spectrograms has been historically problematic, primarily due to phase recovery issues. This study seeks to close this gap by presenting Vocos, a new model that directly generates Fourier spectral coefficients. Vocos not only matches the state-of-the-art in audio quality, as demonstrated in our evaluations, but it also substantially improves computational efficiency, achieving an order of magnitude increase in speed compared to prevailing time-domain neural vocoding approaches. The source code and model weights have been open-sourced.

Published as a conference paper at ICLR 2024
Vocos: CLOSING THE GAP BETWEEN TIME-DOMAIN AND FOURIER-BASED NEURAL VOCODERS FOR HIGH- QUALITY AUDIO SYNTHESIS
Hubert Siuzdak
ABSTRACT
Recent advancements in neural vocoding are predominantly driven by Generative Adversarial Networks (GANs) operating in the time-domain. While effective, this approach neglects the inductive bias offered by time-frequency representations, resulting in reduntant and computionally-inennsiv upsampling operations. Fourier- based time-frequency representation is an appealing alternative, aligning more accurately with human auditory perception, and benefitting from well-established fast algorithms for its computation. Nevertheless, direct reconstruction of complex- valued spectrograms has been historically problematic, primarily due to phase recovery issues. This study seeks to close this gap by presenting Vocos, new model that directly generates Fourier spectral coefficients. Vocos not only matches the state-of-the-art in audio quality, demonstrated in evaluations, but it also substantially improves computational efficiency, achieving an order mag- nitude increase in speed compared to prevailing time-domain neural vocoding approaches. The source code and model weights have been open-sourced at https::/github.. cot wwweoooomeeeeoeeooooooooooooooom ai/vocos
INTRODUCTION
Sound synthesis, the process of generating audio signals through electronic and computational means, has long and rich history of innovation. Within the scope of text-to-speech (TTS), concatenative synthesis (Moulines & Charpentier 1990. Hunt & Black 1996) and statistical parametric synthesis (Yoshimura al.| 1999) were the prevailing approaches. The latter strategy relied on source-filter theory of speech production, where the speech signal was seen as being produced by source (the vocal cords) and then shaped by filter (the vocal tract). In this framework, various parameters such as pitch, vocal tract shape, and voicing were estimated and then used to control a vocoder (Dudley 1939) which would reconstruct the final audio signal. While vocoders evolved significantly (Kawahara et al., 1999, Morise et al. 2016), they tended to oversimplify speech production, generating distinctive "buzzy" sound and thus compromising the naturalness of the speech.
A significant breakthrough in speech synthesis was achieved with the introduction of WaveNet (Oord et al., 2016), a deep generative model for raw audio waveforms. WaveNet proposed a novel approach to handle audio signals by modeling them autoregressively in the time-domain, using dilated convolutions to broaden receptive fields and consequently capture long-rang temporal dependencies. In contrast to the traditional parametric vocoders which incorporate prior knowledge about audio signals, WaveNet solely depends on end-to-end learning.
Since the advent of WaveNet, modeling distribution of audio samples in the time-domain has become the most popular approach the field of audio synthesis. The primary methods have fallen into two major categories: autoregressive models and non-autoregressive models. Autoregressive models, like WaveNet, generate audio samples sequentially, conditioning each new sample on all previously generated ones (Mehri et al. 22016] Kalchbrenner al. 2018 Valin & Skoglund, 2019). On the other hand, nonautoregressive models generate all samples independently, parallelizing the process and making it more computationally efficient (Oord al. 2018 Prenger et al. 2019 Donahue 2018).
Work conducted while Gemelo AI

---

Published as a conference paper at ICLR 2024
Hs
400
200
(a) (1)
(b) sin(wi)
(c) (())
Figure 1: This illustrates the phase wrapping using example sinusoidal signal (b) generated with a time-varying frequency (a). The instantaneous phase, (ott. is shown in (c). The apparent discontinuities observed around and the result of phase wrapping. Nevertheless, when viewed on the complex plane, these discontinuities represent continuous rotations. The instantaneous phase computed (ttt) arg (&()), where (t) denotes the Hilbert transform of st)) sin(wt).
1.1 CHALLENGES OF MODELING PHASE SPECTRUM
Despite considerable advancements in time-domain audio synthesis, efforts to generate spectral representations signals have been relatively limited. While it's possible to perfectly reconstruct the original signal from its Short-Time Fourier Transform (STFT), in many applications, only the magnitude of the STFT utilized, Icading to inherent information loss. The magnitude of the STFT provides clear understanding of the signal by indicating the amplitude of different frequency components throughout its duration. contrast, phase information is less intuitive and its manipulation often yield unpredictable results.
Modeling the phase distribution presents challenges due to its intricate nature in the time-freee freeufryy domain. Phase spectrum exhibits periodic structure causing wrapping around the principal values within the range of (-Tr, (Figure Tp. Furthermore, the literature does not provide definitive answer regarding the perceptual importance of phase-related information in speech (Wang & Lim 1982) Paliwal et al., 2011). However, improved phase spectrum estimates have been found to minimize perceptual impairments (Saratxaga et al., 2012). Researchers have explored the use of deep learning for directly modeling the phase spectrum, but this remains a challenging area (Williamson et al.
2015).
1.2 CONTRIBUTION
Attempts to model Fourier-related coefficients with generative models have not achieved the same level of success has been with modeling audio in the time-domain. This study focuses bridging that gap with the following contributions:
We propose Vocos GAN-based vocoder, trained to produce complex STFT coefficients an audio clip. Unlike conventional neural vocoder architectures that rely on transposed convolutions for upsampling, this work proposes maintaining the same feature temporal resolution across all layers. The upsampling to waveform is realized through the Inverse Fast Fourier Transform.
To estimate phase angles, we propose a simple activation function defined in terms of a unit circle. This approach naturally incorporates implicit phase wrapping, ensuring meaningful values across all phase angles.
As Vocos maintains a low temporal resolution throughout the network, we revisited the need to use dilated convolutions, typical to time-domain vocoders. Our results indicate that integrating ConvNeXt (Liu et al. 2022) blocks contributes better performance.
Our extensive evaluation shows that Vocos matches the tfate-of-the-an art in audio quality while demonstrating counterparts. The source code and model weights have been made spurceource, enabling further exploration and potential advancements in the field of neural vocoding.
order of magnitude increase in speed compared to time-domain
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
GAN-based vocoders Generative Adversarial Networks (GANs) (Goodfellow al. 2014). have achieved significant success in image generation, sparking interest from audio researchers due to their ability for fast and parallel waveform generation (Donahue al. 2018 Engel et al. (018)) Progress was made with the introduction of advanced critics, such as the multi-scale discriminator (MSD) (Kumar et al., 2019) and the multi-period discriminator (MPD) (Kong et al. 2020). These works also adopted a feature-matching loss to minimize the distance between the discriminator feature maps of real and synthetic audio. To discriminate between real and generated samples, also multi-resolution spectrograms (MRD) were employed (Jang et al. 202T)
At this point the standard practice involves using a stack of dilated convolutions to increase the receptive field, and transposed convolutions to sequentially upsample the feature sequence to the waveform. However, this design known to be susceptible aliasing artifacts, and there are works suggesting more specialized modules for both the discriminator (Bak et al. (002)) and generator (Lee) (et al. 2022). The historical jump in quality is largely attributed to discriminators that are able to capture implicit structures by examining imput audio signal at various periods or scales. It has been argued YYou et al. 2021) that the architectural details of the generators do not significantly affect the vocoded outcome, given well-established to these methods, Vocos presents a carefully designed, frequency- -wwere generator that models the distribution of Fourier spectral coefficients, rather than modeling waveforms in the time domain.
discriminating framework. Contrary
Phase and magnitude estimation Historically, the phase estimation problem has been at the core of audio signal reconstruction. Traditional methods usually rely on the Griffin-Lim algorithm (Griffin) Lim 1984), which iteratively estimate the phase by enforcing spectrogram consistency. However, the Grifin-Lim method introduces unnatural artifacts into synthesized speech. Several methods have been proposed for reconstructing phase using deep neural networks, including tikelihood-base approaches (Takamichi al. 2018) and GANs (Oyamada et al. 2018). Another line of work suggests perceptual phase quantization (Kim 2003), which has proven promising in deep learning by treating the phase estimation problem classification problem Takahashi et al., 2018).
Despite their effectiveness, these models assume the availability of a full-scale magnitude spectrogram, while modern audio synthesis pipelines often employ more compact representations, such mel- spectrograms (Shen et al., 2018). Furthermore, recent research is focusing on leveraging latent features extracted by pretrained deep learning models (Polyak et al. 2021 Siuzdak al. 2022).
Closer to this paper are studies that estimate both the magnitude and phase spectrum. This can be done either implicitly, by predicting the real and imaginary parts of the STFT, or explicitly, by parameterizing the model to generate the phase and magnitude components. In the former category. Gritsenko al.] (2020) presents a variant of a model trained to produce STFT coefficients. They recognized the significance of adversarial objective in preventing robotic sound quality, however they were unable to train it successfully due to its inherent instability. On the other hand, iSTFTNet (Kaneko et al.] 2022) proposes modifications to HiFi-GAN, enabling it to return magnitude and phase spectrum. However, their optimal model only replaces the last two upsample blocks with inverse STFT, leaving the majority of the upsampling to be realized with transposed convolutions. They find that replacing more upsampling layers drastically degrades the quality. Pasinn & Schlüter (2022) were able to successfully model the magnitude and phase spectrum of audio with higher frequency resolution, although it required multi-step training (Caillon & Esling 2021), because of the adversarial objective instability. Also, the initial studies using GANs to generate invertible spectrograms involved estimating instantaneous frequency (Engel et al. 2018 8). However, these were limited single dataset containing only individual musical instrument notes, with the assumption of a constant instantancous frequency.
Vocos
3.1 OVERVIEW
At its core, the proposed GAN model uses Fousier-based time- frne-freueency representation as the target data distribution for the generator. Vocos is constructed without any transposed convolutions; instead,
3

---

Published as a conference paper at ICLR 2024
Ingut Reatures
Impui features
ResBlock
ConyNeXi Block
Pxl depthwise Conv.
3xl dilated Conv.
ResBlock
AuA
Lxl pointwise Cocy.
ConvNeX: Block
Transposed Com:,
ResBlock
cap
F
Audio
Audio
(a)
(b)
Figure 2: Comparison of typical time-domain GAN vocoder (a), with the proposed Vocos architec- ture (b) that maintains the same temporal resolution across all layers. Time-domain vocoders use transposed convolutions to sequentially upsample the signal to the desired sample rate. contrast, Vocos achieves this by using computationally efficient inverse Fourier transform.
the upsample operation is realized solely through the fast inverse STFT. This approach permits unique model design compared time-domain vocoders, which typically employ series of upsampling layers to inflate input features to the target waveform's 's resolution. often necessitating upscaling by several hundred times. In contrast, Vocos maintains the same temporal resolution throughout the network (Figure 2 This design, known as an isotropic architecture, has been found to work well in various settings, including Transformer (Vaswani et al. 2017). This approach also be particularly beneficial for audio synthesis. Traditional methods often use transposed convolutions that introduce aliasing artifacts, necessitating additional measures to mitigate the issue (Karras) et al., 2021 Lee et al. 2022). Vocos eliminates learnable upsampling layers, and instead employs the well-establish inverse Fourier transform to reconstruct the original- -cale waveform. In the context of converting mel-spectrograms into audio signal, the temporal resolution is dictated by the hop size of the STFT.
Vocos uses the Short-Time Fourier Transform (STFT) to represent audio signals in the time-frequency domain:
STFT, [m,
Σ x x [n]w(n m]e j2zkn/N
The STFT applies the Fourier transform successive windowed sections of the signal. In practice, the STFT is computed by taking a sequence of Fast Fourier Transforms (FFTs) on overlapping, windowed frames of data, which created as the window function advances or "hops" through time.
3.2 MODEL
Backbone Vocos adapts ConvNeXt (Liu et al. 2022) as the foundational backbone for the generator. It first embeds the input features into hidden dimensionality and then applies stack of ID convolutional blocks. Each block consists of a depthwise convolution, followed by an inverted bottleneck that projects features into higher dimensionality using pointwise convolution. GELU (Gaussian Error Linear Unit) activations are used within the bottleneck, and Layer Normalization employed between the blocks.
4

---

Published as a conference paper at ICLR 2024
Head Fourier transform of real-valued signals is conjugate symmetric, side band spectrum, resulting in re 2 + coefficients per frame. As we parameterize the model to output phase and magnitude values, hidden-dim activations projected into tensor h with ft channels and splitted into:
only single
m, p h[1 (#5n/2 (t) (2)r)//22
To represent the magnitude, we apply the exponential function to m: M = exp(m).
We map p onto the unit circle by caleulating the cosine and sine of p obtain x and y, respectively:
cos
= sin(p)
Finally, we represent complex-valued coefficients as: STFT M. (x (x+ jy).
Importantly, this simple formulation allows to express phase angle sp atan2 x) for any real argument p. and ensures that is correctly wrapped into the desired range (-x, T].
Discriminator We employ the multi-period discriminator (MPD) as defined by Kong al. (2020). and multi-resolution discriminator (MRD) (Jng al. 202T.
3.3 LOSS
Following the approach proposed by Kong al. (020)) the training objective of Vocos consists of reconstruction loss, adversarial loss and feature matching loss. However, we adopt a hinge loss formulation instead of the least squares GAN objective, as suggested by Zeghidour et al. 2021
to(2)
K Σ h
max (0, -)))))
l (x x)
Σ
max (0,1 Dx(x)) + max (0, DR(+))
K
&
where D& is the kth subdiscriminator. The reconstruction loss, denoted as Lmeet is defined as the L1 distance between the mel-scalle scaled magnitude spectrograms of the ground truth sample æ and the synthesized sample: Φ: Limel = [lM(z) - _M I The feature matching loss, denoted as feat calculated as the mean of the distances between the /th feature maps of the kth subdistriminator: Ifeat KL >x D (m) D (+)]
4 RESULTS
4.1
MEL-SPECTROGRAMS
Reconstructing audio waveforms from mel-spectrograms has become a fundamental task for vocoders contemporary speech synthesis pipelines. this section, we assess the performance of Vocos relative to established baseline methods.
Data The models are trained on the LibriTTS dataset (Zen et al. 2019), from which use the entire training subset (both train cleen el and train oteer.... We maintain the original sampling rate of 24 kHz for the audio files. For each audio sample, we compute mel-scaled spectrograms using parameters: /t = 1024, hop,, = 256, and the number of Mel bins is set to 100. A random gain is applied to the audio samples, resulting in maximum level between -1 and -6 dBFS.
We train models up to 2 million iterations, with million iterations per generator and discriminator, During training, we randomly crop the audio samples to 16384 samples and use batch size of 16. The model is optimized using the AdamW optimizer with an initial learning rate of 2c-4 and betas set to (0.9, 0.999). The learning rate is decayed following cosine schedule.
Training Details
5

---

Published as a conference paper at ICLR 2024
Table 1: Objective evaluation metrics for various models, including baseline models (HiFi-GAN,
iSTFTNet, BigVGAN) and Vocos.
UTMOS (1)
VISQOL (1)
PESQ (1)
V/UV F1 (1)
Periodicity (L)
Ground truth
4.058
HiFi-GAN
3,669
4.57
3.093
0.9457
0.129
iSTFTNet
3.564
4.56
2.942
0.9372
0.141
BigVGAN
3.749
4.65
3.693
0.9557
0.108
Vocos
3.734
4.66
3.70
0.9582
0.101
w/ absolute phase
3.590
4.65
3.565
0,9556
0.108
w/ Snake
3.699
4.66
3.629
0,9579
0.102
w/o ConvNeXt
3.658
4.65
3.528
0.9534
0.109
Baseline Methods Our proposed model, Vocos, is compared to: iSTFTNet (Kaneko et al. 2022).
BigVGAN (Lee et al. )022), and HiFi-GAN (Kong al. 2020 These models retrained on the same LibriTTS subset for up to 2 million iterations, following the original training details recommended by the authors. We use the official implementations of BigVGAN" and HiFi-GAN"
and a community open-sourced version of iSTFTNe
4.1.1 EVALUATION
Objective Evaluation For objective evaluation of models, we employ the UTMOS (Saeki et al. 2022) automatic Mean Opinion Score (MOS) prediction system. Although UTMOS can yield scores highly correlated with human evaluations, is restricted to 16 kHz sample rate. To assess perceptual quality, we also utilize ViSQOL (Chinen et al. 2020) in audio-mode, which operates in the full band. Our evaluation process also encompasses several other metries, including the Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. (001). periodicity error, and the Fl score for voiced/unvoiced classification (V/UV FI), following the methodology proposed by Morrison al. (20TT. The results are presented in Table Vocos achieves superior performance in most of the metrics compared to the other models. obtains the highest scores in VISQOL and PESQ. Importantly, also effectively mitigates the periodicity issues frequently associated with time-domain GANs. Big VGAN stands out as the closest competitor, especially the UTMOS metric, where it slightly outperforms Vocos.
In our ablation study, examined the impact of specific design decisions Vocos's performance:
Vocos with absolute phase: earity, scaled to fit within the range of [- T.. This formulation does not give the model inductive bias regarding the periodic nature of phase, and the results show it leads to degraded quality. This finding emphasizes the importance of implicit phase wrapping in the effectiveness of Vocos.
In this variant, we predict phase angles using tanh nonlin-
Vocos with Snake activation: Although Snake (Ziyin et al. 2020) has been shown to en- hance time-domain vocoders such as Big our case, it did not result performance gains; in fact, it showed slight decline. The primary purpose of the Snake function is to induce periodicity, addressing the limitations of time-domain vocoders. Vocos, on the other hand, explicitly incorporates periodicity through the use of Fourier basis functions, eliminating the need for specialized modules like Snake.
Vocos without ConyNeXt: Replacing ConvNeXt blocks with traditional ResBlocks with dilated convolutions, slightly lowers scores across all evaluated metrics. This finding highlights the integral role of ConvNeXt blocks Vocos, contributing significantly to its overall success.

---

Published as a conference paper at ICLR 2024
Table Subjective evaluation metrics 5-scale Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS) with 95% confidence interval.
MOS (1)
SMOS (1)
Ground truth
3.81+0016
4.70+00.1
HiFi-GAN
3.54:10.1
4.49::.11
iSTFTNet
3.57
4.42+0.16
BigVGAN
3.64
4.54
Vocos
3.62::0.15
4.55
Subjective Evaluation We conducted crowd-sourced subjective assessments, using 5-point Mcan Opinion Score (MOS) to evaluate the naturalness of the presented recordings. Participants rated speech samples on scale from ('poor- completely unnatural speech') to 5 ("excellent completely natural speech'). Following (Lee et al.) 2022), we also conducted a 5-point Similarity Mean Opinion Score (SMOS) between the reproduced and ground-truth recordings. Participants were asked to assign a similarity score to pairs of audio files, with a rating of 5 indicating 'Extremely similar' and a rating of 1 representing 'Not at all similar'.
To ensure the quality of responses, we carefully selected participants through a third third-party crowd- sourcing platform. Our criteria included the use of headphones, fluent English proficiency, and a declared interest in music listening as a hobby. total of 1560 ratings were collected from 39 participants.
The results are detailed in Table 2 Vocos performs on par with the state-of-the the--rrttterttttt both perceived quality and similarity. Statistical tests show significant differences between Vocos and BigN in MOS and SMOS scores, with p-values greater than 0.05 from the Wilcoxon signed-rank test.
Table 3: VISQOL scores of various models tested on the MUSDB18 18 dataset. A higher VISQOL score indicates better perceptual audio quality.
Mixture
Drums
Bass
Other
Vocals
Average
HiFi-GAN
4.46
4.40
4.12
4.44
4.54
4.39
iSTFTNet
4.47
4.48
3.80
4,40
4.53
4.34
BigVGAN
4.60
4.60
4.29
4.58
4.64
4.54
Vocos
4.61
4.61
4.31
4.58
4.66
4.55
Out-of-distribution data crucial aspect vocoder ability to generalize to unseen acoustic conditions. In this context, we evaluate the performance of Vocos with out-of-distribution audio using the MUSDB18 dataset (Rafi et al., 2017). which includes a variety multi-tran trati-ttrck music audio like vocals, drums, bass, and other instruments, along with the original mixture. The VISQOL scores for this evaluation are provided in Table 3 From the table, Vocos consistently outperforms the other models, achieving the highest scores across all categories.
FigureB] presents spectrogram visualization of an out- of d-dstribbuionn singing voice sample, as repro- duced by different models. Periodicity artifacts are commonly observed when employing time-domain GANs. Big with its anti-aliasing filters, is able to recover some the harmonics in the upper frequency ranges, marking an improvement over HiFi-GAN. Nonetheless, Vocos appears provide more accurate reconstruction of these harmonics, without the need for additional modules.
4.2 NEURAL AUDIO CODEC
While traditionally, neural vocoders reconstruct the audio waveform from mel-scaled spectrogram an approach widely adopted in many speech synthesis pipelines recent research has started to utilize learnt features (Siuzdak et al. 2022), often in quantized form (Borsos et al. 2022

---

Published as a conference paper at ICLR 2024
In this section, we draw comparison with EnCodec Défossez et al. 2022), open-source neural audio codec, which follows a typical time-domain GAN vocoder architecture and uses Residual Vector Quantization (RVQ) (Zeghidour al. 2021) to compress the latent space. RVQ cascades multiple layers of Vector Quantization, iteratively quantizing the residuals from the previous stage to form multi-stage structure, thereby enabling support for multiple bandwidth targets. In EnCodec, dedicated discriminators are trained for each bandwidth. In contrast, we have adapted Vocos to be conditional GAN with projection discriminator (Miyato Koyama 2018), and have incorporated adaptive layer normalization (Huang & Belongie 2017) into the generator.
Audio reconstruction We utilize the open-source model checkpoint of EnCodec operating at 24 kHz. To align with EnCodec, we scale down Vocos match its parameter count (7. (7.9M) and train on clean speech segments sourced from the DNS Challenge (Dubey et al., 2022). Our evaluation, conducted on the DAPS dataset (Mysore 2014) and detailed Table reveals that despite EnCodec's reconstruction artifacts not significantly impacting PESQ and Periodicity scores, they considerably reflected in the perceptual score, as denoted by UTMOS. In this regard, Vocos notably outperforms EnCodec. We also performed crowd-sourced subjective assessment to evaluate the naturalness of these samples. The results, as shown in T indicate that Vocos consistently achieves better performance across range of bandwidths, based evaluations human listeners.
Table 4: Objective evaluation metric calculated for various bandwidths.
Bandwidth
UTMOS (t)
VISQOL (1)
PESQ (t)
V/UV (t)
Periodicity (L)
1.5 kbps
1,527
3.74
1.508
0.8826
0.215
3.0 kbps
2.522
3.93
2.006
0.9347
0.141
EnCodec
6.0 kbps
3.262
4.13
2.665
0.9625
0.090
12.0 kbps
3.765
4.25
3.283
0.9766
0.062
1.5 kbps
3,210
3.88
1.845
0,9238
0.160
3.0 kbps
3.688
4.06
2.317
0.9380
0.135
Vocos
6.0 kbps
3.822
4.22
2.650
0,9439
0.124
12.0 kbps
3.882
4.34
2.874
0,9482
0.116
Table 5: Subjective evaluation metrics 5-scale Mean Opinion Score (MOS) with 95% confidence interval for various bandwidths.
Bandwidth
Vocos
EnCodec
kbps
2.73:10.20
1.09:10.05
3 kbps
3. 3.84::0.16
1.71
kbps
2.41+0.115
12 kbps
4.00
3.08+00.19
Ground truth
4.09-10.16
End-to-end text-to-speech Recent progress in text-to-speech (TTS) has been notably driven by language modeling architectures employing discrete audio tokens. Bark (Suno Al 2023), a widely recognized spencsource model, leverages a GPT--ttyee decoder-only architecture. with EnCodec's 's 6kbps audio tokens serving as its vocabulary. Vocos trained to reconstruct EnCodec tokens can effectively serve as drop-in replacement vocoder for Bark. We have provided text-to-speech samples from Bark and Vocos on our website and encourage readers to listen to them for a direct comparison
4.3 INFERENCE SPEED
Our inference speed benchmarks were conducted using an Nvidia Tesla A100 GPU and an AMD EPYC 7542 CPU. The code was implemented in Pytorch, with no surdware-specifie optimizations.
*1 listen audio samples at wwt.tttp/////m /ttp::///mmmeeee .g thub. io/vocos/
8

---

Published as a conference paper at ICLR 2024
a) Ground truth
b) iSTFTNet
HiFi-GAN
d) BigVGAN
Vocos
Figure 3: Spectrogram visualization of an duttt--distt singing voice sample reproduced by different models. The bottom row presents zoomed-in view of the upper midrange frequency range.
The forward pass was computed using batch of 16 samples, each one second long. Table presents the synthesis speed and model footprint of Vocos in comparison to other models.
Vocos showcases notable speed advantages compared to other models, operating approximately 13 times faster than HiFi-GAN and nearly 70 times faster than BigVGAN. This speed advantage is particularly pronounced when running without GPU acceleration. This is primarily due to the use of the Inverse Short- Time Fourier Transform (ISFFT) algorithm instead of transposed convolutions. We also evaluate a variant of Vocos that utilizes ResBlock's ' dilated convolutions instead of ConvNeXt blocks. Depthwise separable convolutions offer additional speedup when executed GPU.
Table 6: Model footprint and synthesis speed. xRT denotes the speed factor relative real-time. A higher xRT value means the model can generate speech faster than real-time, with a value of 1.0 denoting real-time speed.
Model
xRT (t)
Parameters
GPU
CPU
HiFi-GAN
495.54
5,84
14.0 M
VGAN
98.61
0.40
14.0 M
ISTFTNet
1045.94
14.44
13.3 M
Vocos
6696.52
169.63
13.5 M
w/o ConvNeXt
4565.71
193.56
14.9 M
CONCLUSIONS
This paper introduces Vocos, novel neural vocoder that bridges the gap between time-domain and Fourier-based approaches. Vocos tackles the challenges associated with direct reconstruction of complex-valued spectrograms, with careful design generator that correctly handle phase wrapping. It achieves accurate reconstruction of the coefficients in Fourier-based fime-freyeency representations.
The results demonstrate that the proposed vocoder matches state-of-the-arr audio quality while effectively mitigating periodicity issues commonly observed in time-domain GANs. Importantly, Vocos provides a significant computational efficiency advantage over traditional time-domain methods by utilizing inverse fast Fourier transform for upsampling.
Overall, the findings this study contribute to the advancement of neural vocoding techniques by incorporating the benefits of Fourier- time-frequency herefrrqueneyyeeee representations. The open-sourcing of the source code and model weights allows for further exploration and application of the proposed vocoder in various audio processing tasks.

---

Published as a conference paper at ICLR 2024
REFERENCES
Taejun Bak, Junmo Lee, Hanbin Bae, Jinhyeok Yang. Jae-Sung Bae, and Young-Sun Joo. Avocodo: Generative adversarial network for artifact-free vocoder. arXiv preprint arXiv:2206. 13404, 2022.
Zalán Borsos, Raphaäl Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Tehoul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:220903344: 2022.
Marina Bosi and Richard E Goldberg. Introduction to digital audio coding and standards, volume 721. Springer Science & Business Media, 2002.
Antoine Caillon and Philippe Esling. Rave: A variational autoencoder for fast and high-quality neural audio synthesis. arXiv preprini arXiv:2111.0500 2021.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 fwelfth international conference quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint arXiv.1802.04208, 2018.
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, et al. Ieassp 2022 deep noise suppression challenge. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 9271-9275 IEEE, 2022.
Homer Dudley. Remaking speech. The Journal of the Acoustical Society of America, 11(2):169-177. 1939.
Jesse Engel. Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. In International Conference on Learning Representations, 2018
Ian Goodfellow,
Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acoustics, speech, and signal processing, 32(2):236-243, 1984.
Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33: 33:13062-13077 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 04144, 2023.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510. 2017.
Andrew J Hunt and Alan W Black, Unit selection in concatenative speech synthesis system using large speech database. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, volume 1, pp. 373-376. IEFE, 1996.
Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolutian spectrogram discriminators for high- figh-fiielity waveform generation. arXiv preprint arXiv:2/06. 07889, 2021.
10

---

Published as a conference paper at ICLR 2024
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410 111100--9...11 PMLR, 2018.
Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. istftnet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6207-6211. IEEE, 2022.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863 863, 2021.
Hideki Kawahara, Ikuyo Masuda- Katsuse, and Alain De Cheveigne. Restructuring speech represen- tations using a pitch-adaptive time-freet aee-freuyyncy smoothing and an unstantaneous-frequueee-- bassd f0 extraction: Possible role of repetitive structure in sounds. Speech communication,, 27(3-4): 187-207, 1999.
Doh-Suk Kim. Perceptual phase quantization of speech. IEEE transactions on speech and audio processing, -344, 2003.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.0445 2022.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11976-1 1986, 2022.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint
2016.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint 1rXiv:1802.05637, 2018
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: a vocoder-basse high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 1977):1877-1884, 2016.
Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive gan for conditional waveform synthesis. arXiv preprint arXiv:21 10. 10139, 2021.
Eric Moulines and Francis Charpentier. Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones. Speech communication, 9(5-6):553-467, 1990.
Gautham J. Mysore. Daps (device and produced speech) dataset, May 2014. URL https://dol
0tte:/10.22112eeee 4660670
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast speech synthesis. In International conference on machine learning, pp. 3918-3926. PMLR, 2018
11

---

Published as a conference paper at ICLR 2024
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (1X.iv:6609003999 2016.
Keisuke Oyamada, Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo, and Hiroyasu Ando. Generative adversarial network-based approach to signal reconstruction from magnitude spectrogram. 2018 26th European Signal Processing Conference (EUSIPCO), pp. 2514-2518. IEEE, 2018.
Kuldip Paliwal, Kamil Wójcicki, and Benjamin Shannon. The importance of phase in speech enhancement. speech communication, 53(4):465-494, 2011.
Marco Pasini and Jan Schlüter. Musika! fast infinite waveform music generation. arXiv preprint arXiv:2208.0811 08706, 2022.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617 3617-3621. IEEE, 2019.
Zafar Rafii Antoine Liutkus, Fabian- Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. Musdb1 8-a corpus for music separation. 2017.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing.
Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022.
Ibon Saratxaga, Inma Hernaez, Michael Pucher, Eva Navas, and lñaki Sainz. Perceptual importance of the phase related information in speech. In Thirteenth Annual Conference of the International Speech Communication Association, 2012.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779-4783. IEEE, 2018.
Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. WayThru Latent speech representation as intermediate features for neural speech synthesis. Proc. Interspeech 2022, pp. 833- 837, 2022. doi: 10.21 (0.21457//nterspeechh h2t.eeersspeeeeeeeeeeee 2022-10797
Suno AI. Bark: Text-prompted generative audio model. https:://gttub..cmmm www../oooo.unoaaaao bark 2023. GitHub repository.
Naoya Takahashi, Purvi Agrawal, Nabarun Goswami, and Yuki Mitsufuji. Phasenet: Discretized phase modeling with deep neural networks for audio source separation. In Interspeech, pp. 2713-2717, 2018.
Shinnosuke Takamichi, Yuki Saito, Norihiro Takamune, Daichi Kitamura, and Hiroshi Saruwatari. Phase reconstruction from amplitude spectrograms based on wes-mm--dstttbbbeee deep neural network. 2018 16th International Workshop on Acoustic Signal Enhancement (IWAENC), pp. 286-290. IEEF, 2018.
Jean-Mare Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5895. IEEE, 2019.
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Dequan Wang and Jae Lim. The unimportance phase in speech enhancement. IEEE Transactions on Acoustics, Speech, and Signal Processing, 30(4))679-611, 1982.
Ye Wang and Mikka Vilermo. Modified discrete cosine transform: Its implications for audio coding and error concealment. Journal of the Audio Engineering Society, 51(1/2):52 61, 2003.
Donald S Williamson, Yuxuan Wang, and DeLiang Wang. Complex ratio masking for monaural speech separation. IEEE/ACM transactions audio, speech, and language processing, 24(3): 483-492, 2015.
Takayoshi Yoshimura, Keiichi Tokuda, Takashi Masuko, Takao Kobayashi, and Tadashi Kitamura. Simultaneous modeling of spectrum, pitch and duration in hmm-based speech synthesis. In Sixth European Conference on Speech Communication and Technology, 1999.
Jaeseong You, Dalhyun Kim, Gyuhyeon Nam, Geumbyeol Hwang, and Gyeongsu Chae. Gan vocoder: Multi-resolution discriminator all you need. arXiv preprint arXiv:2/03.00233. 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint ArXiv: 02882. 2019.
Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it, Advances Neural Information Processing Systems, 33:1583-1594 2020.
13

---

Published as a conference paper at ICLR 2024
A MODIFIED DISCRETE COSINE TRANSFORM (MDCT)
While STFT is widely used in audio processing, there are other time-frequency representations with different properties. In audio coding applications, it is desirable to design the analysis/synthesis system in such way that the overall rate at the output of the analysis stage equals the rate of the input signal. Such systems are described as being critically sampled. When we transform the signal via the DFT, even slight overlap between adjacent blocks increases the data rate of the spectral representation of the signal. With 50% overlap between adjoining blocks, we end up doubling our data rate.
The Modified Discrete Cosine Transform (MDCT) with its corresponding Inverse Transform (IMDCT) have become crucial tool in high-quality audio coding they enable the implementation of critically sampled analysis/synthesis filter bank. A key feature of these transforms is the Time-Domain Aliasing Cancellation (TDAC) property, which allows for the perfect reconstruction of overlapping segments from a source signal.
The MDCT is defined as follows:
2N-1
N
1
[
Σ
x [n] cos
(2)
for k = N and N is the length of the window.
The MDCT is a lapped transform and thus produces N output coefficients from 2N input samples, allowing for a 50% overlap between blocks without increasing the data rate.
There is relationship between the MDCT and the DFT through the Shifted Discrete Fourier Transform (SDFT) (Wang & Vilermo, 2003). It can be leveraged to implement fast version of the MDCT using FFT (Bosi & Goldberg 2002). See Appendix[A.3
A.1 VOCoS AND MDCT
MDCT is attractive in audio coding because of its its efficiency and compact representation of audio signals. In the context of deep learning, this might be seen as reduced dimensionality, potentially advantageous as it requires fewer data points during generation.
While STFT coefficients can be conveniently expressed in polar form, providing a clear interpretation of both magnitude and phase, MDCT represents the signal only in real subspace of the complex space needed to accurately convey spectral magnitude and phase. Naive approach would be to treat raw unnormalized hidden outputs of the network as MDCT coefficients and convert it back to time-domain with IMDCT. In our preliminary experiments we found that it led to slower convergence. However we can easily observe that the MDCT spectrum, similarly the STFT, can be more perceptually meaningful on the logarithmic scale, which reflects the logarithmic nature of human auditory perception of sound intensity. But as the MDCT can take also negative values, they cannot be represented using the conventional logarithmic transformation.
One solution is to utilize a symmetric logarithmic function. the context of deep learning, Hafner et al. 2023) introduces such function and its inverse, referred to as symlog and symexp respectively:
symlog(2) x) sign(z) In( + 1)
symexp(x:) ((xxp))))) 1)
(3)
The symlog function compresses the magnitudes of large values, irrespective of their sign. Unlike the conventional logarithm, it is symmetric around the origin and retains the input sign. We note the cor- respondence with the lal companding algorithm, a well-established method in telecommunication and signal processing.
An alternative approach involves parametrizing the model to output the absolute value of the MDCT coefficients and its corresponding sign. While the MDCT does not directly convey information about phase relationships, this strategy may offer advantages as the sign of the MDCT can potentially provide additional insights indirectly. For example, opposite sign could imply phase difference
14

---

Published as a conference paper at ICLR 2024
Table Objective evaluation metrics for MDCT variant of Vocos compared the ISTFT baseline.
UTMOS (t)
PESQ (1)
V/UV F1 (1)
Periodicity (L)
Ground truth
4.058
Baseline (ISTFT)
3.734
3.70
0.9582
0.101
IMDCT (symexp)
3.498
3.648
0.9569
0.106
IMDCT (sign)
3.536
3.565
0.9547
0.109
of 180 degrees. In practice, we compute a "soft" sign using the cosine activation function, which sup- posedly provides periodic inductive bias. Hence, similar the ISTFT head, this approach projects the hidden activations into two values for each frequency bin, representing the final coefficients as MDCT exp(m)- cos(p).
A.2 RESULTS
Table 7 presents objective evaluation metrics for variant of Vocos that represents audio samples with MDCT coefficients. Both 'symexp' and 'sign' demonstrate significantly weaker performance compared to their STFT-based counterpart. This suggests that while MDCT may be attractive in audio coding applications, its properties may not be as favorable in the context of generative modeling with GANs. The redundancy inherent in the STFT representation appears to be beneficial for generative tasks. This finding aligns with the work of Gritsenko et al. (2020), who discovered that an overcomplete Fourier basis contributed to improved training stability. Furthermore, it is worth noting that the MDCT, being lapped transform, incorporates information from surrounding windows, which effectively act as aliases of the signal. To ensure Time Domain Alias Cancellation (TDAC), the prediction of the coefficients has to be accurate and consistent over the frames.
A.3 FORWARD MDCT ALGORITHM
Algorithm 1 Fast MDCT Algorithm realized with FFT
1: Input: Audio signal x with frame length N
2: Output: MDCT coefficients X
3: procedure MDCT(x)
4: for each frame in & with overlap of N/2 do
5:
< window function
6:
- fxe 55 2mm
Pre-twiddle
7:
< FFT(J)
N-point FFT
f f f (k++
8:
Post-twiddle
I - V 1/ + R() x V2
9:
10:
end for 12: return X
end procedure
15


Reviews:
[{"review_id": "cz27y2Fdj8", "reviewer": "Reviewer_ayuF", "summary": "This paper introduces a GAN-based neural vocoding method that directly reconstructs Fourier spectral coefficients rather than the prevalent time-domain generation. The experiments showcase that recovering phase information not only improves the quantitative metrics but also results in higher human ratings. In addition, its efficient architecture makes it promising to be adopted in real applications.\nThe paper is well-written and the motivation is clearly described. The experiments section is comprehensive.", "questions": "- Picking \"ConvNeXt\" as the backbone looked a bit random to me. Did you try other architectures and decide to use ConvNeXt in the end?\n- Have you run experiments on ConvNeXt in the time domain?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-structured and easy to follow. \n- The motivation of bridging the gap between time-domain and time-frequency-main is sound and the results indicate the proposed method works as expected.\n- The code is open-sourced and audio samples are available.", "weaknesses": "- Novelty is limited considering the ICLR standards. \n- It's not clear to me whether using \"ConvNeXt\" solely in the time-domain is already bringing all the benefits stated in the paper. The experiment of using ConvNeXt in the time-domain seems missing\n- There are a few overclaims in the audio reconstruction task. PESQ/UV F1/ Periodicity numbers of EnCodec still win over Vocos by a large margin at 12.0kbps. Should we still consider \"Vocos notably outperforms EnCodec?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sn9vRLCijg", "reviewer": "Reviewer_7A8e", "summary": "Vocos is a waveform synthesis (vocoder) model operating directly on the frequency scale to estimate Fourier spectral coefficients (magnitude and phase) followed by inverse Fourier transform (iSTFT). Due to the challenges in phase modeling with neural networks, previous non-autoregressive, GAN-based neural vocoders directly generate time-domain waveforms through an upsampling architecture, leading to a slowdown in inference latency (even with the non-autoregressive modeling). Vocos, to the best of the reviewer's knowledge, constitutes the first success in the direct estimation of both the magnitude and phase entirely from the frequency domain, resulting in a highly efficient neural vocoder.", "questions": "Overall, Vocos manifests an exciting step towards enabling a neural vocoder operating entirely in the frequency domain; its fast speed will contribute to accelerating efficient speech synthesis solutions, where the speed of such models has been bottlenecked by the time-domain neural vocoder. But at the same time, I am also interested in the scalability of the method. Specifically, a neural vocoder is also recently viewed as a (de)compression model (similar to VQGAN in the image domain, with EnCodec and DAC[1] as examples) to build the audio generative model in the latent space. For this application, practitioners may want higher quality decoding with increased scale if speed is not a concern. Can we expect that the Vocos (either as a standalone vocoder or the decompression model) can be scaled up further to achieve even better quality? Or, if Vocos exhibits a performance limit at a certain scale, what would be the root cause in the author's opinion?\n\n[1] Kumar, Rithesh, et al. \"High-Fidelity Audio Compression with Improved RVQGAN.\" arXiv preprint arXiv:2306.06546 (2023).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "As stated in the summary, direct phase estimation has historically been known to be challenging. The major contribution of Vocos is being the first model to succeed in high-quality phase estimation for neural vocoding through the careful design of a sinusoidal activation head that handles phase wrapping. Since the feature propagation is entirely in the spectral domain with the same I/O dimension, Vocos is significantly faster than time-domain models and more flexible in adopting modern isotropic architecture (ConvNeXt), previously proposed in other data domains, for neural vocoding tasks. Keeping the scope of a general-purpose neural vocoder is also a plus, where the proposed method is not dependent on a specific input representation (mel spectrogram) and can also be trained with the latent feature as a neural audio codec.", "weaknesses": "While the main focus of the manuscript is realizing an efficient neural vocoder operating in the frequency domain, it would also be interesting to further assess the robustness of the proposed method. Considering that the experimental setup largely follows a previous time-domain neural vocoder (BigVGAN), readers may wonder if Vocos can also achieve similar robustness. The provided samples in the manuscript only contain clean speech samples. To further convince the readers, it would be useful to add non-clean speech and audio samples to the demo (noisy speech, environmental sounds, and music, for example) similar to the results presented in the previous work. The VISQOL score in Table 3 looks promising, so letting the readers form their opinion on the subjective quality by adding such samples will be helpful as well. By adding these samples, we could also confirm that the objective VISQOL scores align with human perception as well.\n\nIn my opinion, the statement \"ConvNeXt blocks can more effectively model spatially local input patterns\" is not fully supported by current results; while it is true that Vocos can easily adopt modern isotropic architectures, the same can also be explored with time-domain neural vocoders. To verify this under scrutiny, adopting ConvNeXt to the time-domain GAN vocoder and measuring the performance gain/loss will be useful (probably with dilation to the depthwise conv in ConvNeXt as done by ResBlock in HiFi-GAN). If the time-domain GAN vocoder shows degradation in performance, it will make the benefits of isotropy in Vocos more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ItMqWMJtF", "reviewer": "Reviewer_cFuH", "summary": "This paper presents an approach to Neural Vocoding with GAN based on generating directly Fourier coefficients instead of targeting time domain signals. Overall, the paper is very well written and provides a very large set of (both qualitative and quantitative) experiments and evaluations, demonstrating the soundness of the proposed approach.\n\nAlthough I think that this work is of very high quality, and I agree overall with the argument made by the authors in the spectral/temporal dilemma in generative models (which has been a central question of study in the audio domain), I am not convinced by the scientific novelty of the proposed method. For instance, the GANSynth model proposed in 2019 (but not cited in this paper) already proposed a GAN model to produce invertible spectral representations, by modeling instantaneous frequencies:\nEngel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., & Roberts, A. (2019). Gansynth: Adversarial neural audio synthesis. arXiv preprint arXiv:1902.08710.\nhttps://openreview.net/pdf?id=H1xQVn09FX\n\nHence, the major novelty proposed by the paper is to use « ConvNeXt blocks », which have themselves been introduced in a previous paper (Liu et al. 2022). Therefore, it seems rather a slim contribution to be accepted in the ICLR conference. I would advise the author to resubmit the paper in a more audio-applicative type of conference.", "questions": "-", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Very large array of experiments demonstrating the quality of the approach.\n- Sound reasoning and scientific method for establishing the overall architecture.\n- Targeting efficient and lightweight models is always a welcome addition in the current trends of deep generative modelling research.", "weaknesses": "- Low amount of scientific novelty\n- Although I agree that GANs are still dominant, neural vocoders are increasingly being driven by diffusion models over the past months.\n- Pictures of spectrograms are usually unhelpful in paper and in this case, I feel it is even harder to truly understand what is the added value of this figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1OK9k6m7W", "reviewer": "Reviewer_9BcD", "summary": "The paper presents a new method for neural vocoding by generateing fourier spectral coefficients instead of generating wave forms in the time-domain, this leads to significant computational speed up.", "questions": "Can you please address the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written.\n- The authors release the source code.\n- The authors present an extensive evaluation of the Vocoder.\n- The proposed models offer significant speed-ups compared to the related work without loss in the quality.\n- The proposed method is very simple and leads to impressive results.", "weaknesses": "1- Limited ablation study: The authors show the effect of replacing ConvNeXt with residual blocks, but no studies are presented on the architecture itself: the effect of the number of layers and hidden dimensions.  The effects of the resolution and overlap of the mel spectrograms.\n\n2- Limited explanation of the backbone architecture: I think the way the backbone network is processing the mel-spectrograms is still ambigious. After digging into the code, it seems that the frequency dimension of the spectrogram is used as input channels, and 1D convolution is performed over the time dimension. The receptive field of each timeframe grows by 7 frames in each layer (right?). Therefore, the decoding of one timeframe depends on the adjacent time frames. This is controlled by the receptive field of the last layer. This raises a lot of questions: (a) Why did you choose 7? (b) how increasing the depth of the backbone network (and therefore the receptive field of the last layer over time) affects the performance. \n\n3- The encodec experiments Table 4 is not clear for me: How did you control the bandwidth for your model? Is it by reducing the mel-spectrogram resolution?\n\n4- In end-to-end text-to-speech: You write, \" Vocos trained to reconstruct EnCodec tokens\". This is not clear: are you using Vocos) to decode EnCodec tokens?\n\n5- (minor) Table 5 Inference speed: Is it possible to include float or multiply-accumulate operations since there is a chance that some FFT operations can be driver (or hardware) accelerated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)