FlowDec: A flow-based full-band general audio codec with high perceptual quality
Authors: Simon Welker
Matthew Le
Ricky T. Q. Chen
Wei-Ning Hsu
Timo Gerkmann
Alexander Richard
YI-CHIAO WU
Venue: ICLR 2025 Poster (2025)
Keywords: audio
audio codec
generative models
flow matching
postfilter
signal enhancement
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: FlowDec is a flow-based postfilter codec for general audio without adversarial training, and a competitive alternative to current GAN-based SOTA codecs.

Abstract: We propose FlowDec, a neural full-band audio codec for general audio sampled at 48 kHz that combines non-adversarial codec training with a stochastic postfilter based on a novel conditional flow matching method. Compared to the prior work ScoreDec which is based on score matching, we generalize from speech to general audio and move from 24 kbit/s to as low as 4 kbit/s, while improving output quality and reducing the required postfilter DNN evaluations from 60 to 6 without any fine-tuning or distillation techniques. We provide theoretical insights and geometric intuitions for our approach in comparison to ScoreDec as well as another recent work that uses flow matching, and conduct ablation studies on our proposed components. We show that FlowDec is a competitive alternative to the recent GAN-dominated stream of neural codecs, achieving FAD scores better than those of the established GAN-based codec DAC and listening test scores that are on par, and producing qualitatively more natural reconstructions for speech and harmonic structures in music.

Published as a conference paper at ICLR 2025
GENERAL AUDIO CODEC WITH HIGH PERCEPTUAL QUALITY
FLOWDEC: A FLOW-BASED
Simon Welker Matthew Le", Ricky T.Q. Chen', Wei-Ning H Timo Gerkmann', Alexander R Yi-Chiao Wu $ *Signal Processing, University of Hamburg, 22527 Hamburg. Germany FAIR / Codec Avatar Labs, Meta, 10001 New York / 15222 Pittsburgh, USA
ABSTRACT
We propose FlowDec, neural full-band audio codec for general audio sampled 48 kHz that combines non-adversarial codec training with a stochastic postfilter based a novel conditional flow matching method. Compared the prior work ScoreDec which is based on score matching, we generalize from speech to general audio and move from 24 kbit/s as low 4 kbit/s, while improving output quality and reducing the required postfilter DNN evaluations from 60 to 6 without any fine-tuning or distillation techniques. We provide theoretical insights and geometric intuitions for our approach in comparison to ScoreDec as well as another recent work that uses flow matching and conduct ablation studies on our proposed components. We show that FlowDec is a competitive alternative to the recent GAN Gon-dnminated stream of neural codecs, achieving FAD scores better than those of the established GAN-based codec DAC and listening test scores that on par, and producing qualitatively more natural reconstructions for speech and harmonic structures in music.
INTRODUCTION
An audio codec is technique aiming to compress audio waveform into compact and quantized representations and to reconstruct the audio waveform based on those encoded representations faithfully. The compact and quantized representations
suitable for efficient transmission and storage, which is essential for mobile communications and live video streaming applications (Kroon
et al. 1986 Salami al., 1994| Rao & Hwang 1996). Different from legacy codecs (Atal & Schroeder 1970 Schroeeer & Atal| 1985] O "''''SShahhssss 1988) which exhibit considerable quality
sacrifice in low-bitrate scenarios, modern codecs achieve lossless (Liehchen & Reznik 2004 Coalson]
2000) or acceptable lossy (Valin et al. 2013 Bessette et al. 2002 Dietz et al. 20155) codings with
2x or 10x compression ratios. However, these codecs usually involve ad hoc designs and extensive manual efforts (Kim & Skoglund| 2024), which hinders the codecs from end-to-end optimizations to achieve high-fidelity audio coding even lower bitrates (e.g. <12kbit/s)
End-to-end (E2E) Neural codecs (Zeghidour et al., 2021 Défossez et al. 2023 Wu et al. 2023
Kumar et al. 2024 have seen surge in interest recent years, particularly due to their usefulness in generative audio tasks such as generating music or speech conditioned on textual description or transcript. These codecs nowadays achieve very good audio quality at bitrates as low as kbit/s, where most classical non-neural codecs fail to produce acceptable results. To achieve high-quality results at low bitrates, most E2E neural codecs employ adversarial training inspired by generative adversarial networks (GANs) (Goodfellow et al. 2020) to recover natural-sounding signals and to avoid artificial artifacts that arise when training only with waveform or spectral losses.
Score-based (diffusion) and flow-baser generative models (Ho et al. 2020 [ong et al. 2021 Lipman et al. 2023) have in recent years taken over many generative application domains from GANs, In this spirit, recently proposed score-based codec is ScoreDec (Wu et al.) (022)), widely applicable generative postfilter for E2E neural codecs. ScoreDec aims to recover enhancing codec outputs, removing adversarial losses when training the E2B model and instead
signals by
Work completed as part of internship at Meta

---

Published as a conference paper at ICLR 2025
training a score-based generative model (Song al. 2021 postfilter. While ScoreDec shows clear advantage in output quality compared to the original codec variants that use adversarial training, it only considers speech signals, was tested only for relatively high bitrate of 24 kbit/s, and most importantly- - has prohibitively expensive inference at a real-time factor (RTF) of 1.7 caused by the need of around 60 DNN evaluations.
In this work, we propose FlowDec, a generative neural codec based on a novel adaptation of conditional flow matching (CFM) (Lipman et al. 2023 Pooladian al., 2023 Tong al., (024). and show that it is a competitive alternative to the current GAN-focused stream of neural codecs for general full-band audio. We address the shortcomings of SeoreDec (Wu al., 2024) by designing and training for general audio beyond only speech, reducing bitrates from 24 kbit/s to below 8 kbit/s, and reducing the needed number of DNN calls from 60 to 6. We design for full-band audio covering the whole range of human hearing (S 20 kHz) with 48 kHz sampling rate, to avoid a significant loss of fidelity due to the total removal of high but audible frequencies as in Défoossez et al. (2023) or
Zeghidour et al. (222)) The key advantage of 48 kHz over 44.1 kHz models such as DAC (Kumar ct al. 2024) are that it is easier to achieve whole-number feature rates (75 Hz vs. 86.13 Hz) and bitrates (7500 vs. 7751.95 bit/s) since 48,000 has simpler divisors.
Our main contributions in this work are: (1) the extension and simplification of prior score-based generative audio enhancement methods (Welker al., 2022: Richter et al. 2023 Wu et al. 2024) with a novel adapted CFM method, with theoretical connections and comparisons to recent works on CFM
(Pooladian al., 2023 Tong et al. 2 ; (2) the application audio coding and extension of the
speech-only ScoreDec Wu et al.) 2024) to general full-band audio at very low bitrates, while reducing the number of DNN evaluations by factor of 10 without fine-tuning or distillation techniques; (3) high-fidelity perceptual quality competitive with a GAN-based state-of-the-art codec (Kumar et al. 2024]. which we confirm with objective metrics and listening tests.
2
RELATED WORK
2.1
NEURAL CODECS
Based on the training objectives, neural audio codecs can be divided into three main categories: auto-encoder (AE), neural vocoder, and postfilter. In the early days, legacy AE-based codecs (Krishnamurthy et al. 1990; Wu et al., 1994, Deng et al. 2210 usually train an AE to reconstruct handcrafted acoustic features and retrieve discrete codes with an independent quantization module on the hidden units which is not globally optimized. and require extensive ad hoc assumptions on audio signals and an additional audio synthesizer. Morishima et al. (9900) propose the first AE speech codec in the waveform domain but do not train the quantizer jointly. The pioneering fully E2E waveform- domain audio codecs incorporate straight-through gradient estimation (Van Den Oord al.. (017) or softmax quantization [Kankanahalii] 2018) for joint AE and quantizer training. However, they suffer from either slow inference from autoregressive decoding or limited quality from the lack of effective waveform losses for von-auvoreeresssvve (NAR) decoding. Recently, given the significant
improvement in NAR audio waveform generation YYamamoto al.
2020 Kumar et al., 2019 Kong
et al. 2020) adopting GANs (Goodfellow et al. 2020), GANO),,AN-bba based NAR audio codecs (Zeghidour
et al. 2021 Défossez et al.) 2023) Wu et al. 2002 Kumar et al.
2024 achieve fast coding, impressive
audio quality, and low bitrates.
By using the high-fidelity audio generations achieved by neural vocoders (van den Oord et al., 2016 Kalchbrenner et al. 2018 Valin & Skoglund 2011a; Kong et al. (2020), methods which reconstruct the audio waveform based on quantized handcrafted acoustic features (Klejsa et al. 2019 Valin & Skoglund, 2019b; Mustafa et al.) I0. codes of conventional codecs (Kleijn et al. [2018), or neural AEs Ww et al. al.] 2023] [San Roman et al. (2024), also achieve impressive coding performance.
Postfiltering (Zhao et al. 2018 8, Deng al., [2200] Biswas & Jia, 2020 Korse et al. 2022) is similar approach, easing the training burden of abstract code-to- tode-to-waveform mapping by utilizing the decoder
of pre-trained codec to generate distorted waveform, which is then enhanced by a postfilter.
2.2 SCORE-BASED GENERATIVE SIGNAL ENHA
Welker et al. (2022) propose SGMSE, a score-based generative model (SGM) for speech enhance- ment (SE), by formulating the speech enhancement task as diffusion process in the complex spectral
2

---

Published as a conference paper at ICLR 2025
DAC
Training
Inference
Audio
Encoder
Quantizer
Decoder
Audio
Discriminator
FlowDec
Inference
Training
Training
Audio
Encoder
Quantizer
Decoder
Audio
Postfliter
Figure I:
Method overview: Codecs such as DAC (Kumar al. 2024) employ adversarial training. using multiple specialized discriminator networks trained jointly with the decoder. Our method FlowDec is trained non-adversarial two-stage fashion, removing these discriminators and instead adding stochastic postfilter that can produce multiple enhanced estimates of the pretrained decoder.
domain. avoid the hoc assumption that the additive noise in noisy speech follows a white Gaus- sian distribution, SGMSE directly incorporates the SE task into the diffusion process by interpolating between clean and noisy spectra, leading to data-dependent prior similar to PriorGrad (gil Lee et 2022). RRchter et al. (2023) propose SGMSE+, extending SGMSE to speech dereverberation and significantly improving its quality by using the more powerful backbone NCSN+ (Song et al. (02I) for the score model. Due to the complex spectral modeling, both magnitude and phase spectra utilized and enhanced, resulting high-quality speech restoration.
Coding artifacts can also be viewed as a special type of noise that should be removed. To take advantage of both E2E and postfilter approaches, SeoreDec (Wu et al. 2024) adopts SGMSE+ as the postfilter for both conventional and neural codecs and achieves human-level speech quality. However, the inference of ScoreDec slow due to the high number of diffusion steps, and the effectiveness of ScoreDec for general audio is unclear. To tackle these issues, propose FlowDec for general audio coding, with significantly reduced runtime cost at real-time factor below 1, and a simplified formulation that requires only one hyperparameter instead of four.
METHODS
We cast the problem of recovering estimate å R of the clean audio x" RR given the code c: = E(x*) from an encoder E as a stochastic inference problem, with the goal of having a model that can provide clean audio estimates & samples from the distribution
f Pdata ( c E(z"))
where Pdata (-Ic) is the conditional distribution of clean audio given the code c. We argue that this treatment is natural, as any encoder E that maps x" E g to a lower-dimensional discrete representation many-to-one mapping: multiple x" will have the same code e. Hence, fulfilling the ideal property D(E(z") x² is formally impossible if D is one-to-one mapping. One could instead construct as an optimal estimator the mean sense by minimizing
min Ez- [dist [dist(D(EE) (D(E(D(E x D
with a pairwise distance dist such as the 2 or I distance. However, it is known that a method trained this way typically does not produce perceptually pleasing signals (Blau & Michaeli 2018 2019) even with domain-specific losses. A popular way around this for neural codecs to employ adversarial training losses (Zeghidour et al. 202 Défossez et al., 20233 Kumar et al. 2024) to shift the distribution of decoded signals closer that of natural signals. While relatively effective, this approach lacks clear interpretability, is limited by the quality the discriminator, and may fail to properly minimize the distance between p(t) and p(=").

---

Published as a conference paper at ICLR 2025
An alternative, which we follow here, is directly construct D as one-to- one-to-many mapping, as done in recent literature on other audio inverse problems such as speech enhancement, dereverberation, and bandwidth extension (Richter et al. 2023] Lemercier al. 2023a b) and most recently for speech coding by Wu et al. (024)). We show a conceptual overview of this idea in Fig. [ We realize this mapping in the form of a stochastic decoder (c(c) 22( Pa(c)), combining deterministic pre-trained initial decoder Do with a stochastic postfilter Q. Defining y = Do(c), St produces conditional samples &- po(Gly) from learned distribution pa(-ly), which approximates the intractable distribution Pdatal-Iy (y) via minimization of a statistical divergence D:
Pg arg min P(gn(-lv), Pdata(-ly)) gia
(3)
We can assume that pn = (2(c)) since Do is known, deterministic and non-compressive. The role of Do is now to provide a decent initial estimate, which may well still suffer from artifacts and is enhanced by Ω to deliver perceptually pleasing results. We choose D as the Wasserstein-2 distance, and practically minimize equation by training aflww model, neural network vo trained with an adapted CFM objective (Lipman et al. (023).
3.1 FLOw MATCHING
Lipman et al. (2023) introduce the idea of Flow Matching, where the goal is to learn a model that can transport samples from tractable distribution (0())) to an intractable data distribution qı (211 = Pdata by solving the neural ordinary differential equation (ODE)
d त.
).(()) = ue ue())(z)),
Φ0(x)= = T0
starting from sample To go- We call de : 10, R- R. the flow with the associated dime-dependent vector field ut [0.1] x I N + RRA, which generates a probability density path
Pt R. + Eso with Pt=o = 20 and Pt=1 Q1- They propose to learn up with the CFM target:
LCFM = (at.)))) [11o((z)) t) u, (x121))
where 01 ~ Qi and L denotes training loss function. A key insight that the conditional Eq. 5 has the same gradicnts an intractable unconditional flow matching objective (Lipman et al. 2023 Eq. 5), and marginalizes to the correct unconditional probability path (z) and flow field 2n (z).
3.2 JOINT FLOW MATCHING FOR SIGNAL ENHANCEMENT
In the original flow matching (Lipman et al. 2023 and score matching (Song et al. 2021 Formulations, zi is sampled independently of 21, typically from a zero-mean Gaussian qu -O, I). |Pooladian et al. (2023) Tong et al. 2024
Unconditional go
Shifted gu( (0())2)))
show that, while the conditional paths pr (x| (+)) fulfill optimal transport (OT) from go qı when go is standard Gaussian, the modeled marginal probability path (xr) generally does not fulfill OT. This can lead high-variance training and low straightness in the learned marginal flow field Ug. and thus to inefficient inference and suboptimal sample quality. To rectify this, both works propose a per- batch approximation to OT between the full distributions, by reordering the pairings in each training batch {<rs,o, ) with optimal couplings determined by OT algorithm on each batch. Effectively, this samples (Fo, 21) ~ qZz,, 21) jointly rather than independently.
Figure 2: Unconditional go (x ver- sus our go (0)o Colored dots rep- resent y, stars are associated z*.
Here, also propose sampling (zo, x1) jointly, but in way that is adapted to enhancement tasks and does not require any OT solvers or extra computations. Concretely, since we have access to the initial estimate y Do(c) (o(E(x*)) )), we choose the probability path
Pr ( |1 y) = (n pr, ot) = N(x + 1(x1 y), - r)²
Note the different notational convention in score-based works, where the meaning of xo and og is reversed.
4

---

Published as a conference paper at ICLR 2025
where E = (dja)(33) is a diagonal covariance matrix. This probability path is linear interpolation between y and z1. with noise linearly decreasing from By to zero. This leads to a coupling between Tp and 21 through y. Namely, go (z v) (z; y, En), i..., the mean of Zo is shifted from O to y, similar to score-based signal enhancement works (Richter et al., 2023 Wu et al.] 2024). Intuitively, while do not use it for inference training, the marginalized (o( is now a mixture of Gaussians, each of variance oy and centered at the respective from the training data, see Fig. 2 When By is well-chosen so these Gaussians have negligible overlap, no minibatch OT is needed as the per-batch couplings can be assumed optimal by construction. We find that the choice of 0g is
important for output quality, see Appendix AA.I more details. The conditional Un can be found via (Lipman et al. 2023 Eq. 15), with the full derivation in Appendix A.2
21 Zz
My 2 21. v)
(7)
1-t
To simplify, since xy can be written in terms of To, we note that
It t + (1 t)
zo (r0ig, My)) E~ N(0.1)
(8)
= t + (1 (1-18 tly + -1-6h t))oa,
(9)
zo y + oyE,
~~ 10,
(10)
which, using that ii x from Eq. 6 leads to the simple joint flow matching loss
2
CJFM =
(0 v0(22, (.
-(=
(y
+
(11)
where D is the training dataset. Note also that this loss removes the numerical instability around f xz of Eq. (7) by reparameterizing in terms of xg and 71- By choosing dy > 0, we en- force the flow field to be contractive mapping. This ensures the ODE for inference is numeri- cally stable and converges locally. Our choice of Pt improves upon SGMSE (Wekeer et al.) 2022 Richter al. 2023 Wu al.) 2024). in that trajectories in our formulation can reach x' ex- actly, which SGMSE fails to do since it does not model the correct go (Lay et al. (023). We also avoid designing and tuning special stochastic differential equations (SDEs) with multiple hy- perparameters and use only one hyperparameter, yy for which we propose data-based heuristic (Appendix A.T). Another recent work for audiovisual speech enhancement by Jung al. (2024) also makes use of CFM, but uses an independent CFM formulation (Tong et al. 2024) resulting in a constant og = a and the target flow field u, being independent of the sampled noise. This leads to non-contractive flow field and the potential for residual noise being left in the estimates since T1 = o > 0, whereas 0 = 0 in our case. We illustrate this qualitatively in Fig. 3 and also show empirically in our results section that, for our postfiltering task, our formulation leads to better quality than both alternatives, at both a low and high number of function evaluations (NFE).
Ours: d, = 0.1(1 (
SGMSE
AloW d, 0.1
Figure 3: Flow field comparison at / = 0.7 for our linear σr (left) versus score-based SGMSE (center) and FloWAVSE with constant o, (right) for a toy problem. The white dot is y. yellow stars pos- sible x², blue lines are sample trajectories, and the background color indicates the density Pt- SGMSE has highly curved trajectories and does not contract xx; FlowAVSI is non-contractive.
In practice, we replace x", y with the feature representations X', Y from an invertible feature extractor and learn the flow in this feature domain. Namely, & is an amplitude-coppressed complex short-time Fourier transform (STFT) (Welker et al. 2022) with compression exponent a 0..,, Appendix AA. for details. We provide to Ug as conditioning via channel-wise concatenation at the input (Richter al. 2023).
After training, the flow model ui together with the ODE models the conditional distribution (x To produce clean feature estimates X ~ Pn, we first sample an initial state (latent) Xn qn (X) Y) and then solve the flow ODE 4 using Ug from t 0 to ODE solver to get X,. We use the Midpoint solver with 3 steps (NFE = 6) unless otherwise noted, due to its improved quality over the Euler solver at low NFE, see Appendix(A.1. Finally, we apply the inverse of the feature extractor + produce the waveform estimate å +--(X²).
with numerical
5

---

Published as a conference paper at ICLR 2025
3.3 NON-ADVERSARIAL CODEC TRAINING
Due to a lack of effective phase losses, NAR audio generative models trained with only spectral losses usually exhibit buzzy noise caused by unsynchronized phases. Many works employ adversarial training to circumvent this and restore more natural-sounding audio. This however requires complex handcrafted nulti-discriminator losses and weightings to avoid unstable training, mode collapse, and divergence, and lacks interpretability (Wu et al. 2024 Lee et al. 2024). In recent years, generative diffusion models have largely superseded GANs for image and audio generation due to easier training and better detail modeling (Dhariwal & Nichol 2021 but have yet to make such a strong impact for audio codecs.
To overcome these issues, remove adversarial training and instead use generative postfilter. We train a deterministic neural codec as the initial decoder Do without any adversarial losses and leave the task of matching the distributions of output audio and clean audio to the stochastic postfilter Ω. The simplest way forward, which we follow, is to take an existing state-of-the-art neural codec such as DAC (Kumar al. 2024) Do and remove all components related to adversarial loss terms.
3.4 UNDERLYING CODEC: IMPROVED NON-ADVERSARIA DAC
In principle, stochastic postfilters such as ours can be trained for any underlying codec to enhance its waveform estimates, shown in Wu al. (2024). We use DAC (Kumar al. (024) as the basis for our underlying codecs due to its status as state-of-tee of-the-art neural codec, as also recently established for speech by Muller al. (2024), and its adaptibility for other sampling rates and bitrates. We remove the adversarial losses and modify some configuration settings listed in Section 4.2
When we first trained this non-adversarial codec, we found that it produced unnatural results and bad ocale-invarian signal-to-distorion ratio (SI-SDR) values around -30 dB, particularly for music. After finding that low frequencies (5 2 kHz) were badly modeled we add a multiscale constant-Q Q transform (CQT) loss, inspired by the high low-frequency resolution of the CQT, frequent use of the CQT in music processing (Moliner et al. 2023), and the multiscale Mel losses used by DAC. As in DAC's multiscale Mel loss, further add a L waveform-domal loss to improve SI-SDR values and phase errors that magnitude- only losses blind to. We demonstrate the effectiveness of these losses in Appendix A.7.2
both the differences of amplitudes and of angaampittudes. We
3.5 FREQUEENCY-DEPENDEN NOISE LEVELS
As noted in Section 3.1 the choice of o, is important for output quality. is well known that the power spectrum of most natural signals follows inverse power law, high frequencies have much lower power than low frequencies. single scalar oy can thus potentially lead to oversmoothing when the added Gaussian noise dominates high frequencies, as also previously observed for images (Kingma& & Gao 2023, Appendix J). To rectify this, we calculate frequency-dependden curves o,(() by performing the heuristic quantile calculation in Eq. (12 independently for each STFT frequency band. Similarly, MBD (San Roman et al.) 2023) proposes a band-depende noise scale but uses only 4 broad Mel bands for this purpose. We demonstrate the effectiveness in Appendix|A.7.. A.7.4
4 EXPERIMENTAL SETUP
4.1 DATASETS
For underlying codec training, we prepare varied combination of datasets containing music, speech, and sounds, which are listed in Table [ As proposed in Kumar al. (2024), we sample audios in type-balanced way during training, i.c., each training batch contains in expectation - the same number of speech files as music files and sound files.
For postfilter training, use the same overall dataset as basis but perform the following additional steps: (1) To avoid slow postfilter training from calling D, in every step, we randomly sample 100,000 clean files x" per audio type and crop out segments with maximum 30-second duration, calculate y = Do(E(z*) )), and store on disk. (2) For the postfilter to learn complex audio scenarios, increase data variety with 100,000 clean 10-second mixtures of all three audio types from the

---

Published as a conference paper at ICLR 2025
Table 1: Datasets used for codec training. Datasets in [brackets] are internal. fm denotes the maximum sampling frequency and "h" is short for hours. For SawCCas-FFreeouun we filter the part of FreeSound contained in WavCaps to keep only the files with commercial ffrinddly licenses. For Common Voice 13.0* we use a custom subset.
Dataset
Duration
fmax
Type
MSP-Podcast (Lotfian & Busso 2019)
103
16 kHz
Speech
Common Voice 13.0* (Ardila et al. 2020
1602 h
16 kHz
Speech
LibriTTS (Zen et al.
2019)
553 h
24 kHz
Speech
EARS (Richter et al.
2023
h
48 kHz
Speech
VCTK 84spk ( Valentim- Botinhao
2017
20h
48 kHz
Speech
LibriVox (Kerns
2014
55611
16 kHz
Speech
Expresso Nguyen et al.
2023)
20h
48 kHz
Speech
InternalSpeech)
1512h
48 kHz
Speech
[Iuser]a|Music]
18949 h
32 kHz
Music
WayCaps-FreeSound FreeSound* (Mei et al.
2024)
1582
32 kHz
Sound
[InternalSound]
5309 h
48 kHz
Sound
subsets described above. We mix each randomly paired three audios in random proportions with mixing coefficients (Wspoech,ee Mmusie.k: Wgound,k) sampled from a Dirichlet distribution Dir (espech = 4, Cruusic = 2, O'audio 1). We repeat all constituent segments shorter than 10 seconds and center-crop all that are longer. This leaves us with 400,000 pairs (2778 hours) of data.
As test set, we use 3,000 random audio samples with 1,000 of each audio type: 500 files from the VCTK test set gelentini-Botinhann 2017) and 500 from the EARS test set (Richter et al. 2024b) for speech, 500 files from MUSDB I8-HQ (Rafii et al. 2019) and 500 from MusicCaps (Agostinelli et al. 2023) for music, and 1000 files from AudioSet (Gemmeke et al. 2017) for sound. To avoid overlap with MusicCaps, we remove all files from AudioSet with music-related tags, but keep tags related to instruments. We crop audios to a 10-second duration. As MUSDB, MusicCaps, and AudioSet are not used for training, we sample from them without regard to train/test splits.
4.2 MODEL TRAINING AND VARIANTS
For our underlying codecs, we use the official code and training settings from DAC Kumar et al. 2024) but remove ad- versarial losses (Section (.3), add a CQT and waveform loss (Section 3.4), and mod- ify the configuration as listed Table 2 We call these underlying codecs NDAC to avoid confusion with the adversarially trained DAC. NDAC-75 is targeted at 48 kHz audio with a whole-number fea- ture rate (75 Hz) and whole-number bi- trates. NDAC-25 is a variant tailored for downstream generative audio tasks, with a lower feature rate (25 Hz) and feature dimension which are advantageous for audio generation due to more efficient memory usage and decreased modeling difficulties. For the CQT loss (Section 3.44 we use the CQT2010v2 implementation of the CQT from the nnAudio Python package with octaves, hop length 256, minimum frequency 27.5 Hz, {16,32,4) 64, 80) bins per octave, with a loss weight of 1 for music samples and 0) for audio and speech samples. For the It waveform loss, we use weight of 50. We train for 800,000 iterations with 0.4 second snippets and a batch size of 72. As baselines, we train DAC-75 and DAC-25, equivalent versions of NDAC-75 and NDAC-25 with the original adversarial losses. To show that the differences between FlowDec and DAC are not just caused by the extra parameters from the postfilter, also train baselines 2xDAC-75 and 2xDAC-25 for which we double the channels of all decoder convolution layers, increasing the parameters by +1 100 M vs. +66 M from the postfilter.
Table 2: Our underlying codec variants, compared to official 44. kHz DAC by Kumar et al. (2024). I's the sampling rate in kHz, H is the hop length in samples, fteat is the feature rate in Hz, n. is the number of code- books, and demb is the latent code embedding dimension. Bitrates are in kbit/s.
Name
Bitrates
f
H
demb
DAC
0.86-7.75
44.1
512
1024
NDAC-75
0.75-7.50
48
640
10
1024
NDAC-25
0.25-4.00
48
1920
16
128
7

---

Published as a conference paper at ICLR 2025
As postfilters, train the following variants based on NDAC-75 and NDAC-25:
1. FlowDec-75m: 75 Hz, multi-bitrate. Trained based on NDAC-75 with bitrates {7.5, 6.0, 4.5,
3.0} kbit/s, by setting the number of codebooks inference to {10, 8, 6, 4). We include only this set of bitrates for ease and speed of training, and because found that the codec does not provide good results below 3b0ts
2. FlowDec- 75s: 75 Hz, single-bitrate. Trained based on NDAC-75 using only the highest bitrate of 7.5 kbit/s. The goal of this variant is to serve baseline for ablations and to investigate the quality gap between a single- and multi-bitrate postfilter.
3. FlowDec-25s: 25 Hz, single-bine bitrat.. Trained based on NDAC-25 with bitrate of 4.0 kbit/s. We do not train for multiple bitrates here as the bitrate and feature rate already very low.
We train all postfilters based on slightly modified NCSN++ architecture (Song al. 202 with 26 M parameters (details in Appendix A.3). We use Adam (Kingmal 2014) a learning rate of 10 for 800,000 iterations, a 2-second snippet duration, and batch size of 64. We track an exponential moving average (EMA) of the weights with decay 0.999 for inference. For every variant, we train one version with global By and one with a wrequency-dependeee (f),), see Section 3.5 We use the rrequency-dependent variants for all results unless stated otherwise. For the global variants, we set oy = 0.66. For the frequency requency-dependent variants, we estimate 768-point frequency curves (f( and smooth them with Gaussian kernel of bandwidth 3. We train further models for ablation studies (Appendix A.7) based on FlowDec-75s.
4.3 OBJECTIVE METRIC EVALUATION
For evaluation with objective metrics we use SI-SDR (Roux et al. 2019), Frechét Audio Distance (FAD) with clap-laion audio embeddings as proposed Gui et al. (2024). frequency-weiihteed segmental oggnalto-noois-ratti (fwSSNR) (Loizou 2013), the neural ITU-T P.804 estimation method SIGMOS (Ristea al.] 2024), and logSpecMSE, i.e., the mean squared error (MSE) of decibel bog-magnitude spectrograms with 32 ms Hann window and 75% overlap. Note that SIGMOS is only valid for speech signals, so we only evaluate it on the speech test audios.
4.4 SUBJECTIVE LISTENING TESTS
Since objective metrics generally do not tell the full story of how method is per- ceived by human listeners (Torcoli et al. 2021), is important also test this per- ceived quality directly. We conduct two MUSHRA-like tests (ITU, 2015) detailed in Table 3 comparing Flow Dec variants against their DAC equivalents. "Test A" is designed to test our main models, and "Test B" to test low feature rate (25 Hz) models. We use Opus (Valin et al.) 2013)
Table 3: Listening test parameters. Bold numbers in parentheses denote the bitrates kbit/s.
Test
Compared methods
FlowDec-75m (7.5, 4.5), FlowDec-75s (7.5),
A
DAC-75 (7.5, 4.5), EnCodec (6.0), Opus (7.5)
FlowDec-25s (4.0), FlowDec-75m (4.5). DAC-25 (4.0), DAC-75 (4.5), Opus (4.0)
B
at the highest used bitrate as the low anchor and include the original audio as the hidden reference. In Test A, we also include the official 48 kHz checkpoint of EnCodec (Détossez al. 2023) at 6.0 kbit/s for comparison. We conduct both tests with 21 random 10-second audios from our test set: 7 from the EARS test set, 7 from MUSDB, and from AudioSet. We ask 15 expert listeners rate each audio on a scale from 0 to 100. We exclude listeners that rated the reference < 90 or the low anchor > for more than 15% of trials, resulting in listeners for Test A and 10 for Test B.
5 RESULTS
5.1
OBJECTIVE METRICS
In Fig. 4 we show the objective metric results of FlowDec-75m and FlowDec-75s compared to EnCodec (48 kHz), DAC-75, 2xDAC-75 and the official DAC 44.1 kHz checkpoint, and also include
8

---

Published as a conference paper at ICLR 2025
FlooDDe--Tm
Den
FeowDDo-225
DAC-75
DAC-25
24DAC-75
EnCodoo
F0Dx100
SI-SDR f
fwSSNR
SIGMOS
15
10.0
17.5
160
3.5
10
7.5
15.0
140
3.0
5.0
2.5
12.5
100
2.5
10.0
0.0
2.0
3.0
4.5
6.0
7.5
3.0
4.5
6.0
7.5
3.0
4.5
6.0
7.5
3.0
4.5
6.0
7.5
3.0
4.5
6.0
7.5
Bitrate /s)
Bitrate (lbbit/s)
Bitrate (kbit//s)
Bitrate (kbitt/s
Bitrate (kbin//s
Figure 4: Mean objective metrics attained by compared methods on the test set at varying bitrates. Colored bands indicate 95% confidence intervals. SIGMOS is speech-only and is calculated only on the speech test files. FAD multiplied by 100 for readability. Numbers can be found in Table 8
Perception-distortiin---
Table 4: FAD 100,
SI-SDR, and mean fwSSNR of FlowDec-75s versus the related Score-
10.0
Dec Wu et al. 2024) and constant 0e (Jung et
6.
7.5
FlowDee-75mn
2024
Best in bold.
SI-SDR
7.5
4.0
45
FlowDeo-75%
5.0
FlowDeo-25i
4.5
Method
FAD x10
SI-SDR
fwSSNR
DAC-75
2.5
4.0
3.0
DAC-25
NFF 6
FlowDec
1.62
7.55
15.46
0.0
7.5
10.0
ScoreDec
145.30
-27.23
3.15
10Dx100
o, = 0.05
28.88
9.95
5,50
0₁ 0.66
29.83
10.10
6.55
Figure 5:
Perception (FAD) distortion
(SI-SDR) - rate tradeoff (Blau & Michaeli] (019) of compared methods. Numbers next points indicate the bitrate kbit/s.
20199)
NFE 50
FlowDec Score
1.34
7.41
15.65
5.73
7.50
14.45
the 25 Hz feature rate models FlowDec-25s and DAC-25 for comparison. Our main model FlowDec- 75m produces the best FAD values by a large margin and also performs best on the SIGMOS OVRL metric. For the intrusive spectral metrics SI-SDR, fwSSNR, and logSpecMSE, retrained DAC generally outperforms FlowDec, though the gap in the perceptually weighted fwSSNR is small. This be expected under the perception--distoriion tradeoff discussed in (Blau & Michaeli 2018 2019): FlowDec favors better perception (FAD) along this tradeoff at the cost of increased distortion (SI- SDR), see also Fig. 55 similar to observations made about ccsse-based models for speech enhancement
(Richter et al. 2023) and JPEG artifact removal (Welker et al. 2024). Furthermore, we see that the single-bitrate FlowDec-75s slightly outperforms Flow 75m at 7.5 kbit/s as expected, and that 2xDAC is slightly better than DAC but does not fundamentally change the qualitative behavior of DAC. For the 25 Hz models, we can see that the general behavior of FlowDec and DAC unchanged, with FlowDec again exhibiting better FAD and SIGMOS.
In Table we compare FAD, SI-SDR and fwSSNR of FlowDec-75s at NFE E {6, 50} against ScoreDec (Wu et al. 2024) and the alternative flow-based formulation with constant a, (Jung et al., 2024). We can see that for NFE 6, FlowDec is clear improvement over ScoreDec which produces unusable results this NFE and also performs significantly better than Jung et al.) (2024) here. At NFE = 50, ScoreDec and FlowDee achieve similar SI-SDR, but Flow performs significantly better in FAD. A full metric comparison table can be found in Appendix [A7.1] Finally, in Fig. 7 we show a qualitative spectrogram comparison of Flow compared DAC for guitar recording, which illustrates better reconstruction of harmonic structures by FlowDec. We show more example spectrogram comparisons, including the worst reconstructions from FlowDec, in Appendix A.8
5.2 SUBIECTIVE LISTENING TESTS
In Fig. show the results from both subjective listening Tests, A and B, as boxplots of MUSHRA scores per method and bitrate. For Test A, we can see that the 4.5 kbit/s variants are rated somewhat lower than the 7.5 kbit/s variants but still achieve good scores compared to EnCodec at 6.0 kbit/s, and

---

Published as a conference paper at ICLR 2025
Test A
Test B
Model
100
100
FlomDee-70m
MUSHIRA Score
1000 d
MUSIRRA Score
c00 ax
Flow:De-776
FlowDee-25e
000
0000
0000
40
C00
8
00
40
DAC-25
20-
EnCoder
Refereace
Opns
FlonDoe-755 (7.5)
Reterence
Flon Deo-Tan- (7.5)
FloeeDee-Tine (4.5)
DAC-75 (7.5)
DAC-25 (4.6)
EaCodee (0.0)
Opea (7.
FborDor-Tee Tomm (4.5)
Relereace
FloeDre-25e (4.0)
DAC-75 (4.5)
DAC-25 (4.0)
Open (4.0)
Figure 6: Subjective listening results from Test A (left) and Test B (right). Numbers in (parentheses) denote the used bitrate in kbit/s. FlowDec is rated par with DAC (Kumar et al. (024). with no significant differences between their score distributions at any given bitrate and feature rate.
the low anchor Opus. We can further see that, at any given bitrate, the score distributions of DAC-75 and
show no significant differences. For Test B with the 25 Hz models, we can again see that DAC and FlowDec generally perform on par, and also that the 25 Hz. models are rated very similarly as their higher feature rate (75 Hz) equivalents at a similar bitrate. In Appendix A.66 we also show results split by audio type, which seem to suggest that FlowDec performs better than DAC for speech samples, slightly worse for sound samples, and on par for music.
5.3 REAL-TIME FACTOR
Reference
FlowDec
DAC
An important property of codec is its runtime. We determine the real-time factor (RTF) of the two NDAC variants and the FlowDec postfilter at NFE E {4, 6, 8} with the midpoint solver 100-$XM4-80GB GPU. We find an RTF of 0.0134 for NDAC-75 and 0.0084 for NDAC-25. For the postfilter, we find that RTF 0.0358 X NFE. At our default setting NFE = 6, this results in a total RTF of 0.2285 for (50D)ec-5((m/s) and 0.2235 for FlowDec- 25s, a significant improvement over the RTF of 1.707 for ScoreDec (Wu et al. 2024).
15
Frequency (aHa)
-25
Amp inplttdde (dB)
10
-50
on an NVIDIA
-100
2
Time (6)
Time (6)
Time (s)
Figure 7: Spectrogram comparison (pre-emphasis of 0.95) of DAC and FlowDec at 7.5 kbit/s for a guitar test audio. FlowDec better preserves har- monics where DAC creates noise-like structures.
CONCLUSION
We presented FlowDec, novel postfilter-bseed neural codec for general audio with high perceptual quality. FlowDec uses novel modification of the flow matching formalism for signal enhancement, which is inspired by previous score- and flow-based generative works for signal enhancement (Richter jet al. 2023 Wu et al., 2024 Jung et al.) 2024) but improves upon them both in terms of theoretical properties and output quality. We showed that FlowDec achieves state-of-the-arr FAD scores for the coding task and, in a listening test, performs on par with the current state-of-the-art GAN-based codec DAC (Kumar al. 2024) bitrates between 4.5 and 7.5 kbit/s. Furthermore, FlowDec also shows promising quality at the very low feature rate of 25 Hz and bitrate of 4.0 kbit/s, which we hope can contribute to more efficient long-range generative audio modeling.
While FlowDec. like DAC, is currently not seeaming-capable due to the noncausal architecture of the used DNNs, our postfilter approach can be modified for causal DNN as (Richter et al., 2024a), which would pave the way for real-time communication and audio streaming applications. We leave this for future work, particularly since there are currently no streaming codecs available that achieve the quality of DAC to our knowledge. Another interesting future direction is the joint training of the initial decoder and the postfilter similar to Lemercier et al, (003b)), which could improve quality but may lead to unstable training. Finally, as the NCSN+ architecture for images, we expect that future work using DNN architectures better adapted to audio signals can further improve the quality of FlowDec.
originally built
10

---

demo page is Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide all necessary details on the mathematical formulations and loss functions in Section B.I and Appendix (A.2, of the network architecture in Appendix [A.3] and the feature representation in Appendix We explicitly describe the proposed modifications to non-adversarial DAC in Section 3.4 and provide all hyperparameters for this modification along with the training details of both this underlying codec and all postfilters in Section 4.2 We provide the full list and details for all datasets, besides internal datasets which at present cannot be open-soureed, in Section 4.1 We note that we used only a small fraction of this total training data for training our FlowDec postfilter, with most being used for training the underlying codecs (see Section 4.1). Our underlying codecs are based on DAC (Kumar et al. 2024) and can straightforwardly be retrained with the public datasets listed in their work, using their available codebase, and the additional implementation details for our proposed CQT loss listed in Section 3.4] To further ensure reproducibility, we have open-sourced our code for FlowDec training and inference, along with pretrained model checkpoints of the FlowDec models listed in this paper, made available at at available at https: https::/sppuhh..gee github. io/F lowDec/
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicI.: Generating music from text. arXiv preprint 1rXi:22301.11355 2023.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerek. Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218 4222,, Marseille, France, May 2020. European Language Resources Association. ISBN 979- 10-1110--55----1 34-4. URL htttts 1/aclanthology
1rec--..rer..... 1250
B. S. Atal and M. R Schroeder. Adaptive predictive coding of speech signals. Bell System Technical Journal, 19(8):1973-1998 1970.
B. Bessette et al. The adaptive multirate wideband speech codec (AMR-WB). IEEE Transactions on Antennas and Propagation (TSAP), 10(8):620-636 2002.
Arijit Biswas and Dai Jia. Audio codec enhancement with generative adversarial networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 356-360. IEEE, 2020.
Yochai Blau and Tomer Michaeli. The perception tradcoff. JEEE/CV Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The lel-diistottionperreppp tradeoff. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 675-685. PMLR, 09-15 Jun 2019.
J. Coalson. Free Lossless Audio Codec, 2000. URL htRLhttss//xpp..or///leeee
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Jun Deng, Björn Schuller, Florian Eyben, Dagmar Schuller, Zixing Zhang, Holly Francois, and Eunmi Oh. Exploiting fime-frequency patterns with LSTM-RNNS for low-bitrate audio restoration. Neural Computing and Applications, 1295)) 107, 2020.
L. Deng, M. L. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. Hinton. Binary coding of speech spectrograms using deep auto-encoder. In ISCA Interspeech. Citeseer, 2010.
11

---

Published as a conference paper at ICLR 2025
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3780-8794, Curran Associates, Inc., 2021.
M. Dietz al. Overview of the EVS codec architecture. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5698-5702. IEEE, 2015.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. AudioSet: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, 2017.
Sang gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffusion models with deta-dependent adaptive prior. In International Conference on Learning Representations, 2022.
Ian Goodfellow, Jean Pougct-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-14) 144, 2020.
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting Frechet Audio Distance for generative music evaluation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3331-333, 2024. doi: 10. D0.109//CASSP44448..222. 10446663.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
ITU. International Telecommunication. Union Recommendation ITU-R BS.1534-3: Method for the subjective assessment of intermediate quality level of audio systems, 2015. URL https
7/ 7/www. i ttu. www.:intt/re.... REC BS. 1534/en
Chaeyoung Jung, Suyeon Lee, Ji-Hoon Kim, and Joon Son Chung. FloWAVSE: Efficient audio-visual speech enhancement with conditional flow matching. ISCA Interspeech, pp. 2210-2214, 2024. doi: 10.21437/mterspeecchh http://terrspecc222227000 2014-701.
N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu. Efficient neural audio synthesis. In International Conference Machine Learning (ICML), pp. 2415-2424, July 2018.
S. Kankanahalli. End-to-end optimized speech coding with deep neural networks. In IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2521-2525. IEEE, 2018.
Jodi Kearns, Librivox: Free public domain audiobooks. Reference Reviews, 28(1):7-8, 2014.
Minje Kim and Jan Skoglund. Neural speech and audio coding, arXiv preprint (eXiv:2408.066955 2024.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Thirth- Conference on Neural Information Processing Systems, 2023.
W. B. Kleijn, F. SC Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters. Wavenet based low rate speech coding. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 676-680, IEEE, 2018.
J. Klejsa, P. Hedelin, C. Zhou, R. Fejgin, and L. Villemoes. High-quality speech coding with sample mn. In IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 7155-7159. IEFF, 2019.
12

---

Published as a conference paper at ICLR 2025
J. Kong, J. Kim, and J. Bae. HiFi-GAN: fidelity speech synthesis. Advances in Neural Information Processing Systems (NeurIPS), 33: 17022-17033, 2020.
Generative adversarial networks for efficient and high
Srikanth Korse, Nicola Pia, Kishan Gupta, and Guillaume Fuchs. PostGAN: A gan-based post- processor to enhance the quality of coded speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 831-835. IEEE, 2022.
A. K. Krishnamurthy, C. Ahalt, D. E. Melton, and P. Chen. Neural networks for vector quantization of speech and images. IEEE Journal on Selected Areas in Communications, 8(8):1449-1457, 1990.
P. Kroon, E. Deprettere, and R. Sluyter. Regular-pulse excitation-a novel approach to effective and efficient multipulse coding speech. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 34(5):1054-1066 1986.
K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. 7. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and Courville MelGAN: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems (NeurIPS), pp. 14910-14921, Dec. 2019.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. In Advances in Neural Information Processing Systems (NeurIPS), NeurIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Bunlong Lay, Simon Welker, Julius Richter, and Timo Gerkmann. Reducing the prior mismatch of stochastic differential equations for diffusion-based speech enhancement. In ISCA Interspeech, pp. 3809-3813, 2023. doi: 00.21437/Ineesspeecch.o 2023-1442
Sang-Hoon Lee, He-Yeong Choi, and Seong-Whon Lee. Periodwave: Multi-period flow matching for high-fidelity waveform generation. arXiv preprint S.Xiv:2408.075447 2024.
Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Analysing diffusion- based generative approaches versus discriminative approaches for speech restoration. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023a. doi: 10.1 1A..1999ASSSP9995.222...... 200.5557..222...0052220
Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. StoRM: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation. IEEE Transactions Audio, Speech, and Language Processing (TASLP), 31:2724-2737, 2023b, doi: 10.1109/TASLP. 2023.3294692.
Tilman Liebchen and Yuriy A Reznik. MPEG-4 ALS: An emerging standard for lossless audio coding. In Data Compression Conference, pp. 439-448. IEEE, 2004.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthe Le. Flow matching for generative modeling. In International Conference Learning Representations (ICLR), 2023.
Philipos C Loizou. Speech enhancement: Theory and practice, 2013.
R. Lotfian and C. Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4):471-45 ectober-Decemaee 2019. doi: 10. 10.1099AAFFC22217.275665
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong. Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A CharGiPT-assistedd weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), pp. 1-15, 2024.
Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with a diffusion model. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1 109/CCSSP4995572000000 wttp:/9935.2222.10000 2029999967..
S. Morishima, H. Harashima, and Y. Katayama. Speech coding based on a multi-layer neural network.
In IEEE International Conference Communications, pp. 429 433. IEFE, 1990.
13

---

Published as a conference paper at ICLR 2025
Thomas Muller, Stephane Ragot, Lactitia Gros, Pierrick Philippe, and Pascal Scalart. Speech quality evaluation of neural audio codecs. In ISCA Interspeech, pp. 1760- 1764, 2024. doi: 10.1445/mmexxpech2024.000 1072.
A. Mustafa, J. Büthe, S. Korse, K. Gupta, G. Fuchs, and N. Pia. A streamwise gan vocoder for wideband speech coding at very low bit rate. In JEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 66-70. IEEE, 2021.
Tu Anh Nguyen, Wei-Ning Hsu, Antony D' Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emmanuel Dupoux. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. In ISCA Interspeech, pp. 4823-4827, 2023, doi: 10. 00/21447/nterspecchh222223 201.3pe6ecch.22119005.
D. O' Shaugghesss. Linear predictive coding. IEEE Potentials, 7(1):29-32, 1988.
Aram-Alexandie Pooladian, Heli Ben- Ben-Hamu, Carles Domingo-Enrin Ennich, Brandon Amos, Yaron Lip- man, and Ricky T. Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28100-28127, PMLR, 23-29 Jul 2023.
Zafar Rafii, Antoine Liutkus, Fabien-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. MUSDB18-HI- an uncompressed version of MUSDB18 18, August 2019. URL BtLthttp::/// //doi
02g/10.5281/ zenooo. 3338373
K. R. Rao and J. Hwang. Techniques and standards for image, video, and audio coding. Prentice- Hall, Inc., 1996.
Julius Richter, Simon Welker, Jean-Maric Lemercier, Bunlong Lay. and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 31:2351-2366 2023. 2023.3285241.
doi: 10.1109/TASLP.
Julius Richter, Simon Welker, Jean-Maric Lemercier, Bunlong Lay, Tal Peer, and Timo Gerkmann. Causal diffusion models for generalized speech enhancement. IEEE Open Journal of Signal Processing, 2024a.
Julius Richter, Yi-Chiao Wu, Steven Krenn, Simon Welker, Bunlong Lay, Shinji Watanabe, Alexander Richard, and Timo Gerkmann. EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation. In ISCA Interspeech, pp. 4877, 2024b. doi: 10.21447/hterrypechh22244 153
Nicolae Catalin Ristea, Ando Saabas, Ross Cutler, Babak Naderi, Sebastian Braun, and Solomiya Branets. Icassp 2024 speech signal improvement challenge. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. SDR - half-baked or well done? In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626 630, 2019. doi: 10. 1091CCSSSP2001 8.888.8.....11...8885.... 8683855.
R. Salami et al. A toll quality 8 kb/s speech codec for the personal communications system (PCS). IEEE Transactions on Vehicular Technology, 43(3):808-816, 1994.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Defossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 1526-1538. Curran Associates, Inc., 2023.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
M. Schroeder and B. Atal. Code-excited linear prediction (CELP): High-quality speech very low bit rates. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). volume 10, pp. 937-940. IEEF, 1985.
14

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Poole. store-based generative modeling through stochastic differential equations. International Conference Learning Representations (ICLR), 2021.
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang. Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Mattco Torcoli, Thorsten Kastner, and Jürgen Herre. Objective measures perceptual audio quality reviewed: An evaluation of their application domain dependence. IEEE Transactions Audio, Speech, and Language Processing ('TALPP), 29:1530-1541 3069302.
2021.
doi: 10.1 10.1109/TASL.P2201
Cassia Milentini-Botinaan Noisy speech database for training speech enhancement algorithms and tts models. In University of Edinburgh. School Informatics. Centre for Speech Technology Research (CSTR), 2017. doi: 10. 10.7488/ds/117.
J.-M. Valin and J. Skoglund. LPCNet: Improving neural speech synthesis through linear prediction. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5895, May 2019a.
J.-M. Valin and J. Skoglund. A real-time wideband neural vocoder at 1.6 kb/s using lpenet. ISCA Interspeech, pp. 3406-3410, 2019b.
J.-M. Valin, G. Maxwell, T. B. Terriberry, and K. Vos. High-quality, low-delay music coding in the opus codec. In 135th AES Convention. Audio Engineering Society, 2013.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukeuoglu. WaveNet: A generative model for raw audio. In Speech Synthesis Workshop, pp. 125. ISCA, Sept. 2016.
A. Van Den Oord et al. Neural discrete representation learning. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based generative models in the complex stft domain. ISCA Interspeech, pp. 2928-29322 2022. doi: 10.21437/ aiterspeech.2022. 10653.
Simon Welker, Henry N Chapman, and Timo Gerkmann. DriftRec: Adapting diffusion models to blind JPEG restoration. IEEE Transactions Image Processing (TIP), 2024.
L. Wu, M. Niranjan, and F. Fallside. Fully vector-quantized neural network-based code-excited nonlinear predictive speech coding. IEEE Transactions Antennas and Propagation (TSAP), (4):482-485 489, 1994.
Y.-C. Wu, I. D. Gebru, D. Marković, and A. Richard. AudioDec: An open-source streaming high- fidelity neural audio codec. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
Yi-Chiao Wu, Dejan Marković, Steven Krenn, Israel D. Gebru, and Alexander Richard. ScoreDec: A phase- preserving high-fidelity audio codec with a generalized ccore-based diffusion post-filter. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 361-365, 2024. doi: 10. 1..109//CASSS48455.... 202.
R. Yamamoto, E. Song, and J.-M. Kim. Parallel WaveGAN: fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203, May 2020.
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. SoundStream: An end-to-end neural audio codec. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 30: 495-507, 2021.
15

---

Published as a conference paper at ICLR 2025
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. In ISCA Interspeech, pp. 1526-1530, 2019. doi: 10.21445//mexxpeechh01199/
Ziyue Zhao, Huijun Liu, and Tim Fingscheidt. Convolutional neural networks to enhance coded speech. IEEE Transactions Audio, Speech, and Language Processing (TASLP), 2764))663-678 2018.
A APPENDIX
A.1 A HEURISTIC FOR CHOOSING By
An important question arising Section 3.1 is how to set By. In Fig. 8 we show the effects of three By settings on simple toy problem. A too-large αy leads to regression to the mean effect, pointing the flow field towards the mean of viable clean audios x for most times i, which results in oversmoothing and bad perceptual quality. On the other hand, a very small By close to 0 does not allow learning the flow field well, as most regions the space have very low probability during training. Similar to the visually guided setting By = 0.4 in Fig. g works well for all of our cases:
we find that the following heuristic
og oF= 3 V - Y 0.997)
(12)
where Q is the quantile operation. Similarly to a root mean squared error (RMSE), this is the root of the 0.997th quantile of squared errors induced by the initial decoder Dy in the feature domain. The constants 3 and 0.997 are inspired by the 3-sigma rule of a Gaussian distribution. The chosen ay. 0.66 for our FlowDec models, then covers all viable estimates X*, except outliers beyond the 0.997th quantile, within the 3-sigma region of the added Gaussian noise around Y.
A.2 DERIVATION OF CONDITIONAL FLOW FIELD
Referring to Section 3.2 we perform the derivation of the target flow field in more detail here. We can find the target flow field ue from our chosen probability path, Pt Eq. ( using (Lipman et al.
2023 Eq. 15):
of (21, v)
(x(1(11,8) 9)
(x (1(11, v)) + (21.111) v)
(13)
(F 8)
-g
(x+ (y + fx: - -y)) + (a11 - y)
(14)
(1 t)oy
F4 y tx + ty (1 f)(11 y)
(15)
1 1-t
-t + y 1x ly
11 1. y + by
(16)
1
T1 Ft.
(17)
1
(18)
which matches the expression zero-mean when their Omin
Lipman et al. 2023 Eq. 21) of the flow field for an unconditional
0. We can further see that
un (x(211))) v)
T1 Zt
(19)
1
F1
(txr (1 t)
(20)
1 1-1
(1 t)z, - (1 - t),00
(21)
1-th
21 To
(22)
16

---

Published as a conference paper at ICLR 2025
t= 0.0
t= 0.3
== 0.6
= 0.95
FV 100
Figure 8: Flow fields for our FlowDec formulation at different times f and settings of the hyperparam- eter oy, illustrated on toy problem. The white dot represents the initial estimate y, the yellow stars represent possible target signals x², and the red cross is the mean of all x". The background shows the probability density Pt and the circle indicates 30, around y.. The flow field for large o, = 1.6 points towards the mean (red eross) for most t, while for o, 0.4, it points towards each viable point much earlier. While a low dy 0.1 leads to the straightest paths, also results in most regions the space having very low probability pe for all / of being sampled during training, which is problematic under model and truncation errors since is much more likely that trajectories fall off the small high-probability regions.
17

---

Published as a conference paper at ICLR 2025
where 21 x² is exactly the target clean signal x* since di = 0, and Zy Ma y. Ey) is sample from Gaussian with mean y (the initial decoder output) and diagonal covariance Ey = diag( ( Hence, we can reparameterize 2o as
Zo = y + oye,
E~~(
(23)
which, together with Eq.
22 and x², leads exactly to the expression used in our loss, Eq. (11
A.3 NCSN++ NEURAL NETWORK CONFIGURATION DETAILS
For our postfilter flow model, we reconfigure the NCSN++ 2-D U-Net architecture (Song et al. 202
used in prior audio works (Richter et al. 2023 Wu et al. 2024). In preliminary investigations, we found that the original architecture can produce Appendix (A.7.55 We found that doubling the channels (128 256) at the first two U-Net depths effectively suppresses these artifacts. An explanation may be that the capacity of only 128 filters in the early layers may not be enough for the increased sampling rate and data complexity (speech + music, sound, speech) compared to Richter et al. (2023). To counteract the increased memory usage, reduce the depth from 4 and reduce the channels depths and 4 from 256 128. We use instead of ResNet blocks per depth as in (Lemercier et al., (202bb). Finally, we remove all attention- layers ensure that the inference runtime is linear in the audio duration. Our architecture has 26 M parameters instead of the original 65 M.
harmonic artifacts in music, see
A.4 FEATURE REPRESENTATION DETAILS
As in related literature RRichter et al. 2023 Wu et al., 2024), we use comppesse and scaled complex spectrograms Xij CFXT as the input and output feature representations of the postfilter network with an invertible feature extractor d:
(<x) X,, == / ,|X,, exp(i <(X;)), X,y : STFF(T()))
(24)
where / denotes the phase of complex number, and STFT is complex- valued short-time Fourier transform (STFT). For this STFT, we use 1534-sample (31 (31.96 ms) Hann window resulting in F 768 frequency bins and a hop length of 384 samples (74.97% overlap). We choose a = 0.3 since we found it to produce better outputs for general audio than the original
0.5 used in Wu ct al. 22024), see Appendix A.7.5 Note that the choice of window length and hop length different from ScoreDec (Wu et al. 2024) (510 15mple Hann window, 320-sample hop length, 37.5% overlap) since we found the increased overlap and frequency resolution to help with output quality. Our choice of window length and hop length the same as in the related 48 kHz speech work by Richter et al.) (2024b). To keep the values of the real and imaginary parts of X constrained to roughly [-1, [-1, we set 0.66, which we determine as the 99.7th percentile of compressed but unscaled STFT amplitudes (i.c., Eq. 24 with 3 = 1) on 2,500 random clean training audio files.
A.S QUALITATIVE OUTPUTS FROM INITIAL DECODER
To show how the enhanced outputs by our FlowDee postfilter, ((Do(c)) compare to the outputs of the initial decoder (c) of the underlying non-adversarially trained codec NDAC-75, show three example spectrograms in Fig. 9 The initial decoder produces overly smooth spectral structures and buzzy noise artifacts. Flow Dec successfully removes these artifacts and replaces them with plausible natural spectral structures, thereby significantly enhancing the audio.
A.6 DETAILED RESULTS FROM SUBJECTIVE LISTENING TESTS
In Fig. 10 we show the score distribution from both MUSHRA-like listening tests (Section 4.4 split by audio type. These results suggest that FlowDec may perform better speech than DAC. particularly for FlowDec- versus DAC-75 at 4.5 kbit/s and that DAC may perform slightly better than FlowDec sound files; score distributions for music very similar.
18

---

Published as a conference paper at ICLR 2025
Reference
Initial decoder Do
Enhanced with Flow
20
Frequeney (kHH))
15
10-
5-
20
Elqueeey (kiie)
15-
Amplitude (d18)
5-
Eequeacy (kif))
15
10-
0-
100
5.0
9.0
9.5
10.0
8.5
9.5
10.0
9.0
10.0
Time (s)
Time (s)
Time (8)
Figure 9: Spectrograms from parts of three audio examples (from top bottom: speech, music, sound) as output by the initial decoder Do of NDAC-75, compared to their enhanced version from 755 The estimates from Du show severe buzzy and unnatural artifacts, which FlowDec successfully replaces with plausible spectral structures.
19

---

Published as a conference paper at ICLR 2025
Test A (Speech)
Test (Speech)
Model
100
100
Now-Doo-5mm
80
MUSHIRA Soune
80
O00 O
FlowDoo-758
C0D d
FlowDee-25e
MUSHRA
DAC-75
40
40
DAC-25
20
20
EnCodee
Referemee
Opus
Referaace Tie (7.3)
Flow Dem (7.5)
FloeDee-75m 75m (4.5)
DAG-75 (7.5)
DAC-75 (4.5)
EnCoder (6.0)
Opas (7.5)
Flo Fem-De-75m (4.5)
Refermnee
FlorDee-2fs (4.0)
DAC-75 (4.5)
DAC:25 (4.0)
Opas (4.0)
(a) Speech samples: Listening test results (MUSHRA score distributions)
Test A (Music)
Test B (Music)
Model
100
100
FlowDee-T3ur
80
MUSHIIA Score
80
Deo-7ee-7
1200
Dee-2ss
MUSHIA
0000
8
00
DAC-75
00
8
DAC-25
20
20
8
EnCodee
Reference
Opus
Reference Dee--ec- (7.5)
Flow Dee--ii-mmm (7.5)
FlowDee 7ie (4.5)
DAC-75 (7.5)
DAC-75 (4.8)
EnCodee (6.0)
Opns (7.5)
FlowDee-Tte (4.5)
Reference
2eeDDee-2is (4.0)
DAC-75 (4.5)
DAC-25 (4.0)
Opes (4.0)
(b) Music samples: Listening test results (MUSHRA score distributions)
Test A (Sound)
Test B (Sound)
Model
100
100
FlowDee--33m
MUSHRA Seore
80
1000
MUSHRA Score
80
8
Flow-ee-7ass
FlowDee:25s
60
000
000
00
60
0
DAC-75
DAC-25
20
20
EnCodre
Referemec
0
Opus
FlowDes-Tte (7.5)
Reference
Flae FeeeDee-Ting Dee-75mi (7.5)
FienDee-Ting 75mi (4.5)
DAC-75 (7.5)
DAC-75 (4.8)
EnCodee (0.0)
Opus (7.5)
Floo Dec-7ne (4.5)
Referance
FlowDec-25e (4.0)
DAC-75 (4.5)
DAC-25 (4.0)
Opus (4.0)
(e) Sound samples: Listening test results (MUSHRA score distributions)
Figure 10: Detailed results from the listening tests (Section 4.4) split by audio type.
20

---

Published as a conference paper at ICLR 2025
Table Mean + 95% confidence interval of objective metrics for FlowDec(-755) compared against baselines using the alternative ScoreDec formulation (Wu et al. 2024 or FlowAVSE (constant-o formulation, each trained on the same data with the same backbone DNN and feature representation. We show results at two different NFE (6 and 50). FAD is multiplied by 100 for readability. For "ScoreDec NC", in contrast
original ScoreDec, we do not use the annealed Langevin corrector (Song et al., 202T) during inference, and instead double the number of predictor steps to achieve the same NFE. We can see that ScoreDec returns unusable estimates at NFE=6. At NFE=50, the metric values are acceptable but still clearly worse than those of FlowDec in FAD, fwSSNR, SIGMOS, and ISSSpecMSE. In SI-SDR and SIGMOS, ScoreDec, and FlowDec achieve similar values NFE=50.
Method
FAD 100
SI-SDR
fwSSNR
logSpecMSE
SIGMOS
NFE 6 FlowDec
1.62
7.55 + 0.25
15.46 + 0.07
80.57 + 1.72
3,48 + 0.03
ScoreDec
145.30
22..23
3.15
4873.42 # 51.92
1.18 0.01
ScoreDec NC
78.71
-5.89 0.19
2484.17 # 28.89 0.10
4.58
1,45 0.01
o, 0.05
28.88
9.95 - 0.21
1613.40 # 33.08
3.00 + 0.02
0.66
29.83
10.10 # 0.22
0.18
1442.94 # 25.52
2.94 # 0.02
NFE = 50
Flow Dec
1.34
7.41 # 0.25
15.65 0.06
81.83 + 2.17
3.44 # 0.03
ScoreDec
5.73
7.50 0.24
14.45 0.09
176.25 4.12
3.51 = 0.03
ScoreDec NC
3.84
7,56 # 0.25
15.00 + 0.08
130.32 + 2.95
3.43 + 0.03
A.7 ABLATION STUDIES
In this appendix section, we conduct several ablation studies further justify the choices have made. We show comparative tables with objective metrics, and spectrograms to illustrate model behaviors qualitatively.
A.7.1 FULL METRIC COMPARISON AGAINST SCOREDEC AND FLOWAVSE
In Table we show objective metric values for FlowDec compared to the prior work ScoreDec (Wu et al. (024)) and FlowAVSE (Jung et al. 2024) at NFE=6 and NFE=50. For the baseline models here, we retrained the model with each alternative formulation while keeping all other settings (data, backbone, feature representation) the same. For FlowAVSE we train one variant with small o = 0.05, and one with the same a, = 0.66 as the ag = 0.66 setting used for FlowDec. As the metrics show, FlowDec works significantly better at NFE=6 where ScoreDec and FlowAVSE fail to produce acceptable results, and also generally outperforms ScoreDec at NFE=50.
A.7.2 NON-ADVERSARIAL DAC WITHOUT ADDED CQT AND WAVEFORM LOSSES
As proposed in Section 3.4 train our underlying advesarraalial codec ("NDAC") based DAC (Kumar et al. 2024) but newly add multiscale constant-Q Q transform (CQT) loss and an L waveform-domas loss, in particular to combat the bad low-frequency preservation of the initial non-adversarial NDAC variants we trained in preliminary experiments. Fig. TT we show this effect qualitatively, comparing the original non-adversarial DAC without added losses (rightmost column) to our proposed underlying codec NDAC-75 (center column), which includes these losses, in the frequency range between 0 and 1500 Hz. The original non-adversarial DAC introduces severe errors and generates very noisy low-frequency spectrum. In comparison, our variant NDAC-75 (center column) does not suffer from these problems in the low-frequency region and produces relatively good estimates.
A.7.3 ADVERSARIAL DAC WITHOUT AND WITH ADDED CQT AND WAVEFORM LOSSES
To further show that the advantages of our method are not caused just by the added CQT and waveform L' loss, as proposed in Section 3.4, we also train a variant of the adversarially trained DAC-75 that includes both those original adversarial losses, the original aon-avveraarial losses (Kumar al.
21

---

Published as a conference paper at ICLR 2025
Reference
NDAC-75
NDAC-75 w/o CQT+wa way
1.5
Frequency (kHt)
1.0
10
0.5
Amplitude (dB)
0.0
1.5
Fequency (kHa)
40
0.5
500
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Time (8)
Time
Time
Figure 11: Low-frequency (0-1500 Hz) spectrum of two music samples (top: vocals, bottom: mixture) from our test set, comparing our underlying codec NDAC-75 against the version we initially trained without added CQT and waveform losses ("w/o CQT+wav").
Table 6: Mcan = 95% confidence interval of objective metrics for DAC-75, compared to a variant ("+Cw") trained with the original adversarial and non-adversarial losses as well as our proposed CQT and waveform losses used for DAC-75 (Section 3.4). Bitrates are kbit/s, and FAD multiplied by 100 for readability.
Method
Bitrate
FAD
S1-SDR
fwSSNR
logSpecMSE
SIGMOS
DAC-75
3,00
9.68
4.66 + 0.18
12.11 # 0.07
80.79 1.41
3.14 # 0.03 3.09 = 0.02
DAC-75 +Cw
3.00
9.39
4.86 :: 0.19
12.21 # 0.07
81.93 + 1.58
DAC-75
4.50
6.80
6,95 + 0.18
13.62 = 0.08
76.95 + 1.32
3.19 # 0.02
DAC-75 +Cw
4.50
6.63
7.17 7.17
13.74 = 0.07
77.81 1.49
3.16 = 0.02
DAC-75
6.00
5.23
8.54 0.18
15.01 + 0.08
74.94 + 1.29
3.19 + 0.02
DAC-75 +Cw
6.00
5.12
8.76 0.19
1.014
75.65 1.42
3.17 # 0.02
DAC-75
7.50
4.15
10.03
16.57 0.09
73.05
1.24
3.19 = 0.02
DAC-75 +Cw
7.50
3.95
10.16
16.65 + 0.07
73.98 1.38
3.19 + 0.02
2024), and our proposed CQT and waveform _ loss. We show the metric results in Table 6 It can be seen that our proposed loss terms seem to improve FAD, SI-SDR, and fwSSNR slightly, and on the other hand, worsen logSpecMSE and SIGMOS slightly. No large differences in any metric can be seen at any particular bitrate, confirming that the strong improvements in FAD and SIGMOS of FlowDec show in Section 5.1 not caused purely by these loss terms being added to underlying codec.
A.7.4 FREQUENCY- O2 VS. GLOBAL oy
In Fig. 12, we compare objective metrics of our main FlowDee-755. 75m and FlowDec-25s variants, both with frequency-dependent yy, against each corresponding variant with a global o, ("ga,")) "), We use each method at bitrate of 7.5 kbit/s for FlowDec-75m and kbit/s for FlowDec-25s. and run inference at different NFE. At a low NFE, we see that the deeeundddeddeedenett og achieves on par or better logSpecMSE and FAD scores, particularly for the 25 Hz models. For the metrics SI-SDR, fwSSNR, and SIGMOS, which choice of a, is optimal seems not as clear. At NFE=4, the global o, variants deteriorate significantly in logSpecMSR, but gain in SI-SDR, indicating that oversmoothing
22

---

Published as a conference paper at ICLR 2025
F 100
SI-SDR f
fwSSNR 7
logSpecMMSB
SIGMOS
15.8
6.4
110
3.50
3
16.6
6.2
105
3.45
2
6.0
15.4
100
5.8
15.2
95
0
2.75
13.4
130
3-
2.50
13.2
3.50
120
2-
2.25
13.0
3.45
110
12.8
1
2,00
12.6
100
3.40
0
1.75
6
50
4
6
50
4
8
50
6
50
4
8
50
NFE
NFE
NFE
NFE
NFE
FlomDee-75m
FlowDee-25e
FeomDe goy
FlowDDe goy
Figure 12: Objective metrics at different NFE, from our main FlowDec-75m (top row) and FlowDec- 25s (bottom row) models, compared against each corresponding global By variant ("goy").
Table 7: Mean + 95% confidence interval of objective metrics for our method compared to the original NCSN+ architecture, and compared to the original choice a = 0.5 contrast to our a = 0.3. We use NFE=6 with the midpoint solver. FAD is multiplied by 100 for readability. Best in bold, second best underlined.
Method
FAD
SI-SDR
fwSSNR
logSpecMSE
SIGMOS
FlowDec-75s with original NCSN++
1.62
7.55 + 0.25
15.46 0.07
80.57 + 1.72
3.48 + 0.03
1.75
7.51 0.25
15.30 + 0.06
79.84 1.76
3.45 + 0.03
with a = 0.5
2.16
7.54 + 0.25
14.49 # 0.09
130.10 + 1.98
3.57 # 0.03
of high frequencies occurring: the frequency-dependent oy variants exhibit this effect much less strongly.
A.7.5 NETWORK ARCHITECTURE AND FEATURE REPRESENTATION
In Table 7 we show metric results of FlowDec-755 compared two ablation model variants: one trained with the original NCSN+ architecture (Song et al. 2021 Richter et al.) 2023), and one trained with the original choice the feature representation parameter
0.5 (Welker al. 2022 Richter et al. [2023] Wu et al.) 2024). We can see that FlowDec- 75s performs best in FAD, SDRSDR and fwSR, and significantly improves upon a 0.5 in logSpec speech-only metric, the 0.5 model achieves the best score, which may hint at 0.5 being more optimal for speech signals; however, all other metries a 0.3 seems to better choice, and it seems to work better overall for general audio.
SIGMOS, which is
A.7.6 COMPARISON OF ODE SOLVERS
In Fig. we show that the numerical Midpoint ODE solver is much more effective than the simpler Euler ODE solver producing high-quality audio at low numbers of DNN evaluations (low number of function evaluations (NFE)). Both solvers perform similarly at high NFE of 50, but Euler generally degrades significantly low NFE (4, 6, 8). While Euler achieves better SI-SDR, it at the same time shows significantly worse fwSSNN and logSpecMME, which indicates spectral oversmoothing (removal of high frequencies). Midpoint performs similarly for NFE=6 for NFE=8 and NFB=50
23

---

Published as a conference paper at ICLR 2025
FA 100
SI-SDR
fwSSNR T
IngSp SroSpeeMMS MSR
SIGMOS
6
200
3.45
4-
7
150
3.40
12
6-
3.35
8
50
4
50
4
50
NFE
NFE
NFE
NFE
NFE
Midpoint
Euler
Figure 13: Objective metrics for FlowDDee-75m at 7.5 khit/s, comparing the Fuler and Midpoint solver for different NFE.
but degrades slightly next possible lower NFE=4, thus confirming our default choice NFE=6 to be a good choice along the tradeoff between output quality and inference speed.
A.8 QUALITATIVE SPECTROGRAM COMPARISONS
In Fig. 14, we show spectrograms comparing FlowDec-75m and DAC-75 on three examples with high harmonic content such speech and isolated music instruments. We can that, for these examples, FlowDec recovers more plausible natural spectral structures, and recovers high harmonics better.
we show the three examples from our test set with the worst logSpecMSE values for FeowDec, and also those with the worst fwSSNR values in Fig. T6 We again compare FlowDec-75m against DAC-75 and also show the output from the initial decoder, NDAC-75. For the IogSpecMSE examples, see that FlowDec either inpaints frequencies that are not there in the clean reference, or wrongly removes high frequencies present in the initial decoder outputs beyond 16 kHz, which may related to training on music data with sampling rate kHz.
For fairness, in Fig. 15
For the example with worst fwSSNR (-3.32) in Fig. T6. we can see that Flow Dec mistakenly filters out most of the strong frequency content around 6 kHz even though is present the initial decoder output, and replaces it with spectrally more complex but wrong structures, indicating that the FlowDec postfilter is mistakenly treating these sounds as artifacts from the initial decoder rather than parts of the target signal. For the other two next- worst fwSSNR examples, FlowDec reconstructs relatively similar estimates as DAC-75, with no particularly implausible structures visible.
A.9 FULL OBJECTIVE METRICS TABLE
In the main paper, we showed objective metrics result visually. For completeness, we list the exact numbers of metric values in Table R
24

---

Published as a conference paper at ICLR 2025
Clean
Flow FlowDec- Dec- 75m
DAC-75
NDAC-75
20.0
17.5
15.0
Froqueny (kitz)
12.5
10.0
7.5
5.0
0.0
16
14
Erqueecc (kHt)
10
10
10
10
16
12
Prequeccy (KH22
10
0
5
10
0
Time (8)
Time (s)
Time (8)
Time (8)
Figure 14: Spectrograms (pre-emphasis of 1.0 applied) comparing FlowDec-75m against DAC-75 and NDAC-75 on three audio files: speech (top), glockenspiel and speech (middle), and acoustic guitar (bottom).
25

---

Published as a conference paper at ICLR 2025
Clean reference
DAC-75
Flow 75m-75m
NDAC-75
1 Ses MSF 415.00
pre MSF 66052
MSF. 419.22
20
Frequency (kHt))
15
10
Clean reference
DAC-75
PlowDee-70i
NDAC-75
LSL 136.62
ISL 634
IogSpecMMS 96.11
20
Frequency (kitz)
15
10
Clean reference
DAC-75
Flow Dew-eec-5m
NDAC-75
log, MSF MSF 25.43
logSper MSE 575.90
SogSSecMS3 MSB 60:01
20
Froquency (kHe)
15
10
10
10
0
10
0
Time (8)
Time (8)
Time (6)
Time (s)
Figure 15: Spectrograms comparing FlowDec-75m against DAC-75 as well as the initial decoder output (NDAC-75) on the three audio files where FlowDec-75m produces the worst logSpecMSE values on the whole test set.
26

---

Published as a conference paper at ICLR 2025
Clean reference
DAC-75
Flow 75m
NDAC-75
SWSSNR - 6.21
frRSNR - -3
SWSSAR 6.51
201
Froquency (kftz)
15
10
Clean reference
DAC-75
FlowDee-7700
NDAC-75
AWSENR 484
AWSNR 4.23
ASSNN 5.82
Frequency (kite)
15
10
Clean reference
DAC-75
Dec-75m
NDAC-75
AWSENR = 7.08
AWSENR = 6.39
GWSENR = 7.88
20
Frequency (kite)
15
10
10
10
10
10
Time (<)
Time (+)
Time (s)
Time (
Figure 16: Spectrograms comparing FlowDee-75m against DAC-75 as well as the initial decoder output (NDAC-75) on the three audio files where FlowDee-75mm produces the worst fwSSNR values on the whole test set.
27

---

Published as a conference paper at ICLR 2025
Table 8: Mean + 95% confidence interval of all metrics shown visually Fig. Best bold, second
best underlined. MBD refers to the method from San Roman et al. 2023 using their official 24 kHz checkpoint kbit/s. Bitrates kbit/s.
Method
Bitrate
FAD
SI-SDR t
fwSSNR
logSpecMSE
SIGMOS
7.50-12.00 kbit/s
FlowDee-75m
7.50
2.09
6.12 + 0.25
15.56 0.06
93.71 1.65
3.39 0.03
Feow5e
7.50
1.62
6.22 # 0.25
15.61 # 0.07
93.65 # 1.72
3.44 0.03
DAC-75
7.50
4.15
9.23
0.19
16.21 # 0.09
85.49 1.24 1.24
3.16 0.02
2xDAC-75
7.50
4.36
9.54 0.19
17.03 0.07
82.16 1.26
3.19
DAC 44.1 kHz
7.75
6.00
9.19
16.46
98.61 # 1.83
3.16
NDAC-75
7.50
34,46
7.49 0.26
17.50 # 0.09
75.07 # 1.17
2.71
EnCodec
12.00
4.08
8.98 0.18
13.66 + 0.13
135.68 2.66
2.61 0.02
6.00-6.03 kbit/s
FlowDec-75m
6.00
2.53
5.16 0.25
14.36
95.11 + 1.68
3.40 + 0.03
DAC-75
6.00
5.23
7.72 0.18
14.70 0.08
88.08 + 1.29
3.16
2xDAC-75
6.00
5.30
8.09 0.19
15.53
84.50
3.18
DAC 44.1 kHz
6.03
7.23
0.19
14.76
100.87
3.16
NDAC-75
6.00
36.80
6.36
15.74
1.25
2.70
EnCodec
6.00
7.35
6.27 0.188
1.17
142.69 2.71
2.41
MBD KHz)
6,00
16.87
0.44
7
782.25 15.9
2.73 0.02
1.31-450kbittt
FlowDec-75m
4.50
3.01
3.57 # 0.24
99.02 # 1.76
3.41 = 0.03
DAC-75
4.50
6.80
6.08 + 0.18
13.33 # 0.08
90.19 1.32
3.15
2xDAC-75
4.50
6.42
6.47 0.18
14.20 0.07
8.34
3.16
DAC 44.1
4.31
9.25
5.74
13.23 0.08
104.08 # 1.85
3.13
NDAC-75
4.50
41.01
5.05 + 0.24
14.41 # 0.08
78.36 1.23
2.68 0.02
2.58-3.00 kbit/s
FlowDee-75m
3.00
4.41
1.10 0.28
11,43 = 0.06
104.62 + 1.87
3,40 + 0.03
DAC-75
3.00
9.68
0.188
11.83 0.07
94.81 1.41
3.10
2xDAC-75
3,00
8.82
4.29
12.74 0.07
91.79 1.43
3.13
DAC 44.1 kHz
2.58
12.82
+.20
11.24 + 0.08
110.00 = 1.88
NDAC-75
3.00
49.07
0.266
12.89 # 0.08
81.75 # 1.28
2.59 + 0.02
EnCodec
3.00
15.66
3.44 0.20
9
153.48 + 2.78
2.10 0.02
4.00 kbit/s (25 Hz)
FeowDe
4.00
2.47
2.21 + 0.31
13.26 + 0.07
98.95 + 1.85
3,42 + 0.03
DAC-25
4.00
5.98
6.15 + 0.21
13.57 = 0.08
89.90 1.34
3.13 + 0.03
2xDAC-25
4.00
5.96
6.49 + 0.21
13.95 # 0.08
90.45 + 1.27
3.17 + 0.02
28


Reviews:
[{"review_id": "nPds693Irg", "reviewer": "Reviewer_ofGs", "summary": "This paper proposes FlowDec, a 48 kHz general audio codec with a flow-matching diffusion post-filter. FlowDec modifies the DAC audio codec with different loss functions, the stochastic post-filter, and frequency-dependent noise. Evaluations demonstrate strong improvements over prior post-filter and diffusion-based methods, but only small improvements over DAC.", "questions": "The NeurIPS 2023 oral by Kingma and Gao unifies under a common framework the diffusion variants of, e.g., flow matching and optimal transport. This could be useful, given that both are discussed in your work. You might also take a look at their Appendix J, which has some discussion of the frequency analysis of diffusion noise that may have some overlap with your proposed frequency-dependent noise.\n\nCan we hear some audio examples? It is difficult to judge the results otherwise, and the MUSHRA test (unlike an AB or ABX test) doesn't provide sufficient granularity to discern statistical significance between, e.g., FlowDec and DAC.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Figures 2 and 3 are great visualizations for the diffusion dynamics and proposed improvements.\n\nGood selection of baselines, including retraining baseline models to account for differences in parameter count.\n\nThe authors mention that they will be open-sourcing their code.", "weaknesses": "You should consolidate the six main contributions. I recommend at most three or four. Some of the contributions currently listed seem minor, or are a side effect of another contribution.\n\nA primary issue with the proposed system is RTF. The logical alternative (DAC) is still orders of magnitude faster. I do still think this is an important paper in helping close the gap.\n\nA minor note, but a 44.1 kHz sample rate is sufficient to prevent audible aliasing above 20 kHz. The other rationales given for using 48 kHz over 44.1 kHz are convincing enough for me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DCKUwTaJG", "reviewer": "Reviewer_VB3E", "summary": "The paper introduces FlowDec, a neural audio codec with a two-stage approach: (1) an autoencoder with residual vector quantization, trained without adversarial loss, and (2) a postfilter that reduces coding artifacts and enhances perceptual quality. FlowDec adapts conditional flow matching for signal enhancement, achieving improvements over previous score- and flow-based models. Both listening tests and objective metrics show that FlowDec provides perceptual quality competitive to state-of-the-art GAN-based codecs.", "questions": "1. Could the authors clarify the main purpose of FlowDec? Why is it presented as a standalone codec rather than positioning the postfilter as an enhancement tool?\n2. What is the rationale for choosing to enhance only outputs from a non-adversarial codec instead of using GAN output as the initial estimate for FlowDec's postfilter? Could the authors specify what benefits NDAC offers over DAC in this proposed two-stage setup?\n3. The STFT settings seem unconventional. A 1534-point FFT would definitely be slower than a 1536-point FFT due to FFT efficiency with highly composite sizes. I assume the goal is to get exactly 768 frequency bins, but why not just use a 1536-point FFT with 769 bins? Overall, this may be negligible given the slower inference, but I'm curious about the tradeoff here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, with clear explanations and informative illustrations that help clarify key points.\n- It provides useful context relative to previous score-based models. It also includes valuable theoretical insights and formalism to support its approach.\n- It performs on par with state-of-the-art GAN codecs and shows slight improvements for speech signals. FlowDec may handle high-frequency harmonics better, where GAN-based codecs often introduce periodicity and harmonic artifacts.", "weaknesses": "The paper introduces FlowDec as a codec, but its primary focus seems to be a flow-based postfilter that enhances outputs from an *underlying* codec. The main weakness of this paper is the decision not to explore a GAN-based codec combined with the proposed postfilter. The paper argues against adversarial training in Section 3.3, stating that the “generated phase may be very different from the original phase.” However, FlowDec itself doesn't seem to preserve phase. Results presented in Section 5.1 show that FlowDec performs worse than GAN-based codecs in terms of SI-SDR and fwSSNR, which are sensitive to phase shifts. This might still be acceptable for a low-bitrate codec, particularly if perceptual quality is prioritized. However, the paper should more clearly justify why a GAN isn't used as the underlying codec. A GAN-based codec with a flow-based postfilter might not only reduce coding artifacts but also achieve higher reconstruction metrics. This would call into question the rationale behind the proposed non-adversarial DAC (NDAC) model. Overall, the distinction between FlowDec as a novel audio codec and its role as an enhancement tool for *any* audio codec could be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iRXczT85aO", "reviewer": "Reviewer_QBQt", "summary": "FlowDec is an improved version of ScoreDec (uses a score-based generative model as a postfilter), by switching the objective to flow matching. It further proposes a joint flow matching objective tailored for the postfiltering task (e.g. mean-shifted noise with frequency-dependent diagonal covariance). This makes it faster than real time (unlike ScoreDec) as a practically viable, full-band, and universal audio codec without adversarial training.", "questions": "There have been several concurrent works adopting flow matching with similar task (i.e. mel spectrogram vocoder and discrete codec decoder), such as RFWave [1] and PeriodWave [2,3]. These work directly utilize flow matching as the decoder rather than postfilter, with several configs being order of magnitude faster than this work.\n\nWhile the direct comparison may not be strictly necessary, can the authors provide conceptual comparisons between the aforementioned methods along with possible advantages of FlowDec?\n\n[1] Liu, Peng, and Dongyang Dai. \"RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction.\" arXiv preprint arXiv:2403.05010 (2024).\n\n[2] Lee, Sang-Hoon, Ha-Yeong Choi, and Seong-Whan Lee. \"PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation.\" arXiv preprint arXiv:2408.07547 (2024).\n\n[3] Lee, Sang-Hoon, Ha-Yeong Choi, and Seong-Whan Lee. \"Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization.\" arXiv preprint arXiv:2408.08019 (2024).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Contrary to adversarial training (a dominant approach in vocoder/codec) which requires domain specific expertise to stabilize training and tune the hyperparameters of multiple losses, FlowDec simplifies the trianing pipeline by elimiating the adversarial losses. In my opinion, this can be considered as one of the first work that achieves competitive quality to GAN-based models with RTF < 1. \n\n* The proposed design choice is well justified overall, both in the theoretical and empericial perspective. It provides enough rigor in experimental detail, including toy experiments and detailed ablation study of each component.", "weaknesses": "* While the evaluation results from the paper look convincing overall, no demo page has been provided at submission. It's hard to form the reviewer's opinion on the subjective quality without access to the demo. I hope the authors can provide the samples for the reader to evaluate the subjective quality themselves. \n\n* Although it improved the RTF by a large margin, it still requires multiple NFE from the postfilter resulting in RTF of 0.22-0.23 which is considerably slower than recent, feed-forward audio codecs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fy5dLjWhNv", "reviewer": "Reviewer_bRLZ", "summary": "The authors introduce FlowDEC, a flow matching-based post-filtering method designed to enhance the audio quality decoded from discrete audio tokens. This method has demonstrated strong objective and subjective results.", "questions": "-  The question arises as to why the waveform is not reconstructed directly from discrete tokens using flow matching. To provide a more comprehensive evaluation, could you elaborate on the potential computational or quality trade-offs between direct reconstruction via flow matching and the method proposed in the paper?\n- Regarding line 168-171, the presence of $p_{data}(\\cdot|y)$ is not clear as it seemingly should just be $p_{data}(x)$. This needs further clarification and justification.\n- From line 220 to 223 and in Figure 2, it may be inappropriate as we cannot obtain $x_1$ for real postfiltering problems. This aspect requires reconsideration or better explanation.\n- In Figure 3, the use of a large $\\sigma_t$ of 0.1 o illustrate that FlowAVSE is non-contractive is not convincing, especially since the original FlowAVSE employs a value of 0.04. It would be beneficial for the authors to provide a clearer explanation for this choice or consider revising the value used. \n- As $y$ is a time-domain signal, it is unclear how to apply frequency-dependent $\\sigma_y$ since it cannot be directly applied. The authors need to provide more details on this aspect.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Possibly the first application of flow matching in postfiltering.\n- Beautiful experiment result plots and strong objective and subjective results.\n- Enhance DAC by incorporating a multi-scale Constant-Q Transform (CQT) loss.", "weaknesses": "- The latency of the proposed method is at least an order of magnitude higher than that of DAC. To facilitate a comprehensive evaluation, it would be beneficial to receive specific latency measurements for both the proposed method and the DAC. Furthermore,  identifying the applications or use cases that could be critically impacted by this higher latency would provide valuable context. Moreover, to support the validation of the method’s performance, it would be advantageous if the authors could provide an online demo that allows for direct comparison of the latency between the two methods.\n- The baselines are limited. There are other diffusion-based works that reconstruct audio waveforms from discrete tokens, e.g. Multi-band diffusion [1]. \n- The concept of selecting a data-dependent prior, as proposed, is not entirely novel, with works such as Priorgrad [2] having implemented a similar strategy. To better understand the relationship between the proposed method and existing approaches, it would be helpful to have a clearer comparison of their similarities and differences. \n- Multi-band diffusion [1] uses frequency-dependent noise levels, which is a notable feature that could be further discussed in comparison to the proposed method.\n\n[1] \"From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion\" by Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez.\n\n[2] \"PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior\" by Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, Tie-Yan Liu", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)