# Modeling strategies for speech enhancement in the latent space of a neural audio codec

Sofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaise

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sofiene Kammoun, Xavier Alameda-Pineda, Simon Leglaise. Modeling strategies for speech enhancement in the latent space of a neural audio codec. 2025. hal-05335192

HAL Id: hal-05335192

https://hal.science/hal-05335192v1

Preprint submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
MODELING STRATEGIES FOR SPEECH ENHANCEMENT IN THE LATENT SPACE OF A
NEURAL AUDIO CODEC

Sofiene Kammoun*

Xavier Alameda-Pineda†

Simon Leglaive*

*CentraleSupélec, IETR (UMR CNRS 6164), France

†Inria at Univ. Grenoble Alpes, CNRS, LJK, France

ABSTRACT

Neural audio codecs (NACs) provide compact latent speech repre-
sentations in the form of sequences of continuous vectors or dis-
crete tokens. In this work, we investigate how these two types of
speech representations compare when used as training targets for su-
pervised speech enhancement. We consider both autoregressive and
non-autoregressive speech enhancement models based on the Con-
former architecture, as well as a simple baseline where the NAC en-
coder is simply fine-tuned for speech enhancement. Our experiments
reveal three key findings: predicting continuous latent representa-
tions consistently outperforms discrete token prediction; autoregres-
sive models achieve higher quality but at the expense of intelligibility
and efficiency, making non-autoregressive models more attractive in
practice; and encoder fine-tuning yields the strongest enhancement
metrics overall, though at the cost of degraded codec reconstruction.
The code and audio samples are available online.¹

***Index Terms***— Speech enhancement, neural audio codec, au-
toregressive modeling, latent representations, discrete tokens.

# 1. INTRODUCTION

Speech enhancement (SE) addresses the problem of estimating a
clean speech signal from a degraded recording affected by additive
noise, reverberation, or other distortions. Although the target output
is often the clean waveform (for listening or downstream tasks),
the enhancement process can be defined in different representation
domains. The choice of representation (waveform, time-frequency,
continuous latent vectors, or discrete tokens) strongly conditions
model design, inference speed, and the quality and intelligibility of
the restored signal.

Classic SE algorithms worked predominantly in the short-
time Fourier transform (STFT) domain, exploiting structure in the
time-frequency plane and statistical assumptions about speech and
noise [1, 2, 3]. Early deep learning approaches to SE followed the
established time-frequency pipeline by estimating time-frequency
masks or magnitudes using discriminative training of neural net-
works [4, 5]. Later, time-domain architectures such as Conv-TasNet
[6] showed that learned encoder-separator-decoder pipelines with
1D temporal convolutions can outperform oracle time-frequency
masking in speech separation, which removes the need for explicit
phase estimation by using trainable filterbanks and time-domain loss
functions. The effectiveness of this time-domain approach was also
confirmed for SE [7].

In parallel, neural audio codecs (NACs), initially introduced
in [8], are being adopted across speech processing frameworks,
where they provide an alternative representation space for speech

signals [9]. NACs compress audio through an encoder-decoder
framework into compact latent sequences represented as continuous
vectors or as discrete tokens produced by residual vector quanti-
zation (RVQ) [10]. The appeal of these representations for down-
stream tasks is due to two key advantages. First, they offer a very
compressed representation of the data, potentially reducing memory
and compute for sequence models. Second, the discrete-token view
makes it straightforward to reuse modeling techniques from natural
language processing (NLP).

Recent audio generative models such as [11] process audio sig-
nals as sequences of discrete tokens, using RVQ-aware and often au-
toregressive (AR) transformer-based architectures [12]. Inspired by
this success, recent works have developed SE methods that generate
clean speech tokens conditioned on noisy speech tokens [13, 14, 15].
While transformers are often assumed to require discrete tokens as
input, their first layer is an embedding layer that produces continu-
ous representations, making it feasible to directly use the continuous
latent representation of the NAC without modifying the rest of the
model. Recent works have started exploring the use of transformer-
based architectures using continuous representations for image gen-
eration [16], speech synthesis [17], speech separation [18], and SE
[19, 20].

Moreover, in general audio generation tasks, AR models are
appropriate when the output length or alignment between input and
output is unknown, which is the case for tasks like text-to-speech
(TTS) [11], prompt-based music [21], and image [22] generation.
However, in SE, the input and output signals (noisy and clean
speech) are of equal length and perfectly aligned, with no absolute
need for diverse predictions. This reduces the intrinsic need for
AR modeling. Still, AR models offer the advantage of modeling
intra-sequence temporal dependencies. In contrast, non-AR models
generate the entire sequence simultaneously, relying only on input-
output dependencies. Recent concurrent studies have also begun
to explore this trade-off for SE [23], [24], offering complementary
insights.

Finally, SE and codecs are two essential components of voice
communication systems, making the study of their interaction prac-
tically relevant. A natural approach to SE using NACs is to fine-tune
the NAC encoder to produce a clean speech latent representation
directly from the noisy input. However, this may compromise the
NAC's reconstruction performance. To our knowledge, this simple
strategy has only received very little attention in prior work [25].

In this work, we focus on three central questions regarding SE
using NACs: (i) whether to model discrete tokens or continuous vec-
tors in the NAC's latent space, (ii) how AR and non-AR models com-
pare, and (iii) how the fine-tuning of the NAC's encoder compares to
adding a separate SE model. Our goal is not to optimize any single
model for peak performance but to provide a fair comparative study
of these signal representation spaces and modeling choices.

¹sofienekammoun.github.io/SE-NAC-25/
## 2. METHOD

We frame our study of SE in NAC latent spaces along three modeling axes introduced above. To structure the discussion, we design a family of models that systematically vary along these axes while keeping other components comparable. All methods share the following common structure. A pretrained NAC maps noisy and clean waveforms into latent representations, either continuous embeddings or discrete tokens. Enhancement models then learn a conditional distribution of clean latents given noisy latents. Depending on the variant, the predictor may operate autoregressively over time or non-autoregressively, targeting either continuous vectors or discrete tokens. Finally, the NAC decoder reconstructs the waveform from the predicted latent sequence. We also compare these models to a simple fine-tuning of the NAC encoder. In the following, we denote by $\theta$ the model parameters for each SE method. All models are trained in a supervised manner by minimizing the negative log-likelihood.

### 2.1. Neural audio codecs

NACs are encoder-decoder architectures built with 1D convolutional networks, but they further introduce RVQ of the latent representation extracted by the encoder [8]. Let $\mathbf{y}_{\text{wav}}$ and $\mathbf{x}_{\text{wav}} \in \mathbb{R}^{d \cdot f_s}$ denote the noisy and clean speech waveforms of duration $d$ in seconds and sampling rate $f_s$ in Hertz, assuming $d \cdot f_s \in \mathbb{N}$ without loss of generality. A NAC can be described with the following operations:

$$ 
\begin{array}{ll}
\text{Tokenize:} & \mathbf{x}_{\text{wav}} \xrightarrow{\mathcal{E}} \bar{\mathbf{x}} \in \mathbb{R}^{L \times T} \quad \mathcal{Q}_{\mathcal{C}} \xrightarrow{\mathcal{D}} \mathbf{x} \\
& \mathbf{y}_{\text{wav}} \xrightarrow{\bar{\mathbf{y}}} \bar{\mathbf{y}} \in \{1, \dots, K\}^{N \times T}, \\
\\
\text{Detokenize:} & \mathbf{x} \xrightarrow{Q_{\mathcal{C}}^{-1}} \hat{\mathbf{x}} \in \mathbb{R}^{L \times T} \quad \mathcal{D} \xrightarrow{\hat{\mathbf{y}}} \hat{\mathbf{y}}_{\text{wav}} \in \mathbb{R}^{d \cdot f_s}, \\
& \mathbf{y} \xrightarrow{\hat{\mathbf{y}}} \hat{\mathbf{y}} \in \mathbb{R}^{L \times T} \quad \mathcal{D} \xrightarrow{\hat{\mathbf{y}}_{\text{wav}}} \hat{\mathbf{y}}_{\text{wav}} \in \mathbb{R}^{d \cdot f_s},
\end{array}
\quad (1) $$

where $\mathcal{E}$ is the encoder that encodes the waveform into an $L$-dimensional downsampled continuous representation of length $T$, $\mathcal{D}$ is the decoder that performs the reverse operation, and $Q_C$ and $Q_C^{-1}$ denote the direct and inverse RVQ modules, which depend on a set of codebooks $C = \{C_n \in \mathbb{R}^{L \times K}\}_{n=1}^N$, with $K$ codebook vectors of dimension $L$ for each quantization stage $n \in \{1, \dots, N\}$. See [8] for a description of RVQ [10] in the context of NACs.

### 2.2. Speech enhancement models

#### 2.2.1. Discrete token representation

We first consider modeling strategies where the noisy and clean speech signals are represented as tokens in the discrete latent domain: $\mathbf{x} = \{x_{t,n}\}_{t=1,n=1}^{T,N}$, $\mathbf{y} = \{y_{t,n}\}_{t=1,n=1}^{T,N} \in \{1, \dots, K\}^{N \times T}$.

**Discrete autoregressive model (D-AR)** This first SE model is AR over time and quantization depth, and can be written using the chain rule of probabilities as:

$$ 
\begin{aligned}
p_{\theta}(\mathbf{x} | \mathbf{y}) &= \prod_{t=1}^{T} \prod_{n=1}^{N} p_{\theta}(x_{t,n} | \mathbf{x}_{1:t-1}, \mathbf{x}_{t,1:n-1}, \mathbf{y}) \\
&= \prod_{t=1}^{T} \prod_{n=1}^{N} \prod_{k=1}^{K} f_{t,n,k}^{(\text{D-AR})}(\mathbf{x}_{1:t-1}, \mathbf{x}_{t,1:n-1}, \mathbf{y}; \theta)^{1\{x_{t,n}=k\}},
\end{aligned}
\quad (2) $$

where $1\{\cdot\}$ denotes the indicator function, and $f_{t,n,k}^{(\text{D-AR})}(\cdot) \in [0, 1]$ denotes the probability that the discrete clean speech token $x_{t,n}$ at time $t$ and quantization stage $n$ equals $k \in \{1, \dots, K\}$, given the

noisy tokens $\mathbf{y}$, the clean speech tokens $\mathbf{x}_{t,1:n-1}$ at time $t$ and quantization stages $1$ to $n-1$, and the past clean speech tokens $\mathbf{x}_{1:t-1}$ at all quantization stages. Therefore, we have $\sum_{k=1}^{K} f_{t,n,k}(\cdot) = 1$. Inspired by the RQ-Transformer model [22] for text-to-image generation and by the Conformer model [26] for speech recognition, in this work, we propose the RQ-Conformer to implement the function $f^{(\text{D-AR})}$. The proposed model is composed of 3 modules: (i) a bidirectional Conformer for processing the noisy speech tokens $\mathbf{y}$, (ii) a causal Conformer to autoregressively process the clean speech tokens over time $\mathbf{x}_{1:t-1}$, and (iii) a causal transformer processing the outputs of both previous modules and the clean speech tokens over the quantization-depth dimension $\mathbf{x}_{t,1:n-1}$ providing the predicted output $f_{t,n,k}^{(\text{D-AR})}$. This choice is motivated by the recurrent nature of RVQ, where the quantization at level $n$ depends on the quantization at level $n-1$. At the input, a trainable embedding layer embeds each discrete token in $\{1, \dots, K\}$ into a continuous vector of dimension $H$ (the internal Conformer dimension).

**Discrete non-autoregressive model (D-NAR)** We now remove the AR dependencies in the previous model described in (2), such that:

$$ p_{\theta}(\mathbf{x} | \mathbf{y}) = \prod_{t=1}^{T} \prod_{n=1}^{N} \prod_{k=1}^{K} f_{t,n,k}^{(\text{D-NAR})}(\mathbf{y}; \theta)^{1\{x_{t,n}=k\}}, \quad (3) $$

where $f^{(\text{D-NAR})} : \{1, \dots, K\}^{N \times T} \mapsto [0, 1]^{T \times N \times K}$ is now simply composed of a bidirectional Conformer model that processes the noisy tokens $\mathbf{y}$ in a feed-forward manner. At the output, the model contains $N$ classification heads that convert the Conformer's output to a sequence of probabilities over $\{1, \dots, K\}$.

#### 2.2.2. Continuous latent representation

We now consider modeling strategies where the noisy and clean speech signals are represented as continuous latent vectors, i.e., $\bar{\mathbf{x}} = \{\bar{\mathbf{x}}_t\}_{t=1}^T$, $\bar{\mathbf{y}} = \{\bar{\mathbf{y}}_t\}_{t=1}^T \in \mathbb{R}^{L \times T}$. So, at a time step $t$, instead of predicting a categorical distribution over the discrete tokens for each quantization stage $n$, the model will directly provide a prediction of the continuous vector $\bar{\mathbf{x}}_t$, given $\bar{\mathbf{y}}$ and potentially additional conditions. Additionally, the input embedding layer becomes a simple linear layer that projects input vectors from dimension $L$ to $H$.

**Continuous autoregressive model (C-AR)** The first model in the continuous case is autoregressive and can be written as:

$$ 
\begin{aligned}
p_{\theta}(\bar{\mathbf{x}} | \bar{\mathbf{y}}) &= \prod_{t=1}^{T} p_{\theta}(\bar{\mathbf{x}}_t | \bar{\mathbf{x}}_{1:t-1}, \bar{\mathbf{y}}) \\
&= \prod_{t=1}^{T} \mathcal{N}(\bar{\mathbf{x}}_t; f_t^{(\text{C-AR})}(\bar{\mathbf{x}}_{1:t-1}, \bar{\mathbf{y}}; \theta), \mathbf{I}_L),
\end{aligned}
\quad (4) $$

where $\mathcal{N}$ denotes a multivariate Gaussian distribution over $\bar{\mathbf{x}}_t$ with mean vector $f_t^{(\text{C-AR})}(\cdot)$ and identity covariance matrix $\mathbf{I}_L$. Here, $f$ denotes a causal Conformer model for processing the noisy speech embeddings $\bar{\mathbf{y}}$ concatenated to the clean speech latent vectors $\bar{\mathbf{x}}_{1:t-1}$. In this approach, we consider temporal dependencies using an AR model over time. However, we assume that all dimensions of the continuous latent vectors $\bar{\mathbf{x}}_t$ are independent. We should note here that to avoid the problem of error accumulation that usually occurs for AR models, during inference only we quantize the input vectors $\bar{\mathbf{x}}_{1:t-1}$ using the NAC's RVQ module before feeding them to the Conformer model.
**Continuous non-autoregressive model (C-NAR)** As in the discrete case, we now remove the autoregressive dependencies in the previous model described in (4), such that:

$$p_{\theta}(\bar{x} | \bar{y}) = \prod_{t=1}^{T} p_{\theta}(\bar{x}_{t} | \bar{y}) = \prod_{t=1}^{T} \mathcal{N}(\bar{x}_{t}; f_{t}^{(\text{C-NAR})}(\bar{y}; \theta), \mathbf{I}_{L}), \quad (5)$$

where $f^{(\text{C-NAR})}: \mathbb{R}^{L \times T} \to \mathbb{R}^{L \times T}$ denotes a bidirectional Conformer model, with the same input layer previously described and a predictive head that generates the mean vector of the Gaussian distribution.

### 2.2.3. Baseline: fine-tuning the encoder

In contrast to the approaches described above, we also consider a simpler baseline where the NAC encoder is fed with the noisy speech waveform and fine-tuned to provide the clean speech latent representation, bypassing the need for an additional sequence model. In the continuous representation setting (C-FT model), the encoder is fine-tuned to predict the clean speech continuous latent representation. The formulation of this model closely resembles (5) with $\bar{y}$ replaced by $y_{\text{wav}}$. In the discrete representation setting (D-FT model), the target is the sequence of clean speech discrete tokens. To bridge the gap between continuous encoder outputs and discrete tokens, we follow the so-called “soft labeling” strategy described in [22]. Specifically, the encoder output vectors are mapped to probability distributions over codebook indices by measuring their Euclidean distance to each codebook vector and normalizing using a softmax function. We use the straight-through estimator for gradient backpropagation [27].

## 2.3. Training and inference

All models considered here are trained in a supervised fashion on paired noisy–clean speech data by maximizing the conditional likelihood of clean speech given noisy speech, which is simply defined by taking the logarithm of equations (2) to (5). When the model is defined over the continuous representation $\bar{x}$, maximizing the likelihood is equivalent to minimizing the mean squared error (MSE). In the other case, when the model is defined over the discrete latent representation $\bar{x}$, this amounts to minimizing the cross-entropy loss.

Independent of the representation space, given a trained model, SE is achieved by taking the argmax over $\bar{x}$ or $\bar{x}$ of the above-defined probabilistic models. In this work, we are not interested in introducing stochasticity in the SE process by sampling the distributions. We would rather estimate the most likely clean speech signal for a given trained model and noisy signal. For continuous-prediction models, the estimated vectors are quantized before being passed to the NAC decoder, ensuring compatibility with the codec reconstruction pipeline, and for discrete-prediction models, the predictions go through the detokenization process described in (1).

# 3. EXPERIMENTS

## 3.1. Experimental setup

**Dataset** LibriMix [32] is a single-speaker noisy speech dataset constructed from clean utterances of LibriSpeech [33] combined with noise samples from WHAM! [34]. It covers a range of signal-to-noise ratio (SNR) values between -6 and 3 dB, simulating challenging noisy conditions. We use the train-360 subset for our experiments, which provides 156 hours of paired noisy/clean speech for training and 4 hours each for validation and testing.

**NAC** For the NAC model, we resort to Descript Audio Codec (DAC) [35], widely used across similar lines of work. We use the 16 kHz variant, which has $N = 12$ quantization stages, with each codebook containing $K = 1024$ vectors of dimension size $L = 1024$, and $T = 50$ for one second of audio. This model is trained on an extensive dataset of speech, music, and environmental sounds.

**Hyperparameters** All enhancement models are based on the Conformer architecture [26] with hidden dimension $H = 384$. The hyperparameters are chosen such that all models contain approximately 6–8 × $10^7$ parameters, on the same order as the NAC encoder. For the continuous models, we use a 16-layer Conformer with a linear prediction head projecting from $H$ to the codec’s continuous representation dimension $L$. For the discrete models, the D-AR variant consists of an 8-layer bidirectional Conformer processing noisy tokens, an 8-layer causal Conformer autoregressively modeling past clean tokens, and a 6-layer causal Transformer along the quantization-depth dimension. The D-NAR model uses an 8-layer Conformer followed by $N = 12$ feed-forward prediction heads, each projecting to a distribution over the corresponding codebook indices. Training is performed on one-second paired speech segments, with a batch size of 32 per GPU (4× NVIDIA HGX A100) for 300 epochs. We use AdamW with $\beta = (0.9, 0.95)$, weight decay 0.05, and a cosine learning rate schedule with 10 warm-up epochs. Following common scaling strategies, the maximum learning rate is set to $0.005 \times$ (batch size/256).

**Metrics** To evaluate the models’ performance, we rely on non-intrusive quality metrics commonly used in the literature on generative SE. For speech quality, we use the DNSMOS P.835 [36], a non-intrusive predictor trained to approximate human ratings. DNSMOS provides scores along three dimensions: SIG (speech quality), BAK (background noise suppression), and OVRL (overall quality). In addition, we include DNSMOS P.808 [37] as a complementary quality estimator. To assess naturalness, we use UTMOS [38], a neural MOS predictor particularly sensitive to artifacts introduced by vocoding or compression, and focuses more on perceived naturalness rather than noise suppression. For speaker similarity, we compute the cosine similarity (CosSim, in %) between embeddings of enhanced and clean speech, extracted using a WavLM-based speaker representation model.<fnref n="2" /> Intelligibility is measured through the differential word error rate (dWER, in %), obtained by comparing Wav2Vec²³ transcriptions of enhanced versus clean speech. In addition to non-intrusive metrics used for SE, we use the intrusive measures PESQ [39] and ESTOI [40] to assess clean-speech reconstruction after training. Finally, to quantify the computation needed for inference, we report the number of floating-point operations (FLOPs) required to generate one second of speech, excluding the NAC encoder and decoder. For reference, the NAC encoder and decoder, respectively, require 25 GFLOPs and 87 GFLOPs.

**Baselines** We compare the proposed models against discriminative approaches for SE. We include several widely adopted discriminative neural networks that learn direct mappings from noisy speech to clean speech in either the time or time-frequency domain: the Deep Complex Convolutional Recurrent Network (DCCRNet) [28], the Deep Complex U-Net (DCUNet) [29], the Dual-Path Transformer Network (DPTNet) [30], and the fully-convolutional Conv-TasNet [6], using models pretrained on LibriMix available online.<fnref n="4" /> In addition, we consider AnCoGen [31], a model designed for the analysis

<footnote n="2">²https://huggingface.co/microsoft/wavlm-base-sv</footnote>

<footnote n="3">³https://huggingface.co/facebook/wav2vec2-base-960h</footnote>

<footnote n="4">⁴https://huggingface.co/JorisCos</footnote>
<table><caption>**Table 1.** SE results (best and second-best scores in each column are bold and underlined).</caption><thead><tr><th>Model</th><th>OVRL (↑)</th><th>SIG (↑)</th><th>BAK (↑)</th><th>P808 (↑)</th><th>UTMOS (↑)</th><th>CosSim (↑)</th><th>dWER (↓)</th><th>GFLOPs (↓)</th></tr></thead><tbody><tr><td>DCCRNet [28]</td><td>2.80</td><td>3.15</td><td>4.03</td><td>3.54</td><td>3.01</td><td>96.6</td><td>11.80</td><td>26</td></tr><tr><td>DCUNet [29]</td><td>2.97</td><td>3.33</td><td>3.96</td><td>3.62</td><td>3.07</td><td>96.7</td><td><strong>10.11</strong></td><td>250</td></tr><tr><td>ConvTasNet [6]</td><td>3.11</td><td>3.39</td><td>4.01</td><td>3.31</td><td>3.27</td><td>96.4</td><td>11.30</td><td>10</td></tr><tr><td>DPTNet [30]</td><td>3.00</td><td>3.32</td><td><u>4.05</u></td><td>3.31</td><td>3.38</td><td>96.6</td><td>10.75</td><td>2</td></tr><tr><td>AnCoGen [31]</td><td>3.00</td><td>3.32</td><td><u>4.05</u></td><td>3.31</td><td>3.38</td><td>96.6</td><td>19.30</td><td>-</td></tr><tr><td>D-AR</td><td>2.90</td><td>3.17</td><td>3.99</td><td>3.53</td><td>2.76</td><td>95.2</td><td>25.09</td><td>5857</td></tr><tr><td>D-NAR</td><td>2.89</td><td>3.18</td><td>3.94</td><td>3.50</td><td>2.72</td><td>95.5</td><td>23.12</td><td>6</td></tr><tr><td>D-NAR*</td><td>2.91</td><td>3.20</td><td>3.94</td><td>3.51</td><td>2.80</td><td>96.4</td><td>15.93</td><td>6</td></tr><tr><td>C-AR</td><td><strong>3.32</strong></td><td><strong>3.61</strong></td><td><strong>4.07</strong></td><td><strong>3.77</strong></td><td><strong>3.61</strong></td><td>96.2</td><td>20.47</td><td>472</td></tr><tr><td>C-NAR</td><td><u>3.25</u></td><td><u>3.56</u></td><td>4.01</td><td>3.60</td><td>3.54</td><td>97.0</td><td>13.48</td><td>6</td></tr><tr><td>C-NAR-FT</td><td>3.24</td><td><u>3.56</u></td><td>4.03</td><td><u>3.67</u></td><td><u>3.60</u></td><td><strong>97.2</strong></td><td><u>11.07</u></td><td>6</td></tr><tr><td>D-FT</td><td>2.84</td><td>3.12</td><td>3.97</td><td>3.40</td><td>2.63</td><td>95.3</td><td>24.42</td><td>0</td></tr><tr><td>C-FT</td><td>3.20</td><td>3.52</td><td>4.00</td><td>3.58</td><td>3.37</td><td>96.8</td><td>12.81</td><td>0</td></tr><tr><td>Mixture</td><td>1.75</td><td>2.46</td><td>1.81</td><td>2.62</td><td>1.51</td><td>93.9</td><td>30.00</td><td></td></tr></tbody></table>

and controllable generation of speech, in particular SE.<fnref n="5" /> To the best of our knowledge, no NAC-based SE methods have publicly released the code or model weights, which prevents direct comparison with such approaches.

## 3.2. Results

Table 1 reports objective evaluations across all models. The most noticeable observation is that models trained to predict continuous codec representations consistently outperform their discrete counterparts, irrespective of AR/NAR design (on average, +0.80 UTMOS and +0.40 SIG, comparing C-AR, C-NAR, and C-FT with their discrete counterparts). This gap is also evident when comparing C-FT against D-FT.

To further investigate the weakness of discrete models, we introduce a variant of D-NAR that uses continuous inputs $\tilde{y}$ instead of discrete noisy tokens $y$. This allows us to measure to what extent the input representation affects the D-NAR model. This model (denoted by D-NAR* in Table 1) performs better than other discrete variants, but still lags behind continuous models, suggesting that conditioning on discrete tokens is only part of the problem and that the main bottleneck lies in the output space and associated loss function used for discrete prediction.

Additionally, whether continuous or discrete, AR models tend to achieve higher quality (DNSMOS, UTMOS) than NAR models, likely due to their ability to capture temporal dependencies during generation. However, they demonstrate degraded speaker similarity and intelligibility, which could be explained by error accumulation over autoregressive steps. Moreover, the quality gains of continuous AR over continuous NAR are modest compared to the significant computational overhead and the higher dWER values.

Finally, we experiment with fine-tuning the NAC encoder alongside training a C-NAR enhancement model. We chose the C-NAR system to explore this fine-tuning strategy because it already provides a strong balance between quality and efficiency. This strategy, denoted by C-NAR-FT in Table 1, yields the best compromise between speech quality, intelligibility, and inference speed.

We also observe that fine-tuning the encoder for SE can compromise the primary role of the NAC as a high-fidelity speech codec, since the modified encoder may no longer reconstruct clean speech accurately. To study this effect, we test the C-NAR, C-FT, and C-NAR-FT models on clean speech, and report ΔPESQ and ΔESTOI

relative to the reference NAC before fine-tuning. Results show that C-FT suffers the largest degradation (ΔPESQ = −0.73, ΔESTOI = −0.03), C-NAR-FT also degrades reconstruction (−0.64, −0.03), while C-NAR preserves fidelity best (−0.32, −0.01). From a practical perspective, this suggests different operating points depending on the target application. For telecommunication scenarios where compression and enhancement must coexist and clean reconstruction remains important, C-NAR is the preferred choice. For applications where SE performance is the sole priority, the C-NAR-FT model provides good performance while being relatively more efficient.

## 4. CONCLUSION

In this work, we investigated SE models in the latent space of NACs, comparing discrete and continuous representations, AR and NAR models, and the impact of codec encoder fine-tuning. Our results highlight the advantages of continuous representations and NAR models, as well as the trade-offs between enhancement performance and codec fidelity. The study was conducted using only a few hundred hours of training data, offering a complementary perspective to related works that often leverage thousands of hours [13, 14, 15]. However, scaling effects may alter the trends we observed in this study. Also, we did not explore the use of semantic tokens, which are also frequently employed in the previously mentioned literature and could yield different conclusions. Finally, our analysis is restricted to the Descript Audio Codec, extending the study to other NACs would help assess the generality of our findings.

**Acknowledgments** This work was performed using computational resources from the Mésocentre computing center<fnref n="6" /> of Université Paris-Saclay, CentraleSupélec, and Ecole Normale Supérieure Paris-Saclay, as part of the DEGREASE project (ANR-23-CE23-0009), funded by the French National Research Agency.

## 5. REFERENCES

[1] S. Boll, "Suppression of acoustic noise in speech using spectral subtraction," *IEEE Trans. Audio, Speech, Lang. Proc.*, vol. 27, no. 2, pp. 113–120, 1979.

[2] R. McAulay and M. Malpass, "Speech enhancement using a soft-decision noise suppression filter," *IEEE Trans. Audio, Speech, Lang. Proc.*, vol. 28, no. 2, pp. 137–145, 1980.

<footnote n="5">⁵Contrary to [31], we use the most recent DNSMOS P.835 model for performance evaluation, which explains the difference in reported scores.</footnote>

<footnote n="6">⁶https://mesocentre.universite-paris-saclay.fr</footnote>
[3] Y. Ephraim and D. Malah, "Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator," *IEEE Trans. Audio, Speech, Lang. Proc.*, vol. 32, no. 6, pp. 1109-1121, 1984.

[4] A. Narayanan and D. Wang, "Ideal ratio mask estimation using deep neural networks for robust speech recognition," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2013.

[5] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, "Speech enhancement with lstm recurrent neural networks and its application to noise-robust asr," in *Int. Conf. Latent Variable Analysis and Signal Separation (LVA/ICA)*, 2015.

[6] Y. Luo and N. Mesgarani, "Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation," *IEEE/ACM Trans. Audio, Speech, Lang. Proc.*, vol. 27, no. 8, pp. 1256-1266, 2019.

[7] A. Défossez, G. Synnaeve, and Y. Adi, "Real time speech enhancement in the waveform domain," in *Interspeech*, 2020.

[8] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec," *IEEE/ACM Trans. Audio, Speech, Lang. Proc.*, vol. 30, pp. 495-507, 2021.

[9] J. Li et al., "Investigating neural audio codecs for speech language model-based speech generation," in *Proc. SLT*, 2024.

[10] B. Juang and A. Gray, "Multiple stage vector quantization for speech coding," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 1982, vol. 7.

[11] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, "Neural codec language models are zero-shot text to speech synthesizers," *arXiv:2301.02111*, 2023.

[12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in *Adv. Neural Inform. Proc. Syst. (NeurIPS)*, 2017, vol. 30.

[13] H. Yang et al., "Genhancer: High-fidelity speech enhancement via generative modeling on discrete codec tokens," in *Interspeech*, 2024.

[14] X. Li, Q. Wang, and X. Liu, "MasksSR: Masked language model for full-band speech restoration," in *Interspeech*, 2024.

[15] J. Yao et al., "GenSE: Generative speech enhancement via language models using hierarchical modeling," in *Int. Conf. Learn. Rep. (ICLR)*, 2025.

[16] T. Li, Y. Tian, H. Li, M. Deng, and K. He, "Autoregressive image generation without vector quantization," *Adv. Neural Inform. Proc. Syst. (NeurIPS)*, vol. 37, 2024.

[17] L. Meng, L. Zhou, S. Liu, S. Chen, B. Han, S. Hu, Y. Liu, J. Li, S. Zhao, X. Wu, H. M. Meng, and F. Wei, "Autoregressive speech synthesis without vector quantization," in *Proc. of Annual Meeting of the Association for Computational Linguistics (ACL)*, July 2025.

[18] J. Q. Yip, C. Y. Kwok, B. Ma, and E. S. Chng, "Speech separation using neural audio codecs with embedding loss," in *Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)*, 2024.

[19] H. Li, J. Q. Yip, T. Fan, and E. S. Chng, "Speech enhancement using continuous embeddings of neural audio codec," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2025.

[20] X. Sun, H. Dinkel, Y. Niu, L. Wang, J. Zhang, and J. Luan, "Efficient speech enhancement via embeddings from pre-trained generative audioencoders," *arXiv:2506.11514*, 2025.

[21] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, "Simple and controllable music generation," *Adv. Neural Inform. Proc. Syst. (NeurIPS)*, vol. 36, 2023.

[22] D. Lee et al., "Autoregressive image generation using residual quantization," in *IEEE/CVF Conf. Computer Vision Pattern Recog. (CVPR)*, 2022.

[23] L. Della Libera, C. Subakan, and M. Ravanelli, "Autoregressive speech enhancement via acoustic tokens," *arXiv:2507.12825*, 2025.

[24] H. Yang, G. Wichern, R. Aihara, Y. Masuyama, S. Khurana, F. G. Germain, and J. Le Roux, "Investigating continuous autoregressive generative speech enhancement," in *Interspeech*, 2025.

[25] J. Hauret, T. Joubaud, and É. Bavu, "Real-time speech enhancement in noise for throat microphone using neural audio codec as foundation model," *arXiv:2508.02974*, 2025.

[26] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al., "Conformer: Convolution-augmented transformer for speech recognition," in *Interspeech*, 2020.

[27] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, "Neural discrete representation learning," in *Adv. Neural Inform. Proc. Syst. (NeurIPS)*, 2017.

[28] Y. Hu, Y. Liu, S. Lv, M. Xing, S. Zhang, Y. Fu, J. Wu, B. Zhang, and L. Xie, "Dccrn: Deep complex convolution recurrent network for phase-aware speech enhancement," *Interspeech*, 2020.

[29] H. Choi, J. Kim, J. Huh, A. Kim, J. Ha, and K. Lee, "Phase-aware speech enhancement with deep complex u-net," in *Int. Conf. Learn. Rep. (ICLR)*, 2018.

[30] J. Chen, Q. Mao, and D. Liu, "Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation," in *Interspeech*, 2020.

[31] S. Sadok, S. Leglaise, L. Girin, G. Richard, and X. Alameda-Pineda, "Ancogen: Analysis, control and generation of speech with a masked autoencoder," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2025.

[32] J. Cosentino et al., "LibriMix: An open-source dataset for generalizable speech separation," *arXiv:2005.11262*, 2020.

[33] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: an asr corpus based on public domain audio books," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2015.

[34] G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux, "Wham!: Extending speech separation to noisy environments," *Interspeech*, 2019.

[35] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, "High-fidelity audio compression with improved RVQGAN," in *Adv. Neural Inform. Proc. Syst. (NeurIPS)*, 2023.

[36] C. K. Reddy, V. Gopal, and R. Cutler, "DNSMOS P. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2022.

[37] C. K. Reddy, V. Gopal, and R. Cutler, "DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2021.

[38] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, "UTMOS: utokyo-sarulab system for voicemos challenge 2022," in *Interspeech*, 2022.

[39] A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra, "Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs," in *IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP)*, 2001.

[40] Jesper Jensen and Cees H. Taal, "An algorithm for predicting the intelligibility of speech masked by modulated noise maskers," *IEEE/ACM Trans. Audio, Speech, Lang. Proc.*, pp. 2009-2022, 2016.