FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
Authors: Luca Della Libera
Francesco Paissan
Cem Subakan
Mirco Ravanelli
Venue: NeurIPS 2025 poster ()
Keywords: Speech coding
Discrete representations
Vector quantization
Generative models
Primary area: deep_learning

Abstract: Large language models have revolutionized natural language processing through self-supervised pretraining on massive datasets. Inspired by this success, researchers have explored adapting these methods to speech by discretizing continuous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, an efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resynthesis and voice conversion at lower bitrates than the current state-of-the-art, while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling. Demo samples and code are available at https://lucadellalib.github.io/focalcodec-web/.

Fodel:odec: Low-Bitrate Speech Coding via Focal Modulation Networks
Luca Della Libera 1.2 Francesco Paissan 3.2,4 *Concordia University MMila-Quebec Al Institute FFoddazione Bruno Kessler
Cem Subakan
Mirco Ravanelli 1.2
$University of Trento
Université Laval
Abstract
Large language models have revolutionized natural language processing through self-supervised pretraining massive datasets. Inspired by this success, searchers have explored adapting these methods to speech by discretizing contin- uous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resyn- thesis and voice conversion at lower bitrates than the current state-of-the-art while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling. Demo samples and code available at https::/lucadelllaib. htwpaaiibbiibubbbooooocccoo web/
Introduction
17 have led significant progress in natural language processing, enabling breakthroughs in tasks such summarization, translation, question answering, code generation, and retrieval. Building on this success, the research community has extended these methods to other modalities, with speech emerging as major area of interest, The impressive performance text-conditioned audio and speech generation models
Recent advancements in large language models 46 10 26
along with recent speech language models 85 22 16 45), highlights the potential of token-based approaches for speech processing.
A key component of these pipelines is the neural audio codec, which compresses speech into tokens that downstream models can process. These tokens must preserve acoustic and semantic information to ensure effective representations for downstream tasks while maintaining high reconstruction quality, Another important requirement a low token rate, As sequence length increases, capturing tong-term dependencies becomes more challenging, and computational costs increase.
Despite recent progress, current codecs still face several challenges. Acoustic codecs 14
34
25
76
achieve high-quality reconstruction but often rely on multiple codebooks, adding complexity to the design of downstream models. Additionally, they typically lack strong semantic representations. Hybrid codecs [86] (37] [6] (8) aim to combine both acoustic and semantic information while maintaining high-quality resynthesis. Still, they often depend complex multi-codebook designs, explicit disentanglement, distillation losses, or supervised fine-tuning. Single-codebook designs [36]
"Coresponnence to: laee
concordia.ca
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Discriminator
Focal Downseale
Transformer
Focal Upscale
Inverse STFT
ConvNet
Quantizer
ConvNeXt
Binary
Compressor
Decompressor
Encoder
Decoder
Figure 1: FocalCodec architecture. The encoder extracts features containing both acoustic and semantic information. These features are then mapped to a low-dimensional space by the compressor, binary quantized, and projected back by the decompressor. The decoder resynthesizes the waveform from these features.
21 25 76 74] offer a simpler architecture but struggle balance compression while maintaining both reconstruction quality and effective representations for downstream tasks, especially at low bitrates. To address these limitations, introduce FocalCodec, an efficient low-bitrate codec based focal modulation [8]] that compresses speech into the space of a single binary codebook. FocalCodec achieves competitive performance in reconstruction lower bitrates than the current ttate-of-the-are under variety of conditions while also preserving sufficient semantic and acoustic information for downstream tasks.
Our contributions are as follows:
We introduce FocalCodec, novel hybrid codee featuring a compesso--quatize--ccomssss architecture that compresses speech using a single binary codebook at ultra-low bitrates (0.16 0.65 kbps).
We propose focal modulation- architecture with strong inductive biases for speech, offering an efficient and scalable solution for tokenization.
We demonstrate the versatility of FocalCodec through comprehensive evaluations of recon- struction quality and performance in downstream tasks, highlighting its potential for both discriminative and generative speech modeling.
Demo samples and code are available at https://hcaaeelllaiiiiihhbbobold
2
Related Work
Acoustic Codecs. Acoustic codecs, built on the VQ-VAE [67 framework, aim for high-fidelity
reconstruction. Notable advancements include hierarchical RVQ 83 lightweight architectures [14),
improved RVQ techniques [34], and efficiency-driven designs
78
57 1. Recent methods explore
scalar quantization 41
79 Mel-spectrogram discretization 4 and novel paradigms like diffusion-
and flow-based decoding 75
80
49 To reduce bitrate without compromising performance, multi-
scale RVQ [63] (2) achieves improved compression by varying frame rates deeper quantizers. However, its hierarchical design adds complexity to downstream applications, as it requires flattening the token sequences. Single-codebook designs (36] 21 25 76 have emerged as a simpler, efficient alternative, delivering robust performance at low bitrates. Our codec aligns with this trend, leveraging novel focal modulation architecture and pretrained self-supervised encoder efficiently unify semantic and acoustic representation learning.
Semantic Codecs. Semantic codecs leverage self- superviede features from large models trained with contrastive (3 or predictive [33.9 objectives and k-means clustering (39 for quantization, either from single layer [50 71 or multiple layers
61). Improvements upon this paradigm include replacing k-means with RVQ 24 90 20 70, noise-aware 42) and speaker-invariant tokenization [ While these approaches effectively capture linguistic and content-related information, they often discard much of the acoustic detail, resulting in low speaker fidelity when vocoder is trained

---

to resynthesize speech directly from these representations. To improve reconstruction quality, 990 20 77 incorporate continuous embeddings capture prosody and speaker traits. However, this defeats the purpose of using speech tokenization for unified semantic and acoustic modeling. In contrast, our codec adopts self-supervised architecture similar to semantic codecs but preserves both semantic content and acoustic detail through its decoupled training strategy, ensuring high-quality reconstruction while preserving the advantages of semantic representations.
design and
Hybrid Codecs. Hybrid codecs combine semantic and acoustic features to balance reconstruction quality and content representation. Some methods [28
employ multiple codebooks to disentangle speech into distinct subspaces, such as content, prosody, and timbre, while others [37] utilize dual encoders to separately capture content and fine-graine acoustic information. Semantic distillation 866 [6] has also been explored to enrich the first RVQ Q codebook with semantic information from HuBERT (2 and WavLM (9) More recently, Parker et al. [8]] trained large-scale transformer- based VQ-VAF, achieving exceptional reconstruction quality at ultra-low bitrates. To enhance semantic content, they employed supervised fine-tuning on force-aligned phoneme data. Our codec also belongs to this category but instead of relying on complex multi-codehook designs with explicit disentanglement, distillation losses, or supervised fine-tuning, is purely based on self-supervised learning. It compresses both semantic and acoustic information into a single codebook, pushing the boundaries of hybrid codec design at low bitrates.
27 88
3 FocalCodee
3.1
Architecture
The proposed codec is largely based on the VQ- VAE framework but incorporates compressor and decompressor modules between the encoder and decoder (see Figure TT. The discriminator is used only during training and is discarded afterward.
Encoder. To build hybrid codec with simple design, without relying on distillation losses or multiple encoders, the encoder must capture both acoustic and semantic information. This ensures high-quality reconstructions and expressive tokens for training downstream models. Self-supervised models like HuBERT and WavLM retain significant acoustic information in their lower layers (9). making them suitable for hybrid codecs. For instance, Baas et al. 22 show that high-quality vocoder can be trained using continuous representations from layer-6 of WavLM-large.. Following this approach, we use the first layers of WavLMMl as our encoder. However, effective quantization is critical for approximating continuous representations with sufficient granularity. Standard k-means clustering typically fails to preserve essential acoustic details [68]. To address this, we introduce compressor-quantien teuuuuniieerdeoooreeepooo design based on focal modulation, which allows for granular quantization that preserves both semantic and acoustic information.
Compressor. The compressor maps the encoder representations to a compact, low-dimensional latent space. Optionally, can perform temporal downsampling to further reduce the frame rate. Prior work typically relies on convolutional, recurrent, transformer- based architectures for compression. In contrast, we introduce a novel focal downscaling module, which combines a downscaling operation with focal block. The downscaling step applics linear projection compress the feature dimension, while a ID convolution can be used instead to additionally downsample along the time dimensions. To better capture periodic patterns, follow 24 and apply Snake activations [89 after the projection.
To build a focal block, we replace the self self-attention mechanism the standard transformer block with focal modulation. Focal modulation [8]] is an efficient alternative to self attentioo that enables fine- to-coarse modeling and introduces useful inductive biases such as translation equivariance, explicit input dependency, time and channel specificity, and decoupled feature granularity. While originally designed for image and video processing, these properties also benefit speech modeling [15). Unlike self-attention, which directly computes token-wise interactions, focal modulation first aggregates the global context and then modulates local interactions based on this aggregated representation. Intuitively, self-attention mixes tokens by first computing pairwise similarities and then aggregating, which can make the result sensitive to a few high-scoring neighbors. Focal modulation inverts this
3

---

order: it first forms a compact, multi-scale summary of the input (local + global context) and then uses this summary to modulate each token. This ensures that interactions are guided by the overall context rather than being dominated by individual tokens, while avoiding quadratic cost.
Formally, focal modulation computes output representation y; for each input feature x; in sequence X1:n as:
1++1 Σ
y: - (xx, Oh
2 o gi
where q()) and h()) are linear projections, and 2 E zn. and g E gi. are the context and gating vectors position i and focal level & {1...., L 1+,,, with @ denoting element-wise multiplication. The context sequence 2i: is obtained via stack of depth-wise convolutions with increasing kernel sizes capture dependencies from short to long range, with average pooling applied the last level feature map to incorporate global information. Then, for each focal level, a point-wise convolution used to compute the gating sequence gi:.- This hierarchical approach, operating at multiple granularities, makes focal modulation well-suited for processing speech features, enabling efficient and scalable representation learning in linear time while preserving long-range dependencies.
Quantizer. FocalCodec maps latent representations from the compressor into the codebook space of single quantizer, eliminating the need for hierarchical designs in downstream models. To achieve this, while maintaining both reconstruction quality and efficiency, the quantizer should satisfy the following requirements: I) given that the original waveform is already significantly compressed into a short sequence of latents, the quantizer must compensate by using a sufficiently large codebook size to reduce the quantization error; 2) the quantizer should make efficient use of the codebook capacity, avoiding under-utilization; 3) code lookup must remain efficient, despite the increased codebook size, to ensure fast inference.
To address these challenges, we employ binary spherical quantization (BSQ) [87. originally introduced for compression of images and videos. To the best of our knowledge, this is the first successful application of binary quantization in the speech domain. BSQ belongs to the category of lookup-free quantization (LFQ) methods 41 82 i.e. it utilizes an implicit codebook, defined as:
C c- VT. VT.J VT.'
1 11"
(2)
which represents an T--dimensional hypercube projected onto unit hypersphere. The codebook size is determined by the latent representation dimension L as |C| = 2 For example, latent representations of dimension 13 correspond to a codebook size of 8192. The quantization process consists of two steps. First, the input vector v of dimension L is normalized to lie on the unit hypersphere:
(3)
|vll2
Second, binary quantization with a normalization factor of VL is applied independently to each dimension of u:
sign (u)
VT.
where sign(.) denotes the sign function, with (0 remapped to to ensure the output always lies on the hypersphere. To make the quantization differentiable, we the straight-through estimator (5).
BSQ offers several advantages over traditional quantization methods. First, the parameter-frea implicit codebook is lightweight and computationally efficient. Second, empirical evidence [87] shows that
the binary quantization bottleneck encourages high codebook utilization, even for large values of L, outperforming other lookup-free methods such as finite scalar quantization (FSQ) [41]. Third, the quantization error is bounded, resulting in faster convergence compared to vanilla LFQ, which does not normalize the representations. Finally, tying the codebook size to the latent dimension helps prevent performance degradation in downstream generative models when using larger codebooks [22
The decompressor reconstructs the encoder continuous representations from the quantizer output. closely mirrors the structure of the compressor, with the downscaling layers replaced by upscaling layers.
Decompressor.

---

Decoder. Most codecs use symmetric architectures, where the decoder mirrors the encoder. How- ever, some works 4 25 37 explore asymmetric designs with larger decoders to improve reconstruc- tion quality. In this work, we adopt an asymmetric design but prioritize the encoder, allocating ~ 5x more parameters to it than the decoder. We argue that a strong encoder is essential for extracting ro- bust, disentangled representations for downstream tasks. Even with high compression rate, smaller decoder can still generate high-quality audio while offering faster inference, which is beneficial for streaming applications. For the decoder, choose the more efficient Vocos [62] architecture over Hifi-GAN [2]. Vocos maintains consistent feature resolution and uses inverse STFT for upsampling, minimizing aliasing and improving computational efficiency. The decoder processes features through ConvNeXt (38] blocks and projects the sequence of hidden representations to Fourier coefficients for waveform reconstruction. The final audio is synthesized using inverse STFT.
Discriminator: Following HiFi-GAN 332 we employ a multi-period discriminator and a multi-
34 25], which utilize multi-resolution and/or STFT-based discriminators in place of a multi-scale discriminator. The multi-resolution and STFT-based discriminators are particularly useful for mitigating over-smoothing artifacts in freuefryyynccy components [34), which are more critical for music and environmental sounds. Since our focus is on speech (i.c. medium frequency range), we stick to the simpler HiFi- -AN setup.
scale discriminator. This approach slightly differs from prior work [83 14
62
3.2 Training
The training process consists of two stages. In the first stage, the compressor, quantizer, and decompressor are jointly trained to reconstruct the encoder continuous representations, ensuring that the tokens retain both semantic and acoustic information from the encoder, which is kept frozen. The training objective includes reconstruction loss and entropy loss. The reconstruction loss is computed the squared 1.2 distance between the reconstructed and original encoder features. The entropy loss, defined as in 82 (87, encourages both confident predictions and uniform code utilization. Note that we omit the commitment loss used in standard VQ, as for BSQ there is no concern of embedding divergence (quantization error is bounded).
In the second stage, the decoder is trained to resynthesize audio from the encoder continuous representations. The training objective includes adversarial loss, reconstruction loss, and feature matching loss, in (32]. However, following (83], we use hinge loss formulation instead of least squares. The reconstruction loss is computed as the L1 distance between the reconstructed and original log-Mel spectrograms, while the feature matching loss is the mean of the distances between the I-th feature maps of the k-th subdiscriminator.
This design allows the second stage to run in parallel with the first, simplifying the training setup. At inference, the same decoder operates on dequantized features produced by the compressor-quantize decompressor pipeline. Because the decompressor is trained to reconstruct the original continuous features from the discrete codes, these dequantized features closely approximate the originals. As result, the decoder maintains strong performance without requiring any additional fine-tuning.
when using dequantized features as input,
This decoupled training approach ensures that both semantic and acoustic information are preserved the tokens, which is crucial for downstream tasks while maintaining high reconstruction quality. trained end-to-end without additional constraints on the hidden representations (e.g. distillation loss), the reconstruction loss prioritizes acoustic features, as observed in 14 34
4 Experiments
4.1 FocalCodec
We train FocalCodec on LibriTTS (84) resampled to 16 kHz. We train three variants of the model with codebook size of 8192 and token rates of 50 Hz, 25 Hz, and 12.5 Hz by adjusting the temporal downsampling factors in the compressor layers to (1, 1, 1). (2, 1, 1), and (2, 2, 1), respectively. These patterns are mirrored in the decompressor layers for upsampling. Information about hyperparameters and training details can be found in Section E.
5

---

4.2 Baselines
We compare our models to recent state-of-the-art low-bitrate codecs across acoustic, semantic, and hy- brid categories. Since this paper fo- cuses on low-bitrate codecs, when multiple quantizers are available, we configure them to achieve a bitrate below 1.50 kbps, ensuring a fair comparison. For acoustic codecs, we compare against En-
Table 1: Codecs considered in our experimental analysis.
Codec
Bitrate (kbps)
Sample Rate (KHz)
Token Rute
Codchooks
Code Size
Params (M)
MACs
(G)
EnCodee
1,50
24
75.0
2 x 1024
128
15
DAC
1,00
50.0
x 1024
8
74
WayLM6.
0.45
1024
Speech Takenizer
1,00
16
50.0
x 1024
1024
108
17
SommatiCodoe
0.65
25.0
x 8192
1536
1033
1599
Mimi
256
Wav
0.48
24
40.0
4096
512
85
BigCodoe
1.04
80.0
x 8192
8
160
61
Codec [4]. D (34, WavTo-
Stable
0.70
15625
kenizer 25 and CigCodec (76).
FocalCodee 69550
0.65
16
50.0
x 8192
13
142
FocalCodec @:25
0.33
25.0
8192
13
144
Among these, BigCodec is the cur- rent state-of-the-an for low-bitrate
FocalCodee
0.16
speech reconstruction quality (74]. We use the official checkpoints for these models. We do not
include the recent TS3-Codec (74), which matches BigCodec performance at an even lower bitrate, as it is not publicly available. However, we contacted the authors to request reconstructed samples for comparison. Additional results related to IS3-Codes can be found in Section G.1
For semantic codecs, we adopt the approach introduced in [71), which quantizes layer-6 representa- tions from ---are using k-means clustering with 512 centroids. These representations are fed into a Conformer [19] encoder to reconstruct continuous representations, followed by HiFi-GAN decoder. This baseline, referred to as M6.M provides a direct comparison between our codec and another model leveraging WavLM layer-6 features but differing in design and training methodology. Since the code and checkpoints for WavLM6-KM are not publicly available, we reimplemented the model using subset of LibriSpeech 447. Note that we do not include additional baselines from this category, as semantic codecs typically underperform in terms of reconstruction quality [48] require much higher bitrates to be competitive in this regard [43. Furthermore, most hybrid codecs are already built on top of semantic representations. Therefore, we prioritize the hybrid category. to which our codec also belongs. For hybrid codecs, we compare against SpeechTook enizer [86. SemantiCodec 37.. Mimi [6), and Stable Codee (48), using their official checkpoints. The configurations and details of each model summarized in Table [ Multiply-accumul operations per second (MACs) are measured using p Additional information about the baselines is provided in Section D
4.3 Speech Resynthesis
We evaluate FocalCodec on speech resynthesis, considering both English and multilingual speech. For English speech, we use LibriSpeech [47] test-clean. For multilingual speech, following (76). we randomly select 100 utterances from each of the 7 foreign languages in Multilingual LibriSpeech [51] (Dutch, French, German, Italian, Polish, Portuguese, and Spanish), resulting in a total of 700 5 We also consider the more realistic scenario of speech contaminated with environmental noise. For this, we use the test splits of VoiceBank (66] and the more challenging LibrilMix, which is constructed by mixing clean utterances from the first speaker of LibriMix [12 with noise from WHAM! 73
We evaluate the models using objective metrics. To measure naturalness, we employ UTMOS (59 for clean speech and DNSMOS (56) for noisy speech. Note that we do not include signal-level metrics such as SNR, PESQ 58 or STOI (64,, as these metrics do not correlate well with perceived reconstruction quality 48, 71). To evaluate speaker fidelity, we compute the cosine similarity (Sim) between speaker embeddings extracted from the reconstructed audio and the target audio. These embeddings are obtained using WavL.M-bas--S differential word error rate (dWER) [2), which measures the difference in word error rate between
99. To assess intelligibility, we compute the
Note that we use the 16 kHz checkpoint, whereas the original results in 34 use the 24 kHz checkpoint.
https///eandoommm//cco/////11111 https:/htugginghccccc//mmorsssst bassss

---

Table 2: Speech resynthesis performance across clean and noisy datasets.
Codee
Bitrate (kbps)
UTMOS
dWER
Sim
Code Usage
Norm. Entropy
RTF
DNSMOS
dWER
Sim
Usage
Code
Norm. Entropy
RTF
1ibrShherch test-clion
Noley VoiceBout
Reference
4.09
0.00
100.0
3.56
0.000
100.0
EnCodes
1.50
8.05
93.8
93.4
82.1
109
2.76
28.16
87.7
77.5
78.1
1,00
20.04
98.7
WavLM6- KM
0.45
3.75
6.20
90.0
26.1
95.1
85
3.06
20.67
82.9
25.8
92.3
Speech Tokenieee
1.00
2.28
5.14
91.6
95.9
97.0
63
2.74
34.51
82.2
88.1
B8.4
42
SemaniCadee
0.65
2.91
8.97
96.0
75.9
94.4
0.62
3.13
31.46
90.6
52.4
92.6
0,78
Way Tokenizer
0,48
11 15
95.4
100,0
96.7
181
3.09
42,12
89.8
94.8
94.0
BigCldee Stable Codeee
1.04
4.11
2.55
98.5
100.
98.6
22
3.19
20.67
92.3
99.8
96.8
17
0.70
4,32
4.97
94.7
98.5
94.7
103
3.33
20.32
88.8
75.7
95.4
FocalCodee
2.18
97.4
100.0
91.3
98.0
96.2
Focal odee 6925
0.33
4.14
3.30
96.3
99.8
98.4
195
3.17
11.25
90.1
89.6
96.0
FocalCodee @ 12.5
0.16
4.22
7.94
93.9
98.2
97.4
208
3.22
27.97
84.7
77,3
95.5
Multtlinuull FibriSpoonk
Noday LiheilMix
Reference
2.84
0.00
100.0
3.73
0.00
100.0
EnCodez
1,50
1.33
29.66
95.5
93.4
79.2
140
2.40
55.17
86.3
84.4
78..7
97
DAC
1,00
1.24
56.08
89.1
100.0
90.0
97
2.40
90.92
76.6
99.1
88.8
WayLM6
26.8
Speech Tokenizee
1,00
1.55
56.32
92.0
96,1
91.0
74
2.58
57.26
82.8
93.5
96.5
SemamiCade
0.65
1.87
36.21
97.7
76.4
94.7
0.74
2.67
51.18
89.9
64.7
9K0.8
91
Mimi
0.69
2.08
30.96
96.7
95.9
89.0
239
2.65
19.14
89.4
90.8
90.1
96.4
BigCodee
1.04
2.86
15:24
99.1
100,0
97.9
24
2.75
$3.26
83.3
100.0
98.2
Stable Codee
0.70
3.47
56.99
95.9
92.9
93.8
144
2,91
43.52
90.0
95.8
93.4
ForalCodee
0.65
12.57
100.0
91.6
100.0
98.5
FocalCodee
0.33
19.78
99.2
97.4
292
90.7
99.6
97.9
FocalCodec 12.5
0.16
3.37
54.15
95.2
96.4
96.9
296
2.92
42.59
88.9
97.2
97.2
164
the reconstructed and target audio, using transcriptions from Whisper small [53]. To ensure fairness in evaluation, we do not use more powerful ASR models (e.g. Whisper large-v3), as these models correct pronunciation mistakes and are more robust noise, potentially hiding flaws in the reconstruction. We also report code usage, i.e. the ratio of unique tokens used to the codebook size (averaged over codebooks for multi-codebook models), and normalized entropy [13 48), where higher values indicate more uniform codebook usage. For inference speed, we measure the real-time factor (RTF), i.e. the ratio of the reconstructed audio duration the processing time. An RTF greater than indicates thatee--haa-rralll of memory.
performance, measured on an NVIDIA V100 GPU with 32 GB
Results are presented in Table 2 FocalCodec shows strong performance across both clean and noisy speech resynthesis tasks. On clean speech, FocalCodec @050 achieves the best trade-off of quality, in- telligibility, and efficiency. Notably, FocalCodec is the best terms of dWER, surpassing BigCodec, which is currently state-of-the-art. It also generalizes well to multilingual speech, obtaining the lowest dWER and high Sim. Note that FocalCodec, WavLM66KM, SpeechTokenizer, BigCodec and Stable Codec were trained exclusively on English speech. In noisy speech resynthesis, FocalCodec again excels, achieving the lowest dWER by a large margin on both VoiceBank and Libri I Mix, while maintaining high speaker similarity. Meanwhile, FocalCodec @ 25 and FocalCodec @ 12.5 exhibit some degradation in dWER and speaker similarity, particularly in multilingual settings, due to their significantly lower bitrates. Nevertheless, despite operating just 0.16 kbps. FocalCodec@ 12.5 remains competitive with several baselines that use much higher bitrates (c.g. EnCodec). It is also worth noting that FocalCodec's UTMOS tends increase at lower bitrates, likely due to the stronger smoothing effect introduced by downsampling. However, UTMOS tends to saturate and may not fully capture perceptual quality [59]. dWER and Sim are therefore essential to provide a more comprehen- sive evaluation. Finally, the high code usage and normalized entropy across all FocalCodec variants indicate efficient token utilization, contributing to their strong overall performance. Additional results on reconstruction quality, including subjective evaluations, streamability and Mel-spectrogram
analysis, can be found in Sections G.2 to G.4
4.4 Voice Conversion
We conduct one-shot voice conversion experiments verify that FocalCodec can effectively dis- entangle speaker information from content despite its single-codebook design. This task involves
7

---

converting speech from a source speaker to an arbitrary target speaker using ref- erence speech from the target speaker. For single-codebook baselines, includ- ing FocalCodec, we use k-nearest neigh- bors search the codec feature space, as in (2). Specifically, we replace each frame the reconstructed feature se- quence (right before the decoder) with the average of the = closest matches in terms of cosine distance from con- tinuous features extracted from the ref- erence. For multi-codebook baselines, instead, we follow the procedure [86]. The source and reference speech are kenized, and the first codebook tokens from the source are concatenated with the second-to-last codebook tokens from the reference. The resulting sequence then forwarded to the decoder. If sequence lengths differ, the reference is truncated or circularly padded as needed. Effective disentan- glement of content and speaker information between first and subsequent codebooks is expected to yield fair voice conversion performance. We conduct voice conversion experiments on VCTK (77). which includes parallel utterances from different speakers. To create the test set, we randomly select an utterance from source speaker, the corresponding utterance from a target speaker, and an utterance with different content from the same target speaker to act as the reference. Among available reference utterances, we select the longest to minimize padding issues. We repeat this process for each speaker, for each of the ~44 parallel utterances, resulting a dataset with 2521 samples. To evaluate performance, we use UTMOS, dWER, Sim, and RTF as defined in Section 4.3
Table 3: One-shot voice conversion on VCTK [77].
Bitrate (kbps)
Codec
UTMOS
dWER
Sim 1 RTF
Reference
4.09
0.00
100.0
EnCodec
1.50
1.24
86.52
72.2
57
1.00
1.25
104.00
67.2
60
WavLM6-KM M6-KM
0.45
2.90
26.68
92.4
57
Speech Tokenizer
1.00
1.49
20.32
81.2
33
SemantiCodec
0.65
2.02
106.00
72.8
0.60
Mimi
0.69
2,40
110.10
89.7
71
Wav Tokenizer
0.48
3.13
43.15
73,4
89
BigCodec
1.04
1.31
99.96
68.9
13
Stable Codec
0.70
3.76
27.63
71.1
65
FocalCodee 6050
0.65
3.38
21.27
92.2
116
FocalCodec @25
0.33
3.40
23.59
92.6
118
FocalCodec @ 12.5
0.16
3.43
29.93
92.6
117
As reported in Table 3 FocalCodec achieves the highest speaker similarity while maintaining good intelligibility, confirming its suitability for voice conversion tasks. This is particularly impressive, especially compared other hybrid codecs like SpeechTokenizer and Mimi, which are explicitly optimized to disentangle semantic information in the first codebook and acoustic information in the following. Despite this, FocalCodec outperforms these models, excelling in both speaker identity preservation and intelligibility, striking remarkable balance of quality. efficiency, and speaker similarity. WavLM6 KM ranks as the second-best performing model, which is expected since shares the same encoder as FocalCodec. In contrast, acoustic codecs struggle with this task, as they do not separate speaker and content information.
4.5 Downstream Tasks
To evaluate the quality of the learned discrete representations, we train downstream models on both discriminative and generative tasks.
Discriminative Tasks. We evaluate performance on automatic speech recognition (ASR), speaker identification (SI), and speech emotion recognition (SER). These tasks allow us to assess token quality along three axes: semantic information retention (ASR), acoustic information retention (SI), and emotion information retention (SER, which requires a non-trivial combination of semantic and acoustic clues). To focus the disentanglement of learned representations, we employ shallow downstream models, aiming to stay as close as possible to linear probing. Following [86], we employ a shallow BiLSTM for all tasks. For ASR, we use LibriSpeech [47] brain-clean-100
and train-clean 360 for training. dev-clean for validation, and test-clean for testing. The word error rate (WER) is reported. For SI, we also use LibriSpeech, grouping utterances from train-clean-100 100 and train-cai caann-ceeenn-666 by speaker ID. Data are randomly split into training, validation and test sets in a ratio of 80% / 10% 10%. The speaker error rate (ER) is reported. For SER, we use the IEMOCAP dataset (7) focusing on four emotions: sadness, happiness, anger, and neutral. Sessions 1-4 used for training, session 5F for validation, and session 5M for testing. The emotion ER is reported. Details about the model architecture, hyperparameters, and training procedure are provided in Section E.2
Table shows the results. In ASR, FocalCodec @ 50 achieves the third lowest WER. While SpeechTo- kenizer and Stable Codec perform slightly better, the former operates at 1.5x higher bitrate using

---

Table 4: Evaluation on discriminative and generative downstream tasks.
Discriminative Tasks
Generative Tasks
Codec
(kbps)
ASR
SI
SER
||
SE
SS
TTS
WER
ER
|DNSMS
dWER
Sim
DDNSMS
dWER
Sim
UTMOS
dWER
Sim
Reference
3.56
0.00
100.0
3.77
0.00
100.0
4.09
0.00
100,0
RaCodee
1.50
27.89
47.00
3.11
57.10
85.9
3.11
78.55
87.3
64.28
83.2
DAC
1.00
35.89
3.27
45,90
3.03
67.65
81.7
2.76
106.00
83.3
1.34
47.05
85.9
Wa-LMM6KM KM
0.45
19.04
22.30
42.90
3.52
22.85
83.6
3.49
76,91
85.0
3.74
38.67
88.7
SpeachTokkeiizer
2.73
41.50
83,99
87.3
89.2
SonantiCodoe
0.65
41.42
15.90
51.60
3,59
102.00
83.5
3.59
123.00
84.4
2 2.82
48.38
91.4
Mimi
0.69
22.98
5.43
44.70
3.30
53.98
84.6
3.41
95.23
1.11
28.63
93.6
WayTokenizer
0.48
35.62
2.44
49.80
3.41
51.75
88.6
3.54
105.00
86.4
3.68
47.56
92.8
BigCodoe
1.04
26.41
2.34
47.50
3.52
26.68
93.2
3.54
89.24
89.4
3.43
54.43
89.4
Stable Codoc
16.85
16.54
103.00
78.2
49.28
88.8
FocalCodec 6i
0.65
17.63
4.48
45.60
3.47
10.93
91.4
3.71
73.87
89.0
4,11
28.
93.3
FocalCodee 0:25
14.74
90.0
85.4
91.6
FocalCodee @8 22.5
0.16
33.24
11.69
46.30
3.58
36.98
86.9
3.57
116.00
80.8
4.12
21.59
90.8
two codebooks, while the latter was fine-tuned on force-aligned phoneme data enhance semantic representations. In contrast, our model is purely self-supervised. In SI, FocalCodec @ 50 achieves a marginally higher error rate (~2%) than codecs such as BigCodec and WavTokenizer. However, these models perform significantly worse in ASR due to being trained solely with reconstruction- objectives. On the other hand, the purely semantic WavLM-KMM codec performs competitively in ASR but exhibits the highest ER in SI despite using the same encoder as FocalCodec. This further confirms the effectiveness of our codec design, as improves WER over WavLM-KM6 while preserving speaker information. Interestingly, Stable Codec also performs poorly in SI, likely because semantic fine-tuning tends to remove acoustic information from the representations. In SER, no codec clearly excels, with FocalCodec @ 50 performing on par with the best models. Overall, Focal- Codec @ 50 shows competitive performance across all discriminative tasks, rivaling hybrid codecs with more complex multi-codebook designs and higher bitrates. The more compressed variants, FocalCodec@*) 25 and FocalCodec@) 12.5, still achieve good performance while operating at ultra-low bitrates.
Generative Tasks, We evaluate performance on speech enhancement (SE), speech separation (SS), and text-to-speech (TTS). For these tasks, employ more powerful raasformer-baaee downstream models, focusing on generation quality. For SE we use VoiceBank [66. To form a validation set, we randomly select two speakers from the training set. The input tokens are extracted from noisy utterances, while the target tokens come from clean utterances. Performance metrics include DNSMOS, dWER, and Sim. For SS, use Libri2Mix (12 train-100, dev, and teat sets. The setup mirrors that of speech enhancement: input tokens are derived from speech mixtures, while target tokens correspond to the two individual sources. For TTS, we use LibriSpeech train-cla caan-clea---ee 100 and train- brain-clead clean- --lea-3660 -360 for training, clean for validation, and test-clean for testing. Note that fest-cleam contains several utterances longer than 20 seconds (~4%), whereas our training splits include almost none. To reduce the mismatch between training and testing conditions, we removed these long utterances from the test set. The input consists of character-bauee text tokens, while the target tokens are derived from the corresponding utterances. Performance is evaluated using UTMOS, dWER, and Sim. Details about the model architecture, hyperparameters, and training procedure are provided in Section E.2
From Table we observe that in SE, FocalCodec@55 50 significantly outperforms all other baselines in terms of dWER. A similar trend is observed for SS, where FocalCodec @ 50 is consistently superior to the other baselines. However, the absolute performance is still far from practical utility, likely due to the loss of information crucial for SS during quantization. As with discriminative tasks, FocalCodec @25 and FocalCodec @ 12.5 show degraded performance, due their ultra-low bitrates. However, this trend is reversed for TTS, with FocalCodec @ 25 achieving the best overall results, followed closely by FocalCodec @ 12.5. This can be attributed to the fact that, in autoregressive modeling, shorter sequences reduce the computational burden and simplify the task of predicting the next token. Both models, operating at frame rate closer to that of text with a single codebook, make next-token prediction easier and more computationally efficient than other methods. This highlights the importance of having compact representations for downstream tasks. Note, however, that we trained on only 460 hours of speech, which explains why TTS performance is not tftte-.f-

---

4.6 Ablation Studies
Due to limited computational re- sources, we perform ablation stud- ies on a smaller variant of Focal- Codec. This variant is similar to the 50 Hz model, with the main dif- ference being the model size, as de- tailed in Section E.. We focus on the clean speech resynthesis task us-
Table 5: Ablation studies LibriSpeech test-cles clnn
47
Compression Block
Downscale Activation
Quantizer
UTMOS
dWER
Sim
Focal modulation
Snake
BSQ
3.73
2.54
95.7
Focal modulation
Snake
FSQ
3.71
2.61
94,8
Focal modulation
Snake
LFQ
3.74
2.75
95.4
Focal modulation
Leaky ReLU
LFQ
3.72
2.85
95.2
Conformer
Snake
LFQ
3.74
3.58
94.3
ing LibriSpeech test-clean [47].
AMP
Snake
LFQ
4.52
Linear
Snake
L.Q
2.55
9.37
82.5
The results shown in Table [
Replacing BSQ with FSQ [41] leads to worse UTMOS, dWER, and Sim. It also results in less
uniform code usage, evidenced by the normalized entropy measured for these two configurations (99.7 for BSQ vs. 97.7 for FSQ). Replacing BSQ with vanilla LFQ results in worse dWER and Sim despite similar UTMOS. Replacing Snake activations with leaky ReLU causes only minor performance degradation. The most significant performance drop occurs when the focal modulation blocks are replaced with Conformer [19] blocks, anti-aliased qulli-periodiccty (AMP) [35 blocks, or linear layers, in this order. This leads to notable decrease in both dWER and Sim. This analysis further validates our design choices, highlighting the importance of the selected components for achieving optimal performance.
5 Conclusions
In this work, we introduced FocalCodec, a low-bitrate single-codebook speech codec that employs novel architecture based on focal modulation. It delivers competitive performance in speech resynthesis and voice conversion at low and ultra-low bitrates while maintaining robustness across diverse conditions, including multilingual and noisy speech. Furthermore, it effectively preserves both semantic and acoustic information, providing powerful discrete representations for downstream tasks. detailed discussion of the limitations and societal impact of this work provided in Sections ^ and B
6 Acknowledgments
We acknowledge support from NSERC, the Digital Research Alliance of Canada (alliancecan. ca), the NVIDIA Academic Grant Program for computing resources, and Translated for funding through the Immediate research grant.
References
(1] Y. Ai, X.-H. Jiang. Y.-X. Lu, H.PP. Du, and Z.-H. Ling. APCodec: neural audio codec with parallel amplitude and phase spectrum encoding and decoding. JEEE/ACM Transactions on Audio, Speech and Language Processing, 32:3256-3200 2024.
[2] M. Baas, B. van Niekerk, and H. Kamper. Voice conversion with just nearest neighbors. In Interspeech. pages 2053-2057, 2023.
[3] A. Bacvski, Y. Zhou, Mohamed, M. Auli. wav2vee 2.0: framework for self-supervised learning of speech representations. International Conference on Neural Information Processing Systems (NeurIPS), pages 12449-12460, 2020. 2020.
[4] H. Bai. T. Likhomanenko, R. Zhang. Z. Gu. Z. Aldench, and N. Jaitly, dMel: Speech tokenization made simple. arXiv preprint arXiv:2407. 15835, 2024.
Y. Bengio, N. Léonard, and Courville. Estimating or propagating gradients through stochastic neurons conditional computation. arXiv preprint 1r08v:1308.343: 2013.
[6] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. AudioLM: A language modeling approach to audio generation. JEEE/ACM Transactions Audio, Speech and Language Processing, 31:2523 2533, 2023.
10

---

[7] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower. S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335-359, 2008.
[8] H.-J. Chang. H. Gong, C. Wang, J. Glass, and Y.-A. Chung. DC-Spin: A speaker-invariant speech tokenizer for spoken language models. arXiv preprint arXiv:2410.24177, 2024.
[9] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, pages 1505-1518, 2022.
[10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, et al. PaI.M: scaling language modeling with pathways. Journal of Machine Learning Research, 24, 2024.
[11] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Defossez. Simple and controllable music generation. In International Conference on Neural Information Processing Systems (NeurlPS), volume 36, pages
2023.
[12] Cosentino, M. Pariente, Cornell, A. Deleforge, and Vincent. LibriMix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005. 010220055.1126 11262, 2020.
[13] T. M. Cover and A. Thomas. Elements of information theory. Wiley Wiley-Interscience. 2006.
[14] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression. Transactions on Machine Learning Research (TMLR), 2023.
[15] L.. Della Libera, C. Subakan, and M. Ravanelli. Focal modulation networks for interpretable sound classification. In IEFE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pages 853-857, 2024.
[16] A. Défosscz, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour. Moshi: specch-text foundation model for real-time dialogue. arXiv preprint 27X00:2410.00037, 2024.
[17] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle. A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang. A. Fan, et al. The Llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
[18] A. Graves. S. Fernández, Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In International Conference on Machine Learning (ICML), pages 369-376, 2006.
[19] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution 2020.
Convolution-augmented augmented transformer for speech recognition. In Interspeech, pages 5036-5040,
[20] H.-H. Guo, Y. Hu, K. Liu, F--Y. Shen, X. Tang, Y.-.. Wu, F.-L. Xie, K. Xie, and K.-T. FireRedTTS: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.00 0028830,300220002 2025.
[21] Y. Guo, Z. Ii, C. Du, H. Wang, X. Chen, and K. Yu. LSCodee:: Low-bitrate and speaker-decoupled discrete speech endec. arXiv preprint 24.iv:2410.15770 2024.
[22] M. Hassid, T. Remez, T. A. Nguyen, Gat, A. Conneau, F. Kreuk, J. Copet, A. Défossez, G. Synnaeve, E. Dupoux, R. Sehwartz, and Y. Adi. Textually pretrained speech language models. In International Conference on Learning Representations (ICLR), 2023.
[23] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. HuBERT: Self- supervised speech representation learning hy masked prediction of hidden units. IEEF/ACM Trans. Audio Speech Lang. Process., 29:3451-3466 2021.
[24] Z. Huang, C. Meng, and T. Ko. RepCodec: A speech representation codee for speech tokenization. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 5777-5790, 2024.
[25] S. Z. Jiang, W. Wang, Y. Chen, M. Fang, J. Zuo, Q. Yang. X. Cheng, Z. Wang R. Li. Z. Zhang, X. Yang. R. Huang. Y. Jiang, Q Chen, S. Zheng, W. Wang. and Z. Zhao. Wav Tokenizer: An efficient acoustic discrete codec tokenizer for audio language modeling. In International Conference on Learning Representations (ICLR), 2025.

---

[26] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamffrdd D. S. Chaplot, D. de las Casas, and E. B. H. others. Mixtral of experts. arXiv preprini arXiv:2401. 00400888 2024.
[27] X. Jiang. X. Peng. Y. Zhang, and Y. Lu. Universal speech token learning via www-bitrtt neural codec and pretrained representations. IEEE Journal of Selected Topics in Signal Processing, pages 1-13, 2024.
[28] Z. Ju, Y. Wang, K. Shen, X. Tan, Xin, D. Yang, Liu, Y. Leng, K. Song. S. Tang. Wu, T. Qin, X.-Y. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao. NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.33000 2024.
[29] J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-.. Mazaré, J. Karadayi, V. Liptchinsky, R. Col- lobert, C. Fuegen, et al. Ligri-Light: A benchmark for ASR with limited or no supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673, 2020.
[30] J. Kim, K. Lec, S. Chung, and J. Cho. CLam-TTS: Improving neural codec language model for zero-shot text-to-speech. In International Conference on Learning Representations (ICLR), 2024.
[31] M. Kolbæk, D. Yu, Z. Tan, and J. Jensen. Multitalker speech separation with utterance-level permutation invariant training deep recurrent neural networks. IEEF//CM Transactions on Audio, Speech, and Language Processing, 25:1901-1913, 2017.
[32] J. Kong, J. Kim, and J. Bae. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In International Conference on Neural Information Processing Systems (NeurIPS), 2020.
[33] F. Kreuk, G. Synnaeve. A. Polyak, U. Singer, A. Défossez. J. Copet. D. Parikh, Y. Taigman, and Y. Adi. AudioGen: Textually guided audio generation. In International Conference on Learning Representations (ICLR), 2023.
[34] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-fidelity audio compression with improved RVQGAN. In International Conference on Neural Information Processing Systems (NeurIPS). 2023.
[35] S. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon. BigVGAN: A universal neural vocoder with large-scale training. In International Conference on Learning Representations (ICLR), 2023.
[36] H. Li, L. Xue, H. Guo, X. Zhu, Y. Lv, L. Xie, Y. Chen, H. Yin, and Z. Li. Single-Codec: Single-codebook speech codec towards high performance speech generation. arXiv preprint arXiv:2406.07422 2024.
[37] H. Liu, X. Xu, Y. Yuan, M. Wu, W. Wang, and M. D. Plumbley. SemantiCodee: An ultra low bitrate semantic audio codec for general sound. arXiv preprint 4rXiv:2405.00253 2024.
[38] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A ConvNet for the 2020s. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] S. P. Lloyd. Least squares quantization PCM. JEEE Trunsactions on Information Theory, pages 129-137, 1982.
[40] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICIR), 2019.
[41] F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: VQ-VAE made simple. In International Conference on Learning Representations (ICLR), 2024.
[42] S. Messica and Y. Adi. NAST: Noise aware speech tokenization for speech language models. In Interspeech 2024, pages 1169-4173 2024.
[43] P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli. DASB discrete audio and speech benchmark. arXiv preprint arXiv:2406. 14294, 2024.
[44] P. Mousavi, J. Duret, S. Zaiem, L. Della Lihera, A. Ploujnikov, C. Subakan, and M. Ravanelli. How should we extract discrete audio tokens from self-supervised models? In Interspeech, pages 2554-2558, 2024.
[45] T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-jussa, M. Elbayad, S. Popuri, P.-A. Duquenne, Algayres. R. Mavlyutov, I. Gat, G. Synnaeve, J. Pino, B. Sagot, and E. Dupoux. SpiRit-LM: Interleaved spoken and written language model. arXiv preprint 45Xiv:2402.05755 2024.
[46] OpenAI. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, J. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023. 2023.
12

---

[47] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur: LibriSpeech: An ASR corpus based on public domain audio books. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 5206-5210, 2015.
[48] J. D. Parker, A. Smirnov, J. Pons, C. Carr, 7. 7ukowwki, Z. Evans, and X. Liu. Scaling transformers for low-bitrate high-quality speech coding. In International Conference on Learning Representations (ICIR)), 2025.
[49] N. Pia, M. Strauss, M. Multrus, and B. Edler. FlowMAC: Conditional flow matching for audio coding low rates. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5, 2025.
[50] A. Polyak, Y. Adi, Copet, Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In Interspeech, pages 3615-3619 3619, 2021.
[51] V. Pratap. Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert. MLS: A large-scale multilingual dataset for speech research. In Interspeech, pages 2757-2761, 2020.
[52] K. Qiu, X. H. Chen, J. Sun, J. Wang. Z. Lin. M. Savvides, and Raj. Efficient autoregressive audio modeling via next-scale prediction. arXiv preprint arXiv:2408.0900 2024.
[53] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), volume 202, pages 28492-28518, 18, 2023.
[54] M. Ravanelli, T. Parcollet, A. Moumen, S. Langen, C. Subakan, Plantinga. Wang. P. Mousavi, L. Della Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang. G. Karakasidis, S.-L. Ych, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Valuaga-Goome S. M. Mousavi, A. Nautsch, H. Nguyen, X. Liu, S. Sagar, J. Duret, S. Mdhaffar, G. Laperrière, M. Rouvier, R. D. Mori, and Y. Estève. Open-source conversational AI with SpeechBrain 1.0. Journal of Machine Learning Research (JMLR), 25(333)11-11, 2024.
[55] M. Ravanelli. T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F, Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio. SpeechBrain: A general-purposs. speech toolkit. arXiv preprint arXiv: 100.200000000 04624, 2021.
[56] C. K. Reddy, V. Gopal, and R. Cutler. DNSMOS P.835: non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In /EEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[57] Y. Ren, T. Wang. J. Yi, L. Xu, Tao, C. Y. Zhang, and J. Zhou. Fewer-token neural speech codec with time-invann invariant codes. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12737-12741, 2024.
[58] A. Rix, J. Beerends, M. Hollier, and A. Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 749-752, 2001.
[59] T. Saeki, D. Xin. W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari. UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In Interspeech, pages 4521 4525, 2022.
[60] M. Schoeffler, S. Bartoschek, F.-. Stöter, M. Roess, S. Westphal, B. Edler, and J. Herre. webMUSHRA. comprehensive framework for web-hased listening tests. Journal of Open Research Software, 2018
[61] J. Shi, X. Ma, H. Inaguma, A. Sun, and S. Watanabe. MMM: Multi-layer multi-residual multi-stream discrete speech representation from self sulf-super learning model. In Interspeech, pages 2569-2573 2024.
[62] H. Siuzdak. Vocos: Closing the gap between time-domain and fourier- based neural vocoders for high- quality audio synthesis. In International Conference on Learning Representations (ICLR), 2024.
[63] H. Siuzdak, E. Grötschla, and I., A. Lanzendörfer. SNAC: Multi-scale neural audio codec. In Audio Imagination: NewrlPS 2024 Workshop Al-Driven Speech, Music, and Sound Generation. 2024.
[64] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen. An algorithm for intelligibility prediction of time frequency weighted noisy speech. IEEE Transactions on Audio, Speech and Language Processing. pages 2125 22166, 2011.
13

---

[65] J. Tian, J. Shi, W. Chen. S. Arora, Y. Masuyama, T. Maekaku, Y. Wu, J. Peng. S. Bharadwaj. Y. Zhao, S. Coreell, Y. Peng. X. Yue, C.-HI H. Yang. G. Neubig, and S. Watanabe. DSPnet-SpeechLMM An open speech language model toolkit. Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies (System Demonstrations,, pages 116-124, 2025.
[66] C. Valentini-Botiin ment methods for noise-robust text-to-speech. In Speech Synthesis Workshop, pages 146- 142-152, 2016.
X. Wang. S. Takaki, and J. Yamagishi. Investigating RNN-based speech enhance-
[67] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In International Conference on Neural Information Processing Systems (NeurIPS), pages 6309-6318, 2017.
[68] B. van Niekerk, M.-A. Carbonneau, J. Zaïdi, M. Baas, H. Seuté, and H. Kamper. A comparison of discrete and soft speech units for improved voice conversion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6562-6566, 2022.
[69] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. Hc, S. Zhao, and F. Wei. Neural codee language models are zero-shot text to speech synthesizers. arXiv preprint xrXiv2230.022111 2023.
[70] Y. Wang. H. Zhan, L. Liu, R. Zeng. H. Guo, J. Zheng. Q. Zhang, X. Zhang, S. Zhang. and Z. Wu. MaskGCT Zero-shot text-to-speech with masked generative codec transformer. In International Conference on Learning Representations (ICIR), 2025.
[71] Z. Wang, X. Zhu, Z. Zhang, Y. Lv, N. Jiang, G. Zhao, and L. Xie. SELM: Specch enhancement using discrete tokens and language models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11561-11565, 2024.
[72] Z.-Q. Wang et al. Sequential multi-frame neural beamforming for speech separation and enhancement. In IEEE Spoken Language Technology Workshop (SLT), pages 905-919111 905-911,,70 2021.
[73] G. Wichern, J. Antogmini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. L. Roux. WHAAM!: Extending speech separation to noisy environments. In Interspeech, pages 1368-1372, 2019.
[74] H. Wu, N. Kanda, S. E. Eskimez, and J. Li. TS3-Codec: Transformer-based simple streaming single codec. arXiv preprint arXiv:2411,18800 2024.
[75] Y.C. Wu, D. Marković, S. Krenn, I. D. Gehru, and A. Richard. ScoreDec: A phase-preserving high-fidlitty audio codee with a generalized score-based diffusion post-filter. In IEEE International Conference on Acousticx, Speech and Signal Processing (ICASSP), pages 361-365, 2024.
[76] D. Xin, X. Tan, S. Takamichi, and H. Saruwatari. BigCodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint X4X::2409005377, 2024.
[77] Yamagishi. C. Veaux, and K. MacDonald, CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit, University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
[78] D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y. Zou. HiFi-Codec, Group-residual vector quantization for high fidelity audio codec. arXiv preprint ar 205.:2300.. 2023.
[79] D. Yang. D. Wang. H. Guo, X. Chen, X. Wu, and H. Meng. SimpleSpeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models. In Interspeech, pages 4398 440284402,4400 2024.
[80] H. Yang, Jang, and M. Kim. Generative de-quantization for neural speech codee via latent difffusion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
[81] J. Yang, C. Li, X. Dai, and J. Gao. Focal modulation networks. In International Conference on Neural Information Processing Systems (NeurIPS), 2022.
[82] L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, Gong, M.-H. Yang. I. Essa, D. Ross, and L. Jiang. Language model beats diffusion tokenizer is key to visual generation. In International Conference on Learning Representations (ICLR), 2024.
[83] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. SoundStream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 495-50 507- 2021.
14

---

[84] H. Zen. V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia. Z. Chen, and Y. Wu. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. Interspeech, 2019.
[85] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang. Y. Zhou, and X. Qiu. SpeechGPT: Empowering large language models with intrinsic cross- modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP. pages 15757-15773, 2023.
[86] X. Zhang, D. Zhang. S. Li, Y. Zhou, and X. Qiu. SpeechTokenizer: Unified speech tokenizer for speech large language models. In International Conference on Learning Representations (ICLR), 2024.
[87] Y. Zhao, Y. Xiong, and Krähenbühl. Image and video tokenization with binary spherical quantization. In International Conference on Learning Representations (ICLR), 2025.
[88] Y. Zheng, W. Tu, Y. Kang, Chen, Y. Zhang, L. Xiao, Y. Yang, and Ma. FreeCodec: A disentangled neural speech codec with fewer tokens. arXiv preprint arXiv:2412.01053 2024.
[89] L. Ziyin, T. Hartwig, and M. Ueda. Neural networks fail learn periodic functions and how to fix it. In International Conference on Neural Information Processing Systems (NeurIPS). 2020.
[90] M. Fajszezak, G. Cámbara, Y. Li, Beyhan, A. Korlaar, F. Yang. A. Joly, Álvaro Mortin-Cortinas
A. Abbas, A. Michalski, A. Moinet, Karlapati, Muszyúíka, H. Guo, Putrycz, Gambino. Yoo.
E. Sokolova, and T. Drugman. BASE TTS: Lessons from building a billion-paramete text-to-speech model on 100k hours of data. arXiv preprint 2rXiv 08093, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section 4
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specificnn specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their hest judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
16

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: see Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: see the project page at http:://ttaallllagggghhhhohoooe. and Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cct cc/pulicc/
guides/Codeeuunnt seiobboicy for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nniss//..//////////e:::eeee cc/public/ qu..dess/ooeeSuubllesssoool..... for more details.
17

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see Section C and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification; we follow common practice the speech processing community, particularly the audio codec and self-supervised representation literature, where large performance differences and the use of established test sets are considered sufficient to indicate significance. In line with prior work, we do report error bars explicitly, obtaining them would require substantial computational resources.
Guidelines:
The answer means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yicld results that are out of range (c.g. negative error rates).
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: see Section F
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies https: pestpa://neuriaa. httpp////eeerppa..pppblll/////// httpppallce/teaeaauaeellllllllll
Answer: [Yes]
Justification: we have reviewed the NeurlPS Code of Ethics and ensured that our research complies with its guidelines in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Section
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks
could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
19

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section ald Section
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttsttt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
20

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy should or should be described.
http:ppp..e///ooeeeerre///////// for what
21

---

A Limitations
Despite its competitive performance. FocalCodec is undertrained compared to other state-of the--ar approaches. While the WayLM encoder benefits from 94k hours of pretraining. the rest of the pipeline was trained on only a few hundred hours of clean English speech. Expanding the datasct include more data, broader range of domains (c.g. multilingual speech, mixtures, etc.) could further improve quality, robustness, and versatility of the model. By comparison, competing methods such as Way VayTokknizerr (8k hours), StableCodec (105k hours), and Mimi (7M hours) are trained on significantly larger and more diverse datasets.
B Societal Impact
We believe this research has the potential for meaningful societal benefits. Ultra-low bitrate speech codecs can significantly reduce the bandwidth and storage requirements for transmitting and storing spoken content. This has practical implications
improving the accessibility and efficiency of voice communication in bandwidth- constrained settings, such as rural or remote arcas, and for enabling on-device speech applications with minimal resouree consumption. However, we also acknowledge potential risks associated with misuse. In particular, voice conversion capabiliitis enabled by FocalCodec could potentially be exploited for malicious purposes, including voice cloning, impersonation, and the creation of deceptive harmful deepfake audio content. To mitigate these risks, we encourage responsible use of this technology and further research into detection and authentication mechanisms to ensure secure and ethical deployment. It is worth noting nevertheless that similar capabilities are already publicly accessible, with models like https//huggingffcc co/ cw/amphion/ Vevo offering few-shot voice conversion.
C Datasets
The following datasets were used in this work:
LibriSpeech 47 is large-scale corpus of English read speech derived from audiobooks in the Libri project. It contains approximately 1000 hours of speech sampled 16 klZz, with predefined training, validation, and test splits. License: CC BY 4.0.
is a corpus designed for text-to-speech research, constructed from the same source as LibriTTS X
LibriSpeech. consists of 585 hours of transcribed speech with predefined training, validation, and test
splits. License: CC BY 4.0.
Multilingual LibriSpeech [51 is extension of LibriSpeech to multiple languages, including Fnglish, German, Dutch, French, Spanish, Italian, Portuguese and Polish. It provides approximately 44,500 hours of transcribed English speech and about 6000 hours from other languages. License: CC BY 4.0.
VoiceBank [66] is dataset primarily used for speech enhancement, including 11,572 utterances from 28 speakers in the training set (noise 0 dB, 5 dB. 10 dB, and 15 dB), and 872 utterances from unseen speakers in the test set (noise at 2.5 dB, 7.5 dB, 12.5 dB, and 17.5 dB). License: CC BY 4.0.
dataset for speech separation and enhancement, created by mixing LibriSpeech utterances with noise from the WHAM! (73 corpus. It provides mixtures of two three speakers at different signal-to-noise ratios. License: MIT.
LibriMix 12
VCTK (77 is corpus of English speech recordings from 110 speakers with various accents. It is widely used for speaker adaptation, tex--to-speech, and voice conversion tasks. License: CC BY 4.0.
IEMOCAP 2 is a dataset designed for emotion recognition, consisting of scripted and improvised dialogues performed by 10 actors. It includes audio, video, and textual transcriptions with emotion labels such as happiness, sadness, and anger. License: httpss//sail.ussc.ed wwwpa///www..ee////////eeeappppp wew.aaa..ppeeeaeehhe
D Baselines
Additional information about the baseline codecs is provided in Table For our WavLM66KM reproduction, we use LibriSpeech train- clean-100 and train-cleen vrain-clean-360 First, we train k-means quantizer with 512 centroids on top of layer-6 representations from WavLM-large. We train on audio chunks of 16,000 samples with a large batch size of 512 for improved stability, and we stop training when cluster centroids stop changing significantly. Then, we train dequantizer to minimize the I.2 loss between quantized and original WayLM features. We employ Conformer [19 encoder with layers, 4 attention heads, hidden dimension of 5 512, and a -orward layer dimension of 512. We train on audio chunks 7040 samples with batch size 16. We use the AdamW (40) optimizer with an initial learning rate of 0.0005, 31 of 0.8, B2 of 0.99, weight decay of 0.01, and dropout of 1. The learning rate is reduced by factor of 0.9 if validation loss does not improve within margin of 0.0025. Gradients are clipped to maximum L2 norm of 5. Training stops when validation
22

---

Table 6: Baseline codecs.
Cunsal
Training
Muitiingual
Checkguint
PaCsin [4]
Oprinnal
DNS, Cummon Viioe, AndioNar FSDSOK, Senzrdo
17c+
Yos
General
creader 34kir
MIT
DAC
No
DAPS, DNS, Cumman VCTK, MUSDB, Jamaddo
10c+
Yos
Genzal
wrightx Incheeph
MIT
Wor N1
No
Gipdpooch, and MusPopuli English for L.sLM pectrining)
Subes of Lphnns aáliiin Light
94k)
Speech
Sscrete- Warkin codoe
Apuchi
Speech Tokenerr No
LibeiSperch
Speech
sgeechskenieer Loog
Apuche
SemmnniClodec 5
No
Gipdppoch subsci of OyenSLR, R. Million Song Dutanct. Mediey MUSDBI is Audioter W CapCaps ViLUSound
2061
Yes
General
Jemanticoder pokercate_ 50
MIT
Mine
Yos
Predominantly English speeih (io addition Libri Light, CipsSpachh, and VinPopali English for WasLM pectrining)
TM
94k)
Likely
Sperch
mim
CC 4.0
Way Tedeenias
No
LieiTTS, VCTK vabut of subiet AudieSee, Janesdo, MUSDB
General
Was Tokkeniace Sayge unily-40iokkes
BixCoded 20
No
LibriSpeech
960
No
Speech
bigcodos.pi gi
MIT
Stahic Codia: 11
Optinoll
Libe-l Muhilingsal I LihelSpooch Enghish
105k No
Sperch
StahillyA
loss does not decrease for several consecutive epochs. Finally, we train a HiFi-GAN VI chunks of 7040 samples with a batch size of 16. We use the AdamW optimizer with an initial learning rate of 0.0002, 31 0.8, B3 of 0.99, and weight decay of 0.01. The learning rate follows an exponential decay schedule with a factor of 0.999. Training continues until perceived audio quality stops improving.
B2 decoder on audio
E Hyperparameters and Training Details
E.1 FocalCodec
The compressor processes 1024-dimensional WavLM features and forwards them through 3 focal downscaling blocks with hidden dimensions of 1024, 512, and 256, respectively. Each block has two focal levels, a window size of 7. focal factor of 2, and a layer scale initialization of 0.0001. A final projection maps the 256- dimensional hidden states to latent representations dimension 13, which are then quantized with binary spherical codehook of 213 8192 codes. The decompressor mirrors the compressor, replacing focal downscaling blocks with focal upscaling blocks to reconstruct the 1024-dimensional continuous representations from the quantized latent codes. We usc weight of 1.0 for the reconstruction loss and weight of 0. for the entropy loss. We train on LibriTTS [84] (585 hours from 2456 speakers) using full utterances rather than fixed-length chunks, which differs from related work. This approach allows us to fully exploit the unlimited receptive field of focal modulation. This is in line with our vision that the encoder should be as powerful as possible to extract high-quality representations, while the decoder can be lightweight and use limited context windows. For this stage, use the AdamW 40 optimizer with initial learning rate of 0.0005, 81 of 0.8. 82 of 0.99, and weight decay of 0.01. The learning rate reduced by a factor of 0.9 if validation loss does not improve within a margin of 0.0025. Gradients are clipped to maximum I.2 norm of 5. Training stops when validation loss does not decrease for several consecutive epochs.
The decoder processes 1024-dimensional WavL. features and forwards them through 8 ConvNeXt blocks with a hidden dimension of 512, feed-forward dimension of 1536, kernel size of 7, and padding of 3. For the STFT. we set the FFT size to 1024 samples and the hop length to 320, The feature matching loss is calculated using 80-dimensional log-Mel spectrograms with the same STFT configuration. The discriminator adopts the convolutional architecture introduced in [32]. We train on LibriTTS using audio chunks of 7040 samples with a batch size of 16. Due to resource constraints, our training is limited to the train-clean: 100 split. We found this amount of data sufficient to obtain high-quality reconstructions. We use the AdamW optimizer with an initial learning rate of 0.0002, 81 of 0.8, f82 of 0.99, and weight decay of 0.01. The learning rate follows an exponential decay schedule with a factor of 0.999. Training continues until perceived audio quality stops improving, which occurs around 3M steps.
For the smaller variant of FocalCodec used in the ablation studies, we employ the same setup with the following modifications: the hidden sizes in the three focal downscaling blocks are reduced from 1024, 512, 256 to 512, 256, 128; the codebook size is decreased to 1024; use HiFi LAFi--AA-VII [Σ] decoder instead of Vocos (62) the model trained on LibriSpeech train-clean-1e clean-100 using batch size of 4.
E.2 Downstream Tasks
Automatic Speech Recognition (ASR). The model architecture is a 2-layer BiLSTM with 512- dimensional hidden states. A CTC [8] head stacked on top and trained to predict either characters BPE units. Experiments use characters and BPE vocabularies of sizes 250, 500, and 1000, with the best result reported, Note that for Mimi and FocalCodec@ 12.5, training on characters is infeasible due to the low token rate (12.5 Hz), which results in hidden sequences shorter than the target, making them incompatible with CTC loss. For all models except Mimi, performance improves monotonically with increasing BPE sizes up 1000.
23

---

while Mini achieves the best results with BPE-500. If the codec employs multiple codebooks, we compute weighted sum of the embeddings from each codebook, with the weights learned during training, done in [9] The embedding layer initialized using discrete embeddings from codec quantizer.
Speaker Identification (SI). The SI setup closely mirrors that of ASR. The only difference is that the BiLSTM output sequence is aggregated using statistics pooling, followed cross-entropy classification head.
Speech Emotion Recognition (SER).
The SER setup is the same as SI, where only the number of output
classes is different.
Speech Enhancement (SE). The model architecture is Conformer 19 encoder with layers, 4 attention heads, a model dimension of 512, and a ferd-forward layer dimension of 2048. Codecs with multiple codebaoks weighted sum of embeddings for the input, with independent linear heads for each codebook in the output. The embedding layer is initialized using the discrete embeddings from the codec quantizer. Training is performed using cross- cotsopy loss between predicted and target tokens.
Speech Separation (SS). The SS selup closely mirrors that of SE, The only difference is that training is performed using entrrpy loss with permutation invariant training [3]] and the number of output heads is doubled to account for predicting two sources parallel.
Text- Text-To--peech (TTS), The model architecture is an autoregressive Llama [17] decoder with 12 layers. 4 attention heads, key-value head, a model dimension of 512, feed-forward layer dimension of 2048, and a base RoPE frequency of 10,000. To provide speaker information, we extract speaker embeddings from the target utterance using WavLM- base [9. finc-tuned for speaker verification. The pooled speaker embedding is prepended to the text embeddings to condition the model on speaker identity, The embedding layer is initialized using the discrete embeddings from the codec quantizer. Training is performed with next-token prediction, where the input sequence consists of pooled speaker embedding, text embeddings, and speech token embeddings. The cross-entropy loss is computed only on speech tokens, while text and speaker emheddings are excluded from loss computation. For inference, we use top-p sampling with p= 0.9 and temperature of 1.0. Following the experimental protocol of (65]. we generated 5 samples per utterance and selected the one with the lowest WER relative to the input text, using Whisper-small [53 to obtain the transcription.
Training Details. For all tasks, we use AdamW (40 optimizer with a batch size of 16, an initial learning rate of 0,0001, B1 0.8, ,, 82 = 0.99, weight decay of 0.01, and dropout of 0.1. The learning rate is reduced by factor of 0.9 if validation loss does not improve within margin of 0,0025. Gradients clipped to maximum 1.2 norm 0.01. Training stops if validation loss does not decrease for several consecutive epochs.
F Implementation and Hardware
Software for the experimental evaluation was implemented in Python using the SpeechBrain 555 toolkit. Each model trained on a single GPU, with choice between V100 GPUs (16 GB) and A GPUs (40 GB), depending on cluster resource availability.
G A Addiional Results
G.1
Comparison to TS3-Codec
TS3-Codec [74] is recent transformer-only architecture designed for low-bitrate streaming speech coding. Despite its lower bitrate and streamable architecture, it remains competitive with BigCodec, the current state- of-the-art. Like FocalCodee, utilizes a single quantizer. However, fully transformer-based architecture prioritizes reconstruction, focusing on acoustic representations. The model was trained on Libri-Light 29) Since the model is not publicly available, we reached out to the authors to obtain reconstructions of the LibriSpeech test-clean for comparison. Table 7 shows the results. FocalCodec 50 surpasses TS3-Codec across all evaluated metrics, while FocalCodec 25, despite operating at significantly lower bitrate, still achieves superior performance terms of UTMOS and dWER. These findings further highlight effectiveness of proposed models.
Table 7: Clean speech resynthesis on LibriSpeech test-cleer clean
47
Bitrate (kbps)
Sample Rate
Token Rate (Hz)
Params (M)
MACs
Codee
(kHz)
Codebooks
Code Size
(G)
UTMOS
dWER | Sim
Reference
4.09
0.00
100.0
TS3-Codec (X2)
0.85
16
50.0
1 131072
16
204
8
3.84
4,51
97.1
FocalCodec @50
0.65
16
50.0
x 8192
13
142
9
4.05
2.18
97.4
FocalCodec @25
0.33
16
25.0
8192
13
144
9
4.14
3.30
96.3
FocalCodec @ 12.5
0.16
16
12.5
1 8192
13
145
8
4.22
7.94
93.9
24

---

G.2 Subjective Evaluation
We conduct a subjective test with 40 participants who rate a total of 10 reconstructions from LibriSpeech employ the MUSHRA (60] format without hidden
clean. Following prior work [14 anchor. Listeners compare multiple versions of an example once, including labeled reference and hidden reference. They are asked the following question: "Please evaluate the quality proximity between an audio sample and its reference. Please listen carefully to the reference audio and then rate the quality of each test audio clip compared to the reference. Use the scale where 0 indicates resemblance to the reference, and 100 means perfectly the same the reference." " Participants various public channels. To keep the subjective test short, we selected subset of baselines based on their overall performance in objective metrics. To ensure that participants spent sufficient time on each listening task, we filtered out submissions where less than 60 seconds were spent on any of the 10 reconstructions. Out of 40 total submissions, this resulted in 33 valid entries. As showcased in Figure 2 FocalCodec achieves extremely low bitrates while maintaining strong performance. In particular, FocalCodee @ outperforms most baselines and remains comparable BigCodec and Stable Codec.
86
37 48
recruited online by sharing a link the test across
Mean Opinion Score
MUSHRA Listening Test
00
00
Heference
100
WavLMG-KM
Mean Opinion Store
Tokeniser
Stable Coder
BicCol
3DO
SpeechTTokenize
PorallCndeelo00
SemantiCoder
Mini
WaYTokenizer
Mini
SemantiiCode
BigCodec
00
1900
Stable Codec
FocaCCodoo/300
LMM-MMKKM
Tolentro
PocalCodecR25
0.2
0.4
0.6
0.8
1.0
FocalCodec112.5
Biirate (kbpe)
0
25 50
75
100
Figure 2: Subjective evaluation from 33 participants averaged over 10 samples. Left. Trade-off between mean opinion score and bitrate. The green dashed line highlights the reference score. Focal- Codec achieves extremely low bitrates while maintaining strong performance. Right. Distribution of mean opinion score. The red lines highlight the mean. FocalCodec @ 50 outperforms most baselines and remains comparable to BigCodec and Stable Codec.
G.3 Streaming Inference
Although our codee is non-causal, it can made streamable via chunked inference. This involves splitting the input signal into fixed-size chunks with a certain amount of overlap to reduce houndary artifacts. To assess the strcamability of our codec, of 3 seconds. The reconstructed chunks stitched together using the overlap-add method with linear fade- in/fade-out. shown in Table (upper section). despite the gap with the full-context model, FocalCodec 50 maintains acceptable performance at 500 milliseconds latency. However, this is still too high for strict real-time use. To enable real time streaming, we replace all non-causal components with their causal counterparts, and we distill the causal WavLM features into the causal model using fcature-mate matching loss during training Additionally, we train the decoder reconstruct 24 kHz waveforms. As shown Table B (lower section), with these adjustments, together with increased model capacity and additional training data, we can achieve competitive performance at 80 milliseconds latency while maintaining a real-time factor suitable for deployment consumer- graude GPUs.
chunk size of 500 milliseconds with ~4% overlap and left context
Table Streaming clean speech resynthesis on LibriSpeech test-clean
47
Bitrate Sample Rate
Token Rate
Latency (ms)
Codec
Codebooks
Causal
UTMOS
dWER 1
Sim f
(kbps)
(kHl)
(Hiz)
FocalCodee @5
0.65
16
50.0
8192
X
Inf
4.05
2.18
97.4
FocalCodec @50
0.65
16
50.0
x 8192
X
500
3.16
4.55
96.9
EnCodec
1.50
24
75.0
2 x 1024
/
20
1.58
8.08
93.8
Mimi
0.69
24
12.5
5 2048
80
3.29
5.73
96.0
CocalCodec 50
0.60
24
50.0
4096
3.87
4.38
96.3
25

---

G.4 Mel-Speetrogram Analysis
Figure ssoows examples of reconstructed Mel-spectrograms from LibriSpeech 47 (left) and Iibri Mix 12 (right), using the lop-performing codees. The reconstructed speech from LibriSpeech is almost indistinguishable from the ground truth. For LibrilMix, the first row shows audio contaminated with noise, while the second row shows the original clean audio. It can be observed that BigCodec. a purely acoustic codec trained for reconstruction, attempts to reconstruct the noise, resulting in poor intelligibility. In contrast. Stable Codec and FocalCodee, which have semantically meaningful representations, are able to perform basic denoising. Notably, FocalCodec assigns more energy the frequency hands corresponding to speech, even more than the original clean audio, leading to improved intelligibility. On the other hand, Stable Codec, while providing good denoising, introduces some artifacts and static noise in the lower part of the spectrogram, which degrades intelligibility.
LibriSpeech
LibrilMix
Inpat.
Targes
BigCodoc
Stuble Codee
FocalCodre
Figure 3: Reconstructed Mel-spectrograms from LibriSpeech 47 (left) and Libri1Mix 12 (right).
26


Reviews:
[{"review_id": "rWOcF0ovm8", "reviewer": "Reviewer_Qfbg", "summary": "This paper proposes FocalCodec, a low-bitrate speech codec based on focal modulation and a single binary codebook. It aims to retain both acoustic and semantic information while simplifying the architecture compared to multi-codebook approaches. The method shows competitive performance on speech resynthesis, voice conversion, and various downstream tasks, particularly at bitrates as low as 0.16 kbps.", "questions": "1. On the explanation of Table 4 (refer to weaknesses).\n2. In training stage 2, the continuous representations before quantization are used as input to the decoder. Could you provide more details on this process? Alternatively, would it be possible to include a \"skip quantization\" option in the code to facilitate testing?\n3. We noticed that low-bitrate codecs in Table 4 generally yield high WER in generative tasks such as TTS. Although your model is trained on less than 500 hours of data, training a higher-bitrate codec (e.g., EnCodec or DAC) or a continuous-representation-based TTS model on the same amount of data would likely yield much better results. What do you think is the main reason for this performance gap? Does it suggest that low-bitrate codecs are inherently unsuitable for generative tasks?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well-written and clearly organized.\n2. The work is technically sound, with comprehensive experiments. It demonstrates strong performance even at extremely low frame rates, including on multilingual and noisy speech reconstruction. It also performs well on voice conversion and is thoroughly evaluated on both discriminative (e.g., ASR) and generative (e.g., TTS) downstream tasks.\n3. The model design and training process are novel in several aspects, such as the use of focal downsampling and BSQ, the parallel two-stage training strategy, and data augmentation to improve noise robustness.\n\n**Weaknesses:**\n\n1. The newly introduced modules, such as the focal module and BVQ, are all based on previous work. Although the final experimental results are good, the work lacks fundamental innovation.\n2. The introduction to the focal modulation in Section 3.1 is not intuitive enough and could be illustrated with diagrams for better understanding.\n3. In Table 4, results on generative tasks show that FocalCodec can yield high dWER while achieving DNSMOS/UTMOS scores close to or even better than the reference. This discrepancy is confusing. It would be helpful to include additional metrics (e.g., MCD for TTS) for a more balanced evaluation.", "was_revised": true, "final_justification": "I appreciate the additional results provided, which clarified my concerns regarding the main experimental results. I will maintain my positive score.", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address the main points raised.\n\n**\"The newly introduced modules, such as the focal module and BVQ, are all based on previous work. Although the final experimental results are good, the work lacks fundamental innovation.\"**\n\nWhile the focal modulation module and binary spherical quantization build upon prior ideas, the key innovation lies in how they are integrated into a simple and effective framework. A key contribution, overlooked in prior work, is that quantizing a pretrained self-supervised model can directly yield tokens that are both acoustic and semantic. This challenges the common belief that one must first train an acoustic autoencoder and then inject semantics via distillation or other complex techniques. Instead, our approach shows that a single quantizer operating on low-level self-supervised features can achieve both goals simultaneously within a low-bitrate, efficient framework. While simple in hindsight, we believe this represents a novel and impactful insight as evidenced by experimental results.\n\n\n\n**\"The introduction to the focal modulation in Section 3.1 is not intuitive enough and could be illustrated with diagrams for better understanding.\"**\n\nWe agree that the explanation in Section 3.1 can be made more intuitive to improve clarity. We will revise the paragraph accordingly in the updated version of the paper.\n\n\n**\"In Table 4, results on generative tasks show that FocalCodec can yield high dWER while achieving DNSMOS/UTMOS scores close to or even better than the reference. This discrepancy is confusing. It would be helpful to include additional metrics (e.g., MCD for TTS) for a more balanced evaluation.\"**\n\nUTMOS measures naturalness of the speech signal, not intelligibility or semantic accuracy. It is entirely possible for speech to sound fluent and pleasant (high UTMOS) even if the content is incorrect or semantically invalid (high dWER). Likewise, UTMOS does not reflect whether the speaker identity is preserved. This is why it is important to report complementary metrics to capture different aspects of generative quality (UTMOS <-> naturalness, dWER <-> intelligibility, speaker similarity <-> speaker identity preservation).\nWe agree that including an additional metric is helpful. In fact, in our updated TTS experiments (see the response to Reviewer mS6Q for more details), we now also report Mel Distance to provide a more comprehensive evaluation. Even with the addition of this metric, the results remain consistent with our previous findings.\n\n| Codec            | UTMOS ↑  | dWER ↓      | Sim ↑     | Mel Distance ↓ |\n| ---------------- | -------- | --------- | -------- | -------------- |\n| EnCodec          | 1.71     | 64.28     | 83.2     | 131.00         |\n| DAC              | 1.34     | 47.06     | 85.9     | 113.00         |\n| WavLM6-KM        | 3.74     | 38.67     | 88.7     | 98.85          |\n| SpeechTokenizer  | 2.69     | 35.46     | 89.2     | 93.51          |\n| SemantiCodec     | 2.82     | 48.38     | 91.4     | 99.47          |\n| Mimi             | 3.11     | 28.63     | **93.6** | 101.00         |\n| WavTokenizer     | 3.68     | 47.56     | 92.8     | 94.52          |\n| BigCodec         | 3.43     | 54.43     | 89.4     | 105.00         |\n| Stable Codec     | 3.19        | 49.28         | 88.8        | 101              |\n| FocalCodec@50    | 4.11     | 28.10     | 93.3     | 90.86          |\n| FocalCodec@25    | **4.16** | **16.75** | 91.6     | 88.01          |\n| FocalCodec@12_5  | 4.12     | 21.59     | 90.8     | **86.14**      |\n\n\n**\"In training stage 2, the continuous representations before quantization are used as input to the decoder. Could you provide more details on this process? Alternatively, would it be possible to include a ‘skip quantization’ option in the code to facilitate testing?\"**\n\nYes, this is already supported. Here's a more detailed explanation:\n- During training, the decoder learns to map continuous WavLM layer-6 features to the corresponding waveform.\n- At inference time, these continuous features are passed through the compressor, quantizer, and decompressor to obtain dequantized features, which are then fed to the decoder. The decompressor is trained to reconstruct the original continuous features from the discrete codes, so the dequantized features closely approximate the originals. As a result, the decoder maintains strong performance even when using dequantized inputs, without requiring any additional fine-tuning.\n\nThis means one can bypass the quantization step entirely and feed the continuous features directly into the decoder. In this configuration, the system effectively runs as a continuous autoencoder, which typically yields higher reconstruction quality than the fully quantized version. We added a sentence in Sec. 3.2 to clarify this aspect.\n\n\n**\"We noticed that low-bitrate codecs in Table 4 generally yield high WER in generative tasks such as TTS... Does it suggest that low-bitrate codecs are inherently unsuitable for generative tasks?\"**\n\nTable 4 suggests the opposite: low-bitrate codecs, when reconstruction quality is sufficiently high, are particularly well-suited for generative tasks, especially in autoregressive settings. In fact, our lowest-bitrate codecs achieve the best performance on TTS. This is largely due to the compactness of the token sequences they produce, which significantly simplifies sequence modeling. \n\nIn contrast, higher-bitrate codecs such as DAC and EnCodec generate much longer sequences, which can pose challenges for autoregressive models, especially when training data is limited. Notably, not only the WER but also other metrics deteriorate for higher-bitrate codecs - for example, EnCodec shows worse results across the board despite its higher bitrate.\nMore broadly, generative modeling over discrete speech tokens involves a fundamental trade-off:\n- High bitrate -> easier to achieve high reconstruction fidelity, but results in long sequences that are difficult to model;\n- Low bitrate -> harder to preserve fine-grained details, but produces shorter sequences that are more tractable for generative models.\nOur results show that a single, low-bitrate codebook can strike an effective balance between reconstruction quality and generative tractability."}, {"review_id": "6nzRcObkX3", "reviewer": "Reviewer_mS6Q", "summary": "This paper proposes FocalCodec, a novel single-codebook hybrid speech codec based on focal modulation networks and binary spherical quantization (BSQ). It achieves low bitrates (0.16–0.65 kbps) while preserving both semantic and acoustic information, outperforming prior work in reconstruction tasks and downstream ASR/SI/SER/TTS tasks. Its architecture combines a frozen WavLM encoder, compressor with focal modulation, BSQ quantizer, decompressor, and an efficient Vocos-based decoder. Extensive experiments demonstrate the performance for low-bitrate speech coding.", "questions": "1. The performance of DAC in this paper is much worse than the original paper. Did you use and evaluate the DAC properly?\n2. The performance on downstream tasks seems limited. The WER for TTS task is around 30, it has room to improve.\n3. This paper use pretrained WavLM feature as input. This limits the generalization ability for codec, which is very important for codec feature.", "limitations": "As mentioned in last session, the author’s results is not aligned with other previous paper. Besides, the downstream tasks’ generated samples are not showed on the demo page. So I highly recommend the author to further check the experimental part and improve downstream tasks’ performance. More details please refer to the Questions part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Ultra-low bitrate performance: This work achieves 0.16 kbps while preserving intelligibility and speaker similarity, surpassing many baselines operating at >1 kbps.\n2. This codec using only one codebook, encoding both semantic information and acoustic information, which is suitable for both recognition and generative downstream tasks.\n3. This paper contains both code and demo page, shows good reproducibility.\nWeaknesses:\n1. DAC baseline evaluation concerns. The reported DAC performance is much worse than in its original paper.\n2. Limited downstream task performance. For example, TTS WER remains around 30, indicating poor intelligibility.\n3. Pretrained WavLM encoder dependency. Using frozen WavLM features as input limits generalization, as it relies heavily on the pretraining domain of WavLM.", "was_revised": true, "final_justification": "I have carefully considered the authors’ rebuttal and appreciate the detailed clarifications and additional experiments they provided. While some concerns remain, the response successfully addressed several critical points I raised. My final decision is to raise my initial score to 4. This paper explores the potential of low bit codec in the downstream tasks, and the authors shows reasonable results on TTS in the rebuttal response.", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address the main points raised.\n\n**\"DAC baseline evaluation concerns. The reported DAC performance is much worse than in its original paper.\"**\n\nIn our experiments, we evaluate DAC in a low-bitrate setting (1 kbps) by using only 2 out of its 32 available codebooks, which significantly degrades performance compared to the 32-codebook variant. Additionally, we use the 16 kHz version of DAC, which has a lower bitrate but underperforms relative to the 24 kHz version used in the original DAC paper. When we evaluate the 24 kHz version using our resynthesis pipeline on LibriSpeech, we obtain the following results:\n\n| Codec | Bitrate (kbps) ↓ | UTMOS ↑ | dWER ↓ | Sim ↑ | Code Usage ↑ | Norm. Entropy ↑ |\n| --------- | ------- | ----- | ----- | ---- | ---------- | ------------- |\n| DAC-24kHz | 1.50    | 1.79  | 12.67 | 92.1 | 99.9       | 86.6          |\n| DAC-16kHz | 1.00    | 1.29  | 20.04 | 89.2 | 100.0      | 91.7          |\n\nThese results are better than the 16 kHz version but still significantly behind other baselines, despite the higher bitrate (1.5 kbps of DAC-24kHz vs 1 kbps of DAC-16kHz).\nWe hope this explanation clarifies the observed performance gap and helps convince the reviewer of the reliability of our evaluation. We clarified this aspect in the paper.\n\n\n**\"Limited downstream task performance. For example, TTS WER remains around 30, indicating poor intelligibility.\"**\n\nWe agree that there is room for improvement in TTS performance. However, it's important to note that our TTS models are trained on only 460 hours of speech -- several orders of magnitude less than what is typically used in large-scale TTS systems. Our goal is to compare codec representations fairly in a controlled setting, rather than to achieve state-of-the-art speech synthesis.\nThat said, we considered the reviewer's suggestion and explored ways to improve TTS quality.\nFirst, we observed that LibriSpeech-460 contains very few utterances longer than 20 seconds, whereas LibriSpeech test-clean includes several such samples (around 4%). To reduce mismatch between training and testing conditions, we removed these long utterances from the test set.\nSecond, following the same experimental protocol from [1], we generated multiple samples (using temperature = 1.0 and top-p = 0.9) and selected the one with the lowest WER relative to the input text. By sampling 5 outputs per input and choosing the best one (using Whisper-small for transcription) we observed improved intelligibility across all TTS models.\nBelow are the updated results, which further emphasize the effectiveness of our codecs.\n\n| Codec            | UTMOS ↑  | dWER ↓      | Sim ↑     | Mel Distance ↓ |\n| ---------------- | -------- | --------- | -------- | -------------- |\n| EnCodec          | 1.71     | 64.28     | 83.2     | 131.00         |\n| DAC              | 1.34     | 47.06     | 85.9     | 113.00         |\n| WavLM6-KM        | 3.74     | 38.67     | 88.7     | 98.85          |\n| SpeechTokenizer  | 2.69     | 35.46     | 89.2     | 93.51          |\n| SemantiCodec     | 2.82     | 48.38     | 91.4     | 99.47          |\n| Mimi             | 3.11     | 28.63     | **93.6** | 101.00         |\n| WavTokenizer     | 3.68     | 47.56     | 92.8     | 94.52          |\n| BigCodec         | 3.43     | 54.43     | 89.4     | 105.00         |\n| Stable Codec     | 3.19        | 49.28         | 88.8        | 101              |\n| FocalCodec@50    | 4.11     | 28.10     | 93.3     | 90.86          |\n| FocalCodec@25    | **4.16** | **16.75** | 91.6     | 88.01          |\n| FocalCodec@12_5  | 4.12     | 21.59     | 90.8     | **86.14**      |\n\n[1] J. Tian, J. Shi, W. Chen, et al. “ESPnet-SpeechLM: An Open Speech Language Model Toolkit”. NAACL, 2025.\n\n\n**\"Pretrained WavLM encoder dependency. Using frozen WavLM features as input limits generalization, as it relies heavily on the pretraining domain of WavLM.\"**\n\nWe would like to note that in our opinion domain limitations apply to all models including ours. For instance, EnCodec performs well on clean speech but struggles in noisy conditions, as it was not trained on such data. Similarly, our method inherits the limitations of the WavLM encoder's pretraining domain, but this is a general characteristic of pretrained models.\nThat said, our quantization framework is not tied to WavLM. We use WavLM in this work due to its strong performance in speech modeling, but the compressor-quantizer-decompressor pipeline is modular and can, in principle, be applied to any encoder. For example, it could be fine-tuned on top of a continuous autoencoder trained for general audio or music."}, {"review_id": "T2GLlL25AU", "reviewer": "Reviewer_UCYX", "summary": "The paper introduces FocalCodec, a novel low-bitrate neural audio codec designed to efficiently compress speech into discrete tokens using a single binary codebook. Unlike previous methods that often rely on complex multi-codebook architectures, FocalCodec employs a focal modulation-based compressor-quantizer-decompressor framework. The use of binary spherical quantization (BSQ), along with certain design details (e.g. the choice of activation function), reflects the thoroughness and novelty of the work.. This design preserves both semantic and acoustic information at bitrates as low as 0.16 kbps, establishing competitive performance with prior codecs in terms of reconstruction quality, voice conversion, and downstream task performance. The authors demonstrate that FocalCodec is effective for both discriminative tasks (like ASR and speaker ID) and generative tasks (like TTS and speech separation), offering a strong balance of compression, quality, and versatility.", "questions": "* Can this method be applied in streaming scenarios? It appears that the focal modulation module is non-causal. Is this structure easy to adapt for streaming use, and how would such a modification affect performance?\n* FocalCodec seems to achieve good performance without relying heavily on complex training strategies. If additional techniques such as semantic distillation were introduced, could the performance be further improved?\n* The use of a single codebook can indeed reduce system complexity; however, it typically requires a larger codebook, which may introduce challenges for subsequent joint modeling of text and speech. In this work, a codebook size of 8192 is used—could this size be further reduced?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "* Strengths:\n    * The idea presented in this paper is novel. Unlike previous works that primarily focus on improving performance through the design of training strategies, this paper explores enhancing speech codec performance from the perspective of model architecture and demonstrates competitive results.\n    * The bitrate achieved in this work is sufficiently low while maintaining acceptable performance, which confirms the feasibility of the proposed approach.\n    * The writing of the paper is clear and easy to follow.\n\n* Weaknesses:\n    * The proposed method should be validated through larger-scale experiments, using a larger training set to further demonstrate its effectiveness.\n    * Subjective evaluations are missing, such as MOS or MUSHRA scores. As the authors mentioned, some objective metrics may not align well with human perception, which further highlights the necessity of including subjective assessments.\n\nOverall, the idea in the paper is good. I would be happy to raise my score if more convincing experimental results or evaluation methods can be provided.", "was_revised": true, "final_justification": "In the rebuttal, the authors supplemented their work with human evaluations and additional experiments, confirming the feasibility of the proposed method in streaming scenarios. They also conducted a more in-depth discussion on issues such as the size of the codebook. Although the method performed slightly worse than BigCodec and Stable Codec in human evaluations, it still demonstrated competitive performance. Therefore, I will maintain my initial score and remain inclined to accept this paper.", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address the main points raised.\n\n**\"Subjective evaluations are missing, such as MOS or MUSHRA scores. As the authors mentioned, some objective metrics may not align well with human perception, which further highlights the necessity of including subjective assessments.\"**\n\nWe conduct a subjective test with 40 participants who rate a total of 10 reconstructions from LibriSpeech test-clean. Following prior work, we employ the MUSHRA format without hidden anchor. Listeners compare multiple versions of an example at once, including a labeled reference and a hidden reference. They are asked the following question: \"Please evaluate the quality proximity between an audio sample and its reference. Please listen carefully to the reference audio and then rate the quality of each test audio clip compared to the reference. Use the scale where 0 indicates no resemblance to the reference, and 100 means perfectly the same as the reference.\" Participants were recruited online by sharing a link to the test across various public channels. To keep the subjective test short, we did not include EnCodec and DAC due to their poor performance based on objective metrics.\nTo ensure that participants spent sufficient time on each listening task, we filtered out submissions where less than 60 seconds were spent on any of the 10 reconstructions. Out of 40 total submissions, this resulted in 33 valid entries. The results reported below confirm that FocalCodec achieves extremely low bitrates while maintaining strong performance. In particular, FocalCodec@50 outperforms most baselines and remains comparable to BigCodec and Stable Codec.\n\n| Codec            | Mean  | 95% CI (Lower-Upper) |\n| ---------------- | ----- | -------------------- |\n| WavLM6-KM        | 34.10 | 31.81 – 36.38        |\n| SpeechTokenizer  | 26.08 | 23.70 – 28.46        |\n| SemantiCodec     | 56.25 | 53.61 – 58.88        |\n| Mimi             | 60.38 | 57.77 – 63.00        |\n| WavTokenizer     | 77.81 | 75.51 – 80.10        |\n| BigCodec         | 92.72 | 91.49 – 93.96        |\n| Stable Codec     | 88.76 | 87.15 – 90.36        |\n| FocalCodec@50    | 80.65 | 78.63 – 82.66        |\n| FocalCodec@25    | 72.25 | 69.92 – 74.59        |\n| FocalCodec@12_5  | 68.99 | 66.53 – 71.46        |\n\n\n**\"Can this method be applied in streaming scenarios? It appears that the focal modulation module is non-causal. Is this structure easy to adapt for streaming use, and how would such a modification affect performance?\"**\n\nThe current focal modulation module, comprising full-context convolutions, linear layers, and global pooling, is non-causal, as is the WavLM encoder due to its use of full-context attention. However, in practice, we find that the effective receptive field of these components is limited to just a few seconds of future context. This allows us to perform streaming inference by processing the audio in overlapping chunks of 500 ms without requiring any changes to the model architecture.\nBelow are the results of chunked inference for the speech resynthesis task on LibriSpeech for FocalCodec@50:\n\n| Latency    | UTMOS ↑ | dWER ↓ | Sim ↑ |\n| ---------- | ------- | ------ | ----- |\n| Infinity   | 4.05    | 2.18   | 97.4 |\n| 500 ms     | 3.16    | 4.55   | 91.0 |\n\nWhile a 500 ms latency is acceptable for semi-streaming applications, it remains too high for real-time use. To address this, we are actively exploring architectural modifications to enable fully causal inference. These changes include:\n1) Replacing full-context attention with chunked or causal attention;\n2) Replacing standard convolutions with causal convolutions;\n3) Replacing global average pooling with running mean pooling;\n4) Distilling the non-causal WavLM features into the causal model using a feature matching loss during training.\n\nPreliminary results reported below suggest that, with these adjustments, and by scaling up model capacity and amount of training hours, we can achieve competitive performance at 80 ms latency, which is the same as Mimi, while maintaining a real-time factor high enough for deployment on consumer-grade GPUs:\n\n| Codec          | Bitrate (kbps) | Codebooks | Latency | UTMOS ↑   | dWER ↓   | Sim ↑      |\n|----------------|----------------|-----------|---------|--------- |--------  |---------  |\n| FocalCodec@50  | **0.60**           | 1×4096    | 80 ms   | **3.87** | **4.38** | **96.3** |\n| Mimi           | 0.69           | 5×2024    | 80 ms   | 3.29     | 5.73     | 96.0     |\n| EnCodec        | 1.50           | 2×1024    | 20 ms   | 1.58     | 8.08     | 93.8     |\n\n\n**\"The proposed method should be validated through larger-scale experiments, using a larger training set to further demonstrate its effectiveness.\"**\n\nAs mentioned in our previous point, for the causal variant experiment, we used a larger training dataset to compensate for the performance drop due to switching from full-context to low-latency settings. Specifically, we experimented at two scales: the medium subset of LibriLight, consisting of approximately 5k hours of English speech, and the full LibriLight dataset, comprising around 60k hours. The table below shows how scaling up the training data improves performance, consistent with the scaling behavior often seen in deep learning.\n\n| Training Hours | UTMOS ↑ | dWER ↓ | Sim ↑ |\n| -------------- | ------- | ------ | ----- |\n| 5k             | 3.78    | 4.98   | 95.7  |\n| 60k            | **3.87**| **4.38**   | **96.3**  |\n\n\n**\"FocalCodec seems to achieve good performance without relying heavily on complex training strategies. If additional techniques such as semantic distillation were introduced, could the performance be further improved?\"**\n\nSince FocalCodec relies on a single shared codebook, it is important to be cautious about the type of distillation applied. For example, semantic distillation (as in Mimi) could enhance semantic content but may degrade acoustic detail due to the shared representation space. In contrast, more balanced distillation strategies that combine both semantic and acoustic supervision (such as jointly optimizing ASR and speaker identification objectives) may help improve both aspects without compromising either. That said, as the reviewer rightly noted, one of the key strengths of our approach is its simplicity. A more straightforward and robust path to improved performance may be to retrain WavLM with increased capacity or on a larger and more diverse dataset. This would likely enhance the quality of the representations while preserving the balance between semantic and acoustic information.\n\n\n**\"The use of a single codebook can indeed reduce system complexity; however, it typically requires a larger codebook, which may introduce challenges for subsequent joint modeling of text and speech. In this work, a codebook size of 8192 is used, could this size be further reduced?\"**\n\nWhile 8192 tokens may seem large, it is modest compared to vocabulary sizes commonly used in modern text LLMs -- for example, LLaMA 3 uses 128k tokens, and Gemma 2 reaches up to 256k. Even within the speech domain, recent codecs like TS3-Codec use vocabularies exceeding 100k tokens.\nTo investigate the impact of reducing the codebook size, we conducted an experiment using a codebook size of 1024 instead of 8192. Results for FocalCodec@50 on LibriSpeech resynthesis are shown below:\n\n| Codebook Size | UTMOS ↑ | dWER ↓ | Sim ↑ |\n| ------------- | ------- | ------ | ----- |\n| 8192          | 4.05    | **2.18**   | **97.4**  |\n| 1024          | **4.14**    | 2.54   | 95.3  |\n\nAs expected, reducing the codebook size primarily impacts speaker similarity, since larger codebooks are better at capturing fine-grained acoustic details such as speaker identity. Notably, this experiment was conducted without increasing model capacity or training data. Therefore, it is reasonable to expect that performance with as few as 1024 codes could be further improved by scaling up the model and training on larger datasets."}, {"review_id": "ZfKX1CwBy6", "reviewer": "Reviewer_p8pB", "summary": "This paper proposed a low bit rate speech codec system called FocalCodec. Different from most of the neural audio codec which comprised of encoder, decoder and quantizer, FocalCodec added focal modulation-based Compressor and Decompressor modules. For quantizer, it used binary spherical quantization method (BSQ). Both focal modulation net and BSQ were originally used for image domain. Comprehensive experiments are conducted showing promising performance of FocalCodec in terms of both reconstruction task and down streaming tasks.", "questions": "Can FocalCodec be applied to real-time down stream tasks? If so, what are the supported latency?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1, The paper is well-written and highly readable. \n2, Technique is sounding. Although initially developed for image and video compression, the proposed methods (Focal modulation net and BSQ) are creatively repurposed to offer a compelling and successful approach to speech codec design. The proposed two-stage training scheme which decouple the optimization for quantization and speech reconstruction modules is also critical for building a high-quality codec system given limited amount of model parameter.   \n3, Experiments are comprehensive, and performance is promising. \n\nWeakness:\n\nFor the experiment part, to demonstrate the effectiveness of focal net, it would be necessary to show the performance comparison of focal net with convolution net and transformer architecture. Moreover, to demonstrate the advantage of the proposed BSQ quantizer, it would be also necessary to compare it with the often-used vector quantization, e.g, the one used in DAC.", "was_revised": true, "final_justification": "This paper proposed a low bit rate speech codec system by adopting focal modulation-based Compressor and Decompressor modules. For quantization, binary spherical quantization method is used to convert continuous embeddings into discrete tokens. Comprehensive experiments have been conducted and promissing results are achieved. During rebuttal phase, the authors clarified the question on possibility of real-time system I raised as well as the weakness of insufficient comparison with other model architecture I mentioned. The clarification is reasonable and answered the questions and cleared my initial concerns. Particularly, for converting the current system into realtime system, they provided extra experiments and results, and show the prominence of real-time capability. After the final review, I stick to my initial score, which is 5-Accetp.", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address the main points raised.\n\n**\"For the experiment part, to demonstrate the effectiveness of focal net, it would be necessary to show the performance comparison of focal net with convolution net and transformer architecture. Moreover, to demonstrate the advantage of the proposed BSQ quantizer, it would be also necessary to compare it with the often-used vector quantization, e.g, the one used in DAC.\"**\n\nWe kindly note that a similar comparison is already provided in Table 5 (Ablation Studies) of the main paper. In this table, we compare FocalNet-based quantizers against strong alternatives, including:\n- Conformer, widely regarded as an enhanced version of the vanilla Transformer and extensively used in speech modeling;\n- AMP blocks, specifically designed for speech synthesis and shown to outperform standard convolutional blocks (e.g., in BigVGAN [1]).\nWe believe these are more representative and challenging baselines than standard Transformer or ConvNet architectures, and our results demonstrate a clear advantage of FocalNet in this context.\nRegarding the quantization method, the same table also includes a comparison between our Binary Spherical Quantizer (BSQ) and Finite Scalar Quantizer (FSQ). FSQ has recently been shown to outperform traditional vector quantization methods, such as the one used in DAC. Therefore, our ablation study already provides evidence of BSQ’s effectiveness over stronger and more recent baselines than standard VQ.\n\n[1] S.-G. Lee, W. Ping, B. Ginsburg, et al. “BigVGAN: A Universal Neural Vocoder with Large-Scale Training”. ICLR, 2023.\n\n\n**\"Can FocalCodec be applied to real-time down stream tasks? If so, what are the supported latency?\"**\n\nThe current focal modulation module, comprising full-context convolutions, linear layers, and global pooling, is non-causal, as is the WavLM encoder due to its use of full-context attention. However, in practice, we find that the effective receptive field of these components is limited to just a few seconds of future context. This allows us to perform streaming inference by processing the audio in overlapping chunks of 500 ms without requiring any changes to the model architecture.\nBelow are the results of chunked inference for the speech resynthesis task on LibriSpeech for FocalCodec@50:\n\n| Latency    | UTMOS ↑ | dWER ↓ | Sim ↑ |\n| ---------- | ------- | ------ | ----- |\n| Infinity   | 4.05    | 2.18   | 97.4 |\n| 500 ms     | 3.16    | 4.55   | 91.0 |\n\n\nWhile a 500 ms latency is acceptable for semi-streaming applications, it remains too high for real-time use. To address this, we are actively exploring architectural modifications to enable fully causal inference. These changes include:\n1) Replacing full-context attention with chunked or causal attention;\n2) Replacing standard convolutions with causal convolutions;\n3) Replacing global average pooling with running mean pooling;\n4) Distilling the non-causal WavLM features into the causal model using a feature matching loss during training.\n\nPreliminary results reported below suggest that, with these adjustments, and by scaling up model capacity and amount of training hours, we can achieve competitive performance at 80 ms latency, which is the same as Mimi, while maintaining a real-time factor high enough for deployment on consumer-grade GPUs:\n\n| Codec          | Bitrate (kbps) | Codebooks | Latency | UTMOS ↑   | dWER ↓   | Sim ↑      |\n|----------------|----------------|-----------|---------|--------- |--------  |---------  |\n| FocalCodec@50  | **0.60**           | 1×4096    | 80 ms   | **3.87** | **4.38** | **96.3** |\n| Mimi           | 0.69           | 5×2024    | 80 ms   | 3.29     | 5.73     | 96.0     |\n| EnCodec        | 1.50           | 2×1024    | 20 ms   | 1.58     | 8.08     | 93.8     |"}]

Author rebuttal:

Meta-review: This paper proposes FocalCodec, a low-bitrate audio neural codec based on focal modulation. Compared to the existing audio neural codecs, FocalCodec achieves low bitrates and only needs a single binary codebook to compress speech between 0.16 and 0.65 kbps. It is shown to be efficiently preserve semantic and acoustic information. The authors conduct experiments to demonstrate its superior performance in speech resynthesis and voice conversion compared to numerous existing approaches. Overall, this is an interesting work that is technically solid. Concerns raised by the reviewers include questions on novelty, generalization capability and interpretation of some experimental results. The authors have cleared most of the concerns in their rebuttal. Overall, the proposed FocalCodec is sufficiently novel. The paper is well written and easy to follow. Experiments are controlled and yet extensive, which include the experimental results reported in the submission and added later on in the rebuttal. All reviewers are supportive of accepting this paper.
Decision: Accept (poster)