FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
Authors: Jiaqi Li
Yao Qian
Yuxuan Hu
leying zhang
Xiaofei Wang
Heng Lu
Manthan Thakker
Jinyu Li
sheng zhao
Zhizheng Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Audio coding
neural audio codecs
speech language model
Primary area: generative models

Abstract: Neural audio codecs are foundational to speech language models. It is expected to have a low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain underexplored. We find that pushing existing audio codecs to very low frame rates loses much semantic information. We suggest that low-frame-rate codecs' limitations are in both insufficient semantic decoupling and insufficient time resolution at capturing transient phonetic details. This paper introduces **FlexiCodec** to address this limitation. FlexiCodec improves semantic preservation with a **dynamic frame rate** approach and introduces a novel architecture featuring an **ASR feature-assisted dual stream** encoding and Transformer bottlenecks.
With dynamic frame rates, it uses less frames at information-sparse regions through adaptively merging semantically similar frames. 
A dynamic frame rate also allows FlexiCodec to support inference-time **controllable frame rates** between 3Hz and 12.5Hz.
Experiments on **6.25Hz, 8.3Hz and 12.5Hz** average frame rates confirm that FlexiCodec excels over baseline systems in semantic information preservation and delivers a high audio reconstruction quality. We also validate the effectiveness of FlexiCodec in language model-based TTS. Demos are available at: https://flexicodec.github.io. Code is available at: https://github.com/amphionteam/flexicodec.

Published as a conference paper at ICLR 2026
FLEXICODEC: A DYNAMIC NEURAL AUDIO CODEC FOR Low FRAME RATES
Jiaqi Li
Yao Qian
Yuxuan Hu? Leying Zhang² Xiaofei Wang² Heng Lu
Manthan Thakker² Jinyu Li Sheng Zhao?
Zhizheng Wu 1,,5,6
'Te Chinese University Hong Kong, Shenzhen, Miirosoot, USA, 3Shanghai Jiao Tong University, Shenzhen Loop Area Institute, CCty University of Macau, "Amphion Technology
Co., Ltd.
ABSTRACT
Neural audio codecs are foundational to speech language models. It is expected have low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain under- explored. We find that pushing existing audio codecs to very low frame rates loses much semantic information. We suggest that low-frame-rate codecs' lim- itations are in both insufficient semantic decoupling and insufficient time reso- lution at capturing transient phonetic details. This paper introduces FlexiCodec to address these limitations. FlexiCodec improves semantic preservation with a dynamic frame rate approach and introduces novel architecture featuring ASR feature-assisted dual stream encoding and Transformer bottlenecks. With dynamic frame rates, it uses less frames information-sparse regions through adaptively merging semantically similar frames. A dynamic frame rate also al- lows FlexiCodec to support inference-tine. controllable frame rates between 3Hz and 12.5Hz. Experiments on 6.25Hz, 8.3Hz and 12.5Hz average frame rates confirm that FlexiCodec excels over baseline systems in semantic informa- tion preservation and delivers high audio reconstruction quality. We also vali- date the effectiveness of FlexiCodec language model-based TTS. Audio demos are available at: https:://fexxicceee.ro
io Code is available at: http:://ttttttcccooommmmo/ootttt
INTRODUCTION
The neural audio codec technique, originally designed for waveform compression (Zeghidour al.
2021 Défossez et al.) 2022), is now widely used in various tasks including Language Model (LM)-
based text-to-speech (TTS) (Wang et al. 2023] Anastassiou al., 2024 Du et al.)
2024a Wang
ct al. 2025b Guo et al, 2024) and multimodal LLMs (Défossez et al. 2024: Zeng et al. 2024a
Ding et al.) 2025). By compressing raw speech into compact discrete tokens, neural audio codecs
enable the application of auto-regressive LLM paradigms to the speech domain.
The standard neural audio codec follows an eecoder--uantizee-ddddddrr architecture. The encoder downsamples the audio waveform into sequence of fixed-rate continuous latent vectors. These vectors are then quantized into discrete indices using Residual Vector Quantization (RVQ), a multi- stage process where each quantizer encodes the residual error of the previous one. In downstream tasks like TTS, the first RVQ layer's 'ss tokens (RVQ-1) are often used drive an autoregressive (AR) language model or an LLM, while the remaining layers (RVQ-rest) can be predicted by autoregressive (NAR) model to add acoustic detail.
However, fundamental challenge arises from the high temporal resolution of these codecs. State- of-the-art models like EnCodec (Défossez et al.) 2022). DAC (Kumar et al. 2024). and Speech- second of
Tokenizer (Zhang al.) 2023a) operate frame rates exceeding 50Hz, representing audio with over 50 tokens. This high density, compared to the 44.5Hz rate of typical text representa-
tions (Wang al. 2024), creates two major problems for AR models: (1) significant computational

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of different audio tokenization methods and their properties.
Method
Frame Rate (Hx)
Dynamic Rate
Controllable Rate
Semantic Augmentation
TTS Oriented?
DAC
75
X
X
X
SpeechiTokenizer
50
X
X
SSL feature (HaBERT))
CodeeStime
40 (Avg)
/ (40,50067,88)) 80HZ options)
x
Mimi
12.5
X
X
SSL feature (WavLM)
DualCodec
12.5/25
X
K
SSL feature (w2v-bert-2)
TaDiCodec
6.25
X
Text
Phoneme Tokens
11.7 (Avg)
X
BPE Text Tokens
4.5 (Avg)
X
FlexiCodec
6.25/8.3/12.5
(Any from 3 to 12.5Hz)
ASR feature
burden due to the quadratic complexity of attention, and (2) severe frame rate mismatch between text and audio modalities that may degrade LLM performance (Zeng et al. 2024b. Wang et al.) 2024).
To mitigate this, recent work has focused on low-frame-rate codecs. Methods like Mimi (Défossez al., 2024) and DualCodec (Li al. 2025a) successfully reduced the frame rate to 12.5Hz by decoupling speech into two streams: semantic stream R)Q-1) derived from self-supervised learn- ing (SSL) models (Chen et al.] [022] Barrault et al., 2023), and an acoustic stream (RVQ-rest) for residual details. In this strategy, the low-rate RVQ-1 tokens encode the core semantic information, which is sufficient for many downstream AR models.
While previous works have proposed 12.5Hz solutions, significant gap remains compared to the ~4.5Hz frame rate of text. Furthermore, research into neural audio codecs operating below 12.5Hz is limited. Our initial experiments revealed that pushing existing codecs to below 12.5Hz leads to significant loss of semantic/phonetie content. We suggest two reasons for this result. First, at very low frame rates, the limited information capacity forces a trade-off between acoustic fidelity and semantic representation. Existing codecs may be primarily limited by an insufficient decoupling of semantics from acoustics. This is exemplified by recent work on syllable-level unit discovery (Cho eet al.] 2024a Baade et al. (024)), which successfully extracts semantic units at 5-8Hz but largely dis- cards the fine-grained acoustic details, prosody, and timing required for high-fidelity reconstruction.
We discuss these works in Section 2.2 Second, we propose that current fixed-rate downsampling can lead to the loss of transient phonetic details, whereas natural speech units like syllables and phonemes are inherently dynamic in their rate of occurrence. Driven by these observations, we have the following motivations for designing our low frame rate codec: (1) Dynamic Frame Rate: A fixed low rate inevitably discards transient phonetic details, whereas a dynamic rate may adapt to the phonetic complexity to encode more details. (2) Richer Semantics: SSL features, trained for reconstruction, can be redundant. Features from an ASR model, trained for text prediction, may offer a more concentrated source of semantic information. (3) Controllable Frame Rate: Existing codecs typically operate at allow users to dynamically trade off performance and efficiency for downstream tasks.
more fixed frame rates. A continuously controllable rate would
In this work, we introduce FlexiCodec. novel low-frame-ratee rtat codec built three principles: dy- namic frame rate, ASR-guided semantics, and frame rate controllability. Instead of a fixed frame rate, FlexiCodec dynamically allocates temporal resolution, using more frames for complex pho- netic segments and fewer for sparse regions like long vowels, syllables and silence. This is achieved through a novel ASR-feature-assiin similar frames. A key benefit of this dynamic approach is that a single model can support a contin- uous range of frame rates (3-12.5Hz) at inference, enabling flexible trade-offs for applications like adaptive signal transmission or variable-complextg TTS on edge devices. Our contributions are:
dual-stream architecture that adaptively merges semantically
We propose FlexiCodec, a novel codec that pushes the boundaries of low-frame-rate audio tokenization. To our knowledge, it is among the first codees to achieve high-quality, re- constructible audio at average frame rates below 10Hz. We also introduce novel method for dynamic frame rate allocation in the low-frame-rate regime (3-12.5Hz), guided by ASR features. A review of recent below 10Hz codecs and dynamic-rate codecs are pro- vided in Sections 2.1 and
We show that FlexiCodec outperforms open-source baselines in semantic intelligibility and acoustic quality. Experiments confirm that our dynamic frame rate strategy improves se- mantic information preservation and allows for controllable frame rate as low as 3Hz. Other
2

---

Published as a conference paper at ICLR 2026
design choices including utillizing an ASR encoder, transformer bottlenecks, and FSQ quan- tization, also contributes to our codec's performance.
We demonstrate FlexiCodec 's utility in flexible TTS system. The model yields competi- tive results at multiple frame rates and is substantially faster than existing methods.
RELATED WORKS
2.1
LOW-FRAME- AATE NEURAL AUDIO CODECS
Neural audio codecs convert continuous speech into discrete tokens. SoundStream (Zeghidour et al. 2021], Encodec (Défossez al., 2022), and DAC (Kumar et al. 2024) focused audio compres- sion, relying on residual vector quantization (RVQ) (Zeghidour et al. 2021) and operating at high bitrates (24kbps) and high frame rates (> 50Hz). WayTokenizer (Ji al.) 2024). TS3-Codec (Wul et al. 2024). SemantiCodec (Liu et al., 2024), and StableCodec (Parker et al.) (024) used a single
VQ FSQ (Mentzer al., 2023) codebook. They delivered good audio quality low bitrates (around 1kbps). but operate at high frame rates (240Hz).
Some recent works develop low-frame-rate codecs. A lower frame rate limits the information amount that can be carried by each RVQ layer tokens. Thus, some works (Défossez et al. 2024 Li al. 2025a) decompose the speech tokens into semantic (RVQ-1) and acoustic (RVQ-rest) to- kens. Mimi codec (Défossez et al.] (024)) was based on Encodec with a higher downsampling rate convolutional encoder. employed semantic distillation (Zhang et al. (023a), a tech- nique that distills RVQ-1 embeddings from SSL features. DualCodec (Li et al. (025a) proposed dual-stream architecture where a semantic stream directly encodes SSL features into RVQ-1 tokens, and acoustic stream encodes RVQ-rest tokens. ALMTokenizer (Yang et al., (025b) proposed query-based compression strategy using a set of learnable query tokens, and designed an auxiliary
MAE loss inspired by SSL models. Concurrent work XY-Tokenizer (Gong et al.) 2025 is 12.5Hz codec encoded with a concatenative dual-stream architecture consisting of Whisper ASR feature and waveform feature.
Recently, TaDiCodec (Wang et al. 2025c) and TASTE TTeng et al. 2025) proposed < <10Hz speech tokens, but operating like text-to-speech (TTS) systems, they require the text transcription assist audio synthesis. Accurate text transcription can be unavailable in some audio coding scenarios. By comparison, our work more similar to a conventional codec and does not require transcriptions.
2.2
DYNAMIC-RATE COMPRESSION OF IMAGES AND AUDIOS
The concept of dynaiccally adjusting token rates based content complexity is emerging trend across image and audio modalities. In the image domain, Bolya et al. (2022) first proposed Token Merging (ToMe)) technique to gradually merge most similar tokens in Vision Transformers (ViTs) to accelerate inference. It used the cosine similarity between the self attention keys to guide the merging process. Dyn'Tok (Zhang al.) (2025b) proposed improved similarity calculation strategy based on CLIP (Radford et al. 12021) semantic representation.
In the audio domain, one research trend has been the syllable-level semantic unit discoveries which are inherently dynamic-rate. SD-HuBERT (Cho et al., (2024b) finetuned HuBERT (Hsu et al.) 2021)) with sentence-level self-distillation, and showed that its features distinguish syllable boundaries. Sylber (Cho et al. (2024)) leveraged HD-HTBERT as pseudo labels, and trained student model with syllable encoding ability around 5 syllables per second. SylBoost (Baade et al. (024)) can discover discrete syllabic units using min-cut algorithm on the feature self-similarity matrix, followed by a k-means clustering. It experimented on 8.33Hz and 6.25Hz units. However, compared to neural audio codec tokens, syllable units only encode coarse semantic information, and necessitate external speech synthesis models to produce audio.
Dynamic-rate neural audio codec is emerging research area. Dieleman et al. (2021) proposed an audio VQ- VAE with run-length encoding, operating at an average frame rate of 75Hz. SNAC (Siuz- dak et al. 2024) created multi-resoluticn codec stream 12, 23, and 47Hz for each RVQ layer. CodeeSlime (Wang et al.) (225a) proposed a two-stage process to firstly train an 80Hz fxed-frame- rate codec, followed by merging similar features into 40Hz rre tokens. Similarly,
3

---

Published as a conference paper at ICLR 2026
Dynamic-Rtate Seet.
A2R Teat.
Semunntte menddoog
Encoder
+
Frume Merging Module
62
FSQ
Stat.
12.58tr decodlngg feat.
GAN loos
alfteo
Speech
Condition
Frame Immerging
Codec Decoder
[1l
Module
Codec Eneoder
Merging Module
RVQ
4 Aal R
Roidual
9
Waveform
Dynomle
Rate Tokena
Waveform
Figure 1: Overview of FlexiCodec. The model encodes speech through two streams. The Frame Merging Modules dynamically reduce the 12.5Hz features into lower frame rates, and the Frame Unmerging Module restores 12.5Hz fixed frame rate. The model is trained end to end.
Cosine similarity of ASR Seat. thrrshhold!
0.33
0,95
0.60
12,51Iz Waveeoro
f
f
f,3
Merged (quantized embeddings)
feat.
Repeat by "length" times
Interleave
f
f
4
Transformer Local Attcation
Trausformer Attention
Unmerged frames (12.5Hx)
Retrieval
frames attributes
Ben=1
(a) Merging Module
(b) Frame Unmerging Module
Figure 2: Detailed views the Frame Merging Module and the Frame Unmerging Module.
introduced dynamic-frame-nate al- gorithms on 75Hz RVQ or single-codebook tokens. Compared with previous works, ours explores more challenging low frame rates (510Hz). Compared with CodeeSlime, TFC and VARSTok, our work uses a pre-trained audio semantic feature extractor to guide the merging process, simplifying the needs for multi-stage training or dynamic programming- based merging algorithms.
TFC (Zhang et al. 2025a) and VARSTok (Zheng 2025
2.3 TEXT-TO-SPEECH SYNTHESIS
Modern Text-to-Speech (TTS) has increasingly shifted from statistical parametric methods to sys- tems based on neural audio codecs. VALL-E (Wang et al. 2023) established an "AR+NAR" frame- work: an AR language model generates codec RVQ-I tokens conditioned on text: an NAR language model predicts remaining RVQ layer tokens in parallel. The AR stage provides essential informa- tion the NAR stage, and the AR stage usually the most time-consuming part. Subsequent works have evolved this paradigm, showing the benefits of semantic-rich RVQ-1 tokens in the AR stage (Borsos et al.) 22023a] Du et al., 2024at Zhang et al. 2025d), and replacements of the NAR discrete token prediction with diffusion- based models (Betker 2023 Du et al. 2024a).
The efficiency of TTS system directly correlates to codec frame rate. A lower frame rate codec can significantly boost training and inference speed. A few works (Du al. 2024b Deng et al.) 2025) have adopted 25Hz frame rate tokens the AR stage, others mostly rely on 250Hz tokens (Guo
al. 2024 Wang et al. 2025b). For the NAR stage, these works mostly keep frame rate 250Hz.

---

Published as a conference paper at ICLR 2026
3 FLEXICODEC: A DYNAMIC LOW-FRAME-RATE NEURAL AUDIO CODEC
The core of our work is FlexiCodec, a neural audio codec designed to operate at very low average frame rates while preserving crucial semantic information. Unlike traditional codecs that use fixed frame rate, FlexiCodee employs dynamic frame rate mechanism that allocates more temporal res- olution information-dena regions of speech and less to information-sparse segments like silence or long vowels. This is achieved through an architecture that leverages pre-trained ASR features to encode semantie-rich RVQ-1 tokens, and to guide the adaptive frame merging process. The overall architecture, depicted in Figure [] follows dual-stream encoding, dynamic frame merging, quantization, and frame unmerging decoding pipeline.
Dual-Stream Feature Extraction FlexiCodec begins by processing 16kHz speech waveform through two parallel encoders to decouple semantic and acoustic information: (1) An ASR Encoder, leveraging the pre-trained ASR model, extracts a sequence of semantic features e, E R'xxd, where T and d' denote the number of frames and the vector dimension. And (2) a convolutional codec encoder, downsampling the waveform to produce sequence of waveform features ea E R2'xd also at a 12.5Hz frame rate. The codec encoder consists of CNN blocks with strides [4,4,5,8,2] to gradually downsample the audio, giving 16000Hz + (4 x 4 x 5 block contains strided ID convolution followed by ResNet He et al. 2016).
2) 12.5Hz. Each CNN
For the specific ASR model choice, we use pretrained Sense Voice-Small (An al. 2024 a 230M- parameter, encoder-only Transformer model trained on 300k hours of data with CTC We use its last layer (excluding its CTC logits prediction layers) hidden state as the semantic feature. The model outputs 16.67Hz features, we downsample it to 12.5Hz using linear interpolation to align the frame rate with the acoustic stream. The ASR model is frozen during codec training.
Dynamic Frame Merging This module compresses the number of frames in sequence, resulting in sequence where some segments have lower than 12.5Hz frame rate. Our goal to compress the frames with a minimal loss of semantic information by preferentially merging frames that are seman- tically similar (redundant). Inspired by DynTok (Zhang et al.) 2025b) that used pre-trained image features for token compression, shown in Figure a). Specifically, let the 12.5Hz ASR feature vectors be (1)..... [T]] We first compute the cosine similarity between adjacent frames: & = cos (e, [1), e, [x 1,... T 1. We then scan from left right form maximal contiguous segments ( i+l,..., whose adjacent similarities exceed a threshold r: min 8, 2 т. All frames in such segment are merged into single frame by averaging, applied to both the semantic and acoustic streams:
reuse the extracted ASR feature to guide our merging process,
2
for =
1 = &
[k[ = E
Ea [k]
= = j j-
where [k] is the k-th entry in the compressed frame sequence in the semantic stream, and éa [k] is the sequence in the acoustic stream. We also record the length of each merged frame lg as attributes, which are required reconstruct the original fix-frame-rate sequence.
We find that directly using the average-reduced sequence can degrade the naturalness of the recon- structed audio, particularly causing unnatural transitions between merged frames. To address this, we adopt transformer with local windowed attention that processes an interleaved sequence of original and averaged frames (Fig(]a) (Yang et al., 2025b Li al., (023). This allows the merged tokens to query their adjacent context and produce refined, context-aware representations. A local attention is favored over global attention because it allows generalization to longer audios despite trained on fixed-length audio segments.
Frame Rate Flexibility FlexiCodec trains with a flexible merging threshold т, sampled across a range. This enables the model to support controllable frame rates at inference by adjusting т. At T 7= 1.0 (no frame merging), FlexiCodec functions as a 12.5Hz rixed-frame-rate codec. At < 1.0 (with frame merging), the average frame rate of the new sequence is lower than 12.5Hz, and is calculated as:
Total number of frames afler merging Setting a lower decreases the average frame rate. Audio duration seconds
Semantic (RVQ-1) Quantization The dynamic-rate ASR features are quantized using a Finite Scalar Quantizer (FSQ) Mentzer al. 2023) to produce discrete semantic tokens (we denote as RVQ-1 tokens). FSQ projects the input representations e, into a D-dimensional low-rank space, where the value of each dimension is quantized using rounding operation ROUND into T levels.
5

---

Published as a conference paper at ICLR 2026
The quantized low-rank vector is subsequently projected back the original dimension é:
c.
éa = (rojup(Cs)
Straight-through estimation (Bengio et al. 2013 is employed to estimate the gradients in the ROUND function. The semantic token index y, is obtained by: 41 E Additionally, we wrap the FSQ block with small ConvNeXt (Liu et al. (022) blocks to increase its representation ability, and apply an I.2 loss Teat to align the semantic token embeddings with the unquantized semantic features.
Acoustic (RVQ-rest) Quantization To encode the detailed acoustic information that is not cap- tured in the semantic tokens, following Li et al. (2025a), we compute residual by subtracting the dynamic-rate ASR feature from the dynamic-rate waveform feature. This residual is then quantized using an (N - -lylayer Residual Vector Quantization (RVQ) (Zeghidour et al. 2021 module to produce acoustic tokens f2.N E -(N-1)xf where T denotes the sequence length after frame merg- ing. We employ quantizer dropout (Zeghidour et al. [02I] during training. That is, only the RVQ-1 to RVQ-n layers are subsequently decoded, where n E N] is randomly chosen. When n 1, only the semantic encoding stream is used. Straight-through estimation (Bengio et al. 2013) is used to backpropagate through the codebook lookup.
To reconstruct the audio, the decoder path should verse the dynamic compression The embedding features from the first n chosen RVQ tokens are added form decoding feature representation. This dynamic-rate sequence is then passed the Frame Unmerging Module, detailed in Figure bb). It uses the frame length attributes to expand the sequence back 12.5Hz. It is followed by another Transformer with local attention to smooth the transitions and refine the feature representations. The resulting feature sequence is then fed into a convolutional codec decoder which mirrors the codec encoder, synthesizing the output waveform.
Frame Unmerging and Reconstruction
FlexiCodec is trained end-to-end with a composite loss function:
L Lreeon AGANCGAN + ARVOCRVO + Afeat Creat
where Grecon is a multi-scale L.1 mel spectrogram reconstruction loss following Kumar et al. (0024)
LGAN contains adversarial and feature matching losses for Multi-Period Discriminator (MPD) (Kong
et al. 2020) and Mesoll-Reoolution Spectrogram Discriminator (MRSD) (Kumar et al., 2024). LRVQ involves LI codebook update loss and a commitment loss for RVQ, whereas the FSQ module does
not require a training loss. Lieat is the L2 feature alignment loss between the RVQ-1 semantic token embeddings and the unquantized semantic features.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Codec Training Configuration We use the 16kHz, 54k-hour Librilight-Large (Kahn et al. 2020) dataset for training. Each model is trained for 800k steps on 8 Nvidia V100 32GB GPUs We use a batch size of 5 samples per GPU; each sample a second audio segment. We use AdamW (Loshchilov & Hutter 2017) optimizer with Ir=1 10-1, betas=(0.8,, 0.99); exponential learning rate with (ammmm-0.999990
In each step, the merging threshold is randomly chosen from 0.7 < 7 < 1.0. We set the maximum frame length la to 8 so that each & would take at most log28 = 3 bits of storage. Each token in the local attention transformer can attend to & = tokens left and right. The FSQ module has D = 5 dimensions each quantized to L = 8 levels, resulting in 85 = 32768 codebook entries. The RVQ RVQ-rest quantization has 24 RVQ layers, 4096 codebook en- tries per layer, and 512 512-dimensional codebook entries. FlexiCodec has 216M trainable parameters, in which the two frame merging modules each is 20M, and the frame unmerging module is 100M.
We have not applied FSQ for acoustic quantization because FSQ is single- quantization. We consider the integration of more advanced multi-layer FSQ (rFSQ) schemes, e.g., (Parker al. 2024). he promising direction for future improvements.
Thii is because the NAR codec decoder can only receive fixed-frame-rate sequence. It is also possible to generate audios directly from dynamic-rate tokens using an AR model, and we leave this as future work.

---

Published as a conference paper at ICLR 2026
Codec Evaluation Metrics We evaluate the 4 to 10 second subset of LibriSpeech-test clean (Panayotov et al. 2015). comprising 1,088 audios. We evaluate one FlexiCodec on three frame rates: 12.5Hz (80ms hop size), 8.3Hz (120ms hop size), and 6.25Hz (160ms hop size). Since FlexiCodec has dynamic, controllable frame rate, we configure its T = 1-0, 0.91, 0.867 for 12.5Hz, 8.3Hz, and 6.25Hz average frame rate, respectively. These т settings have been determined based on trial runs on our test set. Our evaluations consist of the following semantic and acoustic testings:
Semantic testing: To evaluate the preservation of semantic content, we transcribe the codec reconstructed audio using a Hlubert-Large-LS9660 (Hsu et al., 202T) ASR model and compute the word error rate (WER) against ground truth transcriptions. The testings include using RVQ-1 1 alone, and RVQ-1:8 (1 to 8 layers) tokens. Specifically, RVQ-1 tokens' semantic preservation relates to downstream model performance, where AR LMss only access RVQ-1 tokens (Li al. 2025a Zhang et al. 2023a). Evaluating RVQ-1:8 is common choice of RVQ-based codecs, implying the acoustic compression quality (Défossez et al. 2024 Zhang et al.) 2023a).
Acoustic testing: We evaluate the reconstructed audio from RVQI:8 tokens against the original audio. Metrics include the Perceptual Evaluation of Speech Quality (PESQ, narrow band) (Rix
et al. 200T). Mel Cepstral Distortion (MCD) (Kubichek) 1993), speaker similarity (SIM, the cosine similarity between speaker embeddings extracted from a WayLM-large--as (Chen et al.) 2022)
speaker verification model), and speech perceptual quality score from a neural Mcan Opinion
Score (MOS) predictor UTMOS (Saeki et al. 2022).
Additional semantic testing with ASR probing: Additionally, to evaluate the semantic align- ment between FlexiCodec semantic tokens and text, we employ an ASR probing task (Gong et al.] Zhang al. 2023a), adapted from XARES (Zhang et al.) (025c) benchmark. We train (Yang et al. the text transcription given FlexiCodec's 's RVQ-I token embeddings. During training, only the pa- rameters of an MLP adapter are updated. Models are trained on LibriSpeech train-clean- 100 using cross entropy loss. We evaluate the ASR WER on LibriSpeech test-clean. The task's upper bound result is obtained by training with the unquantized Sense Voice- encoder feature. Due to the prolonged evaluation time of this task, we only conduct this task as an extended semantic testing in Section 4.3
2025
downstream Swwen..5-0.5B--
2025a) ASR model which is tasked with predicting
4.2 EXAMINING THE IMPACT OF VERY LOW FRAME RATES
We first investigate the performance of representative audio codecs at very low frame rates. Since open-source baselines operating below 12.5Hz are unavailable, we created three new baseline ver- sions by retraining DAC (Kumar et al. 2024) and DualCodec (Li et al. 2025a) to operate at 12.5Hz, 8.3Hz, and 6.25 respectively. To adapt these systems for lower frame rates, we increased their encoder downsampling rates and enlarged their codebook sizes to be consistent with FlexiCodec. Detailed configurations are provided in Appendix [G.]] Our findings are as follows.
RVQ-1 tokens' semantic information preservation is challenging for very low frame rate codecs. As shown in Figure a)), see that the Word Error Rate (WER), evaluated audios reconstructed from RVQ-1 tokens, increases substantially for DAC and DualCodec when the frame rate drops from 12.5Hz to 6.25Hz. At 12.5Hz, DualCodec achieves 5.93% WER, close to the ground truth (GT) 2.1% %. However, as its frame rate drops to 6.25Hz, its gap with GT significantly increases 31.5% vs. 2.1%, indicating its lower resolution tokens fail to capture the whole phonetic details. Note that DualCodec architecture already improves WER over DAC thanks to its semantic augmentation using SSL features. Figure 3))) confirms this trend, though the performance gap is smaller when using more RVQ layers.
FlexiCodec excels at semantic preservation, especially at the lowest frame rates. In contrast, FlexiCodec maintains low WER across all rates. At the most challenging 6.25Hz average frame rate, FlexiCodec achieves 4.15% WER which is close to ground truth (2. (2) and outperforms the best baseline (31.5%). To explain FlexiCodec's superior semantic information retention, our further experiments suggest it is contributed by a combination of our proposed design choices, including the ASR-assisted encoding architecture, dynamic frame rate merging, FSQ, and Transformer modules. These modules are ablated in Appendix[D We have found that a simple switching from DualCodec's SSL into FlexiCodec ASR feature is very useful and it achieves RVQI WER of 6.0% at 6.25Hz.
7

---

Published as a conference paper at ICLR 2026
Iretrained)
DualCodec (retrained)
FlexiCodec (this
(RVQ())Q1)
(b) Frame vs. (RVQ11))
100
3.35
WER1
WERI (%)
PESQT
3.03
297
FastCotee
525
-1.5
0.42
82
3,00
2.77
245
20
12.6
2.75
PlecCodoz
5:93
PlaxlCodec
0
44.3
2:58
Wohm2
2.50
6.25
8.33
12.50
6.25
12.50
6.25
12.50
Frame Avg (Hz)
Frame Rate Avg (Hz)
Frame Rate Avg (Hz)
(d)
Frame Rate vs. MCD (RVQ1:8)
(e) Frame Rate vs, SIM (RVQ1:8)
(f)
Frame Rate vs. UTMOS (RVQ1:8)
8:0
3.75
Rlssioode
4:23
1.53
0.8
78
0.64
UTMOS1
4.2
MCD
3.25
PrexiCddeg
SIM1
Flasi
4.0
4.02
3.00
2.81
0.7
0108
1.6
0.65
2,50
3.6
6.25
8.33
12.50
6.25
8.33
12.50
6.25
8.33
12.50
Frame Rate Avg (Hz)
Frame Rate Avg (Hz)
Frame Rate Avg (Hz)
Figure 3: Evaluation results on three very low frame rates, Each baseline system has been retrained for each target frame rate using the same recipe FlexiCodec.
Table Relation between FlexiCodec's 's frame merging threshold T, average frame rate, and WER.
FlexiCodec Frame Rate (trames/sacom)
N: camples 1.2001 Arg FlaxCodec rabe: 8.31:2.40
AVG frame rate
3.0
3.6
4.5
7.9
12.5
WER/RVOIN
51.5
29.5
14.4
3.13
2.76
WER(RVQL:88
18.1
8.90
4.38
2.37
2.23
Table 3: Ablation of dynamic frame rate (semantic test).
WERIRVQUJ
WER/RVQ1811
ASR probing WER, (diff.
(diff.)
(dff.)
Upper
HexiCodee: 6HH.3Hz
2.98
2.28
13,0
w/o dynamic frame rate
3.56
(1119% rel.)
2
(16% rel.)
14.5 (112% rel.)
Phoneme Rate (phoname//eccond)
FlexiCodes @6.25Hz "y w/o dynamic frame rate
5.22 (+26%% rel.)
2.73 (+8% rel.)
18.8 18.8 (+21% rel.)
Figure 4: Correlation between Flexi- Codec frame rate and phoneme rate a fixed frame merging threshold т. Each data point is audio in TIMIT dataset, representing the audio's 's average phoneme rate average FlexiCodec frame rate.
Table Ablation of dynamic frame rate (acoustic test).
PESQ1
MCDJ
UTMOS*
SIMI
FlexiCodec 08.31
3.03
3,10
4.21
0.78
dynamic frame rate
FlexiCodee @6.25Hz
2.76
3,42
4.18
0.71
+ who dynamic frame rale
2.76
3.47
4.18
0.70
Further improvements are particularly driven by the dynamic frame rate strategy, which we will detail in Section 4.3
Acuustcc quality metrics show more moderate differences across systems and frame rates. As shown in Figure 3 c-e),, the PESQ, MCD, SIM, and UTMOS metrics show that at each frame rate, FlexiCodec slightly outperforms DualCodec, followed by a larger gap with DAC. We observe that the gap between models does not substantially increase when changing to lower frame rates, but has been seen in the semantic testing. We think that the acoustic fidelity is more constrained by bitrate, and because the bitrates of the three systems the same level, the difference is not pronounced.
8

---

Published as a conference paper at ICLR 2026
4.3 ANALYSIS AND ABLATION OF DYNAMIC FRAME RATE
To understand the mechanism and performance of dynamic frame rate, we conduct a series of anal- yses and ablations shown in Figure 4 and Table 2
Our findings are as follows.
Dynamic frame merging effectively adapts to the underlying phonetic complexity of speech. Figure + shows a strong positive correlation (Pearson r 0.775) between the utterance-level phoneme rate and FlexiCodec's frame rale on the TIMIT (Garofolo et al., 1993) subset. This demon- strates that FlexiCodec dynamically adjusts its token frame allocation proportion to the phonetic density of the audio, assigning more frames to segments with faster speech and fewer frames to slower or silence regions. Such adaptivity enables efficient semantic compression by aligning token rate with semantic information density. The fitted line also shows linear coefficient very close to 0.5, indicating that each merged frame approximately encodes two phonemes. We visualize sev- eral cases by aligning phonemes labels with FlexiCodec frames, which is shown in Appendix [A] It confirms that typical merged tokens include syllables/short words, long vowels, and silence.
Dynamic frame rate enables controllable frame rate as low as 3Hz. As shown in Table 2 the merging threshold 7 controls frame rate and semantic preservation trade-off. Notably, by controlling values at inference time, we can obtain different output sequence lengths, which is a unique feature compared to conventional fixed-frame rrate codecs.
Dynamic frame rate improves semantic information preservation. Tables and compare FlexiCodec and its variants that are retrained with fixed frame rate (FFR). To obtain the FFR vari- ants, we modify their codec encoder strides output static 8.3Hz or 6 6.25Hz, and maintain the total parameter count (detailed in Appendix G.I). 3 shows that the FFR variants have consistently worse WERs than FlexiCodec. The gap is larger at the lower frame rate. Specifically, the 6.25Hz FFR variant increases RVQ-1 WER by a relative 26%, the ASR probing WER by 21 1,, and the RVQ1:8 WER by 8%. These results highlight that dynamic frame rate improves semantic informa- tion preservation by mechanism of phonetic note that dynamic frame rate may perform even better on real-world data, which tend to have longer silence regions that are highly compressible. We leave this investigation as future work.
camplexity-adaptiii
frame rate allocation. We also
In terms of acoustic metrics, Table f shows that PESQ and UTMOS are the same between Flexi- Codec and its FFR variant, while a moderate degradation in MCD and SIM is observed in FFR. This indicates that dynamic frame rate mainly boosts semantic preservation rather than low-level acoustic quality. One possible reason is that the acoustic information density in speech can be misaligned with the semantic information density. For example, semantically unimportant segments may still contain acoustic details like noise, sound and music.
4.4 COMPARISON WITH OPEN-SOURCE CODECS AT VARIOUS BITRATES
In this experiment, we compare FlexiCodec with open-source neural audio codecs spanning various bitrate F and frame rates. Information about the baseline codecs are provided in Appendix G We categorize them into 3 bitrate classes, and FlexiCodec at 12.5Hz, 8.3Hz, and 6.25Hz average frame rates fall into each category, enabling bitrate-consistent comparisons. Table presents the results.
FlexiCodec has state-of-the-us audio quality at various bitrate levels. Examining the acous- tic test scores, at >> Ikbps, FlexiCodec at 12.5Hz (1.2kbps) achieves higher acoustic scores than its 12.5Hz counterparts Mimi, XYTokenizer and DualCodec. It only trails behind DAC which uses a higher bitate (6kbps). At O. 8kbps and 0.6kbps, FlexiCodec also demonstrates superior acoustic qual- ity scores than baselines. These results demonstrate that FlexiCodec has a high bitrate efficiency.
FlexiCodec is competitive higher frame rate systems semantic information preservation. Across the WER semantic test metrics at both RVQI and RVQ188 quantization levels, FlexiCodec consistently achieves competitive or better scores compared to other systems operating at higher frame rates. Notably, FlexiCodec at 6.25Hz attains RVQI WER of 4.15, outperforming larger frame rate models such as SpeechTokeniizr-550000 50Hz, Encodec-75Hz, Way Tawoooinnize-55Ae etc.
3 Bitrate indicates the amount of data stored or transmitted per unit of time, measured in kilobits per second (kbps). It can be calculated by (log, Codebook size) x Frame_rate x Num_rvg A lower bitrate codec has higher compression ratio but usually lower audio quality. Since storing and transmitting each of FlexiCodec's frame length attribute takes log, = 3 hits, have added 3 frame
FlexiCodec's 's bitrate Table 5

---

Published as a conference paper at ICLR 2026
Table 5: Comparison between FlexiCodec and other open-source, neural audio codecs.
RVQ1 BRikbps)
RVQ1:
Semantic Test
Acoustic Test (RVQ1:8)
System
BRikbps/n
WERIRVODL
1E
PESQ
> Hköps Acoustic Bitrate
DAC-75Hz
0.75
6.0% 8q
74M
31.2
2.27
3,77
3.62
2.34
0.90
Encodec-73tB
5q
0.50
4.0/8g
105M
3,36
2.47
3.01
3.501
3.17
0183
Mini- 1atii-122200 12.5Hz
1.1
75M
3.15
2.75
3.56
3.62
0.73
XYTokonizer 12.51l
5q
CnalCCeee--B 12,5Hz
0.19
1.2/8g
84M
5.93
2.26
3.29
4.18
2.81
0185
FlodCodee 122.5Hz
0.23
1.3/
216M
2.76
2.23
3.35
4.22
2,76
0.85
-0.8
Was Www/ooonizaa-75B
0.90
0.90/ 1q
81M
4.57
4.57
2.86
3.98
3.51
0.68
SNAC-12.25. 12.23.7Hzz
0,98
20M
4.21
2.51
3.43
3.61
0.67
XChdec2-50Hz
3.65
TS3Cooku(23)-5000
0.85
0.85 1q
204M
4.09
4,09
2,80
3,80
3.38
0.68
FlexiCodec 08.33z
0.15
0.85/84
216M
2.98
2.25
3,03
4.21
3,10
0.78
TAAB-25Hz 11B-22512-770bps 700lps
0.70 2g
953M
4.19
2.84
4.33
4,98
0.58
TAAE-25Hz.. 25E--251H.400hhp 400hps
0.40
0.40/
953M
5.21
5.21
2.61
4.34
4,10
0.53
ComantiCadco-50HH
0.68 1q
4.97
2.38
3.03
0.63
SemantiCodee-2Se
0.34
0,34 14
187SM
23.8
23.8
1.89
2.93
5.92
0.40
X44 4088z
0.68
0.681
204M
5.14
5.14
2.58
3.67
3.65
0.63
TaDiCades-6.
0.151
9.75
FlooCodde
25H25Hz
0.11
0.64
216M
4.15
2.53
2.76
4.18
3.42
0.71
5
MORE EXPERIMENTS
Downstream TTS experiments. In Appendix B we detail our low-frame-rate TTS system with FlexiCodec (exxiCodec-TTS) Our conclusions are (1) FlexiCodec-THe achieves competitive per- formance with significant speedups over baselines, and (2) higher NAR frame rate is important for high audio quality, but using lower frame rates for AR does not necessarily degrade performance.
Downstream audio understanding experiments. In Appendix C show that FlexiCodec mantic token embeddings surpass other codecs in downstream audio understanding tasks, highlight- ing the potential to FlexiCodec in unified multimodal understanding and generation frameworks.
A Allaioon study on other components of FlexiCodec. In Appendix D we confirm our other design choices like utilizing ASR feature, wansformer-based frame merging and unmerging mod- ules, and FSQ quantization are beneficial to our codec's semantic and acoustic performance.
benchmark the encoding and decoding speed of FlexiCodec. The results show that FlexiCodec has a low Real-Time Factor (RTF) of 0.018 for encoding and 0.006 for decoding across all frame rates. This shows that model is practical and efficient solution for integration into larger pipelines like TTS systems.
Codec efficiency analysis. In Appendix E
Generalization to out-of-domain and multilingual data. In Appendix F evaluate the bustness of our English-traingg model on the diverse, out-of-domain Emilia (He et al. 2024) dataset. The results show that (1) FlexiCodec maintains strong semantic and acoustic performance on out- of-domain English speech, confirming it is not overfitted to audiobook data. (2) While the semantic tokens struggle on unseen languages
zero-shot setting, the full acoustic reconstruction remains competitive with multilingual models. (3) A fine-tuning on unseen language, Chinese, can adapt FlexiCodec's semantic tokens and the merging scheme to the new language.
CONCLUSION
this work, we introduced FlexiCodec, a novel neural audio codec designed for very low frame rate operation. By incorporating a dynamic frame rate, ASR-assisted dual-stream architecture, and Transformer based frame merging/unmerring modules, FlexiCodec effectively preserves semantic information at low rates. Experiments confirm its strong performance in low frame rate and low bitrate speech coding, controllable frame rate, and effectiveness in a downstream TTS system. Lim itaion and future work discussed in Appendix
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide comprehensive details on methodology, train- ing configurations, and evaluation setup throughout the paper and appendices. All datasets used, including Librilight-Large LibriSpeech, and TIMIT, are publicly available. We have also released code and model in the Github repository mentioned in the abstract.
8 ACKNOWLEDDEMENTS
This work is supported by the NSFC Grant 62376237; the Shenzhen Science and Technology Pro- gram ZDS 2DSSYS2223066201100022 2023 Shenzhen Stability Science Program; Program for Guang- dong Introducing Innovative and Entrepreneurial Teams 20232T10X044.
REFERENCES
Keyu An. Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04000 2024.
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024.
Alan Baade, Puyuan Peng, and David Harwath. Syllablelm: Learning coarse semantic units for speech language models. arXiv preprint arXiv:2410.04029 2024.
Loic Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, al. Seamless: Multilin- gual expressive and streaming speech translation. arXiv preprint 2rXiv:2312.05188 2023.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
James Betker. Better speech synthesis through scaling, arXiv preprint arXiv:2305.07243 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pictquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523-2533, 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint 2305022305.09666, 2023b.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505- 1518, 2022.
Cheol Jun Cho, Nicholas Lee, Akshat Gupta, Dhruv Agarwal, Ethan Chen, Alan W Black, and Gopala K Anumanchipalli, Sylber: Syllabic embedding representation of speech from raw audio. arXiv preprini arXiv:2410.07166 2024a.
Cheol Jun Cho, Abdelrahman Mohamed, Shang- Wen Li, Alan W Black, and Gopala K Anu- manchipalli. Sd-hubert: Sentence-level self-distillation induces syllabic organization in hubert. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12076-12080 IEFE, 2024b.

---

Published as a conference paper at ICLR 2026
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.00337. 2024.
Wei Deng. Siyi Zhou, Jingehen Shu, Jinchao Wang, and Lu Wang. Indextts: An industrial-Ievel con- trollable and efficient zero-shot text-to-speech system. arXiv preprint arXiv:2502.05511, 2025.
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete represen- tation learning. arXiv preprint arXiv:2103.0660 2021.
Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audie technical report. arXiv preprint arXiv:2504114255, 2025.
Heinrich Dinkel, Zhiyong Yan, Yongging Wang. Junbo Zhang, Yujun Wang, and Bin Wang. Scaling up masked audio encoder learning for general audio classification. arXiv preprint arXiv:2406.06992 2024.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer 2024a.
based on supervised semantic tokens. arXiv preprint
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.1011 2024b.
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, al. E2 uts: Embarrassingly easy fully non- autoregressive zero-shot tts. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 682-689. IEEE, 2024.
John S Garofolo, Lori Lamel, William M Fisher, David Pallett, Nancy L Dahlgren, Victor Zue, and Jonathan G Fiscus. Timit acoustic-phonetic continuous speech corpus. (No Title), 1993.
Yitian Gong, Luozhijie Jin, Ruifan Deng, Dong Zhang, Xin Zhang, Qinyuan Cheng, Zhaoye Fei, Shimin Li, and Xipeng Qiu. Xy-tokenizer: Mitigating the semantic-acoustic conflict in low-bitrate speech codecs. arXiv preprint WwXiv:2506.23322 2025.
Hao-Han Guo, Kun Liu, Fei- Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fir- eredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint 2409 03283, 2024.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint ar Arvi 16532, 2024.
12

---

Published as a conference paper at ICLR 2026
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Serre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 7669-7673. IEEE, 2020.
Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours asr corpus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10991-10995. IEEE, 2024.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In IEEE PACRIM, pp. 125-128, 1993.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang, Yuancheng Wang, Chaoren Wang, Zhenpeng Zhan, and Zhizheng Wu. Dualcodec: A low-frame-rate, semantically-enhance neural audio codec for speech generation. In Proceedings of Interspeech 2025, 2025a.
Jiaqi Li, Xueyao Zhang, Yuancheng Wang. Haorui He, Chaoren Wang. Li Wang, Huan Liao, Junyi Ao, Zeyu Xie, Yiqiao Huang, et al. Overview of the amphion toolkit (v0. 2). arXiv preprint arXiv:2501.15442, 2025b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19744 PMLR, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley, Semanti- codec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing. 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhoferr Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11 11966, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1000 2017.
Alcksandr Meister, Matvei Novikov, Nikolay Karpov, Evelina Bakhturina, Vitaly Lavrukhin, and Boris Ginsburg. Librispeech-pe: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models. In 2023 IEEE automatic speech recognition and under- standing workshop (ASRU), pp. 1-7. IEEE, 2023.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:23009. 15505, 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
13

---

Published as a conference paper at ICLR 2026
Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. arXiv preprint arXiv:2411.19842 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scatt scale weak supervision. In International conference on ma- chine learning, pp. 188492-28511 PMLR, 2023.
A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In ICASSP, 2001. doi: 10.1199/CASSP.2000 .941023.
Takaaki Sacki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02//2, 2022.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint 23X6:23006008844 2023.
Hubert Siuzdak, Florian Grötschla, and Luca Lanzendörfer. Snac: Multi-scale neural audio codec. arXiv preprini arXiv:2410. 1441114411,1,, 2024.
Liang-Hsuan Tseng, Yi-Chang Chen, Kuan-Yi Lee, Da-Shan Shiu, and Hung-yi Lee. Taste: Text- aligned speech tokenization and embedding for spoken language modeling. arXiv preprint w7Xi0 2025.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 2023.
Hankun Wang, Haoran Wang, Yiwei Guo, Zhihan Li, Chenpeng Du, Xie Chen, and Kai Yu. Why do speech language models fail to generate semantically coherent outputs? modality evolving perspective. arXiv preprint arXiv:2412.1700 7048, 2024.
Hankun Wang. Yiwei Guo, Chongtian Shao, Bohan Li, Xie Chen, and Kai Yu. Codecslime: Tem- poral redundancy compression of neural speech codec via dynamic frame rate. arXiv preprint arXiv:2506.21074 2025a.
Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, al. Spark-tts: An efficient Ilm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint vrxXiv2503.017/00 2025b.
Yuancheng Wang, Dekun Chen, Xueyao Zhang, Junan Zhang, Jiaqi Li, and Zhizheng Wu. Tadi- codec: Text-aware diffusion speech tokenizer for speech language modeling. arXiv preprint drXiv:2508.16700 2025c.
Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez, and Jinyu Li. To3-codec: Transformer-based simple streaming single codec. arXiv preprint arXiv:2411.1880 2024.
An Yang et al. Qwen2.5 technical report, 2025a. URL
15115
Dongchao Yang, Songxiang Liu, Haohan Guo, Jiankun Zhao, Yuanyuan Wang, Helin Wang, Zeqian Ju, Xubo Liu, Xueyuan Chen, Xu Tan, et al. Almtokenizer, A bow-bitrate and semantic-rich audio codec tokenizer for audio language modeling. arXiv preprint arXiv:2504.1034. 2025b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
14

---

Published as a conference paper at ICLR 2026
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glmce-voice: Towards intelligent and human- end-to-end spoken chatbot. arXiv preprint arXiv:2412.02611. 2024a.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong. and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data, arXiv preprint 1. 1.760777 2024b.
Hanglei Zhang, Yiwei Guo, Zhihan Li, Xiang Hao, Xie Chen, and Kai Yu. Unlocking temporal flex- ibility: Neural speech codec with variable frame rate. arXiv preprint (xXi:2255..6844 2025a.
Hongzhi Zhang, Jingyuan Zhang, Xingguang Ji, Qi Wang, and Fuzheng Zhang. Dyntok: Dynamic compression of visual tokens for efficient and effective video understanding. arXiv preprint arXiv:2506.03390, 2025b.
Junbo Zhang, Heinrich Dinkel, Yadong Niu, Chenyu Liu. Si Cheng, Anbei Zhao, and Jian Luan. X-ares: A comprehensive framework for assessing audio encoder performance. In Interspeech 2025, 2025c.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023a.
Xueyao Zhang, Liumeng Xue, Yuancheng Wang. Yicheng Gu, Xi Chen, Zihao Fang, Haopeng Chen, Lexiao Zou, Chaoren Wang, Jun Han, et al. Amphion: An open- spen-source audio, music and speech generation toolkit. arXiv preprint arXiv:2312.099111 2023b.
Xueyao Zhang, Xiaohui Zhang, Kainan Peng. Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, et al. Vevo: Controllable zero-shot voice imi- tation with supe-supe disentanglement. arXiv preprint arXiv:2502.07243, 2025d.
Rui-Chen Zheng, Wenrui Liu, Hui-Peng Du, Qinglin Zhang, Chong Deng. Qian Chen, Wen Wang, Yang Ai, and Zhen-Huu Ling. Say more with less: ferable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. arXiv preprint arXiv:2509.0468 04685, 2025.
A VISUALIZATION OF FRAME MERGING PATTERNS
Figure presents visualizations of FlexiCodec tokens aligned with the TIMIT (Garofolo et al. 1993) dataset phonemes. We show four randomly selected utterances from different speakers, each speak- ing the same content. For each utterance, the inferred content of merged tokens is labeled in green above the token sequence. Our discussions are follows.
Typical merged framees correspond to phonemically coherent units. Frames forming sylla- bles, short or common words (e.g., "su," "your," "all"), long vowels (e.g., "/aa/" in "dark", "water"), and silence intervals ([SIL]) are frequently merged to form single tokens spanning multiple original frame indices.
Merging patterns exhibit strong consistency across different speakers for the same utterance. A ltthough the four speakers differ in voice quality and prosody, the dynamic tokenization captures similar merging structures. For example, phonetic patterns of "she". "had", "su",, "grea", "year" and the trailing silence are common to all four utterances.
The deterministic nature of merging allows interpretability and reproducibility. Since merging boundaries are derived from pretrained ASR features without additional trainable parame- ters, the token boundaries are replicable and interpretable. In this work, reused the same ASR feature for both encoding semantic tokens and to derive the merging boundaries. We leave the exploration of more features as future works.
15

---

Published as a conference paper at ICLR 2026
Sample (a)
Transcript: "She hed your dark wuit greaey wash water all year."
TINIT
she
had
dark
SU
grea
$y
Wa
wa
ter
year
[SIL]
Takens
Similarty Values
sample phonema
This sample ClastCodicc rate: 7.25 framesibec
0.
0.3
2.0
25
300
Sample (b)
Transcript: "She had dark suit wash water year."
TIMIT
[BLI
she
had
dark
Su
grea
SV
wa
fer
al
Year
SILL
Plexicodes
:2 aie
Simllrrty Values
sample
65
10
20
25
Sample (c)
Transcript: "She had your dark suit greasy wash water all year!
TIMIT
she
had
your
S11
grea
Wa
year
SIL
Tokena
13
21
24
Similrity Values
8.12
6.H
6:11
6.1
6:13
12.09
This RestCodac
Sample (d)
Transcript: 'She your greasy wash all year?
Phonemes
(911
she had
dark
sh
all
SSIL
FlexiCodec Takens
iale
Similarity
640
0.8
6.
6.11
0.3
sample
65
2.2
Thre bczzedil
46
Token (frames)
Figure 5: Visualizations of FlexiCodec dynamic-rate tokens aligned with TIMIT dataset phonemes. We visualize four random utterances of different speakers speaking same transcript "She had your dark suit in greasy wash water all year." The inferred content of each merged token is labeled in green font.
16

---

Published as a conference paper at ICLR 2026
Next Tokes Prediction
Gonerated Spoech
Embee & Sum
①. Mil Spectrogram Vocader
Flaci
Spectrograss
FlexsCuder RVQ
LM huaats
Conditional Flow Matching NAR Transformer
Text-to-Semantic L.M
leo-1
Jea-2
Icor1
Dysank-Rum RVQJ Takens
12.5 Tokens
Frame Itate Indicator
Pronut Sperch
(12.5/8.36.25
FlexiCodec
lea-1
lea-2
Iea=1
Repeat times
Transerigt
Dynamic Rate RVQ-1 Tokens
()) TTS AR Stage
(b) TTS NAR Stage
Figure 6: Overview of FlexiCodec-TTS. (a) The AR LM has two lightweight prediction heads to predict FlexiCodec RVQ-1 token and corresponding frame lengths. (b) The NAR Stage predicts continuous features conditioned on stage outputs. We experiment with 2 types of continuous features, 1 50Hz Mel Spectrogram, and ( 12.5Hz FlexiCodec feature.
B DOWNSTREAM EXPERIMENT: TTS
B.1 TTS MODEL ARCHITECTURE
We integrate FlexiCodec within AR+ Matching) TTS framework. Our model archi- tecture is inspired by Cosy Voice (Du et al. 2024a). Modifications have been made in the AR and NAR stages mainly because FlexiCodee produces dynamic frame rate tokens.
AR Stage As shown in Figure ((), this is a Transformer AR model that predicts FlexiCodec RVQ1 tokens. Different from previous systems, our model has two token prediction heads instead of one. This is because each FlexiCodec token includes an explicit "frame length" attribute. Therefore, we treat duration modeling as an auxiliary classification task by adding second prediction head after the last hidden layer. Each prediction head is lightweight linear projector. The LM input sequence is:
S {yuhuenooy
M (Embed(g) (*) Embed((.-1))
E
where S M and E denote the start, mid, and end separators. {yu denotes the embedded the 1-th FlexiCodec RVQI token, and 4-1 the frame length of the t- 1-th RVQI token. For training, we employ next-token prediction with two parallel classification heads attached to the LM's last hidden state, to predict both q;" and f--1. The composite loss is calculated L Lrem XLdur where:
phoneme sequence, d:
f Leem Σ
T log P(u!") PiaL, (.--)): Laur Σ
log P(ee -1 PIaLa did-1)
During inference, the model is prefixed with the phoneme sequence and a prompt speech's RVQI (tokens and frame lengths). The remaining tokens and frame lengths sampled autoregressively.
Frame Rate Flexibility in AR stage To enhance flexibility, the AR model is trained on sequences with either 12.5Hz, 8.3Hz, 6.25Hz average frame rate, by setting various 7 values when obtaining the RVQI tokens. To distinguish these sequences, a dedicated control token is prepended to the first audio token. During inference, the user can select from one of the three control tokens to specify an output frame rate.
NAR Stage this stage, we aim to transform the predicted RVQ-1 sequence into audios with acoustic details. While previous works like VALL-E (Wang al. 2023) and SoundStorm (Borsos)
17

---

Published as a conference paper at ICLR 2026
et al. 2023b) have used NAR models to predict RVQ codec tokens, have opted for predicting
continuous features because low frame rate discrete tokens' acoustic quality upper bound is not as good as high frame rate (previously shown in T Following other works (Le al. 22024 Du et al. 2024a] Zhang et al. 2025d), we employ a conditional flow matching (Lipman et al.) 2022). This allows experiment with two acoustic features with different frame rates, (1) the 50Hz Mel spectrogram (100 Mel bands following (Zhang et al.) 2025d), and (2) the 12.5Hz FlexiCodec feature
(the feature labeled in dark green in Figure (). For (2), equivalent to predicting n
23
layers of RVQ-rest tokens simultaneously.
As shown in Figure 6 the model is flow matching transformer. During training, given a speech waveform and its acoustic feature y1, we randomly select a prefix of y1, denoted as y and aim to reconstruct the other part (denoted as yix conditioned on yi and the FlexiCodec RVQ-1 tokens. To obtain the conditioning FlexiCodec RVQ-1 tokens, we first expand the dynamic-frame-
rate RVQ-1 sequence T/- into 12.5Hz fixed frame rate sequence qi.T using frame lengths li and then linearly interpolat it to match the frame rate of y1. The rest of the formulation follows Zhang et al.] (2025d): Du et al. (2024a), using 300M Diffusion Transformer to approximate an optimal transport path between gaussian noise Yo and spectrogram Y1- Classifier-free guidance (CFG) is applied by randomly dropping conditions during training and combining unconditional and conditional predictions during inference. During inference, we use 15 timesteps, and a CFG guidance strength of 1.5. To decode mel spectrogram, vocoder from the Amphion (Li et al. 2025b Zhang et al. (023b) toolkit.
(s)
pretrained Vocos (Siuzdak, 2023)
Frame Rate Flexibility in NAR stage The NAR model supports decoding from either 12.5Hz, 8.3Hz, or 6.25Hz FlexiCodec RVQ-1 sequence. Its conditioning FlexiCodec RVQ-1 sequences are expanded from either 12.5Hz, 8.3Hz, or 6.25Hz average frame rate, by randomly choosing from corresponding values during training. We do not add additional tokens distinguish these sequences because we find it yields the same results.
B.2 TTS EXPERIMENTS
TTS Training Configuration We use the 16kH7, 50k-hour Libriheavy Kang et al. 2024 dataset for training. Each model is trained for 600k steps 8 Nvidia V100 32GB GPUs. The AR model uses AdamW (Loshchilov & Hutter, 2017) optimizer with Ir=1 x 10-4, betas=(0.5, 0.999); batch size is 96 seconds per GPU; gradient accumulation step is 3. The NAR models uses Ir=7.5 x 10 betas=(0.5, 0.999): batch size is 24 seconds per GPU; gradient accumulation step is 3. The AR model has 250M parameters; the NAR model has 300M parameters.
TTS Evaluation Metrics We use the E2TTS (Eskimez et al. 2024) test suite to test models. Its test data consists of 1,132 (reference, test) pairs from Cborispecch-C-tess--lleee (Meister et al.) 2023 We evaluate both objective and subjective metrics. The objective metrics include word error rate (WER) and speaker similarity (SIM-o). We report the real-time factor (RTF) of each system tested on an RTX5000 GPU. RTF calculated the ratio of the total processing time to the total duration of generated audios. The WER metric uses a LareerttLagg-----S- system (Hsu al. 202T), and the SIM-o calculates the cosine similarity of speech samples of each test utterance against reference clips using MaLL--large-bar speaker verification model. For subjective evaluations, we evaluate Naturalness Mcan Opinion Score (NMOS) and Quality Mean Opinion Score (QMOS). For each MOS, six professional linguistic experts evaluate 10 audios per system. The human evaluation instructions are attached in Appendix [ Baseline systems include
ASR
E2TTS (Eskimez al. 2024) VALL-E (Wang et al. 2023). Cosy Voice (Du et al. 2024a), and SparkTTS (Wang et al. 2025b). FireRedTTS (Guo et al. 2024).
TTS Evaluation Results We present the evaluation results in Table 6 Our analysis is as follows:
FlexiCodec-TTS achieves competitive performance with significant speedups. The FlexiCodec- TTS model with a 50Hz NAR decoder demonstrates highly competitive performance across both objective and subjective metrics. At its 6,25Hz AR setting, it achieves WER of 3.2%, Naturalness MOS (NMOS) of 3.32 and a Quality MOS (QMOS) of 3.40, rivaling or exceeding
hhtpp:://githuu..commm/
18

---

Published as a conference paper at ICLR 2026
Table 6: ITS evaluation results. Systems with are borrowed from official results. Each FlexiCodec-TTS is configured to work on each frame rate option without retraining. The actual average frame rate of FlexiCodec-TTI TTS is calculated on all the generated audios during testing.
Model
AR Frame rale
WER,
SIM-ot
RTFIARX
RTR(AR-NARI
NMOSt
QMOS
E2TTS"
94 (NAR)
2.0
0.68
VALL-E*
4.9
0,50
Voice
50
3.2
0.63
0.51 (7.3X)
0.62 (3,4X)
3.17:0095
3.3210.85
SparkTTS
50
6.0
0.55
0,83 (11X)
0.84 (4.7X)
3,15
3.47
FircRedTTS
25
6.8
0.43
0.35 (SX)
0.47 (2.6X)
2.92::0.55
3.10=10.81
FlexiCodee-TTS
12.5 (12.5 actual)
2.5
0.64
0.15 (2.1X)
0.26 (1.4X)
3.27++0.5
3.30++084
(w/ 50Hz NAR)
8.3 (8.0 actual)
2.5
0.65
0.10 (1.4X)
0.22 (1.2X)
3.22-#0.91
3.2840.84
6.25 (6.3 actual)
3.2
0.65
0.07 (1X)
0.18 (IX)
0.32
3.40++0.78
12.5 (12.5 actual)
3.1
0.51
0.15 (2.1X)
0.19
2.87++0.00
2.92
FlexiCodee-TTS
(w/ 12.5Hz NAR)
8.3 (8.0 actual)
3.6
0.50
0.10 (1.4X)
0.13 (0.7X)
2.92:10,99
2.95
6.25 (6.3 actual)
4.6
0.48
0.07 (IX)
0.10 (0.6X)
2.93+1.06
2.87++0.74
strong baselines like CosyVoice (WER 3.2%, NMOS 3.17, QMOS 3.32) and SparkTTS (WER 6.0%, NMOS 3.15, QMOS 3.47). Crucially, this high perceptual quality is achieved with a sig- nificant reduction in computational cost. The 6.25Hz AR configuration has RTF of just 0.07 for the autoregressive stage, representing a 7.3x speedup over Cosy Voice's AR model. Adding the inference time of 50Hz NAR, the total speedup is also substantial, 3.4x.
FlexiCodec-TTS using 12.5Hz NAR shows degraded performance, but still outperforms baselines like VALL-E and FireRedTTS, while achieving higher speedups. This balance of speed and quality makes FlexiCodec-TTS highly suitable for low-latency, resource-constrainge applications.
A higher NAR frame rate is important for high audio quality, but using lower frame rates for AR does not necessarily degrade performance. A comparison between our two FlexiCodec- TTS configurations reveals the distinct roles of the AR and NAR stages. Using a 50Hz NAR decoder consistently yields superior results over 12.5Hz NAR across all metrics, including WER, SIM-o, NMOS, and QMOS. This confirms that a higher temporal resolution in the acoustic decoding stage is vital for reconstructing fine-grained details and preserving naturalness.
Interestingly, for the AR stage, lower frame rates do not always degrade performance. With 50Hz NAR, the 8.3Hz and 6.25Hz AR configurations achieve comparable or even slightly better WER (2.5 vs. 2.5 vs. 3.2), SIM-o (0.64 vs. 0.65 vs. 0.65) compared to 12.5Hz AR. To explain this phenomenon, suggest that (1) lower frame rate tokens might have better semantic and acoustic information disentanglementt, and (2) a shorter sequence may alleviate the challenge of modeling long-range dependencies, allowing the Transformer to learn more effective attention patterns.
C DOWNSTREAM EXPERIMENT: AUDIO UNDERSTANDING
Table 7: Evaluation on various audio understanding tasks.
Spontlin
Emsttion
Sound
Eavironment classification
Keyword spetting
Speaker counting
Emsotion recog.
Non-speeeh
Language
receg.
Model Dataset
ASVSpud2015
CREMA-D
ESC50
Fluent speech
Libricvunt
RAVDESS
Vocalsund
Voelingua33
feature
Whisper
0.53
Dislicug
0.97
0.76
0.36
0.95
0.68
0.75
0.91
Sanse Vouce- Small
0.95
0.71
0.39
0.49
0.08
0.39
mendela w/ RV Sokiw featurey
NowTokeniras-55H
0.02
0.27
Mimi- 12.51fr
0.03
0.46
0.09
0.33
0.59
0.69
0.35
0.73
0.30
DualClodee 12.5ttz
0.93
0.35
0
0.35
0.32
0.53
0.53
0.28
0.64
FestiCode 12.518x
0.93
0.58
0.17
0.30
0.59
0.43
0.50
0.79
0.72
PlaxiCndee 09 3b8
0.92
0.58
0.18
01:28
0.90
0.43
0.50
0.73
0.609
PlesiCuder 6 6.25
0.92
0.54
0.17
0.27
0.98
0.42
0.49
0.74
0.67
To evaluate the quality of the semantic tokens produced by FlexiCodee, and to validate whether FlexiCodec is suitable for building an audio language model, we test its RVQ-1 token embeddings on a diverse suite of 9 audio understanding tasks from the XARES benchmark (Zhang et al., (025c).
19

---

Published as a conference paper at ICLR 2026
We compare against two types of representations: (1) continuous features from strong speech SSL or ASR models (Whisper [Radford et al. 2023). Dasheng (Dinkel et al. 2024). and Sense Voice- Small (An et al.) ). and (2) RVQ-I tokens from other audio codecs. For each task, train a simple downstream model consisting of a linear classifier on top of the frozen embeddings. The results presented in Table 7
The results show that continuous features from large speech models generally outperform codec tokens, which is expected given their continuous nature. Among the codec models, FlexiCodec demonstrates superior performance across most tasks. Notably, on keyword spotting, FlexiCodec at 12.5Hz and 8.3Hz achieves a score of 0.99, matching the performance of the best continuous feature model. Sense Voice-Small, and significantly outperforming other codecs. This highlights the rich semantic information captured in FlexiCodec's tokens. Furthermore, FlexiCodec excels in emotion recognition, non-speech sound detection, and language identification, consistently surpassing other codee models. Even at very low average frame rate of 6.25Hz, FlexiCodee maintains strong perfor- mance, demonstrating its efficiency and effectiveness in preserving crucial semantic content. These results validate that FlexiCodec's tokens are not only suitable for high-quality audio reconstruction but also serve powerful and compact representation for general audio understanding.
D ABLATION STUDY AND ANALYSIS
D.1 ABLATION STUDY
Table 8: Extended ablation study of FlexiCodec. We underline the results that are significantly de- graded from the baseline.
Label
Ablated Ieem
Codec
Semantic Test
Acoustic Test (RVQ1:8)
WERLRVOI))
WER/RVQ1:81]
PESQI
UTMOS:
MCD
SIM
Baseline
FlexiCodec 6622511
4.15
Frame merging module
wio Transformer module
4.19
2.72
4.08
0.67
B2
Use shared parmss for the two streams
216M
8.24
2.37
2.78
4.20
3.43
0.70
Unmerging
C1
wio Transformrr this module
116M
4.22
3.15
2.56
4.05
3.65
0,70
Quautization
DI
Use VQ instead of FSQ
216M
4,43
2,66
2.74
4.18
3.48
0.68
Semantic feature
Use w2y-bert-2 SSL leat.
216M
27.3
5.88
2.47
3,90
3.62
0.74
Use Whisger-modium feat,
216M
9.83
3,27
2.69
4.11
3,50
0.69
E3
Use avg SenseVoice-Se Voice-Small feat.
216M
4.66
2.55
2.74
4.18
3.44
0.73
Table summarizes further ablation study of FlexiCodec, examining the impact of removing or modifying key components including the Frame Merging and Unmerging modules, quantiza- tion schemes, and semantic feature inputs. Each ablated model is retrained following the recipe of FlexiCodec and are tested on 6,25Hz average frame rates. Key observations are summarized below.
Transformers in Frame Merging and Unmerging Modules are critical for maintaining high acoustic quality. Removing the Transformer refinement from either the frame merging unmerg- ing module leads to noticeable decreases acoustic quality metrics (PESQ, UTMOS) and speaker similarity (SIM), as evidenced by the underlined values in rows B1 and CI. However, the impact on the semantic test scores, especially RVQ-1 WER, very small. These results highlight that the Transformer's 's local attention especially refines acoustic features after compression and expansion, reducing artifacts and unnatural transitions caused by naive averaging and frame repetition.
Using SSL feature does not yield good performance in FlexiCodec, compared to ASR features. As shown row E1, adopting the w2v-bert-2 (Barrault et al. 2023) SSL feature (which worked in DualCodec) in system shows degraded semantic and acoustic results, except that its SIM score improves. We have examined the frame similarity pattern of this feature, and found that its adjacent frames tend to have low similarities, it required
very low threshold
20

---

also tried using Whisper-medium Published as a conference paper at ICLR 2026
0.5 to reduce to 6.25Hz average frame rate. We hypothesize that the SSL feature contains leaked acoustic information like timbre, so the frame merging module may not effectively merge semantically consistent tokens together. As comparison, ASR (row E2) and layer-averaged Sensevoice-Small ASR (row E3). Both perform better than SSL. feature, suggesting that the semantic-rich ASR features codecs like FlexiCodec.
suitable for dynamic- frame-i-
D.2 FACTORIAL ANALYSIS OF FLEXICODEC DESIGN CHOICES
Table Factorial analysis of FlexiCodec design choices in relation to RVQI WER at 6,25Hz aver- age frame rate.
Label
FlexiCodec Design choice
WER(RVOD)
Source of data
Semantic fent
Dynamic rate?
Merging&Unmeerpeen transformer?
FSQ?
in this manuscript
FI
w2y-hert-2 SSI.
31.5
Figure 3 Dua CCnde)))
F2
Sense Voice ASR
X
5.99
Section in lext)
F3
Sense Voice ASR
X
5.40
Ablatioo D1)
F5
Sense ASR
X
5.22
Tabl
F6
Voice
FiexiCodee)
Table presents presents a factorial analysis to examine the impact of four central components: the semantic feature source, the Merging&Unmergingg transformers, the dynamic rate mechanism, and Finite State Quantization (FSQ). We note that 5 of the 6 data points in the table have occured previously in the manuscript, so the table mainly provides a factorial view the data. We compare an average frame rate of 6.25Hz to isolate the incremental lift of each design choice on semantic preservation, as measured by RVQ1 reconstruction WER. Key observations are summarized below:
Changing from SSL to ASR feature is the primary driver for semantic preservation fixed low frame rate. As shown by comparing Fl and F2 in Table 9 the most significant improvement comes from replacing the DualCodcc-style w2v-bert-2 SSL features with Voice ASR features. This single change causes the RVQI WER to decrease from 31.5 to 5.99. This confirms that using more semantic-rich feature derived from ASR is the foundational step for achieving good semantic representation at fixed low rate.
The dynamic rate mechanism key to further improving semantic preservation. As dis- cussed in Section 443 the dynamic frame rate mechanism key to further improving semantic information while allowing controllable frame rates. This can be confirmed by examining F3 vs. F4, and F5 vs. F6. We note that the dynamic frame merging reuses the computed ASR features, thus assumes the benefits of the semantic-rich ASR features.
The Merging Unmerging transformers and FSQ provide incremental benefits. The trans- formers and FSQ also contribute to the final performance. Adding the 140M-parameter of the trans- formers (F2 vs. F3) reduces WER from 5.99 to 5.40, showing that the transformers can also benefit fxed-frame-rate codecs. Similarly, integrating FSQ into the full model (F4 vs. F6) provides the final refinement, lowering the WER from 4.43 to our best result of 4. 15. Overall, our analysis validates that all components work in concert, but the combination of ASR features and our dynamic rate mechanism delivers the core innovation and performance gains.
E THE SPEED OF FLEXICODEC
Table 10 presents a comparison of the encoding and decoding speeds, measured by Real-Time Factor (RTF), for FlexiCodec and other state-of-the-arr neural codecs. RTF measured by the ratio of processing time to audio duration. All models were benchmarked one Nvidia V100 GPU using fp32 precision with batch size 1. The average RTF was calculated over our cropped LibriSpeech- test-clean test set.
The results show that, while FlexiCodec not the most efficient among the evaluated systems, still demonstrates acceptable performance. Its encoding and decoding RTF remains constant 0.018
21

---

Published as a conference paper at ICLR 2026
Table 10: Encoding and decoding speed comparison. All models are run on V100 GPU with fp32 precision and batch size of 1. We report the average Reme-Time Factor (RTF) on our test set.
Model
Encode Params
Encode RTF,
Decode Parums
Decode RTF,
DAC-offiial-75H
22M
0.004
52M
0.002
Encodec- 75Hz
7.4M
0.004
7.4M
0.004
SpeechTokenizerr
68M
0.005
35M
0.007
Wavlokenizee-75tt
8.8M
0.005
0.002
Mimi-12.5Hz
38M
0.003
40M
0.002
SemamiCodee-500tt
725M
0,100
196M
0.650
SemantCoodcc255Ht
1660M
0.120
218M
0,643
XYTokemizer- 12.5Hz
306M
0.022
0.005
DualCodec- 12.5Hz
622M
0.019
53M
0.002
FlexiCodec @ 12.5Hz
289M
0.018
161M
0.006
FlexiCodec @8 8.HHz
289M
0,018
161M
0.006
CroxiCodoc@6.2511
289M
0,018
161M
0.006
(encode) and 0.006 (decode) across different frame rate options (12.5Hz, 8.3Hz, and 6.25Hz). The low RTF values indicate that FlexiCodec only adds a very small overhead to downstream tasks like TTS, whose RTFs usually more than 0.1. This positions FlexiCodec a practical and scalable solution for real-world applications.
TESTING FLEXICODEC ON OUT-OF-DOMAIN AND MULTILINGUAL DATA
As FlexiCodec is trained on an audiobook dataset (LibriLight-Large) we wish to know its perfor- mance out-of-domain (OOD) and multilingual speech data to assess its generalizability. For this evaluation, we use Emilia (He et al. 2024). an in-the-wild multilingual dataset featuring spontaneous and conversational speech. As Emilia does provide official test set, craft randomly choosing audios from its training set. For English, we choose one audio from each shard, forming 1,140 speech samples. For other languages, randomly select 200 samples per language. For each language, we use whisper-large-v3 (Radford et al., (023) and evaluate against the tran- scripts in Emilia dataset's metadata. For FlexiCodec evaluations, we determine values from trial runs on each tested language.
cases by
Table 11: Comparison between neural audio codecs English Emilia dataset. We use 1,140 domly sampled speech from Emilia-EN subset. Emilia contains diverse spontaneous speech which out of domain (OOD) for FlexiCodec.
System
RVQI
RVQU8
OOD?
Semantic Test
Acoustic Test (RVQ1:8)
BR(kbps/nng BR(kbps)
WER(RVQD,
WER/RVQL:1)
PESQU
UTMOS?
MCDJ
SIM1
>- -kóps Acoustic Bitrate
DAC-75Hz
0.75
6.07
14.7
6.26
3.67
2.95
3.26
0.91
Eocode-75HH.
1.50
6.07
10.3
6.30
3.10
2.50
3.59
0.92
0.50
4.01 8q
26.8
7.33
2.70
3.06
4.77
0.84
Mini- 12 12.5Hz
1.1
9.37
2.69
3.11
5.14
0.78
DualCodec- XYTokenizer
1.07 8q
6.20
2.92
3.35
4,46
0.89
FlexiCode: 6 12.5Hz
0.23
1.3
8.25
6.47
3.01
3,61
4,18
0.84
8kbyy
Way TowToonnizee-79H.
0.90
0.90/ 14
10.7
10.7
2.73
3.40
4,80
0.75
XCodec2-50f1 50Hzz
0.80
0.80/
8.00
8.00
2:23
2.70
6.88
0.76
FlexiCode: @8.33
0.15
0.85/ /8g
10.8
6,88
2.64
3,59
4.77
0.77
7khps
TAAE- TAAE-25Hl- 700lps
0.70 24
17.3
17.3
2.47
3,90
6.57
0.56
SemantiCCddee Stdeei
0.68
0.68 /g
13.6
13.6
2.13
2.35
6.60
0.61
FlexiCode: G66
Testing English Out-of-Domain Data performance in out-of-domain English speech. each bitrate category. FlexiCodec achieves com- petitive results especially in semantic preservation. On the English Emilia subset, the WER (RVQ1)
As shown in Table 11 FlexiCodec obtains competitive
22

---

Published as a conference paper at ICLR 2026
Table 12: Multilingual Evaluation of codec systems. Among the compared systems. FlexiCodec and Speech are trained only on English speech.
System
WER(RVQL)
WER J
SIM(RVQ1:3) 1
KO
7.11
JA.
DE
DAC-official-5SHH
35.7
77.5
74.3
60.1
49.6
4.87
25.8
16.9
13.6
11.7
0.92
0.92
0.91
0.92
0.92
SpecchTokenizer50000
100
29.6
15.8
0.84
0.86
Way Toweoennize-7355 7511z
9.19
30,0
23.8
16.8
18.5
9.12
30.0
23.8
16.8
18.5
0.81
0.85
0.72
0.81
0.85
Mimi- Mimi-12.5H4 12.5Hz
7.32
32.5
26.1
16.9
14.4
0.77
0.77
0.74
0.77
0.78
XYTokenizer: 12.51
7.28
30.0
20.2
16.8
18.5
0.91
0.89
0.83
0.81
0.85
DualCodee- 12.5Hz
12.9
47.9
47.7
36.8
28.6
5:43
26.1
18.2
13,9
13.0
0.91
0.89
0.88
0.87
0.89
FlexiColee HHz
100
100
58.5
97.5
52.0
6.29
29.6
18.0
14.5
13.4
0.83
0.83
0.82
0.82
0.85
FlexCCodeec 8.311
100
72.7
88.2
71.0
8.32
27.3
31.8
16.2
14.9
0.74
0.75
0.74
0.73
0.78
FlexiColee 2II. (une 12.5 12.5
8.51
48.5
28.0
72.6
50.0
5.91
27.8
18.8
14.8
11.9
0.88
0.84
0.83
0.82
0.84
FlexCoddo--21 tune 8.3328.3H2
10.7
59.7
34.4
100
65.9
7.30
31.9
31.1
17.9
15.3
0.78
0.78
0.77
0.72
0.76
scores for FlexiCodec at 12.5Hz, 8.3Hz and 6.25Hz are 8.25%, 10.8%, and 12.9%. This perfor- mance is highly competitive with other state-of-the-art codecs (e.g.. DualCodec- 12.5Hz has 14.2%, WavTToeenizer-55H has 10.7%). confirming that FlexiCodec's 's superior semantic preservation is not overfitted to clean, audiobook speech, but handles spontaneous, out-of-domain speech robustly. The acoustic results are robust to OOD data well, showing competitive results to in-domain models like DualCodec and XYTokenizer.
Testing on Multilinguul Data To further probe the generalizability of FlexiCodee, we evaluate its performance on non-English languages from the Emilia dataset. The results are shown in Table 12 Since FlexiCodec is trained exclusively on English speech, this presents a significant OOD chal- lenge. When examining the semantic preservation via WER(RVQ1), our model, along with other English-only models like SokeniTokenizer,, struggles significantly, often resulting in high WERs. This is expected, the semantic tokens optimized for English phonetics and not equipped to represent the nuances of other languages. However, when evaluating the more fine fine-grained grained WER(RVQ1:8) and SIM, which leverages the full acoustic information, FlexiCodec demonstrates good ability to generalize. For instance, FlexiCodec @ 12.5Hz achieves WERs (e.g.. 14.5 in French, 13.4 in German) that are competitive with multilingual models like XYTokenizer and Mimi. The acoustic similarity scores (SIM) also highlight FlexiCodec's cross-lingual generalization.
Adapting FlexiCodec to Out-of-Domain Data We demonstrate that FlexiCodec can be adapted to OOD data by finetuning it with Chinese (ZH) data. To perform this adaptation, continue training FlexiCodec with Chinese data from Emilia dataset for 250K steps. The training data does not overlap with the test set. Results shown "FlexiCodec-IH tune" in Table 12 After fine- tuning on Chinese data, it sees a dramatic improvement in semantic preservation for Chinese (ZH) both 12.5Hz and 8. 8.3Hz. This demonstrates that the semantic tokenizer FlexiCodec can learn the phonetic characteristics of a new language with targeted training; the frame merging scheme applies across languages. Furthermore, it is seen that this adaptation also improves FlexiCodec's generalization ability to other languages, as the fine-tuned model shows notable performance gains in languages it was not explicitly trained on. Notably, "FlexiCodec-ZH tune @ 12.5Hz" improves its WER(RVQ1) score on Japanese (JA) from 58.5% to 28.0% and on Korean (KO) from 100% to 48.5%.
G MORE INFORMATION OF BASELINE CODECS
G.1 RETRAINED BASELINES
DAC: a high-fidelity neural audio codec with open-source recipes. proposed two key improvements over Encodec. First, it addressed the codebook collapse problem by reducing the dimension the latent vector to a small value for quantization. Second, It replaced the ReLU activation function with periodic activation function, offering benefits for reconstructing periodic signals such as speech and music. It is trained on more than 10k hours of data including speech, audio and music.
23

---

Published as a conference paper at ICLR 2026
DualCodec: It is a low-frame-rate neural audio codec specifically designed to enhance speech generation tasks like TTS. introduced dual-stream encoding architecture that processes semantic and acoustic information in parallel. One stream rault et al., 2023) to extract rich semantic features. The second stream encodes acoustic information directly from the waveform. This dual approach ensures that the primary codec tokens seman- tically meaningful and allows low frame rates (12.5Hz or 25Hz). Its waveform encoding stream utilizes the architecture of DAC codec.
pre-traieed SSL model w2v-bert-2 (Bar-
To retrain these two systems, we use their public repositories, and modify their codec encoder strides achieve lower frame rates compared their original versions. The configurations are shown in T3] Their decoders mirror their encoders. We have also modified their codebook sizes to align with FlexiCodec. Specifically, we modify to 32768 RVQ-1 codebook size for DualCodec instead of 16384, and use 4096 codebook size for DAC and DualCodec remaining RVQ layers.
To retrain exed-frame-ras FlexiCodec baselines, we also follow Table 13 modify their strides to achieve 8.3Hz and 6.25Hz frame rate, respectively.
Table 13: The CNN encoder strides settings corresponding to each frame rate configuration retrained codec systems.
Codec frame rate
Encoder stride settings
12.5Hz
[4,4.5,822]
[4,5,6,8,2] 8.3Hz 6.25Hz
[4,5,888,2]
G.2 OTHER BASELINES
Encodec DDéfossez et al. 2022): A high-fidelity neural audio codec. It introduced LSTM bottle- neck layers and multi-scale STFT discriminator on top of SoundStream (Zeghidour et al. 2021).
SpeechTokenize (Zhang al., 2023a): A neural audio codec designed for speech language models. It introduced semantic distillation, where the first layer of its RVQ is trained to match semantic-rich representations from pre-trained SSL model HuBERT.
Mimi (Défossez et al. 2 : A neural audio codec designed for real-time applications. It com- presses 24kHz 24kHz.speech into 12.5 Hz token representation with bitrate of 1.1 kbps. It features split RVQ system, where one quantizer captures semantic information and the remaining layers encode acoustic details. It integrates Transformer layers in its bottleneck and distilling semantic knowledge from an SSL model.
WavTokenizer (Ji et 2024): A codec with single-layer neural quantizer operating with large FSQ codebook, optimized for universal speech processing and compression. It is noted for its efficiency and ease of integration into generative frameworks like GPT-4o.
TS3Codec (Wu et al. 2024): A transformer- neural audio codec with FSQ bottleneck, de- signed for minimal bitrate speech encoding. We acquire inference audios from its audios. Its two model offerings, X2 and X4, have been described in its paper.
SNAC (Siuzdak al. 2024): A codec that extends the standard Residual Vector Quantization (RVQ) framework by implementing quantizers operating at different temporal resolutions (Siuzdak al. 2024). Its coarse tokens are sampled lower frequencies. Each RVQ layer operates 12Hz, 23Hz, 47Hz, respetively.
TaDiCodec (Wang et al., 2025c): A speech tokenizer with 6.25Hz tokens and text-assisted de- coding, designed for text-to-speech synthesis. TaDiCodec has transformer-baset encoder and diffusion-basee decoder. Its diffusion decoder utilizes text transcriptions to enhance semantic; ref- erence speech to enhance timbre. During inference, speech, and use HuBERT-Large-LS9en bitrate speech token bitrate text token theoretical bitrate.
the ground truth audio as reference (Hsu et al.) (02T) to obtain the text. We calculate its
24

---

Published as a conference paper at ICLR 2026
TAAE (Stable-Codec) (Parkrr et al. 2024): transformer- based neural audio codee with FSQ or rFS bottlenecks. It aims to achieve high-quality speech at very low bitrates (0.4kbps and 0. 0.7kbps).
The model is effectively scaled up 1B parameters.
SemantiCodec (Liu et al. 2024): a neural audio codec with a diffusion decoder, operating at very low bitrates between 0.31 kbps and 1.43 kbps. Its training data incorporates speech, sound and music.
XYTokenizer (Gong al. 2025): A dual- 12.5Hz neural speech codec with parallel
semantic and acoustic processing streams, operating at low bitrates. The semantic encoder is initial- ized with frozen Whisper-small weights and includes adapter modules, while the acoustic encoder is trainable from scratch.
Comparison between XY-Tokenizer and FlexiCodec While the concurrent work XY-Tokenizer also targets low-frame-rate coding (12.5Hz) using ASR-augmented semantics, our method differs fundamentally in design philosophy. Beyond the distinction of dynamic versus fixed frame rates, we highlight two key structural differences:
Supervision and Scalability: XY-Tokenizer relies on an explicit ASR loss, utilizing paired text transcripts and frozen 0.5B LLM decoder during training. In contrast, FlexiCodec treats the ASR model solely as frozen feature extractor and utilizes feature reconstruc- tion objective. This allows FlexiCodec to be trained entirely on unlabeled audio data, of- fering greater scalability and reduced training resource requirements. However, the text supervision from XY-Tokenizer may offer higher upper bounds, with an additional benefit of being aligned with an LLM's semantic space.
Disentanglement Strategy: FlexiCodec achieves disentanglement via hierarchical resid- ual structure: the first quantizer layer (RVQ-1) explicitly captures semantics, while sub- sequent layers model the acoustic residuals. This flexible hierarchy allows downstream models to selectively utilize only the semantic tokens (RVQ-1) or the full stack for acoustic synthesis. By contrast, XY-Tokenizer employs concatenative scheme without this hicrar- chical residual decoupling, which requires downstream models to processing the full set of tokens rather than offering flexible, separable semantic layer.
We infer each baseline system from its official checkpoint. Each input audio has been resampled to match the codec's s input frame rate. We also note that Mimi, XY-Tokenizer and SNAC do not support decoding solely from RVQ-1 tokens, so their RVQ-1 semantic tests are left blank in Table
H MORE INFORMATION OF FLEXICODEC
Codec Encoder and Decoder The acoustic codec encoder in FlexiCodec follows DAC, which also used by DualCodec. It has a sequential series of 1D convolutional layers. It begins with an initial convolution with a kernel size of 7, followed by set of residual convolutional blocks. Each block contains two dilated convolutions and skip-connection, followed by strided down-sampling layer. As the signal is down wwwn-sample, the number of channels is doubled. A final ID convolution sets the dimensionality of the output features. The decoder mirrors this architecture, using transposed convolutions in place of strided convolutions to upsample the signal, and reconstructs the final audio waveform. FlexiCodec has codec encoder of 15M, and a codec decoder of 35M.
Transformer Contigurations FlexiCodec has 20M parameter transformer in each frame merg- ing module, each has 6 layers, 512 intermediate dimensions, 2048 FFN dimension, and attention heads. The frame unmerging transformer has 100M parameters, with 32 layers, 2048 FFN dimen- sion, and attention heads. The transformers have bidirectional attentions with rotary postional embeddings. We also find that delaying the parameter updates of these transformers can help train- ing stability, especially the unstable RVQ losses. We recommend bypassing them (setting them as identity functions) in the initial training steps, e.g., in the first 10% training steps.
Loss Weights We use weight of 15.0 for the multi-scale spectrogram loss, 2.0 for the GAN feature matching loss, 1.0 for the adversarial loss, 15.0 for the semantic feature alignment loss Le 1.0 and 0.25 for the RVQ codebook and commitment losses, respectively.
25

---

Published as a conference paper at ICLR 2026
The parameter breakdown of FlexiCodec is shown in Table Additionally, the discriminators have 54M parameters in total, but is not used during inference.
Table 14: Number of parameters breakdown FlexiCodec
Component
Parameters
ASR model
234M (frozen)
Codec encoder
15M
Merging transformer (semantic stream)
20M
Merging transformer (acoustic stream)
20M
FSQ (semantic stream)
26M (w/ ConvNeXt module)
RVQ (acoustic stream)
OM
Unmerging transformer
100M
Codec decoder
35M
Total
216M (trainable), 450M (whole)
I ASR PROBING TASK DETAILS
Following XARES Zhang al. 2025c) benchmark, the ASR probing task adds an MLP adaptor be- fore a frozen Qwen2.5-0.5B model. The LM sequence is a concatenation of the adaptor-transformee speech feature, a separator token, and the ground truth text transcription. The model is trained using Nvidia V100 GPU for 10 epochs, a batch size of 16, a cosine decay learning rate of initially 10-3, and decays to 10- 4 over 10 epochs. During inference, a greedy auto-regressive sampling is used to predict the text tokens given the encoded speech features. The feature from FlexiCodec is obtained by using the FSQ continuous features of RVQ-1 tokens. Each frame repeated by its "length" times to become 12.5Hz sequence. This is provide the frame duration information to the ASR, and because XARES only accepts fixed-frame-rate token sequence.
J LIMITATION AND FUTURE WORK
While FlexiCodec demonstrates strong performance in audio compression, semantic preservation, and flexible frame rate, there remain some limitations worth addressing in future research.
Multilingual and streaming support. First, our experiments and model training have been conducted English speech data. The current framework has not been evaluated adapted for multilingual or code-switched speech, which presents unique challenges in semantic compression and phonetic variability. Extending FlexiCodec to handle multilingual corpora is important direc- tion. Second, FlexiCodec presently operates in a full-utterance mode and does not support stream- ing. We have recently seen unofficial adaptations of Sensevoice-Small ASR model for streaming. so adapting FlexiCodec for streaming is technically feasible. A combination of FlexiCodec's 's low frame rate and streaming capability will enable more efficient downstream tasks.
Improving interpretability. While we have attempted to interpret the frame merging patterns by aligning with phonemes, would be interesting to investigate there mapping between phonemes and the semantic token index as well as its frame length. We plan to investigate this in rule-based speech editing task.
We also plan to investigate the integration of FlexiCodec within unified multi- modal understanding and generation frameworks, such as audio L.Mss. Most current audio LLMS use different features for understanding and generation. We will investigate using FlexiCodec tokens for joint semantic understanding and acoustic generation.
Audio LLM.
Improvements inspired by syllable unit discovery methods. Although FlexiCodec was not explicitly designed to model syllables, our visualization analysis (Figure 4l and 5 suggests that the learned dynamic boundaries align closely with syllabic structures. A promising future direction is to explicitly incorporate syllable-level knowledge to guide the merging process. For instance, future work explore using specialized syllable extractors like Sylber (Cho et al. 2024a) replace
26

---

Published as a conference paper at ICLR 2026
augment the current semantic guidance, potentially achieving even better trade-off between semantic abstraction and acoustic reconstruction.
Improving the dynamic frame rate algorithm. In this work, employed simple and effi- cient greedy left-to-right algorithm for the dynamic frame merging process, based on the knowledge of pre-trained ASR models. This merging process contains learnable parameters. While it is em- pirically effective, more sophisticated strategies could be investigated future work. particularly promising direction replace the current heerristic-based algorithm with learnable, end-to-end merging module. Such module could be implemented as a Transformer network that considers the sequence of ASSesyllabiephooneccccccua By making this module differentiable (e.g., using Gumbel-Softtnn or straight-throuhh estimators for discrete decisions), its parameters could be optimized directly via backpropagation from the fi- nal reconstruction loss. The gradients from the decoder would act as global signal, teaching the merging module to make decisions that are not just locally coherent but are globally optimal for minimizing reconstruction error. This approach offers several potential advantages:
features and outputs sequence of merge decisions.
Tassk-Optimized Segmentation: Instead of relying on proxy metric like cosine simi- larity, the model would learn merging strategy tailored specifically the end-goal of semantic reconstruction or high-fidelity audio reconstruction. It could learn, for instance, that preserving certain transient phonetic features more critical for the decoder than pre- serving long, homogenous vowel sounds, and adjust its merging decisions accordingly.
Beyond Heuristics: A learnable module could capture complex relationships in the data that our similarity threshold cannot. It could become sensitive to prosodic contours, speaker-speciffe timing, and other subtle cues that influence perceptual quality.
Developing such learnable dynamic rate mechanism could potentially achieve an even more pow- erful trade-off between compression and quality.
K DECLARATION OF THE USE LLMs
During the preparation of this manuscript, utilized large language models (LLMs) to assist with improving the grammar and readability of the text. The LLMs were used solely as a writing and editing tool and did not contribute to the generation of any original ideas, methods, or conclusions presented in this paper.
L TTS SUBJECTIVE EVALUATION INSTRUCTIONS
The instructions for our TTS subjective evaluation are shown in Table 15 and 16
Table 15: Evaluation instructions for Naturalness Mean Opinion Score (NMOS)
Score
Description
Unacceptable
The voice lacks authenticity and sounds 100% robotic and mechanical. The voice sounds scripted and not human.
2: Poor
Acceptable
The voice sounds human like, but stiff and scripted at the same time. Mostly natural, with some minor scripted moments.
4: Good
Excellent
The voice feels genuinely human.
Instruction: Does the voice sound robotic
human-like?
27

---

Published as a conference paper at ICLR 2026
Table 16: Evaluation instructions for audio Quality Mean Opinion Score (QMOS)
Score
1: Unacceptable
Poor
3: Acceptable
Good
5: Excellent
Instruction: Does it sound like a high-quality recording, without any technical difficulties (e.g.. echoes, static, interference)?
28


Reviews:
[{"review_id": "tLZkJL4mwe", "reviewer": "Reviewer_PDhk", "summary": "This paper presents a neural audio codec / tokenizer for English-only speech. The primary novelty is:\n\n- the use of a specialised module to dynamically alter frame rate.\n- the utilization of a pretrained and frozen ASR model as the 'coarse' token of the codec, removing the need for semantic distillation.", "questions": "- It seems 'RVQ-rest' is trained with 24 levels, and then inferenced with 8? Are all results using the truncated RVQ? Why was it trained with more levels in this case? This needs elaboration or justification.\n- The design for 'RVQ-rest' utilises RVQ, but the downstream TTS models work on the continuous embeddings from this part of the bottleneck. What is the motivation for using RVQ in this section if you're not going to use the tokens?\n- How is the FSQ quantizer trained? Is it using straight-through gradient estimates?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Writing and presentation is good.\n- Overall technical novelty is incremental, but the changes introduced are worthwhile and justified.\n- Results seem overall fairly good", "weaknesses": "- The authors justify the focus on frame-rate by arguing that alignment with the token-rate of text is important. However, this claim of importance feels anecdotal rather than well evidenced (although it is plausible).\n- (minor) The choice of 'RVQ-1' as the name for the coarse/semantic token of the stream is confusing, given that it is not produced by an RVQ (rather FSQ). I understand why the authors chose this name as there is precedent elsewhere, but a better name is needed.\n- Justification of the particular pretrained ASR model used for 'RVQ-1' exists, but direct comparison with previous semantic distillation methods is absent. This makes it hard to judge the impact of this change.\n- The authors state that \"We have not applied FSQ for acoustic quantization because FSQ is a single-layer quantization, and we have not discovered a multi-layer FSQ practice in literature.\". There is a residual FSQ formulation available in a paper you already cite - \"Scaling transformers for low-bitrate high-quality speech coding\" by Parker et al.\n- There are many comparable baselines with public checkpoints available that are not included in the reconstruction evaluation (especially in the <0.7kbps) section.\n- It's good that the authors included subjective metrics (albeit dissapointingly only for downstream TTS, not for reconstruction), but the sample size is so small that the results are very weak. The conclusions drawn from these results need to be softened greatly, given that none of the differences are statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNTmPk22Hw", "reviewer": "Reviewer_TQuw", "summary": "This paper introduces FlexiCodec, a dynamic neural audio codec targeting particularly low frame rates (<12.5Hz) using a variable frame rate instead of a fixed frame rate, providing improvements particularly in semantic information preservation and acoustic quality. \nIt uses an ASR-feature-assisted dual stream architecture to allocate more frames for complex audio segments and fewer to  information-sparse subsequences as in silences or long vowels by adaptively merging adjacent frames. \nExperiments show FlexiCodec significantly outperforms baselines in semantic preservation at 6.25Hz and 8.3Hz, while also supporting variable inference time frame rates (from 3 to 12.5Hz) to control potential trade-offs, and strong performance in downstream TTS tasks.", "questions": "- The Frame Unmerging Module seems relatively expensive for its task (upsampling repeated frames). Could a more efficient upsampling architecture (like a lightweight convolutional upsampler) potentially achieve similar acoustic quality? \n- Token merging is guided by the pretrained semantic features from the pretrained ASR models. The paper's notes on L430 that acoustic and semantic information density could potentially be misaligned. Could you say more about this potential misalignment and whether for example a merging criterion based on the acoustic stream could mitigate this?\n- The paper notes in the limitations the model is not streaming-capable as it operates on full sequences, but that adaptations for streaming are technically. Frame merging scans from left to right to find \"maximal contiguous segments\", implying a need to look ahead. How would this strategy be adapted for a low-latency streaming implementation without or only a limited look-ahead?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents strong empirical results and practical utility, with appropriate comparisons by e.g. retraining very recent work for lower frame rates. Previous work has primarily focused on lowering frame rates to 12.5Hz but not below. \n- A core contribution is the dynamic frame rate mechanism, which (expectedly) primarily improves semantic preservation rather than acoustic representations\n  - Experiments convincingly demonstrate that this dynamic approach significantly improves semantic preservation: compare for example at 6.25Hz, a 26% relative WER reduction compared to a fixed-rate variant, and compared to recent work like DualCodec (Li et al., 2025) retrained for 6.25Hz improvements to 4.15% WER from 31.5% WER\n  - This variable frame rate at inference (3-12.5Hz) is also shown to provide significant speedups for downstream TTS with reasonable tradeoffs in performance\n- The novel ASR feature-assisted dual-stream architecture is also a strength - using features from a pre-trained ASR model, optimized for text prediction, for better semantic information than standard SSL features as validated by ablation studies in the appendix", "weaknesses": "- A limitation is that the decoder and downstream NAR models cannot operate directly on the variable-rate tokens: tokens must first be upsampled back to a 12.5Hz sequence via frame repetition with a (relatively) large 100M Frame Unmerging Transformer, negating some of the efficiency benefits for the synthesis stage. The Frame Unmerging Transformer, accounts for 100M of the models 216M trainable parameters\n- The presented model relies on large pre-trained ASR model (a frozen 230M SenseVoice model) for the semantic features that guide the merging; other models are not compared, so it is not necessarily clear how dependent performance is on the quality and properties of this model or how generalizeable it would be to other languages or domains with weaker models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SxH50TeiZF", "reviewer": "Reviewer_URnt", "summary": "This paper introduces FlexiCodec, a neural audio codec designed to operate at very low and controllable frame rates. The core contribution is a dynamic frame rate mechanism that adaptively merges semantically similar frames, guided by features from a pre-trained ASR model. This is implemented within a dual-stream architecture (semantic and acoustic) that utilizes Transformer-based modules for merging and unmerging frames. The authors demonstrate that this approach improves the preservation of semantic information. The paper also validates FlexiCodec's effectiveness in a downstream TTS application, showing competitive audio quality with substantial inference speedups in the autoregressive stage.", "questions": "See Weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of using pre-trained ASR features to guide the merging of semantically similar frames is intuitive and well-executed. The results in Figure 3a, showing a dramatic improvement in RVQ-1 WER at 6.25Hz, strongly validate this approach's effectiveness in preserving semantic content on in-domain data.\n2. The design allows for a flexible trade-off between semantic quality and sequence length at inference time by simply adjusting the similarity threshold. This is a highly practical feature for applications with varying computational constraints.\n3. The authors conduct a comprehensive set of experiments for their chosen domain, including detailed ablations of the dynamic rate mechanism (Tables 3 & 4), comparisons with numerous existing codecs, and validation on two distinct downstream tasks (TTS and audio understanding).", "weaknesses": "1. The abstract claims, \"We find that a major challenge for very low frame rate tokens is missing semantic information\". I do not fully agree. Recent work on syllabic / dynamic units, specifically Sylber and SyllableLM (Cho et al., 2024, Baade et al., 2024), showed that at around 6-8 Hz you can still carry the linguistic sequence reasonably well, and what starts to go missing is acoustics/prosody/fine timing, not semantics. Figure 4 of the paper actually supports a syllable-like view: FlexiCodec emits about half the phoneme rate, i.e., around syllabic granularity. The authors should tone down this claim and provide a more nuanced motivation that acknowledges that while fixed downsampling can lose transient phonetic details, the main trade-off at very low rates is often acoustic fidelity vs. semantic representation.\n2. The ASR encoder (Sense Voice-Small) was trained on 300k hours of data , and FlexiCodec itself is trained on Librilight-Large (54k hours of audiobooks). The primary evaluation is on LibriSpeech-test-clean, which is also an audiobook dataset. This creates a risk that the excellent WER results are due to the ASR features being highly specialized for clean, read English speech. The claims of superior semantic preservation would be far more compelling if they were supported by an evaluation on an out-of-domain (OOD) dataset, for instance, a corpus of spontaneous or conversational speech (Emilia). This would test whether the ASR-guided merging generalizes beyond the training domain.\n3. The full FlexiCodec model has 216M trainable parameters, with the Frame Unmerging Module alone containing a 100M parameter Transformer. This is a substantial model. While the paper provides Real-Time Factor (RTF) for the downstream TTS task, it omits the RTF for the codec's own encoding and decoding process. This information is critical for assessing the model's practical usability. A model that is fast for downstream tasks but slow to encode/decode may have limited applications.\n4. Table 3 says: removing dynamic frame rate at 6.25 Hz increases RVQ1 WER and probing WER. That’s good, but this ablation is entangled with (a) ASR features, (b) FSQ, (c) transformer smoothing. Right now we cannot tell whether: 1) ASR features alone at 6.25 Hz already close most of the gap; 2) FSQ is the key piece at low rate; 3) or dynamic merging is the actual differentiator. I would suggest providing a factorial ablation: 1) DualCodec-style SSL features, fixed 6.25 Hz; 2) ASR features, fixed 6.25 Hz; 3)ASR features, dynamic 6.25 Hz; 4) FSQ. That way, we can see the incremental lift of each choice.\n5. The paper states: “To our best knowledge, it is the one of first neural audio codecs under 10Hz… and the first work to explore dynamic frame rate on low-frame-rate neural audio codecs.” But there are concurrent <= 10 Hz speech-token systems (TaDiCodec, TASTE), the authors mention them later, and several dynamic-rate codecs, though at higher base rates. This should be toned down.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpGyVTnTyF", "reviewer": "Reviewer_VFuy", "summary": "This paper employs a dynamic neural audio codec, which adaptively merges segments of varying lengths according to different speaking rates to achieve efficient information compression.", "questions": "How does the proposed method handle fast speech scenarios, where phonemes or syllables occur at very high rates? Can the model still achieve low compression rates while maintaining intelligibility and reconstruction quality in such cases? A discussion or experiment addressing this would be valuable.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The use of ASR features as auxiliary information is well-motivated and enables more effective compression. This design allows the model to merge segments corresponding to the same phoneme, thereby shortening the overall sequence length while retaining essential information.", "weaknesses": "I am not entirely certain about my understanding of the implementation. Does each token need to maintain an additional length information field? If so, should this length information also be considered part of the compressed representation, since it seems necessary for accurate audio reconstruction? Clarifying this design choice and its impact on bitrate or compression ratio would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ElsBOy1K0S", "reviewer": "Reviewer_f4v8", "summary": "This paper introduces FlexiCodec, a novel dynamic-rate neural audio codec that successfully addresses the critical problem of semantic information loss at very low frame rates. Its core ideas are innovative and well-executed, specifically the use of a pre-trained ASR model to guide adaptive frame merging and the implementation of controllable frame rates. Supported by extensive and rigorous experiments against strong baselines, the work presents a compelling case for its effectiveness and makes a significant contribution to the field of low-bitrate speech tokenization for language models.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core contribution of a dynamic and controllable frame rate is highly novel and effectively solves a well-motivated problem. The adaptive allocation of temporal resolution based on phonetic complexity is an elegant and powerful approach to preserving semantics.\n\n2. The architecture is thoughtfully designed, cleverly using a pre-trained ASR model for the dual purpose of providing semantic features and guiding the frame merging process. The inclusion of Transformer modules for refinement is also a crucial detail for ensuring high quality.\n\n3. The evaluation is exceptionally thorough and convincing. The authors compare against fairly retrained baselines, conduct extensive ablation studies that validate key design choices, and demonstrate practical utility in downstream TTS and audio understanding tasks.", "weaknesses": "1. The model's evaluation is confined to English, and its reliance on a language-specific ASR model raises concerns about its generalizability to multilingual settings without significant additional effort or resources.\n\n2. A more detailed analysis of the codec's own computational overhead and latency during encoding/decoding would be beneficial to fully assess its efficiency.\n\n3. The frame merging process relies on a simple, greedy left-to-right algorithm. While empirically effective, it may not be globally optimal, and a discussion of more sophisticated segmentation strategies could have strengthened the work..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ls00kcW9Yc", "reviewer": "Reviewer_U5qj", "summary": "They proposes a new codec with the following innovations:\n1. frame merging based on ASR feature similarity, which leads to very low framerate codes\n2. ASR feature condition, for frame merging, and also for RVQ-1\n\nExtensive experiments show that FlexiCodec achieves good semantic and acoustic information preservation, as well as good performance in downstream NCLM-based TTS task.", "questions": "Why do we use FSQ for RVQ-1, as opposed to VQ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Good angle: enhancing semantic info and representation, as well as low framerate compression are very important tasks\n2. Good approach: leverage ASR features is probably the most straightforward approach to enhancing semantic info. Leveraging ASR features similarity for merging has a byproduct which is that we could adjust the compression rate based on the latency requirement\n3. Very extensive and rigorous experiments: the authors not only did ablation studies on most design choices, standard reconstruction-based eval on WER, and acoustic info, but also shown that the alignment between dynamic merging and phonemes, as well as performance on downstream tasks TTS and audio understanding", "weaknesses": "1. XYcodec also uses ASR features, although in a slightly different way - the ASR module is finetuned during training with transcript - which will likely require more resources. Although in table 5 XYcodec is compared, I'd like to see more discussion on comparing XYcodec and FlexiCodec\n\n2. the use of ASR features put FlexiCodec in a different category than most codec models, because it requires supervised signal. It would enhance the paper if the author can show that English-ASR trained model can also do well on unseen languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)