Gogo: Group-wise granularity-ordered codec for stable and efficient speech generation
Authors: Weidong Chen
Helen M. Meng
Xixin Wu
Venue: ICLR 2026 Poster (2026)
Keywords: speech codec
speech language model
speech generation
text-to-speech synthesis
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Current speech language models require their core component, the speech codec, to discretize continuous speech signals into tokens that not only capture high-level cues for autoregressive modeling but also preserve sufficient acoustic details for perceptual quality. To address this need, we propose Gogo, a group-wise granularity-ordered codec that quantizes each group of frames into tokens arranged from coarse to fine, where coarse tokens encode high-level abstractions and fine tokens progressively recover low-level details. Building on the granularity-ordering property of Gogo, we introduce GogoSpeech, a two-stage speech language model that performs speech generation by first constructing a coarse speech backbone at an extremely low token rate and then enriching the backbone with fine-grained acoustic details. Considering the inherently non-uniform information distribution in speech signals, we further design a Group Relative Policy Optimization (GRPO)-trained token allocator that adaptively allocates token budgets to groups based on group-wise complexity. Experimental results demonstrate that Gogo delivers state-of-the-art reconstruction performance across most metrics at a token rate of 47. Moreover, evaluations on zero-shot text-to-speech tasks show that GogoSpeech enables efficient generation by adaptively reducing the average token rate, and attains state-of-the-art results in long-form speech generation.

Published as a conference paper at ICLR 2026
GoGo: GROUP-WISE GRANULARITY-ORDERBI CODEC FOR STABLE AND EFFICIENT SPEECH GENERATION
Weidong Chen, Helen M. Meng, Xixin Wu" The Chinese University of Hong Kong (wdchen, hmmeng, wuxx) 3se. cuhk, edu. hk
ABSTRACT
Current speech language models require their core component, the speech codec, discretize continuous speech signals into tokens that not only capture high- level cues for autoregressive modeling but also preserve sufficient acoustic de- tails for perceptual quality. To address this need, we propose Gogo, group- wise granularity -ordered codec that quantizes each group of frames into tokens ar- ranged from coarse fine, where coarse tokens encode high-level abstractions and fine tokens progressively recover low-level details. Building on the granularity- ordering property of Gogo, we introduce GogoSpeech, a two-stage speech lan- guage model that performs speech generation by first constructing coarse speech backbone at an extremely low token rate and then enriching the backbone with fine- fnn-grained acoustic details. Considering the inherently non-uniform information distribution in speech signals, we further design a Group Relative Policy Opti- mization (GRPO)-trained token allocator that adaptively allocates token budgets to groups based on group-wii wwie complexity. Experimental results demonstrate that Gogo delivers state-of-the- -fhth--rr reconstruction performance across most metries token rate of 47. Moreover, evaluations on zero-shot text-to-speech tasks show that GogoSpeech enables efficient generation by adaptively reducing the average token rate, and attains state-of-the-art results in long-form speech generation.
INTRODUCTION
Large language models (LLMMs) such as the GPT series (Brown et al. 2020 OpenAl 2024) have demonstrated remarkable capabilities across diverse text-based tasks. Their success has inspired growing efforts to extend the LLM paradigm to the speech modality, leading to the development of speech language models (SLMs) capable of understanding and generating speech (Ye al., 2025b Délossez et al. 2024). A common pipeline for SLMs first discretizes speech into sequences of tokens via an audio codec, then models both text and speech tokens autoregressively. The effectiveness of this approach hinges critically on the codec's ' ability to produce tokens that concurrently contain high-level cues (e.g., content, semantics, and structural attributes) for autoregressive modeling (Ye) et al. 2025a) and low-level details (e.g., acoustic fluctuations) for perceptual quality preservation.
Conventional audio codecs use frame-wise quantization for compression and transmission (Kleijn) al. 2021 Valin et al. 2012). While this enables high-fidelity reconstruction, its strong locality bias limits the codec's 's ability to capture high-level cues needed by SLMs. To address this limitation, recent works have augmented codecs with sule-supe representations (Zhang et al. 2024 Li
jet al., (225)) or automatic speech recognition (ASR) features (Jo et al.) 2025] Zeng et al. explicitly inject high-level linguistic and semantic information into the quantization process. How- ever, the fundamental frame-wise quantization paradigm remains unbroken, inherently limiting the ability to learn high-level information. Besides, little attention has been given to the non-uniform information distribution in speech (Dieleman et al. 2021 Voran (0024). Current approaches gener- ally allocate one same bitrate all segments, which leads to redundant coding and low generation efficiency, especially in less complex segments like silence where low coding rate would suffice.
2025) to
To address the aforementioned limitations, we redesign both the codec and SLM framework. As we first propose Gogo, group-wise camullaitt-orderee codec that processes
depicted in Figure [
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Gogo
Group
Coarse
Fine
Tokenizer
GogoSpeech
Speech Input
Stage
You did it!)
Detokenizer
Text Input
Stage
Speech Ouiput
Budgets
Speech Token
Token
No Need Generate
Token
Allocator
Figure 1: System overview. The shading of a token reflects the granularity of the information encodes. The token allocator allocates different token budgets to different groups within the same utterance based on their complexity. Only one group is plotted for simplicity. Best viewed color.
contiguous frames as groups and generates tokens in a coarse-to-fine order, where coarse tokens capture high-level information and fine tokens are used to progressively restore low-level acoustic details. Building on the granularity-ordering property of Gogo, we develop GogoSpeech, a two- stage SLM for speech generation. In the first stage, high-level speech backbone, which serves as a coarse indicator of the target speech, is predicted at extremely low feature rate (around 14 Hz). This reduced feature enhances the stability of autoregressive prediction and mitigates ror accumulation (Arora et al. 2022 He et al. (021). In the second stage, fine aieeggrinne details are incrementally recovered conditioned on the speech backbone. The feature rate the second stage is restored to standard level to ensure high-fidelity synthesis with precise details. To further enhance efficiency, we propose Group Relative Policy Optimization (GRPO)-trained (Shao et al.) (024) to- ken allocator that dynamically assigns token budgets to every group based on its complexity. thereby aligning computational resources with the non-uniform information density of speech signals.
Our main contributions in this work are fivefold: (1) proposing new speech codec, namely Gogo, featuring group-by- boououu and granularity-ordered tokenization that better addresses the representa- tional requirements of SLMs; (2) constructing
speech language model, namely GogoSpeech, enabling staged speech generation from high-level abstractions to fine acoustic details; (3) develop- ing GRPO- token allocator that optimizes efficiency and quality by aligning token budgets with group-wise complexity; (4) conducting reconstruction experiments to show that Gogo achieves text-to- speech (TTS) to show that GogoSpeech achieves better performance with higher stability and effi- ciency. Demo samples can be found at https: //ttts://haaaeee httpsss/hhppppcooooo htt.:ooorr..ituu..oooooo io/gooo
superior performance compared state-of-the-art codecs; (5) performing experiments
2 RELATED WORK
2.1 NEURAL AUDIO CODECS
Modern neural audio codecs predominantly based the VQ-GAN framework (Esser et al.
2021]. which integrates an encoder, a vector quantizer, and decoder into an end-to-end system. Pioneering works such SoundStream (Zeghidour al. 2021} and EnCodec (Défossez et al., 2022) employ residual vector quantization (RVQ) together with carefully designed discriminators improve perceptual fidelity. DAC (Kumar et al. 2023 further enhances codebook utilization by performing code lookup in a low-dimensional space using cosine similarity rather than Euclidean distance. To improve compatibility with SLMM,, recent studies have explored injecting linguistic or semantic information into the quantization process. One line of work leverages self self-supervised
speech representations from models such as HuBERT (Hsu et al. 2021). WavLM (Chen et al. 2022),
and w2v-BERT (Chung et al. 1021 l). For example, SpeechTokenizer (Zhang et al. 2024 distills
semantic teacher representations into the first stage of RVQ. while Mimi (Défossez et al. 2024
transfers semantic information into single-stagg stage quantizer to decouple acoustic reconstruction from semantic coding. Another line of research integrates ASR features into codec training. Notably, S
tokenizer (Du et al. 2024a) partitions the encoder of a pretrained Sense Voice ASR model (An al.
2

---

Published as a conference paper at ICLR 2026
2024 and inserts quantization layer between its two halves. Despite recent advancements, the frame-wis wise quantization paradigm remains unchanged. Its inherent locality bias limits the codec's ability to learn high-level cues, which essential for stable autoregressive modeling in SLMs.
2.2 SPEECH LANGUAGE MODELS
Generative speech language modeling extends the LLM paradigm to the speech modality by mod- eling discrete speech tokens produced by neural audio codecs or clustering methods such as k- means. Early works such as GSLM (Lakhotia et al. (02T) and SpeechGPT (Zhang et al. (223)) di- rectly train language models on speech tokens, enabling the generation of natural-sounding speech. VALL-E (Wang et al. 2023) adopts autoregressive model generate the first token and non- autoregressive model to predict residual tokens from EnCodec. To address the trade-off between high-fidelity reconstruction and effective autoregressive modeling, AudioLM (Borsos et al., 2023) introduces a hierarchical modeling framework that first generates semantic tokens and then refines them with acoustic tokens. In general, acoustic tokens are designed encode speech trate while preserving as much information as possible, whereas semantic tokens are learned from self-supervised speech models capture phonetic or semantic representations that facilitate speech comprehension (Guo et al., 2025b). This hierarchical paradigm has inspired numerous follow-up works, including AudioPaLi (Rubenstein et al., 2023). J. Moshi (Défossez et al. (024).. and TTS- Llama (Shen et al. 2025). While the hierarchical approach improves stability and controllability, the semantic modeling stage in existing methods typically operates the same token rate as the acoustic modeling stage, which is substantially higher than the token rate used in the text modality.
low bi-
2.3 ADAPTIVE BITRATE IN NEURAL CODECS
The non-uniform information density of speech signals makes constant-bitrate codecs inherently in- efficient. SNAC (Siuzdak et al., 22024 extends RVQ operate at multiple temporal resolutions, yet the bitrate remains fixed across different speech regions. Acoustic BPE (Shen et al., 2024) applies
the byte-pair encoding (BPE) algorithm (Devlin al., 2019) to speech tokens. reducing sequence length and increasing token correlation. More recently. VRVQ (Chae et al.] (025) introduces a variable-bitee strategy into RVQ, allowing the number of quantizers per frame to adaptively determined from predicted importance map. Similarly, TFC (Zhang et al. (2025) dynamically allo- cates frame rates to different regions according to temporal entropy. However, these variable-bitrate codecs do not explicitly couple bitrate variation with reconstruction quality for joint optimization, and their effectiveness in generative tasks within the SLM framework remains underexplored.
3 METHODS
3.1
GoGO
The proposed Gogo, as shown in Figure comprises three main components: an encoder for learn- ing speech representations, a flow-based generative model (Lipman et al. 2023 Tong et al. 2024) for mel-spectrogram reconstruction, and vocoder for converting spectrograms into waveforms. Additionally, Gogo integrates an ASR module and an autoregressive (AR) prior (Wang et al. 2025
Yang et 2025) to enhance the suitability of the learned speech tokens for downstream generation.
3.1.1 WORKFLOW
Given an input waveform w, we first extract its mel-spectrogram I R where ny denotes the number of frames and d the number of mel bins. The spectrogram is then partitioned along the temporal axis into multiple non-overlap groups æ¹ Roxd where g denotes the group size, i E [1, denotes the group index, and ny = 1 denotes the total number of groups. The last group is zero-padded if necessary. Subsequently, each group is concatenated with ny learnable queries o E xd which yields z' E R(s+n,)xd Incorporating queries into quantization has been explored in ALMTokenizer (Yang et al, 2025). TYTok (Yu et al.) 2024). and FlexTok (Bachmann et al. 2025). The extended sequences 2 encoded by Transformer Vaswani et al. 2017) encoder, after which the z' part is discarded and finite scalar quantization (FSQ) (Mentzer et al. 2024) is applied to the positions corresponding the learnable queries, producing speech token indices s'
3

---

Published as a conference paper at ICLR 2026
Input Speech
Speech Token
Vocoder
Reconstructed Speech
Place Holder
Masking Token
Mel
Flow-matching
Only for Training
Noise
x
Grouping
Nosted Dropout
Flatten & Align
ASR
Transformer Encoder
AR Prior
Figure 2: Architecture of Gogo. Here the group size g and the number of speech queries Mg assigned to each group is set to 3 for demonstration. Best viewed in color.
R" and their corresponding embeddings 7' e R xdx,, where d, denotes the hidden dimension. Formally, the tokenization workflow in Gogo is given by:
x Mel(wr),
x Grouping(z:).
z Cat(a', .q)),
FSQ(Encoder(z"),
where Mel(-) denotes the mel extraction operation, Grouping(-) denotes the grouping operation, and Cat(-) denotes the concatenation operation. We omit the batch dimension for clarity. In practice, the effective batch size for the encoder is ny times the number of speech samples loaded in batch.
For reconstruction, 7' are first padded with (g - (t,) placeholder tokens to match the original group length resulting in aligned features F² E Rexds. All groups" z are concatenated along the time axis to generate E xds, which is then fed into a flow-matching model to predict the mel- spectrogram. The final waveform is recovered using pretrained Vocos (Siuzdak 2023) vocoder. Formally, the reconstruction workflow in Gogo is given by:
Cat(q', Placeholders),
Cat(a' ........ .F*),,
Vocoder(Flow
3.1.2 TRAINING OBJECTIVES
Gogo leverages conditional flow matching (CFM) (ippman et al. 2023). which extends the frame- work of continuous normalizing flows (Chen et al. 2018) learn time-dependent vector field that transports simple prior distribution, e.g., the standard normal distribution, to the distribu- tion of the mel-spectrograms conditioned on the features ž. Given a mel-spectrogram #1 and Gaussian noise
(0, 1), we first interpolate between them and produce a noisy spectrogram It = (1 t) + to using a flow time ~ U((0, 1). The conditional vector field for this linear interpolation is v v(2o, F1, t) 0rt/Ot 21 Fy. The flow-masn now-matching model, parameterized by 8, takes (I,, H. t) as input and predicts a velocity vector Vo 3, t). Formally, the CFM objective is defined as simple vector field regression loss:
LCRM ((.,(())) [1v, F, t) v(zo, 1 01
We further introduce two auxiliary modules, AR prior and ASR module, to encourage the speech queries capture temporal dependencies and linguistic information within each group, respectively.

---

Published as a conference paper at ICLR 2026
The corresponding training objectives, CAR and LASR. are formally defined in Appendix B the objective for training Gogo can be written as:
Finally,
Gogo ACrM + AARCAR CASR
where ACFM, AAR, and AASR are coefficients employed to balance different loss components.
3.1.3 GRANULARITY ORDERING
To enforce coarse-to fine ordering in the learned speech queries q and tokens s', we introduce two techniques: nested dropout (Rippel et al. 2014) and loss balancer.
Nested dropout randomly drops tokens in nested fashion during training. Specifically, uni- formly sample the number of tokens to retain, T4 E {1...., ng and replace the last (ng ng) tokens with masking tokens This mechanism drives Gogo to prioritize encoding high-level ab- stractions and essential structural attributes into the earlier coarse tokens to minimize Loogo to the greatest extent, while deferring the challenging and fluctuating details to the later fine tokens. Since later tokens are rarely preserved and receive fewer gradient updates, we further introduce a re- weighting mechanism for compensation. Concretely, the gradient of the j-th speech token scaled by wy = 0/5 (j 1), assigning larger weights to tokens with fewer updates and vice versa. Details of the re-weighting implementation are provided in Appendix
Loss balancer is utilized to adjust the loss coefficients λсEм and AASR- further ensuring that the learned speech tokens are organized in a coarse-to to-fine manner. Specifically, when n& is small, the model should emphasize LASR so that coarse tokens encode richer linguistic content. Conversely, ensure that fine tokens capture more acoustic details.
when nig is large, LCrM should dominate Let Amax and Amin denote the maximum and minimum weighting coefficients, respectively. The loss balancer adaptively adjusts ACFM and dASR as follows:
(na 1)(Amax Amin)
(ne 1)(Amax Amin)
ACPM
Amin
AASR Amax
(5)
ngg
1
3.2 GOGOSPEECH
GagoSpeech Stage
You did it!
Speech Buckbone Construction
Text Input
Speech Prompt
Gogo
Gogo
Speech Outpur
Granularity
GogaSpeech Stage Speech Details Enrichment
Groups
Speech Tokens
Figure 3: Architecture of GogoSpeech. Gogo encodes the speech prompt into speech tokens, which serve as input to GogoSpeech for generating the target speech tokens. The target speech tokens are transformed into waveform by Gogo. For visualization, the number of speech queries n, in each group is set to 10 and the speech backbone defined as the first speech tokens of each group.
3.2. STAGE I: SPEECH BACKBONE CONSTRUCTION
Stacking all s¹ yields 2D token matrix S E _ XM,, The speech backbone is defined as the first b tokens of each group, i.e., S, E g which contains the high-level cues of the speech signal. As shown in Figure 3 given the input text y (y1,.... 1. the autoregressive model in Stage generates the backbone of target speech S. IR,
Uu.....92) 92) and the backbone of speech prompt
5

---

Published as a conference paper at ICLR 2026
group by group, where ny denotes the number of groups in the target speech. Let I(+) denote the operation that flattens token matrix into a sequence. The objective of Stage is to minimize the negative log- ligeeiklihood over the target speech backbone:
Estagel
log P(St. I() St.1r1))
(6)
3.2.2 STAGE II: SPEECH DETAILS ENRICHMENT
In Stage II, GogoSpeech progressively enriches the speech backhone predicted in Stage by adding fine-grained acoustic details group by group. For the i-th group of the target speech, the autore- gressive model in Stage II generates the fine tokens Si,++in, conditioned on all tokens of the input speech prompt S, all tokens of the previously generated groups $ 111-1.: and the speech backbone of the current group S1,1.b The training objective for Stage is given by:
Ag
T2g
Lestage2
log P(S [T(S), ((S15-1,), 11,12-1
(7)
1=
3.3 TOKEN ALLOCATOR
Gogo
Hagp Speech
MSE
ri
A
Gogo
T2
Group
Ag
Computation
YG
Ac
01
Token
02
Allocator
Speech Token
0G
Dropped Token
Figure 4: GRPO-trained token allocator. Throughout the GRPO training. the Gogo is kept frozen.
To improve the efficiency of speech generation, we want to allocate more tokens to acoustically complex groups while assigning fewer tokens to simpler ones, such silence. To enable such adap- tive allocation, we design token allocator that receives the backbone of each group, S; S input and outputs budget & E {0, 1, Bg - b}. indicating the number of fine tokens to be generated for that group in GogoSpeech Stage II. The skipped unnecessary fine tokens are replaced with masking tokens m. The cooperation between the token allocator πw, and GogoSpeech is formulated as:
&
Tw (S.1.b) E {0, 1...., ,,. b},
(8)
b+C.
S1.411444 arg max
II
p
Uj v. (()), r( 1.), S.15-1), Si.bee m.
As shown Figure 4 the token allocator is trained from scratch using a slightly modified GRPO technique (Shao et al. 2024). Given that the output space of the allocator is relatively small, compris- ing (n, - b++1) discrete allocation choices, we enumerate possible outputs (01, o2. ,GG), cor- responding to using from b to n, tokens for reconstructing the input speech. The resulting (ny b++1) reconstructed samples are employed to compute the group scores.
We adopt two reward metrics, including R, which penalizes the number of tokens utilized for construction, and R. which penalizes the distance between the input and reconstructed speech. This joint reward encourage the allocator to learn allocation strategies that achieve high reconstruction fidelity while minimizing the number of tokens consumed. The rewards R,, and R, and the entire reward R defined as follows:
R2 -Nmm (7) Ra -E [|Me(()) Mel(
(10)
R R., AdRd.
(11)

---

Published as a conference paper at ICLR 2026
where Num(E) ( denotes the number of speech tokens used to generate An and X are coefficients used to balance the two reward terms. After obtaining the reward for each allocation choice, the advantage is calculated using group relative advantage estimation (Shao et al. 20224:
Aj R₃ MR) ()d
(12)
Since the token allocator is initialized from scratch, omit the KL penalty term used in the original GRPO framework. The allocator Tu is then optimized by maximizing the following objective:
G
JGRPO E Boyao (01
وoح St 1a)
(13)
G
j=1
EXPERIMENTAL SETUP
4.1 IMPLEMENTATION DETAILS
We extract 100-dimensional log mel-filterbank features from audio resampled to 24 kHz, using hop length of 256 and a window size of 1024, resulting in a feature rate of approximately 94 Hz. We the group size to g 20, and allocate ng = 10 speech queries per group. Therefore, the token 47 Hz. The speech backbone is defined as the first 6 = 3
rate of Gogo is computed as R2y x (94/g) tokens of each group. Thus the token rate of backbone is about 14 Hz. Both Stage I and Stage II of GogoSpeech are initialized from the LLaMA (Grattafiori et al. (024). with the vocabulary expanded to include speech tokens. A pretrained Vocos (Siuzdak 2023) is employed to convert the generated spectrograms into waveforms. Please refer to Appendix C for more detailed model configuration.
During training, we empirically set the weight of loss LAR to AAR = 0.06 and amplify the gradient of the AR prior by factor of 50 to ensure effective updates. The coefficients for the losses LASr and LCFM are dynamically adjusted using the loss balancer defined in Eq, [ with Amin = 0.2 and Amax 1.8. For training the token allocator via GRPO, we set the reward coefficients X, 0.2 and Ad = 1.0 to balance the trade-off between token efficiency and reconstruction quality. Additional hyperparameters, training schedules, and inference configurations are provided in Appendix [
4.2 DATASETS
We train both Gogo, GogoSpeech, and the token allocator on the Emilia dataset (He et al.) 2024),
a large-scale and diverse in-the-wild speech corpus designed for multilingual speech generation. In this work, use its English subset, which contains approximately 50K hours of transcribed speech covering a diverse set of speakers, acoustic characteristics, and background conditions, For evaluating the reconstruction quality of Gogo, we adopt the LibriTTS test-clean set (Zen al. 2019) with 4,837 samples in total. To assess the zero-shot speech generation capability of GogoSpeech, we use the Seed-TTS test-en set (Anastassiou al. 2024), which consists of 1,000 samples drawn from Common Voice dataset (Ardila et al. 2019). All speech samples are resampled to 24 kHz.
4.3 BASELINES AND EVALUATION METRICS
We compare Gogo against multiple codec baselines, including EnCodec DDéoossez et al. 2022
DAC (Kumar et al.] 2023), SpeechTokenizer (Zhang et al. 2024). Mimi DDéoossez et al. 2024
SNAC (Siuzdak et al. 2024), Wav (Ji et al. 2025), 9. MagiCodec (Song et al.] 2025) X-
codec2 (Ye et al. 2025b), TAAE (Parker et al. 2025). and DualCodec (Li et al.; 2025). ). All baseline
results are obtained using their official checkpoints. Details of compared codees Appendix
We compare GogoSpeech with TTS baselines, including FircRedTTS-IS (Guo et al. 2025a). F5-
TTS Chen et al. 22025), XTTS-v2 (Casanova et al. 2024). Llasa (Ye et al.) 2025b). Cosy Voice 2 (Dul et al. 2024b), and VoiceCraft (Peng al. 2024). More details can be found in Appendix
For evaluation, we employ both objective and subjective metrics. For objective assessment, adopt UT-MOS (Sacki et al. (022). DNS-MOS (Reddy et al. 2022), and Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 2001) quantify perceptual quality and speech distortion. Speech
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between different codec models the LibriTTS test-clean set. Bold values indicate the best for each token rate. TPS and FPS denote the number of tokens and frames per second, respectively, #CB denotes the number of codebook employed in each model.
UT
DNS MOS
PESQ WB
PESQ
Model
TPS
FPS
#CB
STOI
SIM
WER
MOS
Ground Truth
4.13
3.83
1.00
4.64
4.55
1.00
5.86
DAC
600
8
3.78
3.75
0.99
3.52
3.85
0.98
6.10
EnCodec
600
75
8
3.13
3.56
0.94
2.74
3.36
0.97
6.24
DAC
150
75
2
1.94
3.27
0.85
1.53
1.95
0.90
10.81
EnCodec
150
75
2
1.57
3.20
0.85
1.54
1.92
0.91
8.98
SpeechTokenize Tokenizer
150
50
3
3.10
3.56
0.85
1.47
1.86
0.90
7.23
Mimi
150
12.5
12
3.88
3.77
0.94
2.67
3.22
0.95
6.54
SNAC
82
47
3
3,80
3.84
0.91
2.23
2.75
0.91
7.47
DAC
75
75
1.33
2.97
0.76
1.18
1.45
0.81
30.06
EnCodec
75
75
1.24
2.69
0.78
1.21
1.45
0.77
33.15
Wav
75
75
4.11
3,65
0.92
2.43
2.96
0.90
8.34
Mimi
75
12.5
6
3.54
3.69
0.90
2.01
2.53
0.90
7.65
SpeechTokknizz
50
50
1.31
3.09
0.68
1.11
1.28
0.67
9.18
MagiCodec
50
4.21
3.96
0.93
2.55
3.18
0.86
7.45
X-codec2
50
50
1
4.17
3.90
0.92
2.45
3.07
0.83
6.40
TAAE
50
25
2
4.27
3,89
0.91
2.14
2.82
0.87
8.18
DualCodec
50
25
2
4.05
3,80
0.89
2.02
2.58
0.89
6.54
Mimi
50
12.5
4
3.16
3.62
0.86
1.64
2.10
0.87
9.24
Gogo
47
47
4.19
3.99
0.92
2.59
3.26
0.91
6.35
intelligibility is measured using Short-Time Objective Intelligibility (STOI) TTaal et al. 2010 and Word Error Rate (WER). addition, speaker similarity (SIM) is calculated to evaluate the accuracy of speaker identity preservation. For subjective evaluation, employ the Similarity Mean Opinion Score (SMOS) and Comparative MOS (CMOS) to assess speaker similarity and relative naturalness, respectively. Detailed definitions for each metric are provided in Appendix G
EXPERIMENTAL RESULTS
5.1 CODEC COMPARISON
We compare our Gogo with range of existing codecs, and the results are summarized in Table I Despite operating at relatively low token rate of 47 tokens per second, Gogo achieves superior performance across multiple metrics compared to codecs operating at 50 tokens per second. DAC and EnCodec achieve the best overall performance when operating the exception UT-MOS and DNS-MOS. However, their performance degrades significantly as the token rate is reduced. Notably, Gogo attains UT-MOS and DNS-MOS scores that even surpass the ground-truth recordings, which we attribute to the generative nature of Gogo's flow-matching decoder, enabling enhanced perceptual quality and improved noise robustness.
high token rate of 600, with
5.2 EFFECTIVENESS OF GROUP-WISE QUANTIZATION FOR AUTOREGRESSIVE MODELING
To evaluate whether group-wer quantization better supports downstream modeling, we train naive autoregressive model to perform AR prediction over speech token sequences, which are con- structed by collecting the j-th token from each group in Gogo, where j E [1, ngJ. For the frame-wise baseline, remove the grouping operation in Gogo and replace FSQ with 10-level RVQ, follow- ing the standard frame- wise guantization scheme. Each RVQ layer generates a frame-level token sequence that encodes the residual information left by all preceding layers. Each sequence is mod- eled autoregressively. and perplexity is reported as an indicator modeling difficulty. Additional experimental details and perplexity computation provided in Appendix
8

---

Published as a conference paper at ICLR 2026
Table 2: Perplexity of autoregressive modeling on speech tokens produced by different quantization schemes. Column headers specity the source of speech tokens, denoting their positions within each group for group-wise quantization their corresponding RVQ layer for frame-wist wise quantization. The boldface denotes the best result. i indicates frame-wist wise quantization with single-layer VQ.
Scheme
2
3
4
5
6
8
9
10
Frame-wise f
247.8
Frame-wise
2.3
25.9
84.4
114.9
189.2
261.2
441.8
442.0
727.8
691.4
Group-wise
0,9
8,5
42.0
96.9
169.5
201.6
204.1
221.9
229.6
228.3
The perplexity results are summarized in Table Group-wise quantization consistently yields lower perplexity across all granularities than frame-wis, quantization, suggesting that the group-we wise to- kens produced by Gogo are more seaneerressiv--riennd and capture temporal dependencies more effectively. We can also that coarse tokens yield substantially lower perplexity than fine tokens in both quantization schemes, confirming that fine-grained acoustic details are more challenging for AR models to predict. These findings further motivate the two-stage design of GogoSpeech, where a high-level speech backbone is generated first, followed by fine- ine-grained detail enrichment.
5.3 WHAT Do GRANULARITY--RDEERS TOKENS ENCODE?
To gain deeper insight into the behavior of group-wise granularity-orderee quantization, we conduct probing experiments on Gogo's tokens different granularities to examine the type of information each token encodes. We probe these tokens using diverse set of acoustic, prosodic, and linguistic features. The probing task is formulated as a regression problem, where the mean squared error indicator of each token's 's representational capacity.
reported by the probing model is used More details of the probing setup are provided in Appendix
Voice to Unveiced Ratio
Unique Word Count
Aejctivee
Total Pronouns
Conjanction
80
Energy Estropy
Total Adveit
255
Shimmer
Piich
Total Namt
Zero Crossing Rate
60
20
Speaking Rate
15-
Temee
00
10
20
10
Figure 5: Performance of granularity- tokens across multiple feature prediction tasks. Re- sults are visualized as stacked area charts, where the x-axis denotes token positions within each group and the y-axis indicates the normalized prediction loss relative to the maximum loss for each feature. A higher value corresponds to greater loss and thus lower predictive performance.
The probing results, presented in Figure 5 reveal a clear progression of information across token granularities. Specifically, the first three tokens primarily capture global and high-level information, such as total duration, boiced-to-unvoies middle range predominantly encode prosodic attributes, including speaking rate, jitter, and shimmer. Finally, the last three tokens are responsible for capturing detailed acoustic information, such as pitch, energy, and spectral centroid. These findings confirm that Gogo's group-wise quantization organizes tokens in a coarse-to-fie to-sse-to-fin manner, allowing high-level linguistic and prosodic cues to be modeled with fewer tokens while reserving fine tokens for detailed acoustic reconstruction.
ratio, word count, and linguistic content. Tokens in the
5.4 ZERO-SHOT TTS COMPARISON
We compare GogoSpeech with several state-of oftte-of-thharn the-art TTS baselines, and the results are summarized In objective evaluations, GogoSpeech achieves the highest SIM and competitive WER
in Table 3

---

Published as a conference paper at ICLR 2026
Table 3: Comparison between different TTS models the Seed-TTS test-en set. The boldface de- notes the best result, the underline denotes the second best. The Real-Time Factor (RTF) is computed by averaging the inference time of 47-character sentence over 100 trials on an H100 GPU.
Objective
Subjective
Model
SIM
WER
SIM
WERt
RTF
SMOS
CMOS
Ground Truth
0.734
2.143
0.809
2.037
4.752
0.000
F5-TTS (Chen et al. 2025
0.647
1.830
0.716
1.812
0.184
4.173
+1.73
XTTS-v2 Casanova al. 2024)
0.463
3.248
0.490
2.292
0.208
2.426
-0.61
Llasa-8B-250 Ye et al.] 2025b
0.574
2.970
0.629
5.947
0.944
3.297
+0.882
Voice (Du et al. 2024b)
0.654
2.380
0.701
2.324
0.549
4.331
+1.638
FireRedTTS- Is Guo et al. 2025a)
0.660
2.170
0.705
2.129
0.506
4.247
+1.634
VoiceCraft Peng el al. GogoSpeech (47 Hz)
2024)
0.470
7.556
0.360
10.25
1,248
2.965
00.51
0.667
2.394
0.725
1.788
0.535
4.381
L.832
w/ Allocator (47 Hz 36 Hz)
0.662
2.469
0.717
1.845
0.455
4.253
+15587
All target speeches corresponding the same prompt speech concatenated, and only constructed ples longer than are retained evaluate stability systems in long speech generation.
F5-TTS is fully -on-autoregressive, while the other systems include autoregressive decoding.
compared the leading systems. For long-form speech generation, it further attains the best SIM and WER, demonstrating the effectiveness of the two-stage design in enhancing generation stability. In subjective evaluations, GogoSpeech achieves the best SMOS and CMOS scores, confirming its ability to preserve speaker identity while maintaining strong intelligibility and overall quality.
5.5 EFFECTIVENESS OF TOKEN ALLOCATOR
We further evaluate the effectiveness the proposed token allocator by comparing GogoSpeech with and without adaptive token allocation. The results presented in the last two rows of Table With the token allocator, GogoSpeech generates on average only 36 tokens per second speech, compared 47 tokens without adaptive allocation. The token allocator significantly reduces the computational cost of speech generation and incurs only a marginal performance degradation in both objective and subjective scores. As for inference efficiency, F5-TTS fully non-autoregressive and therefore naturally achieves the lowest RTF. Among the AR systems, GogoSpeech with the token allocator only slower than XTTS-v2, yet it delivers substantially stronger performance. These results demonstrate that the token allocator achieves a favorable trade-off between generation efficiency and speech quality. Please refer Appendix D for visualization of adaptive allocation.
LIMITATIONS
Despite its strong performance, our system has several limitations. First, placeholder tokens in the How-matching decoder can occasionally introduce artifacts. Second, Gogo operates at a token rate of 47 Hz, which higher than the low-bitrate codecs of 25 Hz. Finally, GogoSpeech is built on Llamm-3.2-11 Alama-3.2-1B-Innttmee and its scalability to larger language models requires further investigation.
7 CONCLUSION
In this paper, we present Gogo, group-wise granularity-ordere codec, and GogoSpeech, a two- stage speech language model. Specifically, Gogo produces snoorerrssive-friendt tokens for each speech group, arranged in order from coarse to fine. Built upon Gogo, GogoSpeech performs speech generation by first constructing a high-level speech backbone, then enriching it with fine-grained details. Furthermore, we proposed GRPO-trained token allocator that adaptively allocates token budgets based on group-wise complexity, significantly reducing the number tokens required for synthesis without sacrificing perceptual quality. Extensive experiments on speech reconstruction and zero-shot texteto-speech demonstrate that Gogo achieves superior performance compared to state- of-the-art codecs, and GogoSpeech delivers high-quality, stable, and efficient speech generation.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work introduces speech generation model capable of producing highly human-like speech and supporting zero-shot voice cloning. While these capabilities advance the state the art, they also present potential risks, including misuse for misinformation, impersonation, harmful synthetic audio. Our research intended solely for legitimate scientific purposes. pro- mote responsible deployment, access control, and careful monitoring of downstream use. We are also exploring complementary safeguards such as speech watermarking and deepfake detection enhance the traceability of gen- erated audio. We encourage the community to adopt similar precautions to ensure that advances in generative speech technology are used ethically and for societal benefit.
other forms of
advocate transparent disclosure of synthetic speech, appropriate
This study also involves subjective listening evaluations for assessing the quality of synthesized speech. All participants were fully informed of the purpose and procedure of the listening task, and their participation was entirely voluntary. Consent was obtained prior to the evaluation. Participants were asked to complete the evaluation in a quiet environment to ensure reasonable listening condi- tions. The study did not collect any personal or identifying information, and no sensitive data or high-risk procedures were involved.
ACKNOWLEDGMENTS
This study was supported in part by both the General Research Fund (Project No. 14202623) from the Research Grants Council and the Centre for Perceptual and Interactive Intelligence, a CUHK-led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission, of the Hong Kong Special Administrative Region Government.
REFERENCES
Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting Hc, Hangrui Hu, Kai Hu, et al. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and Ilms. arXiv preprint urXiv:2407.04055 2024.
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430. 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-muliiinguue speech corpus. arXiv preprint arXiv:1912.066770 2019.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 700-710, 2022.
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amir- loo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. In Forty-second International Conference Machine Learning, 2025.
James Betker. Better speech synthesis through scaling. arXiv preprint arXiv: 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier. Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and lunguage processing, 31:2523-2533, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
11

---

Published as a conference paper at ICLR 2026
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Al- jafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. Xtts: a massively multilingual zero-shot text-to-speech model. Interspeech, pp. 4978-4982, 2024.
Yunkee Chae, Woosung Choi, Yuhta Takida, Junghyun Koo, Yukara Ikemiya, Zhi Zhong, Kin Wai Cheuk, Marco A Martinez-Ramires Kyogu Lee, Wei-Hsiang Liao, et al. Variable bitrate residual vector quantization for audio coding. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, 2025.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale self-supervised pre-training for full stack speech processing, IEEE Journal of Selected Topics in Signal Processing, 16(6):1505- 1518, 2022.
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang. Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, arXiv preprint arXiv:2410.06885. 2025.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training, In IEEE Automatic Speech Recognition and Understanding Work- shop, pp. 244-250, 2021.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. arXiv preprint arXiv:2210.1344 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélic Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- 2024.
logue. arXiv preprint
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I, pp. 4171-4186, 2019.
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete represen- tation learning. arXiv preprint arXiv: /03.0:29/33000089 2021.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint 2rXiv:2407.05400 2024a.
Zhihao Du Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint 2011:22412.10117, 2024b.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp.
2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXi::2407.2178. 2024
Hao-Han Guo, Yao Hu, Fei-Yr Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, and Kun Xie. Fireredtts-Is: An upgraded streamable foundation text-to-speech system. arXiv preprint arXiv:2503.20499, 2025a.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li. Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.0644 2025b.
12

---

Published as a conference paper at ICLR 2026
Haorui He, Zenggiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In IEEE Spoken Language Technology Workshop, pp. 885-890, 2024.
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Exposure bias versus self-recovery: Are distortions really incremental for autoregressive text generation? In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 5087-5102, 2021.
Jonathan Ho and Tim Salimans. arXiv:2207. / 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Sulf-super speech representation learning by masked prediction of hidden units. /EEF/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang, Xiaoda Yang. Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, and Zhou Zhao. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling, In The Thirteenth International Conference on Learning Represen- tations, 2025.
Daejin Jo, Jeeyoung Yun, Byungseok Roh, and Sungwoong Kim. Lm-spt: Lm-aligned semantic distillation for speech tokenization. arXiv preprint arXiv:2506.167 2025.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24188 2024.
W Bastiaan Kleijn, Andrew Storus, Michael Chinen, Tom Denton, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, and Hengchin Yeh. Generative speech coding with predictive variance regulariza- tion. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6478- 6482, 2021.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36:27980-27993,, 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Lingnistics, 9: 9:1336-1354, 2021.
Jiaqi Li, Xiaolong Lin. Zhekai Li, Shixi Huang, Yuancheng Wang. Chaoren Wang, Zhenpeng Zhan, and Zhizheng Wu. Dualcodec: A low-framc-rate, semantically-enhanee neural audio codec for speech generation. arXiv preprint arXiv:2505.13000 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In 11th International Conference on Learning Representations, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017a.
Ulya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017b.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: VQ-VAE made simple. In The Twelfth International Conference Learning Representa- tions, 2024.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2024.
13

---

Published as a conference paper at ICLR 2026
Julian D Parker, Anton Smirnov, Jordi Pons, CJ Canr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low- low-bitrate high-quality speech coding. In The Thirteenth International Conference on Learning Representations, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205 2023.
Puyuan Peng. Ya- Huang, Shang-Wai Wen Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 12442-12462, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53741 2023.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In IEEE international conference on acoustics, speech and signal processing, pp. 886-890, 2022.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746-1754, 2014.
Antony Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. IEFE international conference on acoustics, speech, and signal processing, volume 2, pp. 749-752, 2001.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint 12925,, 2023.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. Interspeech, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Feiyu Shen, Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. Acoustic bpe for speech genera- tion with discrete tokens. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 11746-11750. 2024.
Maohao Shen, Shun Zhang, Jilong Wu, Zhiping Xiu, Ehab AlBadawy, Yiting Lu, Mike Seltzer, and Qing He. Get large language models ready speak: A late-fusion approach for speech generation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, 2025.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint (xXiv:2306.008/4 2023.
Hubert Siuzdak, Florian Grötschla, and Luea A Lanzendörfer. Snac: Multi-scale neural audio codec. arXiv preprini 2rXiv:2410.144/ 2024.
Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du, Ziyang Ma, Jian Wu, Jian Cong, Dongya Jia, Zhuo Chen, Yuping Wang, et al. Magicodec: Simple masked caussian-injected codec for high-fidelity reconstruction and generation. arXiv preprint wrXiv:2506.003: 2025.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective in- telligibility measure for time-frequena weighted noisy speech. In IEEE international conference acoustics, speech and signal processing, pp. 4214 4217, 2010.
14

---

Published as a conference paper at ICLR 2026
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, pp. 1 1-34, 2024.
Jean-Mare Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. IETF RFC 6716, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Stephen D Voran. Why some audio signal short-time fourier transform coefficients have nonuniform phase distributions. In IEEE International Conference on Multimedia and Expo, pp. 1-6, 2024.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.021 11, 2023.
Hanyu Wang. Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. LARP: Tokenizing videos with a learned autoregressive generative prior. In The Thirteenth International Conference on Learning Representations, 2025.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 16133- 16142, 2023.
Dongchao Yang. Songxiang Liu, Haohan Guo, Jiankun Zhao, Yuanyuan Wang, Helin Wang, Zeqian Ju, Xubo Liu, Xueyuan Chen, Xu Tan, al. Almtokenizer: A low-bitrate and semantie-rich audio codec tokenizer for audio language modeling. arXiv preprint arXiv:2504. 103444 2025.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiugiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25697-25700 2025a.
Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng. Haohe Liu, Yizhu Jin, Zheqi Dai, et al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint 0022:22502.04/288, 2025b.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. In International Conference on Learning Representations, 2022.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- -hing Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 17:128940-228666, 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint 1rXiv:1904.000 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Yuxiao Dong, Jie Tang, et al. Scaling speech- text pre-training with synthetic interleaved data. In The Thirteenth International Conference Learning Representations, 2025.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757- 15773, 2023.
15

---

Published as a conference paper at ICLR 2026
Hanglei Zhang, Yiwei Guo, Zhihan Li, Xiang Hao, Xie Chen, and Kai Yu. Unlocking temporal flexibility; Neural speech codec with variable frame rate. arXiv preprint arXiv:250S.16844 2025.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. The Twelfth International Conference on Learn- ing Representations, 2024.
A NESTED DROUPUOT
Due to the nature of nested dropout, coarse tokens more likely be retained during training, whereas fine tokens are more likely to be dropped, leading to fewer effective gradient updates for the latter. To compensate for this imbalance, we rescale the gradient of each token according to its retention probability p. Specifically, the coarsest token, i.e., the first learnable token in each group, is always retained, while the finest token, i.e., the last learnable token, is retained with probability 1/ng. since we uniformly sample the number of tokens to retain. More generally, the retention probability of the j-th token defined as:
Pj
where j [1, ny]-
(14)
n4
We fix the compensation weight for the first token to 0.5, and accordingly define the weight for each token as:
0.5
Wj Pj
where j [1,
(15)
Let the quantized features of the i-th group be q' E R", During the forward pass, we apply the reparameterization trick to scale the token representations as:
0.5 d' L +- o d' +
0.5 0 [bbb detach, Pj
9'
(16)
where ['] denotes the j-th token in group i, and the detach operator indicates that the gradient flow is stopped for the detached variable. By applying Equation 16 the value of each embedding 7' [j] remains unchanged during the forward pass. However, in the backward pass, its gradient scaled by the inverse of Pú to offset the imbalance in Pý. As result, ["] that receive fewer gradient updates are compensated with a larger gradient scale.
B AUXILIARY MODULES
The AR prior takes the quantized representations 7" as input and predicts the feature representation of the next speech token at every position. To enhance training stability, the AR prior is optimized with a mean squared error loss in the feature space. Let f denotes the AR prior with parameters 1)- The AR loss defined
CAR
E,
f (7 5)) 9'li + 1112
(17)
ng
Furthermore, we incorporate an ASR module into the Gogo training pipeline to facilitate the linguis- tic representation learning. The group-wise quantized representations g" E R are concatenated along the temporal dimension to form T. E (mr) The ASR module, denoted as hg takes T. as input and outputs predicted token sequence ÿ (ύ1 ... #n corresponding the ground- truth transcription (VI...., VL). The ASR loss is formulated as the cross entropy loss:
I.
1
CASR = E(E,.))
-
Σ
log ho Mee, T.)
(18)
16

---

Published as a conference paper at ICLR 2026
C MODEL CONFIGURATION DETAILS
For Gogo, the Transformer encoder, ASR module, and AR prior are implemented with 12, 8, and 4 layers of standard Transformer blocks (Vaswani et al. (017). respectively. following the architecture of LLaMA (Grattafiori et al., 2024). Each layer uses a hidden dimension of 512, 8 attention heads, a feed-forward dimension of 1536, Rotary Position Embeddings (RoPE), RMSNorm, and SwiGLU activation. Asymmetric masking is applied in the Transformer encoder such that mel features can attend to each other but not to the speech queries, whereas each speech query has access to all mel features and to its preceding queries. We set the group size to g = 20, and allocate ny = 10 speech queries per group. The feature quantization adopts FSQ with levels [8, 8, 8, 5, 5], yielding an effective codebook size of 12,800. The flw-matchinn model in Gogo is implemented as a latent Diffusion Transformer (DiT) (Peebles & Xie 2023), following the configuration of Chen et al. (025). The only modification is that we set the number of layers to 12. Additionally, the input mel- spectrograms of Gogo are first processed by four ConvNeXt V2 (Woo et al. 2023) layers, followed by a two-layer MLP that projects the features to match the hidden dimension of Gogo.
For GogoSpeech, both Stage I and Stage II are built on top of Llama-3.2- IB Instruct, 3.2t---tet-- with the vocabulary extended to include Gogo's 's codebook tokens, enabling the model to directly perform speech token generation. GogoSpeech is trained under the next-token prediction paradigm to jointly model text and speech tokens. The maximum sequence length is set to 256 and 1024 tokens for Stage I and Stage II, respectively. The speech backbone is defined the first b 3 tokens of each group, which are generated in Stage I. The remaining 7 tokens per group are recovered in Stage II.
For the token allocator, we adopt a lightweight Transformer with 2 layers and the same architectural configuration used in Gogo, followed by a linear classifier to predict the token budget for each group.
D TRAINING AND INFERENCE DETAILS
Gogo, GogoSpeech, and the token allocator are optimized separately using the Adam optimizer (Loshchilov & Hutter, 2017a) on 8 NVIDIA H100 NVI. 94G GPUs. The learning rate is decayed based cosine annealing schedule (Loshchilov & Hutter 2017b). Speech samples longer than 20 seconds or shorter than seconds are discarded during training. More detailed training hyperpa- rameters are shown in Table 4
Table 4: Hyperparameters for training Gogo, GogoSpeech, and the token allocator.
GogoSpeech Stage T/ Stage II
Hyperparameters
Gogo
Token Allocator
Training Epochs
10/5
Update Steps
400k 10k
Warmup Steps
5k / 10k 1152 / 288 Samples
1k 128 Samples le-4
Batch Size
1440 Seconds 2e-4
Learning Rate
5e-4
Optimizer
AdamW
AdamW
Adam
Momentum
31, 32 0.9, 0.999 0.01
31, 82 0.9, 0.95
B 32 = 0.9, 0.999 0.01
Weight Decay
0.01
Learning Rate Schedule
Cosine Annealing
Cosine Annealing
Cosine Annealing
For Gogo inference, we begin from sampled Gaussian noise To and integrate it toward the target distribution æ1 conditioned on the speech tokens s². Following Eq. 1 and Eq. the tokens sf are first converted into aligned representations т. We then employ an Euler ordinary differential equation (ODE) solver to iteratively integrate 0rt/Ot = Ui va(2;; [2,, F. t) from zo to IT, where the flow step & is sampled using the Sway Sampling strategy (Chen et al. 2025) to improve generation performance and efficiency. The resulting mel-spectrooramm I1 is converted into waveform using a pretrained Vocos vocoder (Siuzdak 2023). To balance fidelity and diversity, we apply Classifier-Free Guidance (CFG) (Ho & Salimans) 2022) with a guidance scale of 2. For further stability, Exponential Moving Averaged (EMA) weights (Karras et al. 2024) of the model parameters are used during inference.
17

---

Published as a conference paper at ICLR 2026
For GogoSpeech inference, we employ the standard autoregressive decoding strategy widely used in large language models. To balance generation diversity and fidelity, we set the temperature to 0.8, apply repetition penalty of 1.2 to mitigate degenerate loops, and use nucleus sampling with p 1.0. In Stage of GogoSpeech, decoding for each group employs early stopping, terminating as soon as the maximum token budget assigned by the token allocator is consumed.
For token allocator inference, the first b tokens of each group (three tokens in our case) produced by Stage of Gegohpeech used as input. The allocator then predicts budget for the group, specifying how many fine tokens should be generated in Stage II.
E CODEC BASELINES
EnCodec Défossez et al. 2022 builds upon the residual vector quantization (RVQ) framework and employs single multi-scale STFT-basee discriminator effectively suppress artifacts and enhance perceptual quality. It supports variable bandwidths by selecting different numbers of codebooks during training, providing a flexible solution for speech compression and discretization in speech language models. For evaluation, we adopt the official implementation and pretrained checkpoin
DAC (Kumar al. 2023) adapts advances from the Improved VQGAN (Yu al., 2022) image model to address the codebook collapse problem. It performs codebook lookup in low-dimensional space and replaces Euclidean distance with cosine similarity, improving both stability and quality. DAC further employs mulli-period discriminator in the waveform domain and multi-ban multi- scale STFT discriminator the frequency domain, enabling high-fidelity audio generation. For evaluation, we use the official implementation and pretrained checkpoin
Speech
(Zhang et al., 2024) is specifically designed for speech language modeling, where different aspects of speech are disentangled hierarchically across RVQ layers. Specifically, it em- ploys HuBERT as semantic teacher to distill content information into the first layer RVQ. For evaluation, we use the official implementation and pretrained checkpoin
Mimi (Défossez et al. 2024) takes inspiration from previous work on SpeechTokenizer and uses distillation to transfer high-level semantic information in WavLM into the quantized tokens. Unlike SpeechTokenizerr it distills semantic information into plain VQ and apply an RVQ with 7 levels in parallel, thereby removing the constraint that acoustic information must reside in the residual of the semantic quantizer. For evaluation, we use the official implementation and pretrained checkpoin
SNAC (Suzzdak al.) 2024) extends RVQ by allowing quantizers to operate at different temporal resolutions. Through a hierarchy of quantizers running at variable frame rates, adapts to audio structure across multiple timescales, thereby capturing both coarse and fine details more effectively. Each quantizer applies average pooling for downsampling and nearest-neighhor interpolation for upsampling, enabling efficient compression. For evaluation, we adopt the official implementation and pretrained checkpoin
Tokenzzzze (ji et al. 2025) improves subjective quality using single quantizer with an ex- panded codebook to reduce information loss. It further enhances semantic modeling by introducing attention modules with extended contextual windows in the decoder. Finally, the inverse Fourier transform is employed to reconstruct the final audio directly. For evaluation, we adopt the official implementation and pretrained checkpoin
MagiCodec (Song et al., 2025) is single-layer, streaming Transformer-based audio codec trained with a multistage pipeline to mitigate codebook collapse and improve token efficiency. It introduces Gaussian noise injection and latent regularization techniques to encourage learning lowffrequency semantic representations while preventing overfitting to fegh-freuuncc noise. For evaluation, we adopt the official implementation and pretrained prettriieee checkpoin
httpss//gghub.ccoo//sssengnneg/n zer
18

---

httee///waaiinnaseccce///tttttct Published as a conference paper at ICLR 2026
X-codec2 YYe al. 2025b) integrates semantic and acoustic features into unified codebook using a single-layer FSQ quantizer. A pretrained w2v-BERT serves as the semantic encoder, while an acoustic encoder based on residual convolutional blocks with Snake activations captures fine-grained acoustic details. The semantic and acoustic features are concatenated and served as input to the vector quantizer. For evaluation, adopt the official implementation and pretrained checkpoin
TAAE (Parkrr et al. 2025) introduces a Transformer- meadd codec architecture that scales into the 1B parameter range, enabling state-of-the-art speech quality at extremely low bitrates. Unlike CNN- based codecs that rely on convolutional inductive biases with high parameter efficiency, TAAE lever- ages a more general Transformer architecture for greater scalability and better modeling capacity. For evaluation, we adopt the official implementation and pretrained checkpoint"
DualCodec (Li 2025) a dual-stream codec that jointly models colf-sueeriied and waveform representations within an end-to-end framework. The first RVQ layer directly encodes semantic-rich features from a pretrained w2y-BERT-2 model, while the remaining RVQ layers, along with the encoder-decoder design, follow the DAC framework. This integration enables DualCodec to better preserve linguistic content while maintaining high-fidelin fideliiyyyy reconstruction. For evaluation, we use the official implementation and pretrained checkpoint
F TTS BASELINES
FireRedTTS-1S (Guo al. 2025a a high-quality streamable TTS system that achieves real-time speech generation with low latency under 150ms through text-to-semantic decoding and semantic- to-acoustic decoding. For evaluation, we use the pretrained checkpoin
F5-TTS Chen et al. (2025) is a non-autoregressive TTS model built on flow matching. Instead of relying on complex alignment mechanisms, F5-TTS pads the text input with filler tokens to match the length of the target speech and directly performs denoising to generate speech. For evaluation, we use the pretrained checkpoin
XTTS-v2 (Casanova et al., 2024) is multilingual zero-shot multi-speaker TTS model built upon the Tortoise model (Betker 2023). XTTS-v2 supports 16 languages and achieves state-of-the-art the pretrained checkpoin
results in most of them. For evaluation,
Llasa (Ye et al., 2025b) is a large-scale speech synthesis system that employs a single-layer vector quantizer codec and unified Transformer architecture to fully align with standard LLMs such as Llama. For evaluation, we use the pretrained checkpoin
Cosyvoice 2 (Du et al. 2024b) is a multilingual speech synthesis framework that integrates pre- trained language model for discrete speech token prediction with a chunk-aware flw-mmtching model for speech feature generation. For evaluation,
the pretrained checkpoint
VoiceCraft (Peng et al. 2024) is a token infilling neural codec language mode. employs a token rearrangement strategy with causal masking and delayed stacking, enabling seamless speech editing and zero-shot text-to-speech generation. For evaluation, we use the pretrained checkpoin [6
G EVALUATION METRICS
We employ a comprehensive set of evaluation metrics to assess speech quality across multiple di- mensions, including intelligibility, perceptual quality, content preservation, and speaker similarity.
19

---

Published as a conference paper at ICLR 2026
Short-Time Objective Intelligibilityy (STOI) (Taal al. 2010 is widely adopted metric for eval- uating speech intelligibility. computes the correlation between temporal envelopes of reference and reconstructed signals in short-time segments. The score ranges from 0 to with higher values indicating better intelligibility.
Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 2001) measures perceptual speech quality by comparing the reconstructed audio with the clean reference signal using a perceptual auditory model. We report results under both narrow-band (NB, 8 kHz) and wide-band (WB, 16 kHz) conditions.
UTokyo-Sarul.ab MOS (UT-MOS) (Saeki al. 2022) is automatic MOS predictor trained to approximate human judgments of overall speech naturalness and quality. It provides scalable alternative to subjective MOS tests.
Deep Noise Suppression MOS (DNS-MOS) (Reddy et al. 2022) is a non-intrusive quality metric designed for real-world audio evaluation. estimates perceptual quality directly from the signal without requiring reference audio and has been shown to correlate strongly with human ratings.
Word Error Rate (WER) is used to quantify content preservation and intelligibility at the linguistic level. By default, we adopt HuBERT as the ASR mode [7 to transcribe the reconstructed speech, and compute WER by comparing against the ground-truth transcripts. For zero-shot TTS evaluation on the Seed-TTS test-en set, the WER metric is computed using the provided scrip 18
Speaker Similarity (SIM) measures the degree to which the synthesized speech retains the identity of the original speaker. By default, we follow Wang et al. (2023) and employ WavLM-Large-baar speaker verification mode 19 to extract speaker embeddings from both reconstructed and reference speech, and compute cosine similarity as the final metric. For zero-shot TTS evaluation the Seed-TTS test-en set, the SIM metric is computed using the provided scrip!" LT
Similarity Mean Opinion Score (SMOS) evaluates the speaker similarity between the prompt and the generated speech. Human raters judge the degree of resemblance by considering speaker char- acteristics, style, acoustic properties, and potential background artifacts. SMOS is scored on five-point scale, with higher values indicating stronger similarity.
Comparative Mean Opinion Score (CMOS) measures the relative perceptual quality of synthe- sized sample compared to a reference. Raters assign scores on a scale from -3 to 3, where negative values indicate the synthesized speech is worse than the reference, positive values indicate is better, and 0 denotes parity.
For subjective evaluation, we randomly select 20 samples from the Seed-TTS test-en set and invite 20 listeners to rate the SMOS and CMOS scores for the synthesized samples.
H PROBING EXPERIMENTS
The probing model consists of three fully connected layers with ReLU activations and dropout in between. The hidden dimensions of each layer are 512, 128, and 1, respectively. The probing procedure is formulated as a regression task. Given a speech input, we first extract the target feature to be probed. Next, Gogo is used to quantize the speech and generate multiple groups of tokens. To probe the tokens at position 8, as illustrated in Figure 6 we average the tokens at this position across all groups The resulting averaged representation then fed into the probing model predict the value of the target feature. The mean squared error (MSE) loss of the probing model is reported as an indicator of the representational capacity of tokens at each position, with lower loss implying that the token more readily encodes the probed feature.
We conduct probing experiments on Gogo's gramularity-ordeer tokens using a broad set of features spanning acoustic, prosodic, and linguistic dimensions. Acoustic features include zero-crossing rate, mean pitch, energy entropy, and spectral centroid, which characterize low-level signal proper- ties. Prosodic features include jitter, shimmer, duration, voiced-unvoiced ratio, and speaking rate,
http:////huiiinieeecc////eeeeeer httraars//atttooooo////t/deeeeee htt.b/////..bbb.ooo////ooooooo/a veri vecaiination
htw.://aaannnnnntteeeemmmaam speakerr_____
20

---

Published as a conference paper at ICLR 2026
Speech Input
10
Extracted Features
Gogo
MSR
Granularity
Averaiing
Groups
Speech Tokens
Probing Model
Figure Procedure for probing the information encoded in tokens of different granularities. Throughout the probing model training, the Gogo codec is kept frozen. Here we probe the tokens at position for demonstration.
capturing rhythm, intonation, and phonation stability. Linguistic features include lexical statistics such unique word count and counts of adjectives, adverbs, nouns, verbs, pronouns, and conjunc- tions, reflecting higher-level syntactic content. The probing models are trained on the LibriTTS train-clean- 100 subset for 20k steps using the AdamW optimizer and evaluated the LibriTTS test-clean set. The detailed MSE losses for each feature are reported in Table
Table 5: MSE losses for various probed features using Gogo's granularity-ordered tokens the LibriTTS test-clean set. In our setting, each frame group discretized into 10 tokens. Token position corresponds the coarsest token, whereas position 10 corresponds to the finest; ZRC: zero crossing rate, EE: energy entropy, SC: spectral centroid, J: jitter, S: Shimmer, Ratio: voiced
unvoiced ratio, SR: speaking rate, Count: unique word count, Tadj: total adjectives, Tadv: total adverbs, Tn: total nouns, Tv: total verbs, Tpron: total pronouns, Tconj: total conjunction.
Features
2
3
4
8
9
10
Duration
14.44
14.95
15.87
16.28
16.81
16.45
16.59
16.65
17.23
17.45
ZRC (e-3)
1.113
1.058
1.051
1.060
1.076
1.071
1.101
1.047
1.036
1.038
EE (e-3)
8,409
8.290
8.189
8.257
8.142
8.212
8.101
8.066
7.839
7.771
SC (e-3)
1.150
1.053
1.054
1.031
0.988
0.879
0.866
0.830
0.872
0.933
Pitch
1455
1478
1424
1371
1304
1295
1230
1166
1074
1012
(e-5)
4.000
3.969
3.696
3.603
3,415
3.584
3,445
3.887
3.751
3.563
S (e-4)
3.463
3.497
3.435
3.382
3.321
3.345
3.290
3.377
3.350
3.312
Ratio
0.298
0.297
0.313
0.318
0.332
0.319
0.332
0.323
0.338
0.343
SR
0.414
0.415
0.413
0.406
0.403
0.406
0.411
0.411
0.409
0.409
Count
35.81
36.90
39.01
40.00
41.18
40.34
40.63
40.70
41.96
42.47
Tense
0.240
0.239
0.238
0.237
0.236
0.237
0.237
0.239
0.239
0.238
Tadj
2.229
2.250
2.299
2.329
2.352
2.334
2.337
2.338
2.359
2,370
Tady
1,474
1.486
1.507
1.521
1.532
1.523
1.521
1.526
1.537
1.538
Tn
9.716
9.907
10.32
10.55
10.81
10.67
10.71
10.66
10.88
10.99
Tv
4.708
4.804
4.969
5.023
5.128
5.057
5.097
5.116
5.240
5.282
Tpron
2.766
2.797
2.838
2.847
2.868
2.847
2,853
2.866
2.897
2.906
Tconj
0.967
0.977
0.998
1.005
1.015
1.003
1.006
1.010
1.024
1.027
further investigate how reconstruction quality varies when only subset the granularity- ordered tokens is used. Specifically, progressively retain the first n tokens in each group and reconstruct the input speech, where n E [1, n.y). The results are shown in Figure We observe that the word eиror rate drops sharply when the first few tokens included, indicating that most linguistic content is captured by the coarsest tokens in Gogo. However, when more than six tokens per group retained, the improvement of WER becomes marginal, suggesting that the remaining
21

---

Published as a conference paper at ICLR 2026
fine-grained tokens primarily contribute to perceptual quality and acoustic details rather than lin- guistic content. In particular, both NB PESQ and WB PESQ scores show a marked improvement more than four tokens per group retained. The other objective metrics exhibit generally monotonic improvement as the number of retained tokens increases.
0.8
0.6
NB PESQ
WB PESQ
0.4
STOI
UT-MOS
DNS-MOS
SPK SIM
WER
0.0
3
Figure 7: Normalized performance on the LibriTTS test-clean set with varying numbers of retained
tokens per group
PERPLEXITY EXPERIMENTS
Perplexity (PPL) a standard evaluation metric for language models that quantifies their uncer- tainty in predicting the next token in sequence. Lower PPL indicates higher model confidence and prediction accuracy, whereas higher PPL. reflects greater uncertainty and poorer predictive perfor- mance. Formally, perplexity is defined as the exponentiated average negative log-likelihood of the sequence. In our setting, given a speech token sequence I((S..). where S R"y X"* is the token matrix produced by Gogo and j [1, nly denotes the position of the speech token within each group, we compute PPL. to assess the autoregressive modeling difficulty at different token granularities as follow:
ng log P(S (Scii)
PPL (T(S..)) exp
Σ
(19)
Group-wise Quantization
Granularity
Groups
Speech Tokens
Specch Input Hi
Frame-wise Quantizution
RVQ Layers
[start]
Frumes
Speech Tokens
Autoregressive Modeling
[end]
Perplexity
Figure 8: Procedure for evaluating the perplexity of autoregressive model on speech tokens gener- ated by different quantization schemes. Here we show the evaluation using the 8th token of each group or tokens from Sth RVQ layer for demonstration.
22

---

Published as a conference paper at ICLR 2026
As illustrated in Figure the autoregressive model used in the perplexity experiments is 6-layer LLaMA-style Transformer with hidden dimension of 256. trained on the combined LibriTTS train-clean-Iie 100, train-clean-360, and train-other-5i subsets for 200k steps using the AdamW opti- mizer. Evaluation is conducted on the LibriTTS test-clean set.
J STATISTICAL SIGNIFICANCE ANALYSIS
In Table 6 we report the 95% confidence intervals for all subjective evaluation metrics to charac- terize score variability across listeners. For CMOS, in which listeners provide paired comparative judgments, evaluate statistical significance using the Wilcoxon signed-rank test and include the corresponding p-values. Across models, several systems display statistically significant differences from the Ground Truth, indicated by p-values I 0.05. Our proposed GogoSpeech also achieves a statistically significant positive CMÓS relative to the Ground Truth, and incorporating the token allocator leads to similarly significant improvement.
Table 6: Subjective comparison between different TTS models on the Seed-TTS test-en set. The boldface denotes the best result, the underline denotes the second best. We report 95% confidence intervals for all scores and assess statistical significance (p-value) using the Wilcoxon signed-rank test for CMOS. p-value I 0.05 indicate statistical significance.
Model
SMOS
CMOS (p-value)
Ground Truth
4.752 + 0.342
0.000
F5-TTS (Chen et al. 2025)
4.173 0.513
+1.730 + 0.476 (4.9e-4)
XTTS-v2 (Casanova et al. 2024
2.426 + 0.572
-0.961
1.088 (0.072)
Llasa-8B-2500 Ye et al. 2025b
+: 3
+0.882
0.869 (0.027)
Cosy Voice 2 Du al. 2024b)
4.331 0:435
+1.638
= 0.596 (2.9e-3)
FireRedTTS-IS (Guo et al. 2025a
4.247 0.409
+1.634 = 0.561 (3.0e-3)
VoiceCraft (Peng al. 2024
2.965 + 0.697
-0.751
0.877 (0.170)
GogoSpeech (47 Hz)
4.381 + 0.479
+.832
+ 0.491 (2.7e-3)
w/ Token Allocator (47 Hz > 36 Hz)
4.253 + 0.344
+1.587 0.530 (0.013)
K
ABLATION EXPERIMENTS
Table 7: Ablation study on the auxiliary modules and granularity ordering. Gogo is evaluated on the LibriTTS test-clean set. GogoSpeech is evaluated on the Seed-TTS test-en set.
Gogo
GogoSpeech
Model
UT MOS
DNS MOS
PESQ WB
PESQ NB
STOI
SIM
WER
SIM
WER
Auxiliary Module
Proposed Model
4.19
3,99
0.92
2.59
3.26
0.91
6.35
0.667
2.394
w/o ASR Module
3.97
3.91
0.87
2.63
3.29
0.92
8.66
0.640
4.372
w/o AR Prior
4.14
3.96
0.86
2.61
3.24
0.91
6.42
0.661
2.625
Granularity Ordering
Proposed Model w/o Nested Dropout
4.19
3.99
0.92
2.59
3.26
0.91
6.35
0.667
2.394
4.10
3.89
0.89
2.39
3.16
0.89
6.41
0.657
3.969
w/o Loss Balancer
3.89
3.74
0.82
2.29
3.07
0.85
7.50
0.614
4.608
We perform ablation study all auxiliary modules and design choices in Gogo and fur- ther evaluate their impact on GogoSpeech. The results are summarized Table We can see that removing the ASR module leads to slight improvement in PESQ and SIM for Gogo's reconstruc- tion. However, the absence of ASR guidance significantly degrades the performance of GogoSpeech across all metrics. Eliminating the AR prior has little effect Gogo's reconstruction metrics but
Gogo.
23

---

Published as a conference paper at ICLR 2026
results in noticeable performance drop for GogoSpeech. This suggests that the AR prior primarily benefits the autoregressive modeling of GogoSpeech rather than signal reconstruction. Furthermore, ablate the granularity ordering mechanism. We observe that removing either nested dropout or the loss balancer causes a substantial decline in performance for both Gogo and GogoSpeech, highlighting their importance in learning informative coarse-to-fine token representations.
Table 8: Ablation study on the number of coarse tokens used as the speech backbone on the Seed-TTS test-en set. Using all 10 tokens as the backbone degenerates into single-stage model.
5
6
8
10
SIM
0.647
0.654
0.667
0.663
0,662
0.652
0.657
0.656
0.654
0.642
WER
2.754
2.531
2.394
2.391
2.397
2.441
2.346
2.451
2.460
3.121
GogoSpeech. We further perform an ablation study on the number of coarse tokens used as the speech backbone, with results summarized in Table We observe that increasing b from I to 3 yields substantial improvements in both SIM and WER, indicating that a slightly richer backbone provides more effective guidance for Stage II refinement. When lies between 3 and 5, the performance of both metrics becomes relatively stable, with SIM reaching its peak at = 3 and WER achieving its best value downward trend, suggesting that overly detailed backbones may limit the capacity of Stage II to contribute fine-grained refinements. When b 10, meaning that all tokens are generated entirely within Stage I of GpgoSpeech, the performance drops to its lowest level across all metrics. These results provide strong empirical evidence for the effectiveness of our two-stage design, where compact backbone combined with detail refinement yields the best overall performance.
4. As b increases further from
all evaluation metries exhibit general
Table Ablation study on the token allocator on the Seed-TTS test-en set. Various token allocators, trained under different experimental settings, are applied to the same GogoSpeech model.
GogoSpeech
Model
SIM
WER
GRPO Modifications
Proposed Model
0.662
2.469
w/o Removal of KI. Penalty
0.647
3.452
w/o Exhaustive Enumeration
0.651
2.659
Sensitivity to Reward Weights
Proposed Model (An 0.2, da 1.0)
0.662
2.469
An 0.5, Aa = 1.0
0,656
2.882
An 1.0, aa- 1.0
0.650
3.169
An 1.0, da 0.5
0.642
3.772
An = 1.0, Aa = -.2
0.643
3.528
Reinforcement Learning Objectives
Proposed Model
0.662
2.469
Replace GRPO with DPO (Rafailov et al. 2023)
0.645
3.691
Token Allocator. We further conduct an ablation study to evaluate the effectiveness of our mod- ifications to the standard GRPO algorithm, including the removal of the KL penalty and the use of exhaustive enumeration. We also examine the sensitivity of the token allocator different re- ward weights and compare the GRPO-based allocator with its DPO-based counterpart. The results are summarized in Table 9 We observe that removing the KI. penalty and introducing exhaustive enumeration during allocator training consistently yield performance gains. Moreover, the reward- sensitivity analysis suggests that greater emphasis should be placed on the reward term R., while R,, should be assigned smaller weight. Finally, the last row of shows that the GRPO-based token allocator outperforms the DPO-based approach.
Systematic Evaluations under Consistent Training Conditions To rigorously isolate the impact of our proposed approach and further demonstrate the effectiveness of Gogo, GogoSpeech, and the token allocator, we conduct carefully controlled ablation study. The results are reported in
24

---

Published as a conference paper at ICLR 2026
Table 10: Systematic evaluation of codec and SLM design choices the Seed-TTS test-en set. The
boldface denotes the best result, the underline denotes the second best.
Codec Frame-wise
SLM
Token
TPS
SIM
WER
-rie
Single-stage
Two-stage
Allocator
47
0.592
4.117
47
0.642
3.121
47
0.667
2.394
v
36
0.662
2.469
Table 10 All experiments in the table, including codec and SLM model training, are conducted on the Emilia dataset. The frame-wise and group-wise codecs operate at the same token rate, and all SLM models initialized with Llama-3.2- 1-2-IBtInsttuet. Comparing the first and second rows, both using a single-stage SLM, we observe that the SLM using the group-wise codec (i.e., Gogo) outperforms the one using the frame-wise codec, demonstrating the effectiveness of our group-wise quantization. Comparing the second and third rows, both using the group-wise codec, the two-stage SLM (i.e., GegoSpeech) achieves higher performance metrics than the single-stage SLM, validating the effectiveness of our two-stage design. Finally, comparing the last two rows, the introduction of the token allocator reduces the token rate from 47 Hz 36 Hz while maintaining roughly the same model performance, indicating the effectiveness of the token allocator.
L VISUALIZATION OF TOKEN ALLOCATION
To better illustrate the behavior of the token allocator, we visualize the token allocation results in and Figure Specifically, we present three aligned plots: (1) the original mel-spectrogram of the input speech, where vertical dashed lines indicate group boundaries; (2) the token budget assigned to each group by the allocator; and (3) the reconstructed mel-spectrogram obtained by Gogo using only the allocated tokens. The visualization clearly demonstrates that the allocator adaptively assigns more tokens to acoustically rich regions while reducing the allocation in silent or low-information segments, thereby achieving efficient yet high-quality reconstruction.
M THE USE OF LARGE LANGUAGE MODELS
Large language models employed exclusively auxiliary tools edit and polish text written by the authors. Their usage was limited to improving clarity, grammar, and style of expression. No part of the research ideation, methodology, analysis, results relied LLMs.
25

---

Published as a conference paper at ICLR 2026
Input
Number Token Allocation
3
Group
Reconstruction
Figure 9: Visualization of sample 1580 141084 000085 000000 from the LibrTTTS test-clean set. The token allocator reduces the token rate from 47 Hz to 34.28 Hz.
Input
Number
10
Token
Allocation
Group
Reconstruction
Figure 10: Visualization of sample 7176 000.8888830000011111111100000001 00000000000000000 from the LibrTTTS test-clean set.
The token allocator reduces the token rate from 47 Hz to 38.73 Hz.
26


Reviews:
[{"review_id": "VwfpC4CGsr", "reviewer": "Reviewer_9Pzb", "summary": "This paper proposes Gogo, a novel group-wise granularity-ordered speech codec, and GogoSpeech, a two-stage speech language model for generation. The core idea is to quantize groups of frames into tokens ordered from coarse (high-level information) to fine (acoustic details). GogoSpeech leverages this by first generating a low-rate \"speech backbone\" (coarse tokens) and then enriching it with fine details in a second stage, aiming to improve stability and efficiency. Additionally, a GRPO-trained token allocator is introduced to adaptively assign token budgets to different speech segments based on their complexity, further enhancing efficiency. Experiments show that Gogo achieves strong reconstruction performance, and GogoSpeech obtains good results in zero-shot TTS.", "questions": "Could you please provide direct inference efficiency metrics such as RTF (with hardware specifications) or GFLOPS to allow for a more direct and fair comparison of GogoSpeech's computational cost against other models? How does the overall efficiency compare to models with much lower token rates (e.g., 12.5 Hz)?\nRegarding the TTS comparison in Table 3, can you comment on the fairness of comparing your English-only system to strong multilingual baselines? Could the comparison be strengthened by evaluating against English-only configurations of these models, if available?\nThe token allocator is an interesting component. How do you expect it to perform in a multilingual context? Would different speaking habits, prosody, or phonetic structures across languages (e.g., tonal vs. non-tonal) affect its ability to accurately assess group complexity, or would it generalize well without language-specific fine-tuning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Novelty: The core idea of group-wise, granularity-ordered tokenization is novel and well-motivated. It presents a promising alternative to the dominant frame-wise paradigm.\nSystematic Design: The system is thoughtfully designed. The GogoSpeech model is built logically upon the properties of the Gogo codec, and the token allocator further enhances the system's efficiency.", "weaknesses": "Modeling Frame Rate: The final token rate of 47 Hz (or 36 Hz with the allocator) is presented as efficient. However, this claim is weakened when compared to recent models that operate at much lower rates, such as 12.5 Hz or even 6.25 Hz. The paper does not adequately position its efficiency gains against these highly competitive low-bitrate models.\nLack of Direct Computational Metrics: The paper relies on token rate as a proxy for efficiency. This is insufficient for a fair comparison. Crucial metrics like Real-Time Factor (RTF) on a standardized hardware setup or computational complexity (e.g., GFLOPS) are missing. Without these, the claim of \"efficient generation\" remains largely unverified.\nUnfair Baseline Comparison in TTS Evaluation: In Table 3, the proposed English-only model is compared against several multilingual TTS systems (e.g., CosyVoice 2). This is not a fair comparison, as multilingual models need to handle the complexities of multiple languages, which can compromise performance on any single language compared to a specialist model. This limitation must be explicitly acknowledged and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15KftstidA", "reviewer": "Reviewer_UGuC", "summary": "The paper introduces Gogo, a new speech tokenizer, and GogoSpeech, a two-stage autoregressive text-to-speech (TTS) framework built on top of it.\n\nThe Gogo tokenizer consists of a Transformer encoder that processes grouped mel-spectrogram frames augmented with learnable query tokens, and a flow-based generative model for mel-spectrogram reconstruction. The encoder outputs corresponding to the queries are quantized using Finite Scalar Quantization (FSQ), and combined with filler tokens as conditioning input for the flow model, followed by a pretrained vocoder to synthesize the waveform. To structure token learning, the model enforces a coarse-to-fine hierarchy across the query tokens through asymmetric masking, nested dropout, and adaptive loss balancing, encouraging early tokens to capture global content while later tokens refine acoustic details.\n\nBuilding on this tokenizer, GogoSpeech employs a two-stage generation process: the first stage predicts a coarse speech backbone from text, while the second stage adds fine-grained acoustic details. Additionally, a reinforcement-learned token allocator dynamically determines the number of fine tokens to generate per segment, improving efficiency by reducing the average token rate from 47 Hz to around 35–36 Hz. Experimental results demonstrate the effectiveness of the proposed framework.", "questions": "1. Could the authors clarify how the 4-dimensional grouped input to the Transformer encoder is handled in practice? Specifically, is the `n_g` (group index) dimension flattened into the batch dimension before encoding?  \n\n2. Regarding the autoregressive nature of GogoSpeech, the math in Section 2.2 suggests token-level generation within each group. Could the authors confirm whether the model predicts one token at a time across all groups, or all tokens of a single group before moving to the next? It is also unclear how the “flatten” operation affects the sequence length (e.g., whether it becomes *T × b*).  \n\n3. The demo or samples page does not load properly — I was only able to access the codec comparison samples, not the others.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I like the idea of using learnable queries to capture the information from grouped mel-spectrogram frames — it’s a neat and flexible way to summarize local acoustic context. Grouping the frames itself is also smart, as it naturally reduces the token rate without much loss of information.\n\nThe hierarchical design in the query tokens is particularly interesting. Through asymmetric masking, nested dropout, and adaptive loss weighting, the model explicitly encourages a coarse-to-fine structure across tokens. The probing results in Figure 5 clearly reflect this — earlier tokens capture higher-level linguistic features, while later ones focus more on detailed acoustic attributes like pitch and spectral characteristics.\n\nThe addition of the token allocator is another strong point. It’s an elegant way to make the system adaptive by deciding how many fine tokens to generate per segment, which not only saves computation but also reduces the overall token rate without hurting quality.", "weaknesses": "1. The paper does not discuss the limitations of the proposed methods, such as potential trade-offs in efficiency, reconstruction artifacts, or scalability.  \n\n2. The use of filler or placeholder tokens for the flow-matching decoder may lead to hallucinations or slight pronunciation errors during reconstruction. For example, in the Codec Comparison samples, the first utterance (“he”) sounds closer to “the,” suggesting occasional contextual confusion.  \n\n3. The codec architecture involves three major components — a Transformer encoder, FSQ quantizer, and a flow-matching decoder — where the flow-based component requires iterative inference. This could make the overall reconstruction process slower than other non-iterative codecs.  \n\n4. GogoSpeech employs two fully autoregressive stages (for backbone and fine-detail prediction), both predicting one token at a time. Since these are cascaded, inference time can be considerably higher than single-stage or non-autoregressive systems.  \n\n5. There is no evaluation or comparison of runtime efficiency, inference speed, or computational cost against existing baselines, which makes it difficult to assess the practical performance of the system.  \n\n6. Details of the subjective evaluation are missing. The paper does not specify the number of evaluators, number of test samples, listening protocol, or any statistical testing, which limits the reliability of the perceptual results.\n\n7. Statistical significance analysis is not provided for any of the reported metrics. Including confidence intervals or significance tests would help clarify whether the observed improvements are meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWWo1ASpxP", "reviewer": "Reviewer_Mepz", "summary": "This paper proposes a new speech codec that groups the mel feature frames into groups and maps them to coarse-to-fine scale tokens. The token sequence then go through the two-stage GogoSpeech model and the outputs are decoded by a flow matching model into mel features after which a Vocos vocoder is used to get the wave file. The codec tokens are learned by some query vectors and the model is trained so that the more frequent and more salient features are expressed in the first few tokens and the granularity increases as the number of tokens representing a speech signal increases. The novelty lies in the design of the codec, and the use of some experimental ideas such as asymmetric masking, nested dropout, a token length based loss balancing terms. The model is trained on the Emilia dataset. Llama is used as the LLM backbone for the autoregressive modeling of the tokens. Experiments compare different codecs with the Gogo codec, where the proposed system achieves competitive performance to other codecs. For codec operating at a similar rate, Gogo slightly outperforms the other ones in terms of PESQ narrowband, speaker similarity, and WER. GogoSpeech TTS model also performs better than the baseline TTS models in terms of subjective MOS scores. On objective metrics (WER and SIM), GogoSpeech either slightly outperforms or is on par with the other TTS models mentioned in the paper.", "questions": "1. Do the authors think that speech reconstruction and TTS experiments are sufficient to prove that the proposed codec can be successfully utilized in most multimodal LLMs (particularly speech and text LLMs)?\n\n2. Could you please clarify how the regression probing task is set up? Is it based on masking out certain tokens and reconstructing the same audio? \n\n3. In Figure 1, there is a mention of a token budget. Could you please clarify whether once this token allocator is trained, we get the same number of tokens per group? Or is it dynamic within an utterance, i.e., each group getting different number of tokens?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "originality, \n+ The paper proposes a new speech codec that can represent speech at various granularity. \n\nquality, \n+ Experiments demonstrate positive results at a relatively small token rate. \n\nclarity, \n+ Mostly clearly written, there are extensive appendices that provide further details. \n\nsignificance\n+ Because of the introduction of a new speech codec, the paper is relevant to the speech community as well as the speech + text joint LLM modeling community as most joint models utilize a form of speech codec.", "weaknesses": "- The paper introduces a new codec and experiments positively demonstrate the effectiveness of the codec in audio reconstruction and TTS. As mentioned in the introduction, LLM that accept speech inputs usually make use of a speech codec to tokenize the speech input and hence the paper is relevant to speech LLMs. However, to get a full picture of how this codec would work in a speech + text joint LLM setting (e.g. on a spoken QA task), it could have been better if some further experiments had been included. \n\n- The paper also analyzes what the tokens at different levels learn, which is good. However, it might have been more interesting to see the codec perform in a task where the factors listed in Figure 5 (such as jitter, shimmer, etc.) would more explicitly show its use case. These tasks could have been speech emotion recognition, identification of speech attributes, etc. instead of formulating the probing task as a regression task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNBoQx3nkf", "reviewer": "Reviewer_tQ8y", "summary": "The authors present Gogo, a group-wise granularity-ordered speech codec. The proposed method tokenizes contiguous frames into groups and sequentially orders tokens from coarse to fine granularity. Utilizing Gogo codec, the authors construct GogoSpeech, an LLM-based text-to-speech (TTS) model. GogoSpeech synthesizes speech in two stages: the first stage generates high-level acoustic cues (\"speech backbone\") at a low token rate, and the second stage refines these cues by adding detailed acoustic information at a standard token rate.", "questions": "I have no questions but encourage the authors to improve their presentations and experiments.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "Overall, the proposed approach is interesting. GogoSpeech achieves performance comparable to SOTA TTS models.", "weaknesses": "(1) Paper Organization\\\nThe main text lacks essential ablation studies and related work sections. The current paper structure does not sufficiently support the primary claims regarding the effectiveness of the proposed approach. I this this paper requires major revision to clearly emphasize core contributions and key experiments in the main body, while relegating supplementary details to the appendix.\n\n(2) Questionable Impact of Asymmetric Masking\\\nAppendix Table 6 indicates that removing asymmetric masking leads to negligible performance differences (e.g., WER increases from 2.394 to 2.406, indicating no significant difference, described by the authors themselves as \"slight\"). Given that asymmetric masking is presented as a major component in Section 2.1.3, this result unfortunately undermines its claimed effectiveness. Among the three key techniques introduced in Section 2.1.3, the ineffectiveness of asymmetric masking diminishes the overall contribution.\n\n(3) Insufficient GRPO Details and Experiments\\\nThe authors mention using a \"slightly modified GRPO,\" specifically the removal of the KL penalty and exhaustive enumeration of token budgets. However, critical ablation studies investigating these specific modifications and sensitivity analyses concerning the reward weights are missing. Furthermore, comparisons to alternative reinforcement learning-based objectives (e.g., DOP) are also absent. Such analyses are crucial to substantiate the efficacy and necessity of the proposed GRPO-based token allocator.\n\n(4) Fairness and Depth of Zero-shot TTS Comparison\\\nIn Table 3, GogoSpeech (WER: 2.394) achieves competitive but not consistently superior performance compared to state-of-the-art methods such as F5-TTS (WER: 1.830). Although GogoSpeech slightly outperforms F5-TTS in long-form generation, the overall comparison remains mixed and inconclusive. Due to substantial differences in architecture and training conditions, this heterogeneous comparison does not clearly demonstrate the efficacy of the proposed granularity-ordered tokens.\n\nTo rigorously isolate the impact of the proposed approach, systematic experiments varying frame rates, token rates, and codebook sizes under consistent training conditions are required. While claiming state-of-the-art performance would be important, systematic evaluations provide stronger evidence than heterogeneous comparisons. Additionally, if the authors wish to claim only state-of-the-art results, comparisons with API-based TTS systems (e.g., ElevenLabs TTS, OpenAI TTS, Gemini TTS) would significantly enhance the evaluation's rigor and relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)