T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning
Authors: Nabarun Goswami
Hanqin Wang
Tatsuya Harada
Venue: ICLR 2025 Poster (2025)
Keywords: ASR
TTS
Non-Autoregressive
Conformer
Multitask Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: T2V2 is a unified non-autoregressive model for ASR and TTS using discrete tokens, a shared backbone, and auxiliary tasks like CTC error correction, achieving SOTA TTS and competitive ASR performance.

Abstract: We introduce T2V2 (**T**ext to **V**oice and **V**oice to **T**ext), a unified non-autoregressive model capable of performing both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis within the same framework. T2V2 uses a shared Conformer backbone with rotary positional embeddings to efficiently handle these core tasks, with ASR trained using Connectionist Temporal Classification (CTC) loss and TTS using masked language modeling (MLM) loss. The model operates on discrete tokens, where speech tokens are generated by clustering features from a self-supervised learning model. To further enhance performance, we introduce auxiliary tasks: CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance to improve TTS. Our method is self-contained, leveraging intermediate CTC outputs to align text and speech using Monotonic Alignment Search, without relying on external aligners. We perform extensive experimental evaluation to verify the efficacy of the T2V2 framework, achieving state-of-the-art performance on TTS task and competitive performance in discrete ASR.

Published as a conference paper at ICLR 2025
T2V2: A UNIFIED NON-AUTOREGRESSIVE MODEL FOR SPEECH RECOGNITION AND SYNTHESIS VIA MULTITASK LEARNING
Nabarun Goswami', Hanqin Wang', Tatsuya Harada
! 'The University of Tokyo, Japan
PRIENN. Japan
fnabarungoswami, wang, haradaaemi....utootk....
ABSTRACT
We introduce T2V2 (Text Voice and Voice to Text), unified non- autoregressive model capable of performing both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis within the same framework. T2V2 uses a shared Conformer backbone with rotary positional embeddings to efficiently handle these core tasks, with ASR trained using Connectionist Temporal Classi- fication (CTC) loss and TTS using masked language modeling (MLM) loss. The model operates on discrete tokens, where speech tokens are generated by clus- tering features from supe-supe learning model. To further enhance per- formance, we introduce auxiliary tasks: CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MIM, enabling classifier free guidance to improve TTS. Our method is self-contained. leveraging intermediate CTC outputs to align text and speech using Monotonic Alignment Search, without relying on external aligners. We perform extensive experimental evaluation to verify the efficacy of the T2V2 framework, achieving state-of-the-an performance on TTS task and competitive performance in discrete ASR.
INTRODUCTION
Speech recognition and synthesis are foundational tasks in human-compunn computr interaction, enabling applications ranging from voice assistants to automated transcription services. Traditionally, both tasks rely heavily on autoregressive models, which generate sequences one token at time, resulting in higher latency and limited efficiency (Graves) 2012 Sutskever et al., 2014 Chan et al. 2016 Wang et al. 2017 et al. 2019 Casanova et al.) 2024). Recently, -oo-autoregressive models
have gained traction by reducing inference latency while maintaining robustness in both TTS and
RSR et al. 2020 Kim et al. 2022] Lee et al.) 2023 Casanova et al.] 2022] Graves et al.) 2006
Higuchi al. 2020 2021a)) Another promising approach the of discrete speech tokens, made
possible by advances in neural audio codecs(Zeghidou et al.] [202T] Kumar et al. 2024) (acoustic
tokens) and clustering features from large-scale self-supervised speech modeis
al. 2021
Baevski et al. 2020) (content tokens). These discrete tokens offer advantages in sequence modeling, particularly when paired with transformer-like architectures. Discrete tokens enable efficient stor- age and transmission. ideal for large datasets. They also serve as intermediate representations that capture acoustic and linguistic information while being less speaker-specine aligning well with the unified modality in our framework.
Despite the recent success of non-autoregressive models in both ASR and TTS, integrating these tasks into a unified framework remains a significant challenge. A few works (Rubenstein et al.) 2023 Wang et al. 2024 Maiti et al. 2024 Yang et al., 2024a Toyin et al., (024) have explored unified modeling of speech-text related tasks, but they are limited to AR encoder-decoder or decoder only methods. However, existing NAR models typically handle ASR and TTS separately, missing opportunities to share representations and leverage joint training improve performance across tasks. While multitask learning offers potential for parameter sharing, the performance of individual tasks might suffer due the inherent complexity of training multiple tasks simultaneously. Au-

---

Published as a conference paper at ICLR 2025
thors in Xiujuan & Zhongke (2004). detail these challenges and common optimization approaches like scalarization and Pareto optimality. Another key challenge NAR CTC-based ASR is the conditional independence assumption of CTC loss(Graves et al. 2006), which can lead emros in transcription. Several works have tackled this problem with varying degrees of (Higusci 2020; 2022 Chi et al. 202T; Nozaki & Komatsu 202T). Moreover, NAR TTS models rely heavily y on accurate alignment between text and speech, making external alignment tools(McAulific al. 2017) necessary in most systems. unified approach that can address both the independence limitations of CTC in ASR and the alignment challenges in TTS is needed. Last but not least, the use of discrete speech tokens, which have shown success TTS (Borsos al., 2023a] Rubenstein et al. 2023 Wang et al. 2023 Casanova et al. 2024). has not been fully explored in (Sh(Ch ct al. 2023b Yang et al. 2024b), leaving gap developing efficient models that can handle both tasks with the same set of representations. Addressing these challenges, while reducing reliance on external aligners and maintaining competitive performance, forms the core motivation of our work.
al.
To address these challenges, propose T2V2, non-autoregrresive model that unifies ASR and TTS using shared representations. T2V2 models content tokens with CTC-based training for ASR and conditional masked language modeling approach for TTS, converting these tokens acoustic tokens and ultimately to speech via the SoundStorm core of T2V2 is Conformer architecture with rotary positional embeddings (RoPE), capturing both local and long-range dependencies in speech and text. For TTS, Monotonic Alignment Search (MAS) is applied intermediate CTC outputs, eliminating the need for external alignment tools and ensuring consistency across tasks.
et al. (023b) and codec decoder. The
To improve speech generation, we employ classifier-free guidance(Ho & Salimans, 2021 using an unconditional masked speech model to iteratively refine outputs. For ASR, we address CTC's inde- pendence limitation through CTC eтгог correction task, refining outputs with speech embeddings to recover time-step dependencies and enhance transcription accuracy. T2V2 efficiently handles both tasks within unified framework, with auxiliary tasks like CTC error correction and unconditional speech MIM boosting performance. By leveraging discrete content tokens, our method improves sequence modeling and unifies speech and text processing. Our key contributions are as follows:
1. We propose T2V2, the first, to the best of our knowledge, unified NAR model for ASR and TTS and operates on discrete speech tokens derived from self-supervised models.
2. We leverage Monotonic Alignment Search (MAS) with intermediate CTC outputs for TTS alignment without relying on external tools for self-contained text-speech alignment.
3. We introduce a CTC error correction formulation to refine raw CTC outputs, improving ASR performance within the unified framework.
4. Extensive experimental validation confirms that T2V2 markedly improves synthesized speech robustness achieving state-of-the-art TTS performance and competitive discrete ASR performance.
2 MULTITASK T2V2 FRAMEWORK
The overview of our method is shown in Figure The following sections provide detailed de- scription of each component and task.
2.1 NOTATION AND TERMINOLOGY
Here we introduce the notation used throughout the paper. The inputs to our model are discrete speech content tokens, denoted by S = {21, s2. where N is the number of tokens in the speech sequence, and corresponding text tokens, represented as T = { .2..... ., L with being
the text sequence length. These tokens are first converted into embeddings through their respective embedding layers, resulting in Zg for speech embeddings and Z, for text embeddings.
During training, subsets of tokens masked depending the task, and the masked tokens, along with their corresponding embeddings, are denoted with a superscript M, such as S. and Z M The task-specific masking schemes are detailed in Section 2.5
2

---

Published as a conference paper at ICLR 2025
Cero
Cues
Manutoeds Alggeeoee Soveen
Neno
Hareent
Reons
Conformer RoFE
(3.2,3...)
Length Turgets
CTC Cerrctiinn
Speach
Core Teaks
Length Prodiction
Ausilary Tosko
Mad
O
Tokens
g
Yopoo (M) IT9
Logils
Legand
O
= LLngth
Yex CTC LeCgte
Tome Mankedl CTC Outguta
Figure 1: Overview of T2V2 architecture, best viewed in color. The parameters of the Conformer are shared across all tasks. The task specific heads with same color also share their parameters.
The shared Conformer encoder backbone is represented as F. and the task-specific heads include the CTC prediction head, Herc. CTC correction head, HCORR, speech prediction head, HSPEECH. and the length prediction head, MLEN-
2.2 CONFORMER WITH ROPE BACKBONE
In our proposed method, all tasks share a single backbone, FF, based on the Conformer (Gulati et al., 2020) architecture with rotary positional embeddings (RoPE) (Su et al. 2024]. The Conformer combines convolutional layers, which capture local dependencies in the input sequences, with self- attention layers, which model long-rand raage dependencies both speech and sequences.
Rotary Positional Embeddings (RoPE) further enhance the model's 's ability to process variable-length sequences by encoding positional information directly within the self-attention mechanism. Unlike traditional positional encodings, RoPE handles long sequences more effectively, making it especially beneficial for speech and text tasks that require both ane-grained temporal modeling and global context understanding. This combination has been demonstrated to be particularly effective for speech recognition (Li et al., 202T) and synthesis (Borsos et al.) (023b) tasks. Thus we utilize this combination as the backbone efficiently share parameters across multiple tasks, improving both performance and flexibility in multitask learning.
2.3
CORE TASKS
2.3.1
SPEECH RECOGNITION WITH CTC
For the speech recognition task, the speech embeddings Zs. are passed through the shared Con- former backbone F, and then through the CTC text prediction head Here. The Connectionist Tem- poral Classification (CTC) framework (Graves al. 2006) is used align the input speech with the target transcription without requiring explicit frame-level alignment. The predicted output logits over the vocabulary at each time step are given by:
Yere Hclc(F(Zs)),
where Yere represents the CTC logits.
The CTC loss Lere, is computed by marginalizing over all possible alignments between the input sequence and the target transcription T:
Lere
log
Σ
P(a(Yerc),
aEA(T))
where A(T) is the set of all valid alignments of the target sequence T. This formulation allows the model to predict the text sequence while accounting for varying lengths between the input speech and the target transcription.

---

Published as a conference paper at ICLR 2025
2.3.2 TEXT TO SPEECH AND LENGTH PREDICTION
For the tex--to-speech (TTS) task, we compute the alignment score matrix M RRL by extractiing the log probabilities of the text tokens T from the CTC logits YeTe E RXV
as: Vn E {1,..., NY, vl (1..... L).
M YeTC,
(3)
Next, we apply the Monotonic Alignment Search (MAS) Kim et al,, 2020 to M to obtain the alignment matrix A E {0, 1)NxL The MAS establishes a monotonic mapping between the text tokens and the speech time steps. For details, we refer the reader to Kim et al. (2020).
The imput to the Conformer backbone is the point-wise addition of the embeddings of the upsampled text tokens Tup = Upsample A), Z and the masked speech embeddings Z X = Z +
z The backbone output is passed through the speech prediction head MSPEECH. and the loss is computed using cross-entropy focused the masked positions:
LTTS =
Σ E.M
S: log P(HsPEECU (F (((Xrss)):,
(4)
where M is the set of masked positions, represents the target speech tokens, and P(-) is the predicted probability.
For the length prediction task, the target token lengths L, are derived from the MAS alignment matrix A. Text tokens are passed through the length prediction head MLEN. and the loss is computed using an L1 objective:
LLEN Σ (LLEN(F(TT) llo([,)]-
(5)
2.4 AUXILIARY TASKS
2.4.1 CTC ERROR CORRECTION
In traditional CTC-based models, outputs at each time step are conditionally independent (Graves) al. 2006), limiting their ability to capture long-term dependencies and recover from early errors. To address this, introduce CTC eтror correction task that refines initial CTC predictions using confidence based masked intermediate CTC outputs and acoustic context Zs. Unlike Mask-CTC based approaches (Higuchi et al. 2020 2021b), which operate reduced intermediate CTC outputs and explicitly handle substitutions, insertions, and deletions, our method is more closely related to Nozaki & Komatsu (202T); Chi al. (202T) and operates un-reduced outputs, allowing implicit handling of all error types as well as leveraging full acoustic context, thereby relaxing the indepen- dence limitations in the correction phase. For a more detailed discussion in relation the related works, please refer to appendix I.I
The CTC logits Yere, are detached from the computational graph, and the argmax is taken to pro- duce the raw CTC predictions Yerc- A subset of the tokens in Yere is then masked create the masked CTC outputs Y-C- These masked CTC outputs are embedded into Z and point-wise added with the unmasked speech embeddings Zs, resulting in X represents the combined input for CTC error correction, which are passed through the shared Con- former backbone F, and forwarded to the CTC correction head, HCORR. where the loss is computed using the CTC loss function:
2 + Zs. where Xoor
YoUr HCORR (F((
Lerc = log Σ P(a(Yaun)
nEA((T)
By correcting the CTC predictions in this manner, we expect to mitigate the potential negative impact of multitask learning ASR performance and improve transcription accuracy.
2.4.2 UNCONDITIONAL. SPEECH MASKED LANGUAGE MODEL.
Classifier Free Guidance (CFG) (oo & Salimans 202T) has been used diffusion models to balance conditional and unconditional outputs each time step. This approach has also been successfully
4

---

Published as a conference paper at ICLR 2025
applied to CFG to improve the TTS performance by incorporating an unconditional speech only MLM.
models in image generation, such as Muse (Chang et al. 2023a) We adapt
For speech MLM training, the masked speech embeddings Z passed through the shared Con- former backbone, F. The output is then used to predict the speech tokens at masked positions using cross-entropy loss, similar the text-to-speech task but without text conditioning and reusing the same speech head HsPEECC:
YMLM HSPEECH (F(Z))),
(8)
LMLM Σ
S, S, P(YMLM.))
(9)
where Mspeech is the of masked positions S are the target speech tokens, and MLMMM,))) is the predicted probability at position i. The goal during training is to reconstruct the masked speech tokens from the unmasked speech embeddings.
2.5 MASKING SCHEME
In our multitask framework, we employ different masking schemes depending on the task, as the requirements for masking vary. For the CTC error correction task, we use uniform masking strat- egy, while for the TTS and speech MLM tasks, we utilize more aggressive cosine-based masking strategy. Below, we explain the rationale for each scheme.
CTC Error Correction Task: Uniform Masking During training for the CTC eгror correction task, the model refines low-confidence tokens without starting from fully masked tokens. A uniform masking strategy is applied, where the mask is sampled as:
Moon Bernouli(pp, p~ (0, 1),
(10)
targeting random number of tokens from the CTC output Yere for correction. This approach ensures the model focuses on low-confidence tokens while maintaining enough context for effective refinement.
TTS and Speech MLM Tasks: Cosine Masking Schedule For the TTS and speech MLM tasks, start with fully masked sequences and progressively unmask tokens using a cosine masking schedule inspired by MaskGIT (Chang et al. 2022). At each step, the mask is determined as:
U(0, π
Mspecch~ Bernoulli(p),
cos(u),
(11)
2
where u is sampled from a uniform distribution over [0, ]. and p is the cosine of u. This sched- ule mimics the inference iterative refinement where more tokens are masked initially and reduces masking as the model refines the outputs.
2.6 OVERALL TRAINING OBJECTIVE
In our framework, adopted the simplest weighted-sum scalarization approach for multi-objective optimization, assigning equal weights to all task-specific losses from eqs. obtain a single objective as follows:
and to
(12) This choice was based on the assumption that the tasks are cooperative rather than conflicting and the combined loss ensures that the model learns to balance multiple tasks efficiently, allowing it to leverage shared knowledge across tasks while optimizing performance for each individual task.
Liotal = Lerc - CrTS + LMLM + LCTC-cor + CEEN,
2.7
INFERENCE
2.7.1 SPEECH RECOGNITION
During inference, the speech embeddings Zs are passed through the Conformer backbone into the CTC head to generate logits. Low-confidence tokens are identified based on threshold т, masked and refined iteratively by combining masked outputs Y-M with the speech embeddings Zs- Refinement continues until token confidence scores exceed 7 or for a fixed number iterations. The final transcription is obtained through CTC decoding of Yor.
5

---

Published as a conference paper at ICLR 2025
2.7.2 TEXT TO SPEECH
For TTS, text tokens are passed through the backbone into the length prediction head predict log lengths, which are then converted to integer lengths as Innd = cxx [ These predicted lengths are used upsample the text tokens, Tup Upsample(TT Lewea). which are combined with fully masked speech embeddings M. and passed through the backbone into the speech predic- tion head. During generation, tokens are iteratively unmasked based on their confidence scores, starting with fully masked tokens. The unmasking follows the cosine schedule, where the masking probability each iteration is given by pt cos 2T
Additionally, for Classifier-Fres Guidance (CFG), we forward Z without text conditioning to obtain the unconditional logits MLLM- The final output logits are then a weighted combination of the conditional logits YrTs, and the unconditional logits YMLM. as follows:
Ypeed (1 + A) YrIS- A YMLM.
(13)
where À is the guidance weight controlling the balance between conditional and unconditional out- puts. This process yields the final speech sequence Spred.
3 EXPERIMENTS
To verify the effectiveness of our proposed methods, we perform extensive experimental evaluations. In this section, first describe the training details followed by the various experimental results in the subsequent subsections.
3.1 MODEL ARCHITECTURE AND TRAINING DETAILS
3.1.1 TRAINING INFRASTRUCTURE AND SETTINGS
All our models were implemented using the Pytorch(Paszler et al. 2019) framework and trained on NVIDIA H100-80G GPUs with training, we utilized deepspeed RajihannddR et al. 2020) ZeRO optimization. For all experiments, we utilize the Adam optimizer Kinggna & Ba 2015) with a peak learning rate of 2.5e linearly warmed up the first 4K iterations and decayed over the remaining iterations with cosine schedule. The betal and beta2 of the optimizer are set to {0.8, 0.99), with no weight decay, and clip the gradient norm with maximum of 0.5.
et al.) 2019) precision. For efficient distributed
3.1.2 TRAINING DATASETS
For training all models, utilized the 60K hour LibriLight(Kahn et al. 2020) dataset. Ad- ditionally, for punctuated and cased transcriptions, we utilized the transcriptions provided in the LibriHeavy(Kang al. 2024) dataset. We mainly the 509 hour small subset of the LibriHeavy dataset for our experiments. We filtered out samples shorter than 0.2s and longer than 20s for better GPU utilization. Further, we removed samples which violate the input-target length constraints of the CTC loss. Following this we are left with around 500 hours of speech data that we use for the training. We perform all our experiments on 16Khz speech.
3.1.3 MODEL. ARCHITECTURE
For all experiments (unless specified otherwise), we used the same conformer backbone with layers with hidden size of 384, 8 attention heads, 1536 linear size, and convolution kernel size of We utilize Rotary Position Embedding (RoPE) (Su et al.] 2024). For each of the task-specific heads, we utilized the same structure of linear layer with the same hidden size, followed by GELU activation and layer normalization, and finally an output linear layer to map to the respective output dimensions. We trained the models on the small subset of the LibriHeavy dataset for 30K iterations with a batch size of 64.
To convert the content tokens generated by proposed method to acoustic tokens from the codec, we utilized the masked iterative generative model, SoundStorm (Brsoss et al. (0236). We repro- duced this model by utilizing conformer backbone with 12 layers with hidden size of 1024, 16

---

Published as a conference paper at ICLR 2025
attention heads, 4096 linear size, and convolution kernel size of 5 with RoPE, followed by linear output heads for each RVQ level in the codec. We also trained this model on the LibriLight dataset with 15 second segments for 50K iterations with batch size of 640.
3.1.4 TOKENIZERS
For the acoustic tokenizer, we use utilize a 12-level RVQ- RVQ-based codec following the architecture of Descript Audio Codec (DAC) (Kumar al. 2024), trained on the LibriLight dataset with batch size of 144 for 200K iterations using the hyperparameters from For the content tokenizer, we utilize the publicly available HuBERT large checkpoint trained on the LibriLight datase Further we trained 1024-cluster K-Means tokenizer on the train-clean- 100 subset of the aibriSpeech"Panayyon 2013) dataset. Both the acoustic and content tokenizers produce tokens the rate of 50Hz, which simplifies the implementation. Finally for the text tokenization, we used simple uff-8 byte based representation. Byte representation has the advantage of having a compact output layer, requiring minimal text pre-processing and ability to handle any character and language. We did some basic pre-proeessing to expand certain common word contractions and numbers as is standard practice in ASR or TTS training.
et al.)
3.2 EVALUATION DATASETS AND METRICS
To evaluate the performance of our proposed method, we utilized the following datasets and metrics. For evaluation of the ASR task, we utilize the LibriSpeech test-clean subset. For the TTS task, we randomly sampled 40 sentences from the LibriSpeech test-clean subset with at least 20 words in each sentence as the text input and sampled 3-8 second segments from all 20 speakers of the (Mysoee 2014) dataset as the speaker prompt for zero-shot TTS evaluation.
For evaluating the ASR performance, we utilized the Word Error Rate (WER) and Character Error Rate (CER). Since, our model is trained on punctuated and cased inputs, for the evaluation, we normalize the predicted sentences be removing all punctuation except the apostrophe and converting to upper-case following the format of the LirbiSpeech transcripts.
We utilized several automatic metries for zero-shot TTS evaluation, which included UTMOS (Sacki et al. 2022) for speech quality, Speaker Encoder Cosine Similarity (SECS) using the wavim-base plus-sv mode F for speaker similarity, Character Error Rate (CER) using the hubert-large-I9960 ft mode for percieved intelligibility and robustness. To effectively evaluate the zero-shot TTS performance, we finally perform a subjective evaluation. For this evaluation, we utilize one sample per speaker from the evaluation data described above, and present samples from two systems along with the text input and ask the raters to score between (+2, (order methods is randomized), based naturalness, acoustic quality, and human likeness. At least raters rate each sample and this gives us the Comparative mean opinion score (CMOS). Similarly we perform subjective evaluation for Speaker similarity Comparative mean opinion score (SCMÓS), the difference being for this evaluation the reference speaker prompt is also presented to the raters and asked to compare which of the two shown methods is more similar the reference. The rating scale and number of raters per sample are same as the CMOS evaluation. For all scores, we compute the 95% confidence interval by bootstrapping and measure the statistical significance with the Wilcoxon signed-ram -rak test (p-value).
To evaluate latency, we conducted controlled inference runtime (IR) experiments. For TTS, we mea- sured end-to-end runtime (IR-e2e) and text-to-content token runtime (IR-t2c) using 405-character sentence. For ASR, we measured IR with a 60-second sample. All measurements were averaged over 100 trials on single H100 GPU.
3.3 BASELINES
To test the effectiveness of our method, we utilize several state-of-the-ae baselines across various speech synthesis and recognition paradigms.
"https//hugggnnffcccccm
'https//hugginnfacc.co
7

---

Published as a conference paper at ICLR 2025
1. Speech Synthesis (a) Non-iterative: HierSpeech++<Lee et al.
2023). YourTTS et al. 2022
(b) Diffusion: StyleTTS2 (Li 2024)
(c) Autoregressive: XaTSooasanova et al.
2024 WhisperSpeecH
(based on (Borsos
et al. 2023a Kharitonov al. 2023)
2. Speech Recognition
(a) Transducer based non-discrete ASR: Zipformer-Tranee Transducer Yao et al. 2023 6
(b) Discrete ASR: Coformer-CTC trained on discrete content tokens.
We utilize publicly released checkpoints of all the above methods except SoundStorm, which we reproduce and train on the same dataset our method and with similar parameter count.
3.4
ZERO-SHOT TEXT TO SPERCH SYNTHESIS
3.4.1
ABLATION STUDY
Table 1: Zero-shot TTS ablation study for different tasks.
Task Setting
UTMOS
CER
SECS
SMLM, CORR
4.39 # 0.04
0.95
0.94 # 0.01
W SMLM, w/o CORR
4.41 + 0.04
1.08
0.94 + 0.01
w/o SMLM, w/o CORR
4.39 + 0.03
0.82
0.94 # 0.01
To study the effect of each component of our proposed method for the TTS task, we conduct ex- tensive ablation study. First we check the performance of introducing the auxiliary CTC-correction and Speech MLM tasks. For this evaluation, we predict the content tokens in single pass through the network. The results in Table indicaae that the auxiliary tasks have minimal impact on the core TTS task and the performance maintained.
Table 2: Zero-shot TTS ablation study for dif- ferent number of iterations.
Table 3: TTS ablation study for CFG weight A.
UTMOS
CER
SECS
Iters
UTMOS
CER
SECS
0.0
4.43 = 0.03
1.12
0.94 0.01
4.39 + 0.04
0.95
0.94 0.01
1.0
4.43 + 0.02
0.55
0.94 - 0,01
4
4,43 # 0.03
1.12
0.94 + 0.01
1.5
4.40 0.04
0.95
0.94 0.01
8
4.41 + 0.03
1.23
0.94 0.01
2.0
4.42 + 0.02
0.69
0.94 + 0.01
Next we evaluate the number of iterations for generating the content tokens. For this task we do not use CFG. The results reported in Table [ show that our method is able to achieve very good performance even with a single step. Increasing the number of iterations slightly improves the UTMOS, while slightly degrading the CER. For subsequent experiments, we utilize 4 iterations, which provides a balance between the UTMOS and CER scores.
Finally we evaluate the effect of CFG weight, A. From the results in Table 3 we can see that using CFG significantly improves the CER while maintaining the speech quality with = 1.0 giving the best CER score. The degradation observed in single-pass inference with SMLM is marginal (Table []. and is compensated by the iterative inference with CFG, which fully utilizes SMLM's benefits, significantly improving CER for TTS.
3.4.2 MAIN RESULT
Finally, with the best settings from the ablation study, we compare our method with state-of-the-art Our method significantly outperforms baselines trained on similar-scale data.
methods Table 4
8

---

Published as a conference paper at ICLR 2025
Table 4: sero TTS performance comparison. Methods with * indicate multilingual models. UD refers to Unpaired Data while PD refers to Paired Data in hours.
UD
PD
UTMOS
|CR|
SECS
IR-e2e (s)
IR-12c (s)
Large scale paired data * XTTS*
500k
2.8k
4.46 = 0.02
0.88
0.94 + 0.01
0.16 0.00
27k
4.12 # 0.07
0.78
0.93 0.01
2.60 = 0.03
WhisperSpeech
60k
60k
3.95 # 0.11
0.66
0.93
0.01
17.91 0.04
2.84 + 0.01
Small scale paired data
YouTTT**
689
3.69 + 0.08
2.02
0.90 - 0.02
0.11 = 0.00
StyleTTS2
94k
245
4.43 + 0.03
1.59
0.91
0.02
0.27 = 0.00
Ours
60k
500
4.43 # 0.02
0.55
0.94 + 0.01
0.57 = 0.00
0.06 + 0.00
Additionally, surpasses systems trained on thousands of hours of paired speech, demonstrating superior robustness in terms of the CER score and data efficiency. A point to note here is that the UTMOS and SECS metrics are more influenced by the SoundStorm and Codec rather than our method. IR-t2c shows that T2V2 is ~ 5 orders of magnitude faster than the AR WhisperSpeech baseline and surpasses other AR methods in IR-e2e.
We present the result of the subjective evaluation in Table From the results can see that in terms of CMOS, our method is at par with all the state-of-the-art methods since the p-value is greater than 0.05 indicating no statistically significant difference. While for the SCMOS, our method outperforms XTTS and WhisperSpeech while being at par with HierSpeech++ and StyleTTS2. We encourage the readers listen the samples provided in the supplementary materials.
Table 5: Comparative MOS Speech Quality (CMOS) and Speaker Similarity (SCMOS) on
scale (-2, +2). p-value < 0.05 indicate statistical significance.
CMOS (p-value)
SCMOS (p-value)
HierSpeech++
+0.10 + 0.25 (0.337)
+0.12 + 0.26 (0.287)
XTTS
-0.13 # 0.28 (0.418)
-0.30 + 0.22 (0.007)
StyleTTT2
+0.16 + 0.25 (0.271)
+0.14 + 0.24 (0.201)
WhisperSpeech
-0.11 = 0.27 (0.490)
-0.63 + 0.21 (1.5e=))
Ours
0.00
0.00
3.5
AUTOMATIC SPEECH RECOGNITION
3.5.1 ABLATION STUDY
To evaluate the impact of various components and inference settings ASR performance, ducted an ablation study. In Table 6 we examine the effect auxiliary tasks on ASR performance, noting that the CTC Correction task notably regulates the model and improves raw ASR performance from the CTC head without correction. While the SMLM task alone slightly degrades performance, the CORR task compensates, allowing iterative correction to significantly enhance ASR results. Next, investigated the confidence threshold and number of iterations for CTC error correction (Table 7). A threshold of 0.7 with 16 correction iterations yielded the best results. Increasing the iterations slightly raised the IR, but the process remained efficient.
Finally, we analyzed how iterative correction handles all error types (Table Using the best config- uration from Table observed consistent relative improvements across substitutions, insertions, and deletions, demonstrating the effectiveness of our proposed mechanism.
3.5.2 MAIN RESULT
Finally we compare our method with the baselines Table 9 We show the results of the non- discrete reference model T also trained with casing and punctuation. As can

---

Published as a conference paper at ICLR 2025
Table 6: ASR ablation study for different tasks.
Table 7: ASR ablation study for different cor- rection thresholds and iterations.
CER
WER
W SMLM, w CORR
2.732
8.651
Corr. Thresh Iters
CER
WER
IR(s)
W. SMLM, w/o CORR
2.949
9,428
w/o SMLM, w/o CORR
2.886
9.120
w/o CORR
2.73
8.65
0.32 = 0.02
0.8
2.73
8.44
0.40 = 0.03
Table Individual emorr type improvements.
0.8
4
2.72
8.37
0.39 = 0.03
0.8
8
2.72
8.33
0.42 = 0.03
Sub
Ins
Del
0.7
8
2.72
8.29
0,42 # 0.03
w/o CORR
1.300
0.090
0.140
0.7
16
2.71
8.27
0.47 0.03
1.255
0.082
0.135
0.7
32
2.71
8.27
0.60 = 0.03
CORR
43.466
48.89%
43357%
see discrete ASR still under-performs compared to non-discrete AR when compared with the discrete baseline CConforme-CCCC) we achieve comparable performance. We further verified how our method scales with scaling the data. For that we trained larger version of our model from scratch with layers on the LibriHeavy large subset (50K hours) and we see that method outperforms the CTC baseline with same 10-layer conformer backbone. Thus validating the effectiveness and scaling capabilities of T2V2.
transducer, however
Table 9: ASR results for models trained with punctuation and casing. The publicly released models for Zipformer- Zipformer-Transduce. are used for the evaluation, while Conformer-CTC is trained by us.
Libriheavy Subset
CER
WER
IR (s)
Non-discrete ASR (BPE encoding) Zipformer-Tran Transducer
small large
2.01
5.33
1.49 + 0.07 1.51 0.14
Sopformer-Transdiee
0.66
1.99
Discrete ASR (Byte Encoding)
Conformer-CTC
small small
2.69
8.28
0.32 # 0.02
Ours
2.71
8.27
0.47 0.03
Conformer- CTC
large
1.53
4.36
0.34 + 0.02
Ours
large
1.31
4.09
0.55 0.02
FUTURE PERSPECTIVES AND LIMITATIONS
In the current work, focus T2V2's capability in the two core tasks of ASR and TTS, however, the multitask training paradigm could potentially allow the model to learn high performing and general purpose representations speech and text which could be used for other downstream tasks as well as enable training modality agnostic language models. As for the limitations of the T2V2, we see that discrete ASR still underperforms compared to models trained with continuous features, while for TTS, we rely on separate content token to acoustic token translation, which might limit the quality of synthesized speech.
CONCLUSIONS
In this work, we introduced T2V2, the first unified non-autoregressive model capable of handling both automatic speech recognition and text-to-speech synthesis. By leveraging discrete tokens from self-supervised models and incorporating auxiliary tasks like CTC error correction and uncondi- tional speech MLM, T2V2 effectively improves performance across both ASR and TTS. Addition- ally, the use of Monotonic Alignment Search ensures celf-contained approach to text-speech align- ment without relying external aligners. Our experimental results demonstrate significant gains in TTS quality while maintaining competitive ASR performance, highlighting the potential of unified models for multitask learning in speech processing.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our proposed method achieves high quality synthetic speech a target speaker's voice with short sample, thus it is paramount address the broader impact of research. Our method may be potentially misused by malicious parties, towards this end we verified that our speech could still be detected fake speech by a third party detector
REPRODUCIBILIITY STATEMENT
For reproducibility, we have carefully described the model architecture, optimization tasks, training masking schemes, loss functions, and inference procedure in Section [ Additionaly, implementa- tion and training hyperparameters, and all pre-trained models have been described in Section 3.1 including their links the footnotes.
ACKNOWLEDGMENTS
This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, 1, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 13:12449-12460, 2020.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint rrXiv23305.09636 2023b.
Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. B processing for neural networks. 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 88 91, 2019. doi: 1LL109/ARITH..11.00000
Edresson Casanova, Julian Weber, Christopher D Shulby, Arnalo Candido Junior, Eren Gölge, and Moacir A Ponti. YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2709-2720. PMLR, 17-23 Jul 2022. URL http://///ooodddooooooo wgw...iiig.... mmg.mml.... wttp:smmmrpprr.sseeememmmm http://1166//ccaaaaa2222222 html
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Al- jafari. Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. Xtts: massively multilingual zero-shot text-to-speech model, 2024. URL wtt.hhtts://// 04904
William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4960-4964. IEBE, 2016.
Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. I Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325. 2022.
hhtps://deeet..esembll.i/

---

Published as a conference paper at ICLR 2025
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming- -ssuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 4055-4075 PMLR, 23-29 Jul 2023a. URL https: hettps://procecinnn/
www.mmr.ppeee.. wwwees/v2000 chang233n htmlll
Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe. Exploration of efficient end-to-end asr using discretized input from seff-supervised learning. In INTERSPEECH 2023, pp. 1399-1403, 2023b, doi: 10.2 1117///meessseecc22222-----
Ethan A. Chi, Julian Salazar, and Katrin Kirchhoff. Align-refine: Non-autoregressive speech recog- nition via iterative realignment. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies. pp. 1920-1927, Online, June 2021. Association for Computational Linguisties. doi: 10. 12.18863/v1/222 http://2......llnnnn .naacl- mainl 154. URL. https::/aat antho ogy org. orgg2021112222 2021. nwa.cacclll -ma in. 154
Alex Graves. Sequence transduction with recurrent neural networks arXiv:1211.3711, 2012.
arXiv preprint
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Anmol Gulati, James Qin, Shung-Chene Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
anavolution-augmeent transformer
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask- ctc: Non-autoregressivve end-to-end asr with ctc and mask predict. In Proceedings of the 2020 International Conference on Spoken Language Processing (INTERSPEECH), 2020.
Yosuke Higuchi, Nanxin Chen, Yuya Fujita, Hirofumi Inaguma, Tatsuya Komatsu, Jacsong Lee, Jumon Nozaki, Tianzi Wang, and Shinji Watanabe. A comparative study on non-autoregresssvv modelings for speech-to-text generation. In 2021 IEEE Automatic Speech Recognition and Un- derstanding Workshop (ASRU), pp. 47-54, 2021a. doi: 10.1 10.1109/ASRUS11 502.2021..12222 002199681157.
Yosuke Higuchi, Hirofumi Inaguma, Shinji Watanabe, Tetsuji Ogawa, and Tetsunori Kobayashi. Improved mask-cte for non-autoregressive end-to-end asr. In ICASSP 2021 2021 IEEE Interna- fional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8363-8367, 2021b.
doi: 10.11091CCSSPP39222 010.722...22.999 2021.9414198.
Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, and Shinji Watan- abe. Bert meets ctc: New formulation of end-to-end speech recognition with pre-trained masked language model. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 202/ Workshop on Deep Generative Models and DownstreammApllicatinns. 2021. UtL/httss:://peeeeeeee
net/
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and AbdeIrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmante Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673 IEEE, 2020.
12

---

Published as a conference paper at ICLR 2025
Wei Kang, Xiaoyu Yang. Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: a 50,000 hours ast corpus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 10991-109955. IEFE, 2024.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High- fidelity text-to-speech with minimal supervision. Transactions of the Association for Computa- tional Linguistics, 11 11:170 11:1703-1716 2023.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Sys- tems, 33:8067-8077, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp. 5530-5540. PMLR, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL.http:
6980
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024.
Sang-Hoon Lee, Ha- Ha-Yeong Choi, Seung-Bin Kim, and Seong-W Whan Lee. Hierspeech++:: Bridg- ing the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.124:5 2023.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 6706-6713, 2019.
Shengqiang Li, Menglong Xu, and Xiao-Lei Zhang. Conformer-basee end-to-end speech recog- nition with rotary position embedding. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 443-447, IEEE, 2021.
Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. In Proceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Soumi Maiti, Yifan Peng, Shukjae Choi, Geen Jung, Xuankai Chang, and Shinji Watanabe. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024 2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2024. 1224.1044712..
2024. doi: 10. 10.1199/CASSS
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proc. Interspeech 2017, pp. 498-502, 2017. doi: (0.21437/lnerspeechh2011 http:rspeeechh.1111111 1386.
Gautham Mysore. Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech?-a dataset, insights, and challenges. IEEE Signal Processing Letters, 22(8): 1006-1010, 2014.
Jumon Nozaki and Tatsuya Komatsu. Relaxing the conditional independence assumption of etc- based asr by conditioning on intermediate predictions. In Interspeech 2021, pp. 3735-3739, 2021. doi: 10.21 10.21443//nnerspeechh2222 2011--111
13

---

Published as a conference paper at ICLR 2025
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 JEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
ISBN 9781728199986.
Yi Ren, Chenxu Hu, Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tin-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 04558, 2020.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint S.Vvv:230..19255, 2023.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: U'ooyoo-SrrLab System for VoiceMOS Challenge 2022. In Proc. Inter- speech 2022, pp. 4521-4525, 2022. doi: 10.21457/lnerspeehhh22----
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. ISSN 0925-2312. doi: http:://dbi.oo//1.0016j.....eeee sciencedirech com/
127063. URL https: ///w///wwwww www..
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 14, pp. 112--112, Cambridge, MA, USA, 2014. MIT Press.
Hawau Olamide Toyin, Hao Li, and Hanan Aldarmaki. STTATTS: Unified speech-to-text and text-to-speech model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2024, pp. 1853-6863, Miami,
Florida, USA, November 2024. Association for Computational Linguisties. URL https: //ac 1/aclanthologe hww.hholooy.rrg/22222222 findi nnsdinngsennnll 401
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL
02111
Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3709- 3716, 2024. doi: 10.1109/IASLP22024333333
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703. 10135, 2017.
Lei Xiujuan and Shi Zhongke. Overview of multi-objective optimization methods. Journal of Systems Engineering and Electronics, 15(2):142-146, 2004.
Runyan Yang, Huibao Yang, Xiqing Zhang, Tiantian Ye, Ying Liu, Yingying Gao, Shilei Zhang, Chao Deng, and Junlan Feng. Polyspeech: Exploring unified multitask speech models for com- petitiveness with single-task models. arXiv preprint arXiv:2406.07801, 2024a.
14

---

Published as a conference paper at ICLR 2025
Yifan Yang, Feiyu Shen, Chenpeng Du, Ziyang Ma, Kai Yu, Daniel Povey, and Xie Chen. Towards universal speech discrete tokens: A case study for ast and tts. In ICASSP 2024-2024 IEEE In- ternational Conference Acousties, Speech and Signal Processing (ICASSP), pp. 10401-10405. IEEE, 2024b.
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representations, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
A APPENDIX
A.1 CTC ERROR CORRECTION AND RELATED WORK
A.1.1 RELATION LLIGN-REFING
Align-Refine (Chi et al. 2021) and our method share the conceptual similarity of utilizing an itera- tive refinement paradigm. However, our approach differs significantly in its formulation:
Separate Decoder vs. Direct Approach: Align-Refine employs a separate non-causal trans- former decoder for refinement, integrating encoder features via cross-attention. In contrast, our method formulates refinement as a separate task by reusing the same backbone encoder and directly combining intermediate CTC outputs with speech tokens for refinement. This eliminates the need for additional modules such as decoders or cross-attents atteentoo layers.
A.1.2 RELATION TO SELF-CONDITIONED CTC
Our method also shares similarities with seff-Conditioned CTC (Nozaki & Komatsu 2021 in lever- aging intermediate CTC outputs, but there notable differences:
Intermediate Conditioning: Self Coff-Conditioned CTC applies intermediate CTC logit embeddings via additional linear projection layers intermediate encoder stages, making well-suited for ded- icated ASR models. However, in our multi-task scenario, where the backbone supports both ASR and TTS tasks, such intermediate conditioning might not generalize well. Our approach avoids additional layers, maintaining lightweight design.
Iterative Refinement: Unlike Self-Conditioned CTC, which does not support iterative refinement, method incorporates iterative refinement core mechanism.
15


Reviews:
[{"review_id": "VaAxXJ2lwk", "reviewer": "Reviewer_quVm", "summary": "This paper addresses the challenge of unifying automatic speech recognition (ASR) and text-to-speech synthesis (TTS) using discrete speech tokens derived from self-supervised models. The authors introduce a CTC error correction task to enhance ASR performance by refining raw CTC outputs. Additionally, they propose a monotonic alignment search (MAS) technique with intermediate CTC outputs to achieve TTS alignment without relying on external tools. Experiments were conducted on the LibriHeavy dataset, with both objective and subjective evaluations of the synthesized speech.", "questions": "* (Lines 47-49): \"While multitask learning offers potential for parameter sharing, the performance of individual tasks might suffer due to the inherent complexity of training multiple tasks simultaneously.\" — The authors should provide references to prior work to support this claim. Additionally, if multitask learning poses such challenges, how were the authors able to handle both text-to-speech and speech-to-text effectively in the proposed unified model?\n\n* Simplifying Figure 1 would make it easier for readers to understand the proposed framework.\n\n* (Line 64): Why was the Conformer architecture with rotary positional embeddings (RoPE) chosen? Could the authors expand on the reasoning for using RoPE? Does RoPE offer benefits for both ASR and TTS tasks?\n\n* (Line 261): It is unclear how the authors optimized the individual losses within their multitask learning framework. Was any weighting applied to the individual losses, and how did each loss contribute to the optimization of each task?\n\n* There is some confusion regarding the dataset subsets used to train the unified model. In lines 311-315, the authors mention a 509-hour subset of LibriHeavy, while in lines 484-485 they refer to training a larger version of the model on a 50K-hour LibriHeavy subset for discrete ASR. What is the relationship between these two training stages? In the second ASR training, is the model initialized from the first training stage, or is it a separately trained ASR model?\n\n* In Table 8, could the authors include baselines for the discrete Zipformer-CTC model for completeness?\n\n* How does the proposed unified model perform on other TTS attributes, such as pitch, volume, and speed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This paper explores a challenging problem that enables the unification of ASR and TTS tasks, allowing the framework to benefit from the underlying complexities of each task.\n* The claims were tested, demonstrating improvements in TTS performance as intended, while achieving comparable performance for the ASR task.", "weaknesses": "It is unclear how much each task contributed to the overall improvement across tasks. Additionally, the performance of the proposed model on different attributes—such as pitch, volume, and speed of synthesized speech—has not been clearly detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8tbf3dYcVu", "reviewer": "Reviewer_pcMe", "summary": "The paper introduces a unified non-autoregressive model designed to perform named T2V2 for both of ASR and TTS task. The backbone of T2V2 is Conformer with RoPE which operates on discrete speech tokens. The model achieves the tasks the it's designed for by applying CTC for ASR and conditional masked LM approach for TTS, along with auxiliary tasks like CTC error correction and nconditional\nspeech masked LM. With extensive experiment and evaluation, the author achieves competitive TTS and ASR performance. The main contributions claimed in the paper include:\n* The paper proposes the first unified framework based on discrete speech tokens handling ASR and TTS task.\n* The introduction of CTC error correction auxiliary task potentially addresses conditional independence limitation in CTC objective and improves ASR performance.\n* The MAS-based alignment solution is innovative and allows for effective TTS alignment without external alignment tools.", "questions": "1. What's the rational of doing the token length prediction by feeding text embedding through conformer backbone? This is odd because this is only task/step that conformer has to directly deal with text embedding and there seems to be risk of confusing the model.\n2. The length prediction loss is not included in the final training objective in eqs 16, how is the length prediction head trained?\n3. When the paper introduces CTC error correction task, it is stated in section 2.4.1 that the purpose is to address the independence assumption in CTC loss. From the mechanism of CTC error correction, it doesn't seem like the training breaks such assumption because the CTC logits was already detached from the computational graph. The author should elaborate more how this method essentially improves the performance. And how is it different from simply masking ground truth text embedding (or randomly augmented text) and do error correction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper presents a unified framework capable of both ASR and TTS, which may streamline resource use and advance multitask learning for speech models.\n* The proposed auxiliary tasks such as CTC error correction and unconditional speech MLM demonstrate a solution for future work to address challenges in multitask learning.\n* The paper provides comprehensive experimental results include some ablation studies to justify the design choice and demonstrate T2V2's effectiveness.", "weaknesses": "* There is lack of discussions about or comparison with other multi-task models equipped with ASR & TTS capabilities such as [1] [2], etc. The discussion of related work isn't sufficient enough.\n* The author states that the motivation of adopting non-autoregressive approach is the superior decoding time, however, there is lack of comparison of inference efficiency compared with its autoregressive counterparts.\n* Presentation needs to be improved: for example: there is a mixed terminologies of \"CTC token\" \"Text token\" \"Content token\" \"Acoustic token\" used in Figure 1 and section 2/3, there should be a clear definition of the core terms used in the paper.\n\n\n\n[1] Toyin, Hawau Olamide, Hao Li, and Hanan Aldarmaki. \"STTATTS: Unified Speech-To-Text And Text-To-Speech Model.\" arXiv preprint arXiv:2410.18607 (2024).\n\n[2] Yang, Runyan, et al. \"PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models.\" arXiv preprint arXiv:2406.07801 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pm6MFxKmX0", "reviewer": "Reviewer_D6js", "summary": "This paper proposed a T2V2 framework that uses a shared Conformer backbone to jointly train non-autoregressive ASR and TTS tasks with Connectionist Temporal Classification (CTC) and masked language modeling (MLM) loss. Two axillary losses are additionally introduced to improve the model performance, which are CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance. In the proposed framework, Monotonic Alignment Search (MAS) was used with intermediate CTC outputs for TTS alignment without relying on external tools. The experiments show that T2V2 markedly improves synthesized speech robustness achieving state-of-the-art TTS performance and competitive discrete ASR performance.", "questions": "The idea of unifying ASR and TTS in one model is interesting. However, my personal feeling is that the work is still in an early stage and is not ready for publication in ICLR as a top conference. Here are my concerns on the paper.\n\n1. The clarity of the paper needs improvement.\n- The paper involved many topics in the field of ASR and TTS such as discrete tokens, vocoder, non-autoregressive generation, etc. I understand it is hard to clarify each topic in detail. I'd suggest the authors to at least detail the used method for the claimed contributions. For example, MAS is claimed to be a major contribution. However, it is not clear how MAS is applied to the framework. How does it compare to the external alignment? In addition, I am wondering how it is compared to CTC alignment, which also provides speech-text alignments for non-autoregressive ASR (R. Fan, W. Chu, P. Chang and J. Xiao, \"CASS-NAT: CTC Alignment-Based Single Step Non-Autoregressive Transformer for Speech Recognition,\" ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 5889-5893, doi: 10.1109/ICASSP39728.2021.9413429.). \n- It is suggested to clarify the loss for each head in Figure 1, especially the conditional TTS and unconditional MLM. Please differentiate the input and output, maybe with different colors?\n- The experiments of hyperparameter tuning can be presented in the appendix.\n\n2. Validation and performance\n- A very important issue of the paper is the comparison of latency. One of the advantages of using both non-autoregressive generation and discrete token for ASR is the inference speed. The authors have no such comparison. In addition, the paper seems to use iterative generation during inference, the model should be rigorously classified as semi-autoregressive model instead of non-autoregressive model.\n- The CTC error correction does not seem to be novel except that it is multi-tasked training with TTS loss. The Lcorr is not in-depth explored in this study. For example, according to the description of the paper, only the substation errors in CTC logits are considered to be corrected, while deletion and insertion errors are considered in literature. (Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa and T. Kobayashi, \"Improved Mask-CTC for Non-Autoregressive End-to-End ASR,\" ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 8363-8367, doi: 10.1109/ICASSP39728.2021.9414198.)\n- For TTS, the ablation shows that the impacts of MLM and CORR are low. It is interesting to see the performance of training the model with TTS only. This would provide us insights whether the joint ASR and TTS modeling is beneficial.\n- For ASR, the performance on test-clean doesn't seem to be competitive, compared to non-discrete ASR (almost twice of the WER). The advantage of discrete ASR is not stated in the paper.\n- What is the motivation to use CER for the English ASR evaluation?\n\nOverall, the paper has large room for improvements.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper are:\n\n1. The first paper to unify ASR and TTS in one model that operates on discrete tokens.\n2. Extensive experiments to show the performance of the proposed model.", "weaknesses": "The weaknesses of the paper are:\n\n1. Need improvement in clarity.\n2. Lacking validations on metrics like RTF or latency.\n3. The performance does not seem to be competitive.\n\nSee below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5K99yhhx9z", "reviewer": "Reviewer_AXxb", "summary": "This paper proposes a unified non-autoregressive model to perform both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis tasks. CTC error correction and classifier free are used to improve ASR and TTS performance respectively. Intermediate CTC outputs are used to align text and speech by Monotonic Alignment Search, without relying on external aligners.", "questions": "1. What is the time increase for 16 iterations compared to 1 iteration in Table 7?\n2. What is the performance of the large model for TTS tasks?\n3. Table 2 seems to have the best overall performance with iter=1.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting to use the intermediate CTC output to align text and speech via monotonic alignment search without relying on an external aligner.\n2. The CTC error correction method is useful and may be applied to other state-of-the-art ASR systems.\n3. The paper is well-structured and written.", "weaknesses": "1. Figure 1 is too complicated, with the arrows flowing in interlocking directions, so that readers may feel difficult to follow it.\n2. The content tokens employed in the T2V2 model are derived from HuBERT. However, converting these tokens into acoustic tokens and subsequently into speech requires additional components, namely a semantic-to-speech model (specifically SoundStorm) and a codec decoder. In my view, the efficacy of the speech synthesis process is largely dependent upon the performance of SoundStorm.\n3. The models being compared in the TTS experiment should have models that only predict content tokens + SoundStorm, which is the most similar setting to T2V2 + SoundStorm.\n4. In Tables 1 and 6, when comparing the performance of (with SMLM, without COR) and (without SMLM, without COR), it appears that SMLM harms both TTS and ASR tasks.\n5. The change in Corr. thresholds and iterations in Table 7 doesn't seem to make the CER change much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)