T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning

Authors: Nabarun Goswami, Hanqin Wang, Tatsuya Harada
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: ASR, TTS, Non-Autoregressive, Conformer, Multitask Learning

Abstract:
We introduce T2V2 (**T**ext to **V**oice and **V**oice to **T**ext), a unified non-autoregressive model capable of performing both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis within the same framework. T2V2 uses a shared Conformer backbone with rotary positional embeddings to efficiently handle these core tasks, with ASR trained using Connectionist Temporal Classification (CTC) loss and TTS using masked language modeling (MLM) loss. The model operates on discrete tokens, where speech tokens are generated by clustering features from a self-supervised learning model. To further enhance performance, we introduce auxiliary tasks: CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance to improve TTS. Our method is self-contained, leveraging intermediate CTC outputs to align text and speech using Monotonic Alignment Search, without relying on external aligners. We perform extensive experimental evaluation to verify the efficacy of the T2V2 framework, achieving state-of-the-art performance on TTS task and competitive performance in discrete ASR.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730134110873

thread_type:
official_review

<thread_object>
creator: reviewer_axxb

<content>
summary:
This paper proposes a unified non-autoregressive model to perform both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis tasks. CTC error correction and classifier free are used to improve ASR and TTS performance respectively. Intermediate CTC outputs are used to align text and speech by Monotonic Alignment Search, without relying on external aligners.

strengths:
1. It is interesting to use the intermediate CTC output to align text and speech via monotonic alignment search without relying on an external aligner.
2. The CTC error correction method is useful and may be applied to other state-of-the-art ASR systems.
3. The paper is well-structured and written.

weaknesses:
1. Figure 1 is too complicated, with the arrows flowing in interlocking directions, so that readers may feel difficult to follow it.
2. The content tokens employed in the T2V2 model are derived from HuBERT. However, converting these tokens into acoustic tokens and subsequently into speech requires additional components, namely a semantic-to-speech model (specifically SoundStorm) and a codec decoder. In my view, the efficacy of the speech synthesis process is largely dependent upon the performance of SoundStorm.
3. The models being compared in the TTS experiment should have models that only predict content tokens + SoundStorm, which is the most similar setting to T2V2 + SoundStorm.
4. In Tables 1 and 6, when comparing the performance of (with SMLM, without COR) and (without SMLM, without COR), it appears that SMLM harms both TTS and ASR tasks.
5. The change in Corr. thresholds and iterations in Table 7 doesn't seem to make the CER change much.

questions:
1. What is the time increase for 16 iterations compared to 1 iteration in Table 7?
2. What is the performance of the large model for TTS tasks?
3. Table 2 seems to have the best overall performance with iter=1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695708277

thread_type:
official_review

<thread_object>
creator: reviewer_d6js

<content>
summary:
This paper proposed a T2V2 framework that uses a shared Conformer backbone to jointly train non-autoregressive ASR and TTS tasks with Connectionist Temporal Classification (CTC) and masked language modeling (MLM) loss. Two axillary losses are additionally introduced to improve the model performance, which are CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance. In the proposed framework, Monotonic Alignment Search (MAS) was used with intermediate CTC outputs for TTS alignment without relying on external tools. The experiments show that T2V2 markedly improves synthesized speech robustness achieving state-of-the-art TTS performance and competitive discrete ASR performance.

strengths:
The strengths of this paper are:

1. The first paper to unify ASR and TTS in one model that operates on discrete tokens.
2. Extensive experiments to show the performance of the proposed model.

weaknesses:
The weaknesses of the paper are:

1. Need improvement in clarity.
2. Lacking validations on metrics like RTF or latency.
3. The performance does not seem to be competitive.

See below for details.

questions:
The idea of unifying ASR and TTS in one model is interesting. However, my personal feeling is that the work is still in an early stage and is not ready for publication in ICLR as a top conference. Here are my concerns on the paper.

1. The clarity of the paper needs improvement.
- The paper involved many topics in the field of ASR and TTS such as discrete tokens, vocoder, non-autoregressive generation, etc. I understand it is hard to clarify each topic in detail. I'd suggest the authors to at least detail the used method for the claimed contributions. For example, MAS is claimed to be a major contribution. However, it is not clear how MAS is applied to the framework. How does it compare to the external alignment? In addition, I am wondering how it is compared to CTC alignment, which also provides speech-text alignments for non-autoregressive ASR (R. Fan, W. Chu, P. Chang and J. Xiao, "CASS-NAT: CTC Alignment-Based Single Step Non-Autoregressive Transformer for Speech Recognition," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 5889-5893, doi: 10.1109/ICASSP39728.2021.9413429.). 
- It is suggested to clarify the loss for each head in Figure 1, especially the conditional TTS and unconditional MLM. Please differentiate the input and output, maybe with different colors?
- The experiments of hyperparameter tuning can be presented in the appendix.

2. Validation and performance
- A very important issue of the paper is the comparison of latency. One of the advantages of using both non-autoregressive generation and discrete token for ASR is the inference speed. The authors have no such comparison. In addition, the paper seems to use iterative generation during inference, the model should be rigorously classified as semi-autoregressive model instead of non-autoregressive model.
- The CTC error correction does not seem to be novel except that it is multi-tasked training with TTS loss. The Lcorr is not in-depth explored in this study. For example, according to the description of the paper, only the substation errors in CTC logits are considered to be corrected, while deletion and insertion errors are considered in literature. (Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa and T. Kobayashi, "Improved Mask-CTC for Non-Autoregressive End-to-End ASR," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 8363-8367, doi: 10.1109/ICASSP39728.2021.9414198.)
- For TTS, the ablation shows that the impacts of MLM and CORR are low. It is interesting to see the performance of training the model with TTS only. This would provide us insights whether the joint ASR and TTS modeling is beneficial.
- For ASR, the performance on test-clean doesn't seem to be competitive, compared to non-discrete ASR (almost twice of the WER). The advantage of discrete ASR is not stated in the paper.
- What is the motivation to use CER for the English ASR evaluation?

Overall, the paper has large room for improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699757358

thread_type:
official_review

<thread_object>
creator: reviewer_pcme

<content>
summary:
The paper introduces a unified non-autoregressive model designed to perform named T2V2 for both of ASR and TTS task. The backbone of T2V2 is Conformer with RoPE which operates on discrete speech tokens. The model achieves the tasks the it's designed for by applying CTC for ASR and conditional masked LM approach for TTS, along with auxiliary tasks like CTC error correction and nconditional
speech masked LM. With extensive experiment and evaluation, the author achieves competitive TTS and ASR performance. The main contributions claimed in the paper include:
* The paper proposes the first unified framework based on discrete speech tokens handling ASR and TTS task.
* The introduction of CTC error correction auxiliary task potentially addresses conditional independence limitation in CTC objective and improves ASR performance.
* The MAS-based alignment solution is innovative and allows for effective TTS alignment without external alignment tools.

strengths:
* The paper presents a unified framework capable of both ASR and TTS, which may streamline resource use and advance multitask learning for speech models.
* The proposed auxiliary tasks such as CTC error correction and unconditional speech MLM demonstrate a solution for future work to address challenges in multitask learning.
* The paper provides comprehensive experimental results include some ablation studies to justify the design choice and demonstrate T2V2's effectiveness.

weaknesses:
* There is lack of discussions about or comparison with other multi-task models equipped with ASR & TTS capabilities such as [1] [2], etc. The discussion of related work isn't sufficient enough.
* The author states that the motivation of adopting non-autoregressive approach is the superior decoding time, however, there is lack of comparison of inference efficiency compared with its autoregressive counterparts.
* Presentation needs to be improved: for example: there is a mixed terminologies of "CTC token" "Text token" "Content token" "Acoustic token" used in Figure 1 and section 2/3, there should be a clear definition of the core terms used in the paper.



[1] Toyin, Hawau Olamide, Hao Li, and Hanan Aldarmaki. "STTATTS: Unified Speech-To-Text And Text-To-Speech Model." arXiv preprint arXiv:2410.18607 (2024).

[2] Yang, Runyan, et al. "PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models." arXiv preprint arXiv:2406.07801 (2024).

questions:
1. What's the rational of doing the token length prediction by feeding text embedding through conformer backbone? This is odd because this is only task/step that conformer has to directly deal with text embedding and there seems to be risk of confusing the model.
2. The length prediction loss is not included in the final training objective in eqs 16, how is the length prediction head trained?
3. When the paper introduces CTC error correction task, it is stated in section 2.4.1 that the purpose is to address the independence assumption in CTC loss. From the mechanism of CTC error correction, it doesn't seem like the training breaks such assumption because the CTC logits was already detached from the computational graph. The author should elaborate more how this method essentially improves the performance. And how is it different from simply masking ground truth text embedding (or randomly augmented text) and do error correction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702266331

thread_type:
official_review

<thread_object>
creator: reviewer_quvm

<content>
summary:
This paper addresses the challenge of unifying automatic speech recognition (ASR) and text-to-speech synthesis (TTS) using discrete speech tokens derived from self-supervised models. The authors introduce a CTC error correction task to enhance ASR performance by refining raw CTC outputs. Additionally, they propose a monotonic alignment search (MAS) technique with intermediate CTC outputs to achieve TTS alignment without relying on external tools. Experiments were conducted on the LibriHeavy dataset, with both objective and subjective evaluations of the synthesized speech.

strengths:
* This paper explores a challenging problem that enables the unification of ASR and TTS tasks, allowing the framework to benefit from the underlying complexities of each task.
* The claims were tested, demonstrating improvements in TTS performance as intended, while achieving comparable performance for the ASR task.

weaknesses:
It is unclear how much each task contributed to the overall improvement across tasks. Additionally, the performance of the proposed model on different attributes—such as pitch, volume, and speed of synthesized speech—has not been clearly detailed.

questions:
* (Lines 47-49): "While multitask learning offers potential for parameter sharing, the performance of individual tasks might suffer due to the inherent complexity of training multiple tasks simultaneously." — The authors should provide references to prior work to support this claim. Additionally, if multitask learning poses such challenges, how were the authors able to handle both text-to-speech and speech-to-text effectively in the proposed unified model?

* Simplifying Figure 1 would make it easier for readers to understand the proposed framework.

* (Line 64): Why was the Conformer architecture with rotary positional embeddings (RoPE) chosen? Could the authors expand on the reasoning for using RoPE? Does RoPE offer benefits for both ASR and TTS tasks?

* (Line 261): It is unclear how the authors optimized the individual losses within their multitask learning framework. Was any weighting applied to the individual losses, and how did each loss contribute to the optimization of each task?

* There is some confusion regarding the dataset subsets used to train the unified model. In lines 311-315, the authors mention a 509-hour subset of LibriHeavy, while in lines 484-485 they refer to training a larger version of the model on a 50K-hour LibriHeavy subset for discrete ASR. What is the relationship between these two training stages? In the second ASR training, is the model initialized from the first training stage, or is it a separately trained ASR model?

* In Table 8, could the authors include baselines for the discrete Zipformer-CTC model for completeness?

* How does the proposed unified model perform on other TTS attributes, such as pitch, volume, and speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732009538364

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Inference Runtimes

comment:
**Inference Runtime Comparisons**

We conducted inference runtime comparisons as follows:

**TTS**: Time taken to generate speech for a 405-character text input averaged over 100 trials on an H100 GPU

**ASR**: Time taken to recognize a 60s speech input averaged over 100 trials on an H100 GPU

**Results:**

**Inference runtimes for the text-to-content tokens stage.**
| Model                                | Runtime       |
|:-------------------------------------|:--------------|
| WhisperSpeech (AR)                        | 2.842 ± 0.007 |
 | Ours (iter=1)       | 0.013 ± 0.000 |
| Ours (iter=4)       | 0.033 ± 0.000 |
| Ours (iter=4; CFG) | 0.057 ± 0.000 |

**Inference runtimes for end-to-end text to speech waveform.**
| Model                                | Runtime        |
|:-------------------------------------|:---------------|
| StyleTTS2                            | 0.268 ± 0.003  |
| YourTTS                              | 0.110 ± 0.004  |
| HierSpeechPP                         | 0.164 ± 0.004  |
| XTTS_v2                              | 2.597 ± 0.029  |
| WhisperSpeech                        | 17.911 ± 0.042 |
| Ours (iter=1)       | 0.528 ± 0.001  |
| Ours (iter=4)       | 0.546 ± 0.000  |
| Ours (iter=4; CFG) | 0.568 ± 0.000  |

**Inference runtimes for ASR and number of correction iterations.**

| Iterations                                | Runtime        |
|:-------------------------------------|:---------------|
| w/o CORR                            | 0.32 ± 0.02  |
| 1                              | 0.40 ± 0.03  |
| 4                         | 0.39 ± 0.03  |
| 8                              | 0.42 ± 0.03  |
| 16                        | 0.47 ± 0.03 |
| 32                        | 0.60 ± 0.03 |

**Inference runtimes for ASR compared with baselines.**

| Model                                | Runtime        |
|:-------------------------------------|:---------------|
| Zipformer-transducer-small                            | 1.49 ± 0.07 |
| Zipformer-transducer-large                             | 1.51 ± 0.14  |
| Conformer-CTC-small                         | 0.32 ± 0.02  |
| Ours-small                              | 0.47 ± 0.03  |
| Conformer-CTC-large                        | 0.34 ± 0.02 |
| Ours-large                        | 0.55 ± 0.02 |

We have added these in the coressponding tables in the updated paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732009691749

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes in Revised Paper

comment:
**Summary of Changes in Revised Paper**
- Slight rewording of the claims. (Sec. 1)
- Update and simplify the figure. Separated the individual tasks to clearly show the input-output relationships without complicated interlocking arrows. Color-coded and added legend for the various inputs, outputs, and intermediate outputs. (Fig. 1)
- Add inference runtime measurements and comparisons. (Tables 4, 7, and 9)
- Include references to other AR multi-task ASR-TTS models. (Sec. 1)
- Include additional CTC correction references. (Sec. 1)
- Include a discussion on the advantages of discrete ASR. (Sec. 1)
- Include references and discussion about choosing Conformers with RoPE for ASR and TTS tasks. (Sec. 2.2)
- Include details on how MAS is applied to our framework. (Sec. 2.3.2)
- Include a discussion about the CTC correction task and the independence assumption and positioning of our CTC correction formulation's novelty compared to Mask-CTC and Improved Mask-CTC and relation to Align-Refine and Self-conditioned-CTC. (Sec. 2.4.1)
- Include references and discussion about multi-task learning challenges and optimization of losses. (Sec. 1 and Sec. 2.6)
- Update the final loss to include the length prediction objective. (Eq. 16)
- Added Table 8 to present a detailed breakdown of error types—substitutions, insertions, and deletions. This table highlights consistent improvements across all error types with our proposed correction mechanism and updated the description. (Sec. 3.5.1, Table 8)
- Include clarification about training from scratch on the large subset for the scaling experiment. (Sec. 3.5.2)
- Include a discussion about marginal performance degradation due to the SMLM task in single-pass inference, and how iterative inference with CFG (enabled by the SMLM task) significantly boosts CER for TTS and the CORR task compensates for it in ASR. (Sec. 3.4.1 and 3.5.1)
- Add an appendix to further elaborate on the similarities and differences between our CTC correction and Align-refine and Self-conditioned-CTC. (Appendix A)

**NOTE**: The changes in the paper are colored in **blue** text for easy visibility.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a non-autoregressive system for joint TTS and ASR. To enable training of both tasks, CTC alignments, discrete tokens, and various auxiliary losses are used.

I recommend an acceptance for its potential improve both ASR and TTS, especially for efficient inference.

However, I do want to remind the authors that all reviewers unanimously agreed that there is a lack of proper attribution of the improvements. In simple terms, we do not know which parts of the system are necessary and which parts of the system are good-to-have's. The ablation study in Table 8 does not answer this fundamental question.

Reviewers pcMe, D6js, and AXxb raised issues about writing. Issues include organization, vague descriptions, and imprecise use of terms, hampering reproducibility. Making significant changes to the original submission and patching things here and there during a short period of time is typically discouraged. I'd recommend the authors to carefully re-think the organization and contribution of the work.

I want to highlight one change that are not yet implemented, an issue raised by Reviewer pcMe. The use of "content tokens" and "acoustic tokens" is problematic, and is perhaps best to call them HuBERT tokens and DAC tokens to avoid ambiguity and misunderstanding. In particular, HuBERT tokens carry not just "content" (linguistic content I suppose) but many other information. See the two papers below. 

* Yeh and Tang, Estimating the Completeness of Discrete Speech Units, 2024
* Choi et al., Self-Supervised Speech Representations are More Phonetic than Semantic, 2024

Coupled with text tokens, it becomes difficult to keep track of the types (in the sense of programming language) of all the input and output, intermediate or not. It's probably best to explicit detail the set of values that a variable can be. For example, HuBERT tokens should have values in $\\{1, ..., 1024\\}$ and when they are embedded they should be vectors in $\mathbb{R}^d$.

Reviewers have provided valuable feedback, and I encourage the authors to consider them for improving the paper.