A$^2$-Flow: Alignment-Aware Pre-training for Speech Synthesis with Flow Matching

Authors: Sungwon Kim, Sang-gil Lee, Alexander H. Liu, Joao Felipe Santos, Mikyas T. Desta, Sudheer Kovela, Rafael Valle, Bryan Catanzaro
Primary area: generative models
Keywords: Generative Pre-training; Flow Matching; Alignment Learning; Discrete Speech Units

Abstract:
Recent advances in speech synthesis have enabled highly natural and speaker-adaptive speech generation by leveraging large-scale transcribed datasets. However, requiring tens of thousands of hours of annotated speech is impractical in low-resource settings. Existing pre-trained speech models often utilize masked speech inpainting for pre-training and show strong performance on various speech generation tasks using limited task-specific data. Nonetheless, these models still require external alignment mechanisms or extensive additional training to learn alignment for alignment-aware tasks, such as text-to-speech (TTS). In this paper, we propose A$^2$-Flow, an alignment-aware pre-training method for flow matching models in speech synthesis. A$^2$-Flow integrates alignment learning directly into the pre-training process using discrete speech units, enabling the model to efficiently adapt to alignment-aware tasks without the need for separate alignment mechanisms. By embedding alignment learning into pre-training, A$^2$-Flow facilitates alignment-free voice conversion (VC) and allows for faster convergence during TTS fine-tuning, even with limited transcribed data, making it highly suitable for low-resource scenarios. Experimental results show that A$^2$-Flow superior zero-shot VC performance compared to existing models and matches state-of-the-art TTS performance using only a small amount of transcribed data. Moreover, we demonstrate that A$^2$-Flow can be more efficiently applied to alignment-aware speech synthesis tasks than existing pre-training methods, providing a practical and scalable solution for high-quality speech synthesis across diverse settings.

Decision: reject
Reviewer scores: 6, 5, 8, 3

Reviews and discussion:
<thread>
creation date:
1730376791125

thread_type:
official_review

<thread_object>
creator: reviewer_ddfz

<content>
summary:
The paper introduces A2-Flow, a alignment-aware pre-training method for flow matching models in speech synthesis. This approach facilitates alignment-free voice conversion and enhances convergence speed during TTS fine-tuning, making it particularly effective in low-resource settings.

strengths:
1. Good presentation.
2. One framework can solve VC and TTS tasks.

weaknesses:
1. It is hard to understand what alignment-aware pre-training is and its advanteges.

2. It is difficult to grasp the difference between using de-duplicated HuBERT tokens and text for pre-training. In my view, the only advantage is support for voice conversion tasks. However, the idea of a single model addressing multiple audio tasks has been widely discussed in previous works, such as VoiceBox, AudioBox. Thus, it is challenging to view this as a novel contribution.

3. For the TTS task, this work still requires a sentence duration predictor to control duration. Such strategies have been widely used in diffusion-based TTS systems, including SimpleSpeech, DiTTo-TTS, and E3TTS, and so on.

I am happy to discuss this paper further with the authors and other reviewers during the discussion stage.

questions:
(1) Please explain the advanteges of alignment-aware pre-training?

(2) Can you summarize your core contributions? For example, list them as 1,2,3... From this version, I cannot understand your core contributions.

(3) VC task does not need any alignment information, because the output duration can always the same as the input audio. Your TTS model still need additional sentence duration prediction model. So, why do you propose alignment-aware pre-training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730408166414

thread_type:
official_review

<thread_object>
creator: reviewer_hyvy

<content>
summary:
The paper introduces A2-Flow, an alignment-aware pre-training method for flow matching models in speech synthesis. A2-Flow integrates alignment learning directly into the pre-training process using discrete speech units extracted from HuBERT. This approach enables the model to efficiently adapt to alignment-aware tasks such as text-to-speech (TTS) and voice conversion (VC) without the need for separate alignment mechanisms. The authors demonstrate that A2-Flow facilitates alignment-free VC and allows for faster convergence during TTS fine-tuning, even with limited transcribed data. Experimental results show that A2-Flow achieves superior zero-shot VC performance compared to existing models and matches state-of-the-art TTS performance using only a small amount of transcribed data. Additionally, the method is effective in multilingual settings, highlighting its scalability and practical applicability in low-resource scenarios.

strengths:
1. Originality: The paper presents a novel approach by integrating alignment learning directly into the pre-training phase using discrete speech units. This method addresses the limitations of existing models that require external alignment mechanisms or extensive additional training.

2. Quality: Extensive experiments support the effectiveness of A2-Flow. The model demonstrates superior performance in zero-shot voice conversion and matches state-of-the-art results in TTS with significantly less transcribed data.

3. Clarity: The paper is generally well-organized and provides detailed explanations of the methodology, including the integration of discrete speech units and the flow matching framework. The inclusion of diagrams and ablation studies aids in understanding.

4. Significance: A2-Flow offers a practical solution for high-quality speech synthesis in low-resource settings. By reducing the dependency on large amounts of transcribed data, it has substantial implications for multilingual and low-resource language applications.

weaknesses:
1. Limited Baseline Comparisons: While the paper compares A2-Flow with several existing models, a more comprehensive comparison with other state-of-the-art methods, especially in multilingual settings, would strengthen the evaluation.

2. Discussion of Limitations: The paper acknowledges some limitations, such as reliance on self-supervised speech units, but does not deeply explore potential solutions or future work to address these issues.

3. Ablation Study Depth: The ablation studies could be expanded to include more variations in hyperparameters and architectural choices to better understand their impact on performance.

4. Methodology Clarity: Certain aspects of the methodology, like the training details of the total length predictor and the timestep shifting strategy, could be explained in more detail to enhance reproducibility.

questions:
1. Total Length Predictor: Could the authors provide more details on how the total length predictor is trained and integrated into the TTS fine-tuning process? 
2. Multilingual Evaluation: Have the authors considered evaluating A2-Flow on additional low-resource languages or dialects to further demonstrate its effectiveness across diverse linguistic settings?
3. Comparison with Alignment-Free Methods: How does A2-Flow compare with other alignment-free speech synthesis methods in terms of computational efficiency and inference speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675514116

thread_type:
official_review

<thread_object>
creator: reviewer_stad

<content>
summary:
The paper proposes a new way to pre-train an alignment-aware diffusion-based speech synthesis model. Instead of pre-training the model to inpaint the masked region without any conditions, the authors propose to train the models conditioned on deduplicated discrete HuBERT tokens and force the models to learn the alignment during training. The results show that the models can perform comparably to those without pre-training, all achieved without large-scale transcribed datasets. Additionally, the model can perform voice conversion directly after pe-training with high robustness and similarity to the reference speaker.

strengths:
* **Originality**: The paper proposes a novel way to learn the alignment between phonetic representations and speech and shows that this pre-training scheme can transfer to high-quality text-to-speech synthesis with a small amount of transcribed data, while previous methods that do not incorporate alignment pre-training fails to do.

* **Quality**: The paper has compared various models and conducted extensive experiments to examine the effectiveness of its models against several baselines, including E2TTS, without pre-training in a small amount of transcribed data and multi-lingual settings. 

* **Clarity**: The paper has an excellent presentation of its methods and results, avoiding overcompcaliting with mathematical definitions, making it fairly easy to follow. 

* **Significance**: The paper has shown that it is possible to achieve comparable performance to the most recent duration-free E2TTS without a large amount of transcribed data.

weaknesses:
The major weakness is the contribution of this work seems incremental and does not demonstrate significant improvement over previous methods based on the argument presented in the paper. It is unclear what this pre-training scheme brings compared to previous methods such as SpeechFlow and E2TTS. There are a few arguments where the proposed alignment-aware pre-training can provide some advantages over previous methods. However, none of these seems to be fully supported by the results in the paper and existing literature:

1. It can be argued that this pre-training scheme allows simple transfer learning without transcription of speech data for high-quality TTS. However, most existing large-scale TTS models do not rely on human-labeled data but instead use automatically transcribed data. For example, Chen et al. 2024 [1] trained E2TTS on Emilia [2], which is a 100k-hour multilingual dataset with transcriptions obtained through the ASR model WhisperX. Chen et al. 2024 show that E2TTS trained on ASR-transcribed data can obtain a WER of 2.95, which is not significantly different from 2.2 in the paper. Similarly, NaturalSpeech 3 [3] has achieved a WER of 1.81, even lower than ground truth with ASR-transcribed data. It is unclear whether this method is better than training directly on ASR-transcribed data from a practical point of view, either in terms of quality or training resources. For example, in Figure (b), E2TTS trained on transcribed data only needs to be trained for 700k steps to have comparable performance of A2-Flow pre-trained for 700k steps plus 150k fine-tuning steps. Since E2TTS can perform well on the ASR-transcribed dataset, this suggests that A2-Flow takes more time and resources for training while achieving similar performance as E2TTS. It would be helpful for the authors to demonstrate other performance advantages of A2-flow, such as comparing it to E2TTS trained on an ASR-transcribed dataset (which can also be considered unlabeled), or comparing the total amount of time and resources required for training (i.e., to show that the entire pipeline of pre-training + fine-tuning of A2-flow is more efficient than transcribing + training for E2TTS). 

2. It can be argued that this pre-training scheme allows for transferring learning in low-resource settings where transcribed data is limited. However, this advantage is not well supported in the paper either. It is well known that fine-tuning for a different language requires much less data than training it from scratch [4,5]. It is unclear whether this method is better than training E2TTS on a transcribed English dataset (which is enormous) and then fine-tuning it on a low-resource dataset (for example, French with 280 hours of data). 

3. It can be argued that this method removes alignment labels for TTS models compared to SpeechFlow. However, it is unclear whether the alignment labels are hard to learn to begin with. Forced alignment can be obtained with a CTC-based ASR model, and it is shown in HuBERT and WavLM that CTC-based ASR models can be trained with sufficiently high quality with limited labeled data (960 hours). The contribution of this work can be further enhanced if the author shows that SpeechFlow fine-tuned on CTC-based alignments obtained from a fine-tuning HuBERT on 960 hours of data is worse compared to A2-Flow without pre-defined alignments. 

In addition, the subjective evaluations are rather weak. The authors only compared 19 samples of E2TTS and 9 samples of SpeechFlow and concluded that the model is comparable to E2TTS and better than SpeechFlow in Table 2, and no statistical significance is reported for this result. Most recent papers [1, 3] use at least 40 samples (including the E2TTS paper), so 9 samples are insufficient to establish statistical significance. Moreover, no subjective evaluation is conducted for naturalness but it is a common practice for evaluating the true performance of speech synthesis models, as most current objective evaluation metrics only correlate weakly with MOS. Since the authors claim to have reproduced E2TTS and SpeechFlow in the paper, the paper's argument can be further enhanced by comparing to more samples of E2TTS and SpeechFlow, even with reproduced models. 

**References**

[1] Chen, Y., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., ... & Chen, X. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv preprint arXiv:2410.06885.

> Note: This work was published after the ICLR deadline. However, since this work contains reproduced results from a cited paper (E2TTS), I believe this reference is relevant to this review. 

[2] He, H., Shang, Z., Wang, C., Li, X., Gu, Y., Hua, H., ... & Wu, Z. (2024). Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361.

[3] Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., ... & Zhao, S. (2024). Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100.

[4] Lux, F., Koch, J., & Vu, N. T. (2022). Low-resource multilingual and zero-shot multispeaker TTS. arXiv preprint arXiv:2210.12223.

[5] Debnath, A., Patil, S. S., Nadiger, G., & Ganesan, R. A. (2020, December). Low-resource end-to-end sanskrit tts using tacotron2, waveglow and transfer learning. In 2020 IEEE 17th India Council International Conference (INDICON) (pp. 1-5). IEEE.

questions:
1. Is the HuBERT used in the paper trained in English only? If so, how does it work to reconstruct speech in other languages? Which layer is used?

2. What is the model used to compute SECS-O? Is it the WavLM-ECAPA large fine-tuned? 

3. Why only conduct subjective evaluations on 19 and 9 samples even though you have reproduced E2TTS and FlowSpeech? Why is there no subjective evaluation of naturalness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697125781

thread_type:
official_review

<thread_object>
creator: reviewer_98gk

<content>
summary:
A2-Flow is a zero-shot TTS model based on flow matching, which relies on self-supervised speech units from HuBERT for pre-training to implicitly align speech and duration. Subsequently, A2-Flow can perform speech synthesis without the need for separate alignment mechanisms  through fine-tuning with a small amount of text-speech paired data. Due to pre-training, A2-Flow can achieve alignment-free voice conversion and allows for faster convergence during TTS fine-tuning.

strengths:
A2 Flow proposes using semantic tokens derived from unlabeled data for pre-training, aiming to align speech and text. This approach allows for zero-shot text-to-speech (TTS) synthesis using only a small amount of paired text-speech data, eliminating the need for separate alignment mechanisms. Additionally, due to the presence of pre-trained alignment, alignment-free voice conversion becomes possible.

weaknesses:
The paper is somewhat helpful for community advancement, but I believe additional experiments could further demonstrate its validity and effectiveness. For example, some experiments like MOS, CMOS, and SMOS could be added to prove its effectiveness from a subjective perspective.

questions:
- Can you explain how your overall speech duration predictor is designed, and whether it shows good robustness for shorter and longer texts? Additionally, when dealing with longer or shorter texts, is the alignment between your speech and text still stable?
- As iterations increase, will your WER and SECs-O metrics maintain a good level? Will there be a tendency to forget the alignment mechanism, or will they align with the E2TTS level? In this setting, will the improvements from pretraining still be superior to models like E2TTS?
- I've noticed that after feature extraction with HuBERT, there is a de-duplication step. Is this step intended to further align the text and semantic tokens? Would removing this step have a significant impact on the results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732692951219

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment

comment:
We sincerely thank all reviewers for their constructive feedback and positive evaluation of our work. Based on your suggestions, we have conducted additional experiments and provided detailed clarifications to strengthen our paper. Below, we summarize our updates and responses to the reviewers' comments:

Additional Experimental Results:

* We have included the results of additional experiments requested by the reviewers in Tables 6, 7, and 8 (Appendix). Specifically:
To evaluate speech naturalness, we used UTMOS, a model trained to predict MOS as a proxy for naturalness, and reported the average UTMOS score for the entire evaluation set (over 1000 test samples). This metric is now included alongside existing evaluation metrics in Tables 6, 7, and 8, providing a robust and reproducible measure of speech naturalness.
* Table 7 addresses Reviewer staD’s concern that our results might stem from additional fine-tuning iterations compared to E2TTS. To clarify, we provide a comparison of E2TTS and A$^2$-Flow across various training iterations, including a configuration where A$^2$-Flow is fine-tuned with the same amount of transcribed data (40K hours) as E2TTS, referred to as A$^2$-Flow-T. The results demonstrate that A$^2$-Flow-T achieves superior SECS-O scores compared to both the original A$^2$-Flow-Flow and E2TTS.
* Table 8 presents an ablation study to assess the impact of different discrete units used during pre-training. The results show that our pre-training method remains effective when fine-tuned on LibriTTS, regardless of the unit representation, highlighting the robustness of our approach.

Detailed Methodological Clarifications:

* Total Length Predictor: In response to comments from Reviewer 98gk and Reviewer HYvY
, we have added a detailed explanation of the Total Length Predictor’s structure and specifications in Section A.2.1 of the Appendix.
* Timestep Shifting: As suggested by Reviewer HYvY
, we have expanded our explanation of timestep shifting in Section A.2.2, supported by Figure 4 and Algorithm 1 for clarity.
* Human Evaluation: At the suggestion of Reviewer staD, we have provided additional details about our human evaluation methodology, including compensation, evaluator selection, and evaluation criteria, in Appendix A.5.

Core Contributions: To reiterate the core contributions of our work:

* Background: E2TTS, which learns text-speech alignment without a phoneme duration predictor, outperforms most zero-shot TTS models. However, E2TTS requires significant computational resources, large amounts of transcribed data, and extensive training iterations, making them challenging in resource-constrained settings.
* Proposed Approach: We propose an alignment-aware pre-training method that effectively initializes TTS models within the E2TTS framework to model internal text-speech alignment without requiring a duration predictor. 

Our contributions are as follows:
* We demonstrate that using de-duplicated unit sequences instead of text during pre-training enables efficient alignment learning and adaptation to limited paired <text, speech> data during fine-tuning.
* We show that our pre-training method enables high-quality zero-shot voice conversion without requiring additional fine-tuning.
We highlight the language-agnostic nature of our method, demonstrating its applicability across multiple languages.
* Compared to state-of-the-art zero-shot TTS models, our approach achieves superior zero-shot TTS performance using just 1% of the transcribed data. Additionally, it is the first pre-training approach to achieve such results while learning internal alignment without relying on a phoneme duration predictor.

We appreciate the reviewers' valuable feedback, which has helped us improve the clarity and rigor of our work. Thank you for your time and effort in reviewing our submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306463507

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Providing subjective evaluation results (CMOS, SMOS)

comment:
The discussion period is nearly over, but we would like to provide the subjective evaluation requested during the rebuttal process and hope that reviewers can refer to it in their final evaluation.

In our initial submission, we compared our model with all 9 samples from the SpeechFlow demo page and 19 samples from the E2TTS demo page for direct comparison with the benchmark models. In situations where subjective evaluation with models that are not open-sourced is limited, we followed the same approach as existing published papers, comparing our results with the samples available on the demo pages to ensure a direct comparison with the official models rather than reproduced ones. (We would like to reiterate that our subjective evaluation compared to the state-of-the-art E2TTS paper is not significantly lacking, considering that NaturalSpeech 3 uses 20 samples for CMOS and 10 samples for SMOS.)  Furthermore, as a proxy for the subjective evaluation of speech naturalness, we provided UTMOS results for all samples in the test set, comprising over 1,000 samples, during the rebuttal process. 

Furthermore, to strengthen the subjective evaluation results of the paper, we have added additional subjective evaluation in the official comment. These subjective evaluations compare our model with reproduced baselines following **Reviewer staD**’s requests. The baselines are as follows:  

1. **E2TTS (SpeechFlow pre-training)**: SpeechFlow pre-trained with 40K hours of unlabeled data and fine-tuned on 500 hours of LibriTTS data using the E2TTS method. This corresponds to "SpeechFlow-E2" in Table 1 of the original paper.  
2. **E2TTS (Baseline)**: The reproduced E2TTS model trained with 40K hours of data over 700K steps.  

For CMOS and SMOS subjective evaluations, we used Amazon Mechanical Turk as an additional platform. 50 random samples were selected from the LibriSpeech test-clean dataset used for the paper's evaluation, and these were used for comparative evaluation.  

For the CMOS evaluation, 100 raters were asked to evaluate three random samples out of the 50, comparing two models on a scale of -3 to 3. The instruction given was: "**How do the two audio samples compare in terms of pronunciation accuracy and speech quality?**" Transcripts were provided, and the samples from the two models were presented in a random order for each evaluation.  

For the SMOS evaluation, 150 raters were asked to evaluate three random samples. A reference audio was provided along with the generated sample, and raters scored the similarity on a scale of 1 to 5 based on the instruction: "**How similar are the given audio samples to the provided reference speech in terms of prosody and the speaker's voice characteristics?**" Transcripts were not provided during SMOS evaluations.  

In the CMOS evaluation, positive scores indicate that E2TTS (A²-Flow pre-training) outperforms the baseline, while negative scores indicate inferior performance. For SMOS, higher scores indicate greater speaker similarity.  

### Results: Subjective and Objective metrics on LibriSpeech test-clean dataset

| Model                            | Unlabeled | Labeled | **CMOS**               | **SMOS**  | WER | SECS-O|
|----------------------------------|--|--|-----|--------------------|--------------------|-----|
| E2TTS (A²-Flow pre-training)     |40000|500  | **0.0**                | **3.84 ± 0.08**        | 2.2 | 0.704 |
| E2TTS (SpeechFlow pre-training)  |40000|500     | **-1.31 ± 0.19**      |**3.70 ± 0.08**       | 107.7 | 0.690
| E2TTS                            |0|40000     |**-0.19 ± 0.16**    | **3.68 ± 0.08**        | 1.9 | 0.696|

According to the results, as shown in Table 2, the **A²-Flow method demonstrated higher performance in SMOS, similar to the observed difference in SECS-O compared to the SpeechFlow pre-training method and E2TTS trained from scratch.** While E2TTS (SpeechFlow pre-training) achieved a high SECS-O score due to its strong speaker similarity to the reference in objective metrics, its SMOS score was comparable to that of E2TTS. However, it failed to learn alignment, resulting in a WER exceeding 100. Similarly, **its CMOS score showed a significant deficiency compared to the A²-Flow pre-training method.** Compared to E2TTS, the **A²-Flow pre-training method achieved a slightly higher CMOS score**, further reinforcing the validity of A²-Flow as a meaningful pre-training approach based on this subjective evaluation.

We would be truly grateful if these results and the responses provided to each reviewer are taken into account during the final evaluation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
A2 Flow introduces an approach by using semantic tokens derived from unlabeled data for pre-training, aligning speech and text. This enables zero-shot text-to-speech (TTS) synthesis with minimal paired text-speech data and allows for alignment-free voice conversion. The integration of alignment learning directly into the pre-training phase using discrete speech units addresses the limitations of existing models that require external alignment mechanisms or extensive additional training. The paper is well-organized, providing detailed explanations of the methodology, including the integration of discrete speech units and the flow matching framework. Diagrams and ablation studies enhance understanding.

However, the work appears incremental and does not show significant improvement over previous methods like SpeechFlow and E2TTS. A more comprehensive comparison with other state-of-the-art methods, especially in multilingual settings, would strengthen the evaluation. The paper acknowledges some limitations, such as reliance on self-supervised speech units, but does not deeply explore potential solutions or future work to address these issues. The ablation studies could be expanded to include more variations in hyperparameters and architectural choices to better understand their impact on performance.