Unlocking Speech Instruction Data Potential with Query Rewriting

Authors: Yonghua Hei, Xuming Hu
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal language model; Large Speech Language Model; Datasets

Abstract:
End-to-end Large Speech Language Models (**LSLMs**) demonstrate strong potential in response latency and speech comprehension capabilities, showcasing general intelligence across speech understanding tasks. However, the ability to follow speech instructions has not been fully realized due to the lack of datasets and heavily biased training tasks. Leveraging the rich ASR datasets, previous approaches have used Large Language Models (**LLMs**) to continue the linguistic information of speech to construct speech instruction datasets. Yet, due to the gap between LLM-generated results and real human responses, the continuation methods further amplify these shortcomings. Given the high costs of collecting and annotating speech instruction datasets by humans, using speech synthesis to construct large-scale speech instruction datasets has become a balanced and robust alternative. Although modern Text-To-Speech (**TTS**) models have achieved near-human-level synthesis quality, it is challenging to appropriately convert out-of-distribution text instruction to speech due to the limitations of the training data distribution in TTS models. To address this issue, we propose a query rewriting framework with multi-LLM knowledge fusion, employing multiple agents to annotate and validate the synthesized speech, making it possible to construct high-quality speech instruction datasets without relying on human annotation. Experiments show that this method can transform text instructions into distributions more suitable for TTS models for speech synthesis through zero-shot rewriting, increasing data usability from 72\% to 93\%.  It also demonstrates unique advantages in rewriting tasks that require complex knowledge and context-related abilities.

Decision: reject
Reviewer scores: 6, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730274157964

thread_type:
official_review

<thread_object>
creator: reviewer_pew1

<content>
summary:
This paper addresses text normalization in Text-to-Speech (TTS) through Large Language Model (LLM) utilization, aiming to better capture contextual information. By leveraging multi-LLM query rewriting and multi-agent annotation, the authors propose an innovative data generation framework that enables end-to-end speech language models (LSLMs) to better follow spoken instructions. This approach aligns synthetic speech and textual information with TTS models, achieving improvements in data usability and quality while minimizing reliance on human annotation. However, the method remains primarily a data augmentation strategy limited to specific TTS model, and lacking comparision with typical text normalization method for TTS.

strengths:
The paper tackles text normalization for TTS using a text-based LLM framework, potentially improving contextual understanding and data synthesis efficiency for LSLM training.

weaknesses:
- Limited Scope: The focus is primarily on input text rewriting for TTS, which is essential for text normalization but less relevant to speech language models (SLMs) training. The framework generates instruction data but does not advance model training methods.

- Limited Technical Contribution: This work is mainly a data augmentation strategy, using TTS for producing instruction data in a speech-question/text-response format without introducing new training algorithms for LSLMs.

- Lack of Baseline Comparison: Text normalization in TTS has a rich history, especially in industrial applications. Prior studies, such as [1] and [2], propose alternative normalization techniques. The only rule-based baseline provided here involves basic numeric conversion, which oversimplifies comparison and limits validation of this method against established techniques.

- Insufficient Details on Model Training and Evaluation: The results section ("Use synthetic data to fine-tune LSLMs") lacks clarity regarding fine-tuning methods. For example, Table 4 lists "Qwen2-Audio-7B-Instruct" as the model, but it remains unclear if it was continually fine-tuned. Additionally, input-output specifics for this model are not provided.

- Dependency on Specific TTS Model: The study employs only Parler TTS for text synthesis. It’s uncertain how the framework would perform if a phoneme-based TTS model were used, particularly regarding out-of-vocabulary issues.

- ASR Model Dependency: The method’s efficacy in cases where ASR systems fail to recognize rare words is uncertain, raising questions about the proposed solution's handling of out-of-vocabulary cases


References
1. Ebden, Peter, and Richard Sproat. "The Kestrel TTS text normalization system." Natural Language Engineering, 21.3 (2015): 333-353.
2. Zhang, Hao, et al. "Neural models of text normalization for speech applications." Computational Linguistics, 45.2 (2019): 293-337.

questions:
Some suggestions are listed below: 

- Figure 1: Highlight differences using color or bold font for better visibility.

- Figure 2: Text readability is hindered by font size; should consider increasing.

- Table 8: The caption appears incorrect and should be reviewed for accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551622552

thread_type:
official_review

<thread_object>
creator: reviewer_fvbn

<content>
summary:
The paper addresses the enhancement of instruction-following capabilities in speech language models through the creation of an improved synthetic speech instruction dataset. To achieve this, the authors propose a method to rewrite text instructions, aligning them more closely with the typical training distribution of TTS models. This rewriting framework utilizes multiple LLMs and incorporates annotation and validation of synthesized speech via a combination of ASR and embedding methods. Experimental results on several text-based QA datasets demonstrate that this approach enhances the usability of synthetic data and improves the preservation of linguistic information within synthesized speech.

strengths:
1. Generating high-quality synthetic speech instruction data is a valuable task for the community, yet it remains relatively underexplored.
2. The paper introduces a versatile framework that integrates existing LLMs, ASR, and text embedding methods in a plug-and-play manner, requiring no additional training aside from the knowledge fusion component.
3. Results from automatic metrics indicate that the proposed framework enhances the quality and usability of synthetic data.

weaknesses:
Quality

1. The paper begins by highlighting the gap between LLM-generated responses and human responses, but it’s unclear if their framework effectively addresses this. Human responses often include disfluencies or may involve mid-sentence question reformulations. A more rigorous human-in-the-loop evaluation would be beneficial to assess if fine-tuning on their synthetic speech data genuinely enhances the speech language model's ability to follow "spoken" human instructions.
2. The quality of synthetic data is primarily assessed using automated metrics. While I recognize the potential cost, a relevance judgment by human evaluators could provide valuable insights into whether the synthetic data aligns with how humans would expect to interact with a speech model. Given that the core objective is to improve the model's ability to follow human instructions, human-in-the-loop evaluation is essential.

Clarity

1. The paper could benefit from improved readability, particularly with the equations. Including backpointers to the sections where variables are first introduced would aid in following the mathematical formulation.
2. There are typos, such as "finitune" on page 8. A thorough proofreading is recommended to address these grammatical issues.
3. It’s unclear whether all embedding methods in Table 5 are used, or if only a subset based on Table 6 is applied.
4. The reference to "71% to 93%" lacks context. Clarifying what this range represents would improve comprehension.

questions:
Refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698605378

thread_type:
official_review

<thread_object>
creator: reviewer_2kw3

<content>
summary:
This paper presents a query rewriting framework for rewriting text queries such that TTS models are able to easily process and synthesize. To accomplish this, several LLMs are prompted to rewrite the input query such that things like abbreviations and formulas would be written in a way that a TTS model can understand. Multiple ASR models are used to filter the TTS outputs and match them against the original input to make sure that they are aligned. In order to increase the diversity of the speakers, an LLM is prompted to generate speaker descriptions which are passed to the TTS model to control the speaker identity and style. Knowledge Fusion is used to improve the model's performance on rewriting challenging queries. Experiments show that their technique improves the linguistic alignment between the text and generated speech while

strengths:
Paper presents a detailed infrastructure for generation and filtering synthetic datasets for speech instruction datasets. Results show consistent improvement over naive TTS generation. Experimental results show across multiple datasets that the generated speech is higher quality across WER, SIM and PASS and that their technique improves downstream performance on NarrativeQA.

weaknesses:
One of the main issues with this paper is the Main Result in Table 4 and Section 5.3. Authors only show that performance improves for exactly 1 model and on exactly 1 dataset. This is not sufficient evidence to claim that this technique extrapolates to other downstream tasks. This is very strange, since the authors clearly had all of these other datasets available that they could have used for evaluation. 
This table is also not very well explained; there is no experimental setup section about how this experiment was performed. 

Parts of this manuscript are poorly written and explained. Lots of typos and grammatical errors. (See questions)
In the second part of Section 5.2, the authors say they use ROUGE-L to "evaluate the quality of the generated results", however, it doesn't seem like this metric is used anywhere. 

The implementation details in section 5.1 on page 7 also state that top-p for generation is 10 (I am assuming 10%), which is extremely low, this might as well be greedy sampling.

Furthermore the use of Knowledge Fusion is not very well explained. Section 4.3 states that they use Meta-Llama-3-8B-Instruct on line 266/267 as the backbone model to train "the model"? In Figure 3, it seems like they just train a Llama model, how does this integrate with the existing rewriting models in the results sections, where something like "Phi3+Llama3+Qwen2+KF" is written? How do these LLMs integrate with each other in the pipeline?

Furthermore, the abstract, introduction and conclusion all repeat this statement of improving data usability from "71% to 93%" (the conclusion actually says 91%)  but it is unclear where this number actually comes from. 

Ablations are incomplete. In particular, it is unclear whether this technique improves over just having the LLMs rewrite the queries and using raw TTS after that.

questions:
On line 053, you say that these training methods "treat the speech modality as files rather than instructions", what does this mean?
Furthermore, you state that the integration of "traditional tasks in the speech domain (such as speaker classification, speech entity recognition, etc.)" led to the trained models to "lack the ability to follow speech instructions and lead to hallucinations due to the severe bias in task distribution." Where is any reference for this phenomenon? 

In Figure 1, it is stated that these are depictions of several recognition errors in ASR models. However, the second error: "Sentence Pattern Error" is strange, it doesn't seem like there is any error other than capitalization? Is the error that the bot didn't put the "?" at the end? But semantically, the text is a question, not a sentence, so I am not sure what the error here is.

Typos and grammatical mistakes throughout the manuscript:
* Extra period in Equation (1) on line 147, it should not be there since line 149 is the continuation of the sentence afterwards. Also an extra period in Equation (2), (3), (4)
* Line 161, "oringal" instead of "original"
* Incorrectly capitalized "We" near the end of line 192 in section 4.1
* The sentence at the end of line 199 in Section 4.1 is grammatically incorrect: "And the quality for $s_o$ could get by" should be something like "The quality of $s_o$ is given by" or something like that?
* The sentence spanning lines 203 and 204: "In the case of using the same similarity calculation method, since the performance of this evaluation approach depends on the orthogonality of the ASR models’ performance." is grammatically incomplete, it should probably say something like "This metric requires that the ASR models' performance be uncorrelated with each other so that we do not see consistent ASR errors on the same input."
* Missing spaces around the period on line 212/213 in section 4.2
* Missing space after "Section 4.2." on line 261 in Section 4.3
* Missing space after the comma on line 283 in Section 5.1
* Caption for Table 1 should state what Num is instead of just stating the names of the columns.
* Missing space on line 308 between "Table 2" and "provides"
* On line 352, "ROUGEL" should be "ROUGE-L"
* On line 406, "fintinue" should be "finetune"
* Caption for Table 8 is just wrong, it isn't "Examples of speech style descriptions generated by GPT-4", it seems like someone copy and pasted the caption for Table 2 and didn't edit it afterwards.
* In Section E.1, License is spelled "Lince" many times
* Extra period on line 335/336 "alignment.," 
* Missing spaces "thenear" on line 545 in the Reproducibility statement and "thetransparency" on line 555/556 in the Ethics and Ethics Statement. Also, the title of the Ethics statement is "Ethics and Ethics Statement", shouldn't this just be "Ethics statement"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714553800

thread_type:
official_review

<thread_object>
creator: reviewer_tug5

<content>
summary:
This paper presents a novel framework for constructing high-quality speech instruction datasets using query rewriting with multi-LLM knowledge fusion. The key contributions are:

- A query rewriting framework that leverages multiple LLMs to rewrite text instructions to better match TTS model training distributions while preserving semantics.
- A multi-agent annotation and validation approach using multiple ASR models and embedding-based similarity metrics to ensure quality.
- A knowledge fusion method that combines strengths of different LLMs to handle challenging rewriting cases.

strengths:
- Novel and practical solution to an important problem in speech instruction dataset creation.
- Comprehensive evaluation across multiple datasets and metrics.
- Detailed ablation studies validating each component.
- Cost-effective compared to human annotation.
- Strong experimental results showing clear improvements in both data quality and downstream task performance.
- Good technical novelty in combining multiple LLMs and agents for robust performance.

weaknesses:
- Limited analysis of failure cases and error patterns.
- No direct comparison with other query rewriting methods from adjacent domains.
- Validation relies heavily on embedding similarity - could benefit from human evaluation.
- Parameter sensitivity analysis missing (e.g., impact of different thresholds).
- Scalability and computational costs not thoroughly discussed.

questions:
/
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a novel framework for constructing high-quality speech instruction datasets, focusing on query rewriting to enhance the ability of TTS models (Text-to-Speech models) to process and synthesize text instructions. The key contributions include:
- Query Rewriting with Multi-LLM Knowledge Fusion: The paper introduces a query rewriting framework that leverages multiple LLMs to rewrite text instructions, aligning them with typical TTS model training distributions while preserving semantics. This enhances the linguistic alignment between text and generated speech. The framework employs multi-agent annotation using multiple ASR models and embedding-based similarity metrics to validate and ensure high-quality, well-aligned speech output. A knowledge fusion method is used to combine the strengths of various LLMs, addressing challenging rewriting cases and improving the overall model’s ability to handle diverse queries and contexts.
- Efficacy of proposed method is showed empirically: the paper demonstrates that the proposed framework improves data usability and the quality of synthetic speech. This is achieved while minimizing the need for human annotation, making the approach cost-effective and scalable. Experimental results also show that the approach enhances linguistic information preservation and the overall quality of the generated synthetic speech across multiple datasets. The alignment between synthetic speech and text instructions also helps speech language models (LSLMs) follow spoken instructions more effectively.
- To enhance speaker diversity, the method uses an LLM to generate speaker descriptions that control the speaker identity and style in the generated speech, contributing to more natural and varied speech outputs.


Strength of this paper
- cost efficiency and practicality: The paper introduces a cost efficient (compared to human annotation), practical solution to a significant problem in speech instruction dataset creation at scale. It presents a versatile framework that integrates existing LLMs, ASR, and text embedding methods in a plug-and-play manner, without requiring additional training aside from the knowledge fusion component.
- Empirical evidence to support the effectiveness of proposed method: the authors provided experiment results showing improvements in data quality (e.g., WER, SIM, PASS) and downstream task performance (e.g., on NarrativeQA)


Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.
- Insufficient validation and experimental scope: The paper primarily relies on embedding similarity metrics, which could be complemented with human evaluations to better assess the quality of the results. The main result is limited to a single model and dataset, which is insufficient to generalize the approach to other downstream tasks. There is a lack of clarity on experimental setup and additional datasets that were available but not used. Besides, ablation studies are incomplete, particularly regarding comparisons with simpler alternatives (e.g., using raw TTS outputs without the rewriting process, or a wide range of query rewriting methods from adjacent domains); the study is limited to text normalization in TTS, focusing on input text rewriting rather than advancements in speech language models (SLMs) training or model evaluation.
- Marginal Novel: the method is marginal novel, as many components (e.g., TTS for audio/speech data synthesis, ensembling multiple LLMs and agents) are common practices. This work combines these pieces to carefully generate a dataset with good quality. One possibility to make the contribution more novel and generalizable, is to provide thorough analysis or discussion on the success and failure cases, help to readers better understand the method's limitations and future applicability. 
- Scalability and dependency concerns: Scalability and the computational costs of the proposed method are not thoroughly discussed. This work is dependent on specific TTS models and ASR models, which raises questions about the approach’s performance with other models.