DNASpeech: A Contextualized and Situated Text-to-Speech Dataset with Dialogues, Narratives and Actions

Authors: Chuanqi Cheng, Hongda Sun, Xinrong Hu, Rui Yan
Primary area: datasets and benchmarks
Keywords: Text-to-Speech, Voice Generation, Prompt

Abstract:
In this paper, we propose contextualized and situated text-to-speech (CS-TTS), a novel TTS task to promote more accurate and customized speech generation using prompts with Dialogues, Narratives, and Actions (DNA). While prompt-based TTS methods facilitate controllable speech generation, existing TTS datasets lack situated descriptive prompts aligned with speech data. To address this data scarcity, we develop an automatic annotation pipeline enabling multifaceted alignment among speech clips, content text, and their respective descriptions. Based on this pipeline, we present DNASpeech, a novel CS-TTS dataset with high-quality speeches with DNA prompt annotations. DNASpeech contains 2,395 distinct characters, 4,452 scenes, and 22,975 dialogue utterances, along with over 18 hours of high-quality speech recordings. To accommodate more specific task scenarios, we establish a leaderboard featuring two new subtasks for evaluation: CS-TTS with narratives and CS-TTS with dialogues. We also design an intuitive baseline model for comparison with existing state-of-the-art TTS methods on our leaderboard. Comprehensive experimental results demonstrate the quality and effectiveness of \dataname, validating its potential to drive advancements in the TTS field.

Decision: reject
Reviewer scores: 5, 6, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729563912600

thread_type:
official_review

<thread_object>
creator: reviewer_rjhl

<content>
summary:
This work introduces a new TTS task called **Contextualized and Situated Text-To-Speech** (CS-TTS), which incorporates contextual descriptions into speech generation, aiming to enable TTS models to produce more expressive speech. As the lacking of CS-TTS datasets, they created a new CS-TTS dataset called **DNASpeech**. Each speech sample in DNASpeech is accompanied by three types of contextualized prompts: **Dialogues** provide conversational context, **Narratives** describe the environment surrounding the speaker, and **Actions** detail the speaker’s actions and expressions. For dataset construction, they developed an automated annotation pipeline, with human evaluations to validate the approach.

Additionally, they established a leaderboard to assess the performance of current TTS systems, and introduced a baseline model adapted to the CS-TTS.

strengths:
1. This work introduces a new dataset called DNASpeech, specifically created for the innovative CS-TTS task.
2. An automatic annotation pipeline is presented, utilizing techniques likes OCR, speech denoising, ASR to ensure the quality of produced dataset. Additionally, human evaluation is conducted to confirm the effectiveness of the pipeline.

weaknesses:
## Weaknesses 

1. The paper asserts that contextualized descriptions lead to more accurate and expressive speech generation. However, there is only one experiment validating the effectiveness of CS-TTS, and it shows no significant improvement when using contextualized descriptions. For example, in evaluating the alignment between speech and environmental information, the MOS-E score gap between prompt-based TTS methods and non-prompted TTS is less than 0.1. StyleTTS, the best non-prompted TTS model, performs comparably to the prompt-based models. This makes it difficult to confirm the quality of the proposed dataset and the effectiveness of CS-TTS.
2. The work introduces a TTS dataset where each sample includes three types of contextual prompts : **Dialogue**, **Narrative**, and **Action**. It is claimed that **Action** describes the speaker's actions and expressions, while **Narrative** provides environmental context, as mentioned in lines 74-76. However, based on the descriptions and examples given in the paper, these categories are difficult to differentiate. Take Figure 1 for an example, the Action "showing the gun to JURORS" corresponds to the Narrative "He picks up a revolver...", which is also an action. There is also confusion as to why the speaker's emotions are categorized under Action. Furthermore, in lines 142-144, the authors state that MEAD-TTS highlights environmental information (MEAD-TTS seems to focus on Action since it uses templates like "A <gender> says with a <emotion level> <emotion> tone" to write fine-grained prompts), yet in Table 1, MEAD-TTS's prompt is categorized under Actions, not Narratives, which contradicts the definitions provided in lines 74-76. Additionally, it seems like DailyTalk's focus is more on Dialogues than Narratives, given that DailyTalk focuses on chat history.
3. One of the core contributions of the work is the automatic annotation pipeline for building the CS-TTS dataset from movie scripts. However, the description of this pipeline is unclear, making it difficult to understand how different type of prompts are extracted from movie scripts.

## Suggestions 

1.  It would be better to use $\cite$ or other citation formats instead of $\citet$ when the cited paper is not the subject or object in a sentence. It becomes harder to read when citations are embedded in the main text. For example, in lines 32-34, it could be:
> Text-to-speech (TTS) aims to convert input text into human-like speech, attracting significant attention in the audio and speech processing community (Shen et al. 2018; Ren et al. 2020; Shen et al. 2023; Ju et al. 2024).
2. Many TTS-related datasets are mentioned in the related work section. However, the descriptions seem to be directly copied from the original papers, resulting in inconsistent types of information for each dataset. The writing does not highlight the core differences between previous work and this paper. It would be better to reorganize Section 2.1.
3. To improve the integrity of the experiments, it would be helpful to explain the source of the human evaluators and what the interface or  instructions shown to the evaluators.
4. When citing papers published at conferences, it is better to reference the conference version rather than the arXiv version. For instance, FastSpeech2 was published at ICLR 2021, but this paper cites its arXiv version.

## Typos

1. In Table 1, "MM-TTS" should be "MEAD-TTS", as the former refers to the TTS system and the latter is the name of the dataset.
2. A period is missing at the end of line 361.

questions:
1. Does **Dialogue** specifically refer to the immediate sentence before the user speaks, or is it a summary of the entire chat history?
2. Will all three types of contextual prompts be provided to the proposed baseline model at once, or only one at a time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730471860272

thread_type:
official_review

<thread_object>
creator: reviewer_zxkp

<content>
summary:
This paper introduces a novel text-to-speech (TTS) dataset, DNASpeech, which is designed to support contextualized and situated TTS (CS-TTS) tasks. The dataset includes high-quality speech recordings annotated with Dialogues, Narratives, and Actions (DNA) prompts, aiming to enhance the accuracy and expressiveness of speech synthesis. Experimental results are provided to validate the quality and effectiveness of DNASpeech.

strengths:
1. This paper proposes the "DNASpeech" dataset. It addresses a significant gap in existing TTS datasets by providing a rich dataset with contextualized and situated prompts, which is crucial for advancing TTS research.

2. The paper describes the comprehensive automatic annotation pipeline that aligns speech clips with detailed dialogue, narrative, and action descriptions, which is a complex and valuable contribution.

weaknesses:
1. In Sec. 3.2, the authors individually apply information extraction for both speech and scripts in the movie in step 2. Then in step 3, they attempt to align them in two stages.
* 1.1 Why "more than 800 million potential matches are required"? Since you can align the movie and script by movie titles or other meta information. And for the "DNA" prompt, why did the authors choose to extract them from the scripts with such a heuristic algorithm？ What is the accuracy of the alignment? Other methods such as extracting speech attributes directly are not considered.
* 1.2 "Following the script writing paradigm, we extract four key elements from each movie script: Dialogues Narratives, Actions, and Characters." How do you extract them? Please illustrate it in detail.
* 1.3 All data is from the movie. So there is a risk of domain bias. Because the movie can not cover all diverse accents, languages, or speaking styles.

2. This paper is an extension of textual-prompt-based text-to-speech synthesis. The authors propose to extend the descriptive prompt of speech to three dimensions: 1) dialogue, 2) narrative, and 3) action. However, it is only an incremental work of the existing prompt-tts paradigm by extending the annotation pipeline. So it lacks novelty.

3. In the experiments:
* 3.1 It is better to categorize into three types: 1) None-Prompt TTS, 2) natural language description prompt-based TTS, and 3) speech prompt-based TTS.
* 3.2 The method for CS-TTS in Sec. 4.1.2 is not clear. What is for "but includes classification tasks for emotion, pitch, energy, and speed during training"? For emotion, how do you obtain the label? Furthermore, it is not clear how the author leverages the "DNA" prompt as a condition to guide the generation process.
* 3.3 It lacks an ablation study for the attribution controllability for the proposed "DNA" attributes.

4. The obtained dataset contains about 18 hours including 2395 distinct characters, indicating that only 0.45 minutes for a single character. It is small to train a good TTS system.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639533217

thread_type:
official_review

<thread_object>
creator: reviewer_n5vg

<content>
summary:
This paper introduces DNASpeech, a novel contextualized and situated text-to-speech (CS-TTS) dataset that incorporates comprehensive descriptive prompts aligned with speech data. The dataset contains "DNA" prompts - Dialogues (conversational context), Narratives (environmental scenes), and Actions (speaker's expressions/actions) - along with high-quality speech recordings. DNASpeech includes 2,395 distinct characters, 4,452 scenes, and 22,975 dialogue utterances totaling over 18 hours of speech. The authors developed an automated annotation pipeline for aligning speech clips, content text, and descriptions. They also established a leaderboard with two evaluation subtasks: CS-TTS with narratives and CS-TTS with dialogues. The paper proposes a baseline model and demonstrates DNASpeech's effectiveness through extensive experiments comparing it with existing TTS methods.

strengths:
1. The paper introduces a novel and valuable contribution to TTS research through DNASpeech, a contextualized and situated text-to-speech dataset that incorporates comprehensive "DNA" (Dialogues, Narratives, Actions) prompts.

2. The authors develop an innovative automatic annotation pipeline that enables efficient multifaceted alignment among speech clips, content text, and corresponding descriptions, making the dataset construction process systematic and reproducible.

3. The dataset is substantial and diverse, containing 2,395 distinct characters, 4,452 scenes, and 22,975 dialogue utterances with over 18 hours of high-quality speech recordings, providing rich resources for TTS research.

4. The paper establishes a clear evaluation framework through a leaderboard featuring two specific subtasks (CS-TTS with narratives and CS-TTS with dialogues) and provides an intuitive baseline model for comparison.

weaknesses:
1. Although the authors compared the dataset and two models on the other datasets, the dataset's reliance on movie scenes rather than real-world scenarios might limit its applicability to authentic speech patterns and natural conversations.
2. The experimental evaluation metrics are somewhat limited, primarily focusing on MOS scores. Additional objective metrics could provide more comprehensive performance assessment such as spectral distortion or character error rates.
3. The paper lacks detailed analysis of the baseline model's architecture choices and their impact on performance.
4. The comparison with existing methods could be more extensive, particularly in analyzing how different types of prompts affect the speech generation quality. 
5. Similarly, for the public dataset comparison, the author did not select the SOTA models for the comparison. It would be great to see the comparison against them.

questions:
Have you guys conducted ablation studies showing the individual contributions of Dialogues, Narratives, and Actions to the overall speech quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655070679

thread_type:
official_review

<thread_object>
creator: reviewer_swyl

<content>
summary:
This paper introduces DNASpeech, a novel contextualized and situated text-to-speech (CS-TTS) dataset designed to enhance TTS performance by incorporating prompts from dialogues, narratives, and actions (DNA). DNASpeech provides rich multimodal prompts aligned with speech clips, filling a gap in current datasets that lack comprehensive contextual information for TTS tasks. The dataset includes 2,395 distinct characters, 4,452 scenes, and 22,975 dialogue utterances, along with over 18 hours of high-quality speech recordings. To validate DNASpeech, the authors propose a leaderboard and evaluation benchmarks featuring two subtasks: CS-TTS with narratives and CS-TTS with dialogues. The experimental results demonstrate the potential of DNASpeech to drive advancements in controllable and expressive TTS systems.

strengths:
The integration of dialogues, narratives, and actions (DNA) as contextual prompts is an innovative addition to existing TTS datasets, providing richer and more varied situational context. The dataset is constructed using a detailed and well-validated annotation pipeline, with emphasis on quality control through denoising, ASR verification, and manual assessment. The alignment method that combines both coarse-grained and fine-grained techniques is robust and well-implemented. The dataset addresses a crucial need for more context-aware TTS systems and offers a structured evaluation through the established leaderboard. This contribution is likely to inspire further research in controllable TTS using diverse prompts. Additionally, the paper provides a thorough explanation of the dataset construction and the challenges involved, with the use of visual aids to depict the pipeline and dataset characteristics aiding in comprehension.

weaknesses:
The experiments primarily focus on validating the dataset using specific subtasks (narratives and dialogues), but they could benefit from broader model diversity and more diverse metrics beyond MOS evaluations. Including results from a larger variety of baseline models would make the evaluation more comprehensive. While the paper claims that DNASpeech can generalize well for different TTS tasks, the experimental evidence supporting this claim is limited, and testing with a wider set of models and comparing performance on tasks beyond CS-TTS (e.g., emotional TTS) would strengthen this assertion. Additionally, the dataset's reliance on movie scripts might limit its applicability for general conversational TTS, as the movie-based context might not fully represent day-to-day conversational dynamics.

questions:
Can the authors provide further insights into how well the dataset generalizes to tasks beyond CS-TTS, such as emotional speech synthesis or audiobook narration?

How do the authors address potential biases introduced by the use of movie scripts as the primary source for dataset construction?

Would additional metrics beyond MOS, such as word error rate (WER) or speaker similarity, provide more insights into the dataset's applicability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730869047677

thread_type:
official_review

<thread_object>
creator: reviewer_zq6r

<content>
summary:
This paper proposes a new TTS task and benchmark to produce contextualized and situated synthetic speech using dialogues, narratives and actions.

strengths:
The new dataset would be useful to the TTS community and will be made available
The data creation methodology seems reasonable as does the design of the DNA Speech model.

weaknesses:
1. The precise meaning of situated and contextualized is not very clear from my reading of the paper. Further, it is not very clear how actions in particular aid in situated and contextualized TTS. 

2. From the data pipeline, it is not clear whether the obtained subtitles exactly match the speech, or are machine generated in some way. There seem to be many automated portions, for example, obtaining subtitles through OCR, getting Dialogues, Actions, Narratives and Characters from the original movie scripts, speech denoising etc. For all of these steps, there are no objective measures of quality reported, which casts doubt on the quality of data used.  Furthermore, the only quality evaluation used involves training a TTS models using DNASpeech and evaluating it. 

3. The proposed ASR filtering based on Whisper could be potentially aggressive because the authors remove all non perfect matches. This means that the data obtained is selected based on Whisper's biases for movie transcription, which is not ideal.

questions:
1. Three listeners is a small number for a TTS MOS test. Did the authors conduct a wider test ?
</content>
</thread_object>

</thread>


Metareview: