SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Toward Cutting-Edge Speech Generation Methods

Authors: Wen Huang, Yanmei Gu, Zhiming Wang, Huijia Zhu, Yanmin Qian
Primary area: datasets and benchmarks
Keywords: dataset, deepfake detection, anti-spoofing, speech generation

Abstract:
As speech generation technology continues to evolve, the risk of misuse through deepfake audio has become a pressing concern, which underscores the critical need for robust detection methods.  However, many existing speech deepfake datasets fall short in terms of size, diversity, and linguistic coverage, limiting the ability of models to generalize effectively to unseen deepfakes. To address these limitations, we present SpeechFake, a large-scale dataset specifically designed for speech deepfake detection. With over 3 million deepfakes totaling more than 3,000 hours of audio, SpeechFake was generated using 40 different speech generation tools, including cutting-edge techniques, and spans 46 languages. This paper provides a detailed overview of the dataset’s composition and statistics, emphasizing its scale and diversity. Additionally, we establish baseline results for SpeechFake and explore how factors such as generation methods, language diversity, and speaker variation influence detection performance. We believe SpeechFake will be a valuable resource for advancing speech deepfake detection research, offering opportunities to explore new detection strategies and improve model robustness across diverse and evolving generation techniques. The dataset will be publicly available soon.

Decision: reject
Reviewer scores: 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730625994205

thread_type:
official_review

<thread_object>
creator: reviewer_evny

<content>
summary:
The paper introduces a new dataset, SpeechFake, for training and evaluating speech deepfake detectors.  The dataset is larger than previous such datasets, and includes deepfakes generated by a larger number (40) of speech generation systems, including text-to-speech (TTS), voice conversion (VC), and neural vocoders (NV).  The paper also provides baseline results using existing deepfake detection models, demonstrating that training the detectors on SpeechFake is more effective across multiple evaluation sets than training on a previous such dataset, ASV19, and measuring the detection performance across speakers, languages, and deepfake generation methods.

strengths:
+ The paper addresses an important problem.

+ The study of performance across multiple dimensions (languages, speakers, seen/unseen generation methods) is valuable.

+ The dataset will probably be useful for research beyond deepfake detection, like data augmentaiton.

weaknesses:
- The paper doesn't include enough empirical comparisons to make a convincing case for the value added of the new dataset.  It includes a comparison of training on the new dataset vs. on ASV19, but not vs. other existing datasets, or vs. combinations of existing datasets, which might provide a similar generalization benefit.

- The contribution is a bit narrow for ICLR.  A dataset + baseline paper like this would be a better fit in a speech conference or related workshop.

- The description of dataset construction needs more detail, e.g. how multiple speaker IDs were sampled and generated with each model.  Ideally the paper would include an "algorithm" for sampling examples using each method.

questions:
- How is the quality filtering done?  The text states "generated speech with noticeable distortions, excessive noise, or unnatural artifacts is discarded".  Is this done automatically, and if so how?

- The detection performance results in Tables 5 and 6 are much better for all settings than in Tables 3 and 4.  Why is that?  For example, why are the English and Chinese results in Table 4 much better than those in the BD-EN and BD-CN columns of Table 3?

- The extremely low EERs in Table 5 suggest that the multilingual dataset is too easy and has very limited value for deepfake detection research.  Is this a misunderstanding of the results?  Please correct me if I'm missing something.

- For the settings with very low EERs, it would be helpful to include confidence intervals or significance estimates.  It is hard to tell whether we can really draw conclusions based, for example, on the differences between speaker settings in Table 6.

More minor details:

- The title is a bit hard to parse.  I think the word "toward" should probably be something else, e.g. "generated using".

- The third contribution in the bullet list (at the end of the introduction) doesn't seem like an independent contribution (more like an elaboration of the others).

- In Table 1, what does "-" mean?

- The first sentence in Related Work suggests that TTS and VC are the only two speech generation tasks (though NV is added on later).  However, there are other tasks like speech content editing, style conversion, and enhancement.

- The related work sub-section on existing datasets doesn't explicitly state what is new/different about the proposed one (though this is briefly stated elsewhere).

- In the appendix, what does "speaker information" include?  The description is "details about the speaker or the generated voice identity", but I'm not sure what this means.

- In the appendix, the figures and surrounding descriptions comparing waveforms/mel spectrograms are unclear and make questionable statements (as far as I can tell) about the examples shown.  For example, A.2 states that the mel spectrograms show "variations in energy distribution, frequency patterns, and clarity".  What exactly is meant by this?  Another similar statement is "the resulting mel-spectrograms share some common characteristics, particularly in their overall structure and energy distribution across frequencies".  What does "overall structure" and "energy distribution" mean, and how do you measure that they are similar here vs. in other examples across multiple speakers?  Yet another statement of this type is about "... distinct characteristics, such as pitch, timbre, and pacing, which are clearly reflected in the waveforms and mel-spectrograms".  How do you measure the timbre in the waveforms or mel spectrograms?  On a more minor note, labeling the axes in the waveform and spectrogram plots would help.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686900690

thread_type:
official_review

<thread_object>
creator: reviewer_cq2e

<content>
summary:
The paper introduces SpeechFake, a large-scale multilingual dataset designed explicitly for speech deepfake detection. It offers a vast collection of over 3 million samples (amounting to 3,000 hours of audio) generated using 40 advanced speech synthesis tools across 46 languages. The paper provides a comprehensive baseline evaluation for SpeechFake, analyzing factors like generation methods, language diversity, and speaker variation.

strengths:
- SpeechFake Dataset: A large-scale and diverse dataset for speech deepfake detection, encompassing a wide range of speech generation methods. The dataset consists of 3 million utterances, totalling 3,000 hours. It would be beneficial if the authors open-source the entire dataset (including hidden), rather than just a subset.

weaknesses:
- Multilingual Focus: While the work prioritizes multilingualism, it primarily focuses on English and Chinese. Training datasets are lacking for 37 out of the 46 languages, with only a set of 5,000 utterances released for testing purposes for each of these 37 languages.
- Coverage of Speech Generation Tools: Although the dataset includes 40 different speech generation tools, it does not encompass all current techniques, nor does it provide a way to account for future techniques that may emerge. This limitation suggests that the dataset may not remain fully relevant over time and could require updates or recreation as the field advances.

questions:
- Generation Process Details: It would be helpful if the authors could clarify how they determine the number of utterances generated using different speech generation tools for a particular original utterance.
- Choice of Test Sets (Table 3): Referring to Table 3, could the authors provide insights into why test sets from 2019 were selected rather than more recent versions? In fact, evaluating on all available benchmarks might have provided a more comprehensive analysis.
- Performance on "Other" Datasets (Table 3): Referring again to Table 3, could the authors explain why the BD (combined set) does not consistently achieve better performance on the "Other" testing datasets compared to its individual subsets?
- Selection of 40 Systems: What was the rationale behind selecting 40 speech generation systems? It would be valuable to understand if all these systems can produce natural-sounding speech.
- Human Verification of Generated Samples: It would also be useful to know if the dataset was subjected to any human verification, as generative models often produce a variety of artefacts, including silences or truncated words. Verifying whether the generated samples are indeed suitable deepfake candidates would add credibility to the dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730837147700

thread_type:
official_review

<thread_object>
creator: reviewer_icvc

<content>
summary:
The paper introduces a novel database for speech deepfake detection, comprising a large-scale collection of samples generated by 40 different speech synthesis tools across 46 languages. Benchmark results accompany the dataset, with detailed analyses provided on performance across various subsets, each with a specific focus.

strengths:
The proposed database stands out for its extensive coverage of languages and speech generation tools, providing a robust foundation for deepfake detection research. Additionally, the dataset includes rich metadata, which is invaluable for in-depth analysis of anti-spoofing performance or targeted optimizations. The exploration of generator types and the effects of language and speaker variability is particularly engaging and relevant to recent advances in the field.

weaknesses:
While the dataset is comprehensive, the overall scope is closely aligned with previous work in the field, as acknowledged in Table 1. Although the paper offers a solid contribution, it represents an incremental rather than groundbreaking addition to the anti-spoofing community. The reviewer appreciates the authors' efforts but suggests that more extensive analyses or use cases could further strengthen the contribution. Potential extensions might include exploring additional anti-spoofing model variants, conducting detailed error analyses, or leveraging the dataset for novel applications using pre-trained anti-spoofing models.

questions:
- As noted in the weaknesses, additional analyses would elevate the paper’s contribution. The inclusion of diverse use cases for the dataset could also significantly enhance its impact and utility.
- Why do TTS-Tortoise and VC-OpenVoiceTTS contain significantly more data than other systems?
- Why is BD-UT excluded (or separated) from the result discussion table? Including access dates or version details for the API services would also be advisable, as these may vary over time.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a large synthetic speech data set, with a particular focus on a wide range of synthesis models and languages.

I recommend a rejection because of a lack of clear demonstrated value, as pointed out by all reviewers.

Novelty of a data set paper should be demonstrated differently compared to a paper that proposes an approach to solving a problem. This paper did well demonstrating the wide variety of synthesis models and languages, but fails to show the value (i.e., challenge) that the data set brings. The problem is a combination of model choice and writing. The paper should really stress test the best models out there to demonstrate a clear challenge. Reviewers have given valuable suggestions, and the authors are encouraged to improve the paper based on the feedback.

The authors are also encouraged to improve the reproducibility of the data set so that others are able to automate/reproduce the process in future settings.