Deepfake Defense: Constructing and Evaluating a Specialized Urdu Deepfake Audio Dataset (2024)

Abstract: Deepfakes, particularly in the auditory domain, have become a significant threat, necessitating the development of robust countermeasures. This paper addresses the escalating challenges posed by deepfake attacks on Automatic Speaker Verification (ASV) systems. We present a novel Urdu deepfake audio dataset for deepfake detection, focusing on two spoofing attacks – Tacotron and VITS TTS. The dataset construction involves careful consideration of phonemic cover and balance and comparison with existing corpora like PRUS and PronouncUR. Evaluation with AASIST-L model shows EERs of 0.495 and 0.524 for VITS TTS and Tacotron-generated audios, respectively, with variability across speakers. Further, this research implements a detailed human evaluation, incorporating a user study to gauge whether people are able to discern deepfake audios from real (bonafide) audios. The ROC curve analysis shows an area under the curve (AUC) of 0.63, indicating that individuals demonstrate a limited ability to detect deepfakes (approximately 1 in 3 fake audio samples are regarded as real). Our work contributes a valuable resource for training deepfake detection models in lowresource languages like Urdu, addressing the critical gap in existing datasets. The dataset is publicly available at: https://github.com/ CSALT-LUMS/urdu-deepfake-dataset.

heading: 1 Introduction
text: Automatic Speaker Verification, a method for biometric person recognition, has gained popularity in recent years. However, this surge in popularity has also given rise to new challenges in the form of spoofing or deepfake attacks. Initially coined on Reddit in 2017, the term ’deepfake’ (Bitesize, 2019) denotes the application of deep learning techniques for face swapping in videos. Presently, the term has evolved to broadly encompass any audio or video manipulation where key attributes are digitally altered or swapped using artificial intelligence (AI)
technologies. The ASVspoof community classifies these attacks into two main categories: logical access, involving deepfake-generated audios, speech synthesis, and voice conversion, and physical access, which includes replay attacks and impersonation (Wang et al., 2020b).
Deepfakes, a complex way of manipulating media, make fake content easier to generate and harder to detect. Speech synthesis models now allow the creation of deepfakes that are undetectable by the human ear or even verification systems (Mirsky and Lee, 2021). In 2019, impostors leveraged AI-driven software to replicate the voice of a corporate executive, orchestrating a fraudulent transfer of USD 243,000 (Stupp, 2019). This incident underscores the imperative of developing robust methods to accurately identify deepfake audio in order to counteract such fraudulent activities. In a behavioral study, Kobis et al. (2021) revealed that people cannot easily detect deepfakes, yet they perceive that they can. Thus, these fake audios have the potential to spread misinformation, create mass panic and havoc, malign personalities, and change narratives. Moreover, beyond this social impact, deepfakes have the power to break through systems protected by voice recognition through the spoofing attacks listed above. Considering the adverse effects of deepfake audios, it is crucial to develop systems capable of discerning between real and deepfake audio. The ASVspoof challenge, a community-led initiative, promotes the development of such countermeasures against deepfakes and audio spoofing (Wu et al., 2015; Kinnunen et al., 2017; Todisco et al., 2019; Yamagishi et al., 2021).
Countermeasures against deepfakes include detection algorithms designed to identify features in deepfake audios. The physical attributes of sound, encompassing pitch, texture, loudness, and duration, can now be accurately replicated in artificially generated deepfake audios. To detect the features that differentiate bonafide (actual utter-
14470
ances of the people) and fake audios, the model needs to train on a large amount of data (Azeemi et al., 2022). These differentiations are based on spectral and temporal differences and micro features (Delgado et al., 2021; Dhamyal et al., 2021; Tak et al., 2020). Widely used datasets created for this purpose include WaveFake (Frank and Schönherr, 2021), FakeAVCeleb (Khalid et al., 2021), and the ASVspoof dataset (Wang et al., 2020b) itself. These datasets, from high-resource languages, exemplify the large amount of data required to train deepfake detection models. Unfortunately, in lowresource languages, this large amount of data is unavailable. To cater to this lack of data in Urdu, we create and evaluate a dataset that can be used to train against spoofing attacks.

heading: 1.1 Contributions
text: The presented research offers the following contributions:
• We present an audio deepfake dataset, containing 20,451 utterances of bonafide and 16,830 utterances of deepfake audio, to train detection models in Urdu, a low-resource language. The dataset is hosted on a publicly accessible repository1.
• We assess the dataset through human evaluation and discover that about one out of every three audio samples goes undetected by individuals as being fake. This finding carries implications for the potential spread of misinformation.
• We evaluate the dataset qualitatively and qualitatively. Qualitative measures include examining the variations in the relative distribution of deepfake-generated and real audios using t-SNE plotting and comparing L2 norms between bonafide audios and each set of deepfake audios. For quantitative analysis, we calculate the Equal Error Rate (EER) across various speakers and spoofing attacks.

heading: 2 Related Works

heading: 2.1 Deepfake Detection Models
text: The field of audio deepfake detection has seen remarkable growth recently, focusing on using machine learning to differentiate real speech from synthetic audio (Wu et al., 2020; Wang et al., 2020a;
1Public dataset repository: https://github.com/ CSALT-LUMS/urdu-deepfake-dataset
Chen et al., 2020). This research typically follows either a conventional pipeline method, combining feature extraction with classification, or newer endto-end methodologies that process raw audio data directly for both tasks.
A key hurdle in this domain is the extensive data required for training advanced deep learning Textto-Speech (TTS) models (Ping et al., 2017; Shen et al., 2017; Sotelo et al., 2017; Tachibana et al., 2017; Wang et al., 2017). Research has shown high efficacy for multi-speaker TTS models, especially when data for a specific speaker is limited (Latorre et al., 2018; Luong et al., 2019). The study by Luong et al. (2019) emphasized the superiority of multi-speaker models using oversampling techniques in scenarios with sparse data. While undersampling generally showed negative impacts, ensemble methods were noted for their ability to improve speech naturalness, albeit at the cost of higher computational resources (de Korte et al., 2020).
Furthermore, the majority of research and competitions in audio deepfake detection, such as ASVspoof and ADD, are focused on English and Chinese, reflecting a language bias due to easier data collection (Wang et al., 2020b; Yi et al., 2022).

heading: 2.2 Deepfake Detection Datasets
text: The creation of robust TTS datasets is vital for the development of effective detection models. These datasets should be of high quality, featuring diverse speakers, accurate transcripts, and ample audio content per speaker (Bakhturina et al., 2021). Best practices for TTS dataset creation underscore the necessity for error-free, clear recordings, uniformity in tone and pitch, comprehensive phoneme representation, and overall naturalness. Rigorous quality assessments, such as examining the length of clips and transcripts and inspecting spectrograms for noise, are also advised to maintain dataset integrity (coq, 2023).
Recent trends in audio deepfake research include using alternative data sources to address the lack of target data. Efforts to build TTS datasets through community-driven or automated collection and transcription processes have been observed (Gutkin et al., 2016; Xu et al., 2020; Wibawa et al., 2018). However, these methods might result in datasets with lower recording quality and naturalness, which could impact the effectiveness of TTS models when compared to traditional datasets (Guo
et al., 2022). Recently, the focus on enhancing TTS systems for under-resourced languages has gained traction. Researchers are exploring how well-structured datasets in various languages can improve TTS for languages with scarce resources. Techniques like cross-lingual transfer learning and multilingual TTS are being investigated for this purpose (Azizah et al., 2020; Tu et al., 2019; He et al., 2021), aiming to democratize TTS technology and extend its reach to a wider range of languages and dialects.

heading: 2.3 Benchmark Dataset
text: The Phonetically Rich Urdu Speech Corpus (PRUS) and the PronouncUR lexicon are crucial resources for developing and benchmarking Urdu Text-to-Speech (TTS) systems, particularly in the context of audio deepfakes in Urdu, a low-resource language.
PRUS, consisting of 70 minutes of transcribed read speech, with its comprehensive phonetic coverage, including 62 out of 67 total phonemes in Urdu and a wide array of tri-phonemes, offers a detailed representation of Urdu’s phonetic diversity. This corpus, balancing high-frequency word focus with practical dataset size, serves as an ideal benchmark for phonetic diversity and quality assessment in TTS systems. Figure 1 shows a snippet of PRUS corpus and its phoneme counts (PC).
PronouncUR’s lexicon, encompassing approximately 46,000 words and covering 64 out of 67 phonemes, provides a broad spectrum of Urdu sounds. Its phoneme frequency distribution and expert tagging make it invaluable for evaluating
TTS system comprehensiveness and phonetic accuracy.
The availability of PRUS and PronouncUR open up the opportunity to develop benchmark datasets for audio deepfakes in languages like Urdu. These resources are not only vital for TTS system development but also offer a framework for detecting and authenticating audio deepfakes, addressing a significant challenge in digital communication in low-resource languages.

heading: 3 Methodology
text: To create the text corpus for the dataset, we randomly select sentences from reputable Urdu news sources. We then analyze the phonemic structure of the text corpus, ensuring its alignment with natural language patterns. Statistical measures confirm the dataset’s phonemic cover and balance. For the spoofing attacks, advanced text-to-speech models Tacotron and VITS TTS are utilized to generate deepfake audios. Figure 2 highlights the steps taken in dataset construction.

heading: 3.1 Phonemic Analysis of the Datasets
text: The text corpus (referred to as the news corpus here onwards) for our dataset has been curated by randomly selecting 495 sentences from reputable Urdu news sources, with permission. Given the rich phonemic inventory inherent in the Urdu language (Raza et al., 2009), it is imperative to ensure that our dataset possesses a comprehensive phonemic cover and balance. To achieve this, we conduct a careful analysis to ascertain the presence of all possible phonemes within the text and to verify
whether their frequencies aligned with those observed in natural language (Zia et al., 2018).
To establish the phonemic fidelity of our dataset, we conduct a comparative analysis with established Urdu corpora known for their adherence to Urdu’s phonemic distribution patterns. Notably, we employ the Phonetically Rich Urdu Corpus (PRUS) (Raza et al., 2009) and PronouncUR (Zia et al., 2018) as references.
In our linguistic research, we conducted a comparative analysis of phoneme ranks across two different corpora: the PRUS Corpus and the PronouncUR Corpus, each compared against the News Corpus. We formulate the null hypothesis stating no significant correlation between the phoneme distributions of the two datasets. The visual data from the line graphs illustrate a striking similarity in phoneme distribution in both comparisons. Figure 4 shows the phoneme rank comparison between PRUS Corpus and the News Corpus, while Figure 5 shows the phoneme rank comparison between PronouncUR training lexicon and the News Corpus. This visual correlation is statistically substantiated by Spearman’s Rank Correlation Coefficient. It can be understood as ranging from no association (coefficient = 0) to a perfectly monotonic relationship (coefficient = –1 or +1). We observe values of 0.977 for the PRUS Corpus comparison and 0.958 for the PronouncUR comparison, both suggesting exceptionally strong positive monotonic correlations. These high coefficients are coupled with
near-zero p-values, confirming that these correlations are statistically significant and not products of chance. Spearman’s metric was particularly apt for these analyses as it adeptly captures monotonic relationships without the need for data normality, and it remains robust in the presence of outliers.
In addition to comparing the News Corpus with established Urdu corpora, we conducted a detailed phonemic analysis comparing the PRUS Corpus and the PronounceUR Corpus. The results of this comparison are visualized in Figure 3 showing the rank correlation of phonemes between the two corpora.
The Spearman’s Rank Correlation coefficient of 0.956 and Kendall’s Tau coefficient of 0.845 both indicate a strong positive correlation between the phoneme ranks in the PRUS and PronounceUR corpora. The Average Rank Difference of 3.34
suggests a close similarity in the rank order of phonemes between the two datasets. These results further confirm the consistency and reliability of phoneme usage patterns across different linguistic resources.
Our investigation extended to lexical distribution via Zipf’s Law, which posits an inverse relationship between word frequency and its rank in a corpus. Analyzing our dataset against this law, we observed a distribution pattern closely aligning with Zipfian expectations. The linear regression analysis of the log-log plot, as illustrated in Figure 6, yielded a slope of -0.8676, close to the ideal Zipfian slope of -1, and an R-squared value of 0.9595. These results underscore a strong adherence to Zipf’s Law, indicating a natural linguistic patterning within the Urdu news corpus. This adherence not only highlights the corpus’s linguistic representativeness but also validates its utility for computational linguistics research. The close alignment with Zipfian expectations reinforces the dataset’s suitability for exploring language models and comprehension studies, affirming its value in linguistic and phonemic research endeavors.
Furthermore, the strength of these relationships is reinforced by Kendall’s Tau Coefficient. It can again be understood as ranging from no association (coefficient = 0) to a perfectly monotonic relationship (coefficient = –1 or +1). We observe values of 0.888 for the PRUS comparison and 0.841 for the PronouncUR comparison. These coefficients mirror the strong positive correlations indicated by Spearman’s, and their very low p-values support the notion of a significant, non-random association between the phoneme ranks in the respective corpora. The conservative nature of Kendall’s Tau makes it a suitable choice for the datasets, especially considering that it is less influenced by small sample sizes and the non-parametric nature of the data.
Additionally, the Average Rank Difference metric complements these findings, showing minimal discrepancies in phoneme rankings between the PRUS Corpus and the News Corpus at approximately 2.66, and a slightly larger yet modest variation of approximately 4.04 when comparing the PronouncUR Corpus to the News Corpus. Despite the slight differences indicated by this metric, the strong Spearman’s and Kendall’s correlations confirm a general consistency in phoneme rank order across the examined linguistic resources. The coefficients and p-values from both hypothesis tests indicate a significant correlation, thereby rejecting the null hypothesis.
The integration of Spearman’s Rank Correlation, Kendall’s Tau, and Average Rank Difference in these analyses provides a robust, multifaceted validation of the initial graphical observations. It collectively supports the conclusion that there is a substantial overlap in phoneme usage patterns within the compared linguistic resources. While the PronouncUR Corpus exhibits a slightly greater
variability in phoneme rank compared to the PRUS Corpus, both corpora maintain a significant parallelism with the News Corpus, underscoring the reliability of phoneme usage patterns across different linguistic datasets. Table 1 and 2 summarize the results of the phonemic analysis.

heading: 3.2 Spoofing Attacks
text: We create a dataset consisting of a combination of bonafide and deepfake audios. In order to achieve this, we choose two advanced text-to-speech (TTS) models, Tacotron (Wang et al., 2017) and VITS TTS (Kim et al., 2021), to generate the deepfake audio. This selection is based on their demonstrated effectiveness in processing the Urdu language, essential due to its complex phonetic structure, and the popularity of these models in deepfake generation. Additionally, these models represent the cutting edge in TTS technology, providing high-quality, realistic audio outputs. The choice of two distinct models, one based on a sequenceto-sequence model with attention (Tacotron) and the other on a Conditional Variational Autoencoder with Adversarial Learning (VITS TTS), allowed for a comprehensive exploration of audio deepfake generation methodologies. The models have been fine tuned to work on Urdu datasets.

heading: 3.2.1 Spoofing Attack 1: Tacotron
text: Tacotron serves as an end-to-end text-to-speech (TTS) model based on the sequence-to-sequence (seq2seq) paradigm with an attention mechanism. In our study, we train and utilize a Tacotron model to generate deepfake audios. This model incorporates PronouncUR (Zia et al., 2018) as a pronunciation lexicon, functioning as a grapheme-tophoneme (G2P) converter. During the training process, sentences from the PRUS corpus (Raza et al., 2009) are initially passed to PronouncUR to convert them into a string of phonemes, which are then
fed into the pre-trained Tacotron model.

heading: 3.2.2 Spoofing Attack 2: VITS TTS
text: VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-toSpeech) stands as an end-to-end text-to-speech model that combines an encoder and vocoder. In our study, VITS TTS serves as the second attack method. This attack analyzes input text using natural language processing (NLP) techniques to extract linguistic features, including phonemes, stress patterns, and intonation. To train the VITS TTS model, we use the list of sentences from the PRUS Corpus (Raza et al., 2009), along with their corresponding audios.
We train the Tacotron and VITS TTS models on the voice of 17 individuals separately. We then generate the deepfake audios through the trained models. These audios were then compared with the bonafide audios.

heading: 3.3 Training Data Collection
text: We train Tacotron and VITS TTS on the PRUS corpus audios. To achieve this, we select a sample of 20 student volunteers who record the 708 sentences from the PRUS corpus. Each speaker receives a set of pre-recorded audios, articulating every sentence of the PRUS corpus. Participants attentively listen to each audio before reproducing the sentence in their own voice. We also document the laptop make, model, and headphones used by each speaker during recording, and they are instructed to record in a quiet, closed environment. Upon completing the recording stage, we carefully choose a sample of 17 speakers (7 female, 10 male) with high-quality complete audio recordings to advance to the next phase of the experiment, and get written consent for the public sharing of their recordings (and derivatives) for research.

heading: 3.4 Generation of Deepfake Audios
text: We assign a unique speaker ID to each speaker based on their training order. This ensures distinct identification while preserving anonymity for the public dataset release. We generate deepfake audios using the final checkpoint of each model, using the 495 sentences of the News Corpus for both attacks. The speakers also record the bonafide audios of the News Corpus. This process yields PRUS and News Corpus recordings as bonafide audios and two sets of deepfake audios (one for each attack) for each speaker. In Figure 7, the distribution of bonafide and deepfake utterances in the final dataset is depicted. The duration and lengths of audios for each split are shown in Table 4. The dataset is segmented across 8, 4, and 5 speakers for training, development, and evaluation, respectively.

heading: 3.5 Evaluation of the Dataset
text: To understand the differences in the bonafide and deepfake audios in the dataset, it is important to analyze the spectral composition of these subsets. We visualize these subsets by obtaining the Mel Frequency Cepstral Coefficients (MFCCs) of each
audio. MFCCs are a representation of the shortterm power spectrum of a sound signal. They are commonly used in audio processing and speech recognition. We reduce the dimensions of MFFC features through the tree-based t-SNE algorithm — with a perplexity value of 40 as suggested in (Wang et al., 2020b) and plotting the reduced dimensions. Figure 8 shows the scatter plot of the processed features for each subset. The colors represent different subsets of the dataset, i.e. bonafide audio (green), VITS TTS deepfake audios (yellow), and Tacotron deepfake audios (blue). The smaller clusters within each subset represent individual speakers. We notice differences in the position and distribution of each attack as compared to the bonafide audios. Both deepfake subsets exhibit considerable overlap with the bonafide audios, especially those generated using the Tacotron model, highlighting the spectral similarity between these subsets.
In addition to computing t-SNE of the Melfrequency cepstral coefficients (MFCCs) from the audio samples, we also calculate the L2 norm of the MFCCs to compare bonafide recordings with those generated by the Tacotron and VITS TTS models. Figure 9 illustrates a notable trend: Tacotrongenerated audios exhibit a smaller disparity from bonafide audios compared to VITS TTS-generated audios.
We further evaluate the quality of the generated audios by running it on AASIST-L and RawNet2. AASIST-L (Jung et al., 2022) is a lightweight end-to-end audio anti-spoofing model that can efficiently model spoofing artefacts in temporal and spectral domains. RawNet2 (Tak et al., 2021) is an end-to-end convolutional neural network for audio anti-spoofing. We obtain an overall equal error rate of 0.495 and 0.524 through AASIST-L for audios generated through TTS and Tacotron respectively. The EER breakdown for each speaker through AASIST-L and RawNet2 is presented in Table 5. The EER score for AASIST-L varies from 0.44 to 0.58 depending upon the quality of the generated audios for each speaker. This range for the
EER score indicates that the real and fake audios cannot be distinguished reliably.

heading: 4 Human Evaluation

heading: 4.1 User Study
text: To assess the quality of our dataset, we employ a human evaluation-based approach. Participants in our study listen to a set of 30 random audios in a controlled environment and classify each as either Fake (deepfake) or Real (bonafide). We employ a convenience sample of 100 participants between the ages of 10 to 48, with a male-to-female ratio of 70-30, with varying tech literacy. The participants are paid PKR 500 per evaluation (approximately 10 minutes) Each random sample of 30 audios includes 10 random bonafide audios, 10 Tacotrongenerated, and 10 VITS TTS-generated audios.
We conduct the evaluation in a controlled environment to eliminate biases stemming from variations in speaker quality. During the assessment, we ask each participant to listen to each audio and give the following instructions: "The audio sample that you will listen to is audio produced by humans or produced artificially by artificial intelligence. Please listen to the audio sample and determine whether the voice is artificially generated or is uttered by a person, judging only on the basis of the voice you hear. You can listen to it as many times as you like. And then share your reasons for the classification." Each participant categorizes each audio in the assigned group of recordings into two distinct groups, real or fake. We document their reasons for classifying the audios as fake or real. We observe that most participants base their judgment on factors such as audio distortion and length. Audios containing longer sentences with minimal pauses for breath are often categorized as deepfake generated.

heading: 4.2 Analyzing User Study Results
text: The evaluation results, illustrated by the ROC curve in Figure 10, shed light on how well human participants performed in distinguishing between genuine and deepfake audio samples at various classification thresholds. The ROC curve, plotting True Positive Rate against False Positive Rate, indicated a moderate level of discriminative performance with an Area Under the Curve (AUC) value of 0.63.
This AUC suggests that individuals demonstrated a limited ability to detect deepfakes, with approximately 1 in 3 fake audio samples being misidentified as real. When considering the consequences of such limitations in distinguishing between genuine and manipulated content, especially
in contexts like political situations or audio leaks in Pakistan, there is a heightened risk of misinformation spreading. This misinformation could contribute to a climate of mistrust, political polarization, and potentially erode public confidence in state institutions.
The societal impact of these findings on democracy underscores the need for more robust detection mechanisms to mitigate the potential threats posed by deepfakes. Developing reliable methods to differentiate between genuine and manipulated content becomes crucial for safeguarding public trust, political discourse, and the integrity of democratic processes.

heading: 5 Limitations and Conclusion
text: In presenting our Urdu deepfake detection dataset, we recognize limitations and suggest areas for future improvement. The dataset currently emphasizes two text-to-speech (TTS) synthesis methods—Tacotron and VITS TTS. Expanding to a broader range of TTS techniques in future iterations will enhance deepfake detection. The dataset’s reliance on a convenience sample leads to a gender imbalance in the speakers, highlighting the need for a more diverse dataset in future work. Additionally, our dataset primarily covers logical access scenarios; future research could include physical access scenarios for added detection challenges. In conclusion, our dataset lays a solid foundation for deepfake detection research in the Urdu language. Addressing the outlined limitations and pursuing future research directions will further enhance the dataset’s value and contribute to the advancement of deepfake detection technologies in low-resource languages.

heading: 6 Ethical Impact
text: Deepfakes pose risks of spreading misinformation, causing panic, damaging reputations, and manipulating narratives. While improving detection models is a key solution, it inadvertently fosters the development of more sophisticated deepfake generation models that can evade detection. The creation of extensive deepfake audio datasets raises ethical concerns as it may inadvertently contribute to refining audio deepfake generation techniques. Responsible management of such datasets is crucial to address potential ethical challenges in their deployment.

heading: A Reproducibility and Hyperparameters
text: The hyperparameters used for training and evaluation of the TTS models are added below. Table 6 contains the hyperparameters for Tacotron and Table 7 shows the hyperparameters for VITS TTS.

heading: B Datasets and Evaluation Model
text: We use the PRUS Corpus available under the Creative Commons license, which allows distribution, remixing, tweaking, and building upon the work, as long as we credit the creators for the original creation. We use PronouncUR and AASIST-L available under the MIT License.

heading: Future Work
text: In presenting our Urdu deepfake detection dataset, we recognize limitations and suggest areas for future improvement. The dataset currently emphasizes two text-to-speech (TTS) synthesis methods—Tacotron and VITS TTS. Expanding to a broader range of TTS techniques in future iterations will enhance deepfake detection. The dataset’s reliance on a convenience sample leads to a gender imbalance in the speakers, highlighting the need for a more diverse dataset in future work. Additionally, our dataset primarily covers logical access scenarios; future research could include physical access scenarios for added detection challenges