I Can Hear You: Selective Robust Training for Deepfake Audio Detection

## Abstract

Abstract Recent advances in AI-generated voices have intensified the challenge of detecting deepfake audio, posing further risks for the spread of scams and disinformation. To tackle this issue, we establish the largest public voice dataset to date, named DeepFakeVox-HQ , comprising 1.3 million samples, including 270,000 high-quality deepfake samples from 14 diverse sources. Despite previously reported high accuracy, existing deepfake voice detectors struggle with our diversely collected dataset, and their detection success rates drop even further under realistic corruptions and adversarial attacks. We conduct a holistic investigation into factors that enhance model robustness and show that incorporating a diversified set of voice augmentations is beneficial. Moreover, we find that the best detection models often rely on high-frequency features, which are imperceptible to humans and can be easily manipulated by an attacker. To address this, we propose the F-SAT : F requency- S elective A dversarial T raining method focusing on high-frequency components. Empirical results demonstrate that our training dataset boosts baseline model performance (without robust training) by 33%, and our robust training further improves accuracy by 7.7% on clean samples and by 29.3% on corrupted and attacked samples, over the state-of-the-art RawNet3 model.

## 1 Introduction

AI-generated voices have become increasingly realistic due to larger datasets and enhanced model capacities (Ju et al., ; Neekhara et al., ), and they have been used in many important applications (Calahorra-Candao & Martín-de Hoyos, ). However, the success of AI-synthesized human voices poses significant security risks, including deepfake voice fraud and scams (Tak et al., ; Sun et al., ; Yang et al., ). A recent CNN report reveals a fraud in Hong Kong where a finance worker sent $25 million to scammers after a video call with a deepfake ‘chief financial officer’. The voice was created by an AI model, highlighting the risk of such technology.

Figure: Figure 1: The distribution of deepfake samples over predicted scores using the state-of-the-art detector (Jung et al., ) trained on the In-the-Wild dataset (Müller et al., ), with a decision boundary at 0.5. Tests on original, corrupted, attacked, and real-world deepfake audio reveal significant shifts in prediction scores, highlighting that training solely on current public datasets without robust training methods leads to poor performance.
Refer to caption: x1.png

Figure: Figure 2: We apply a high-pass filter to audio samples to remove low-frequency components. The x-axis represents the center frequency of the filter applied. Notably, there is a marked decline in detection performance for real audio starting at 4000 Hz and for fake audio at 6000 Hz.
Refer to caption: x2.png

Due to the importance of this problem, a number of work has investigated detecting AI-generated audio. Despite previously reported high detection accuracy on public datasets (Todisco et al., ; Frank & Schönherr, ), existing deepfake voice detectors perform poorly under real-world conditions (Xu et al., ; Müller et al., ; Radford et al., ). This is because the established benchmarks are often trivial, small, outdated, and homogeneous. Consequently, models trained and validated solely on these datasets fail to generalize to more diverse and challenging real-world deepfake samples. Moreover, deep learning models for audio are particularly vulnerable to adversarial attacks (Szegedy et al., ) (Zhang et al., ), where an attacker can subtly alter audio inputs in ways that are imperceptible to humans but mislead models into incorrect classifications.
Figure 1 illustrates a dramatic shift in the models’ prediction scores when exposed to these factors, underscoring the need for more robust training methodologies.

To address the above limitations, we first created the largest deepfake audio dataset to date, *DeepFakeVox-HQ*, including 270,000 high-quality deepfake samples from 14 diverse and distinct origins. We show that simply training on our collected dataset can produce new state-of-the-art models.

Moreover,
we find that even the state-of-the-art AI-voice detection models often depend on high-frequency features to make decisions (see Figure 2), which are imperceptible to humans. On the other hand, the low frequency signals can be heard by humans but are not relied on by the model to make predictions. As a result, natural corruptions in high frequency or attackers can easily manipulate the model by changing the high frequency signals, reducing the detection’s robustness.

In an initial study, we observed that standard adversarial training on raw waveforms not only fails to bolster robustness but also diminishes performance on unattacked data.
To address these shortcomings, we propose Frequency-Selective Adversarial Training (*F-SAT*), which focuses on high-frequency components. Since our adversarial training is targeted, we can mitigate specific vulnerabilities without touching the true features at lower frequencies, thus enhancing the model’s resilience to corruptions and attacks while maintaining high accuracy on clean data.

Visualizations and empirical experiments demonstrate that using only our training dataset, we can produce state-of-the-art models, achieving a 33% improvement on the out-of-distribution portion of our test set, which includes 1,000 deepfake samples from the top five AI voice synthesis companies and 600 samples from social media. Additionally, by incorporating random audio augmentations, our model achieves the highest accuracy across 24 different types of corruptions. Furthermore, after applying F-SAT, our model further achieves a 30.4% improvement against adversarial attacks in the frequency domain and an 18.3% improvement against unseen attacks targeting raw waveform data in the time domain.

## 2 Related Work

AI-synthesized human voice:
AI voice synthesis generally falls into two categories: text-to-speech (TTS) and voice conversion (VC). TTS systems convert written text into spoken audio using a desired voice. This process typically involves three main components: a text analysis module that transforms text into linguistic features, an acoustic model that converts these features into a mel-spectrogram, and a vocoder. Some of the leading TTS models include StyleTTS (Li et al., ), VoiceCraft (Peng et al., ), XTTS (Casanova et al., ), and Tortoise-TTS (Betker, ), and are known for their ability to produce high-quality audio that closely mimics real human speech.

VC models, in contrast, take an audio sample from one person and transform it to sound like another person for the same speech content. Recent VC approaches primarily operate within the mel-spectrum domain (Ju et al., ; Shen et al., ; Popov et al., ), using deep neural networks to shift the mel-spectrograms from the source to the target voice.

Detection Method:
AI deepfake detection models based on deep learning can be grouped into two main categories: those processing raw audio end-to-end and those analyzing spectrum. The first category includes models like RawNet2, which employs Sinc-Layers (Ravanelli & Bengio, ) to extract features directly from waveforms, and RawGAT-ST, which utilizes spectral and temporal sub-graphs (Tak et al., ).
RawNet3 (Jung et al., ), which begins by using parameterized filterbanks (Zeghidour et al., ) to extract a time-frequency representation from the raw waveform and then is followed by three backbone blocks with residual connections, a structural approach that sets it apart from ECAPA-TDNN (Desplanques et al., ).These models process the audio data in its raw form to capture nuanced details directly impacting model performance.

The second category of AI deepfake detection models involves transforming raw audio into spectrograms for analysis. This process utilizes extracted features such as Mel Frequency Cepstral Coefficients (MFCCs) (Sahidullah & Saha, ), Constant Q Cepstral Coefficients (CQCCs) (Todisco et al., ), bit-rate (Borzì et al., ), and Linear Frequency Cepstral Coefficients (LFCCs) (Li et al., ). The analysis of these features is conducted using traditional machine learning methods like Gaussian Mixture Models (GMMs) (Todisco et al., ) or advanced neural networks such as Bidirectional Long Short-Term Memory (Bi-LSTM) (Akdeniz & Becerikli, ), ResNet (Alzantot et al., ), and Transformers (Zhang et al., ). These methods enable deeper and more intricate pattern recognition, enhancing the model’s ability to identify and classify deepfake audio accurately.

Adversarial Attack:
Adversarial training, originally developed for image processing systems (Madry et al., ; Mao et al., ), has been increasingly adapted to the audio domain (Chen et al., ), particularly to enhance the robustness of applications such as Automatic Speech Recognition (ASR) and speaker verification systems. This method aims to mitigate vulnerabilities to deceptive audio inputs, thereby improving security (Hussain et al., ). Another defensive strategy employs diffusion models to counter adversarial audio attacks (Wu et al., ), effectively smoothing out perturbations and hindering attackers from altering audio signals without significantly compromising signal quality.

While these techniques enhance robustness, they hurt the detection on unattacked audio, highlighting a trade-off between robustness and accuracy (Zhang et al., ; Tsipras et al., ). This compromise is particularly critical in scenarios that demand high accuracy and user satisfaction. Furthermore, the generalizability of adversarially trained models to new and unseen attacks remains limited (Rajaratnam et al., ), raising questions about their effectiveness in rapidly evolving threat environments.

## 3 DeepFakeVox-HQ

### 3.1 Training dataset

In this section, we introduce a new training dataset and a rigorous test set. In contrast to prior dataset, our dataset is large, diversified, realistic, and up-to-date, as shown in Table 1. Prior detectors show poor generalization capabilities in realistic settings, as shown in Figure 3. Both our training and testing datasets integrate the latest advancements in AI voice synthesis technologies. Additionally, the testing dataset includes several new models not covered in the training dataset, specifically designed to test the generalization ability of our detection systems.

**Table 1: Comparasion of Deepfake audio Dataset**
| Dataset | #Real Utt | #Fake Utt | Language | Conditions | Year | #Vocoder | Fake Type |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ASVSpoof19 | 12k | 109k | English | Clean | 2019 | 17 | TTS, VC |
| ASVSpoof21 | 22k | 589k | English | Clean, Noisy | 2021 | UNK | TTS, VC |
| WavFake | 14k | 90k | English | Clean | 2021 | 7 | TTS |
| ADD2022 | 61k | 251k | Chinese | Clean | 2022 | UNK | TTS, VC |
| In-The-Wild | 20k | 12k | English | Clean, Noisy | 2022 | 19 | TTS |
| LibriSeVoc | 13k | 79k | English | Clean | 2023 | 6 | TTS |
| Our Train | 690k | 640k | English | Clean, Noisy | 2024 | 40 | TTS, VC |
| Our Test | 3k | 3k | English | Clean, Noisy | 2024 | 15 | TTS, VC |

High quality deepfake samples: The limitations of existing public datasets, which often lack high-quality deepfake samples, can potentially impair model performance. To address this, we have investigated more than 30 recent advancements in Text-to-Speech (TTS) and Voice Conversion (VC) models developed in the past few years. Our training dataset now includes deepfake audio samples generated using the top seven TTS models: MetaVoice-1B (Liu et al., ), StyleTTS-v2 (Li et al., ), VoiceCraft (Peng et al., ), WhisperSpeech (Radford et al., ), VokanTTS, XTTS-v2 (Casanova et al., ), and Elevenlabs. We use four datasets—VCTK (Yamagishi, ), LibriSpeech (Panayotov et al., ), In-The-Wilds (Müller et al., ), and AudioSet (Gemmeke et al., )—to generate deepfake audio. These datasets include both clean, high-quality and noisy, low-quality real audio, ensuring that the deepfake audio produced is highly diverse and accurately reflects real-world conditions.

Figure: Figure 3: Performance of the RawNet3 baseline model on various datasets. ‘Ours (train, w/o new)’ represents our training dataset after removing all high-quality deepfake samples.
Refer to caption: x3.png

Reference data: For both fake and real audio, having only high-quality samples is insufficient. A broader diversity of samples is essential for the training dataset. Thus, for real audio, we utilize portions from six public audio datasets: VCTK, LibriSpeech, AudioSet, ASRspoof2019 (Todisco et al., ), Voxceleb1 (Nagrani et al., ), and ASRspoof2021 (Liu et al., ), with half consisting of clean audio and the other half of noisy audio. For fake audio, we include two low-quality fake audio datasets: WaveFake (Frank & Schönherr, ) and ASRspoof2019 to further enhance the diversity of the training material.

### 3.2 Testing dataset

Our test dataset comprises approximately 7,000 samples, balanced equally between real and fake audio. Real audio samples are sourced from recent celebrity speeches and conversational videos on platforms such as YouTube. For the fake audio, we not only utilize samples created using the seven latest TTS models but have also expanded our dataset to include contributions from eight of the most advanced AI voice synthesis models or commercial software currently available, namely CosyVoice (Du et al., ), Resemble, Speechify, LOVO AI, Artlist, and Lipsynthesis. Additionally, this set includes fake audio directly collected from social media platforms like YouTube and X, further enriching the dataset with a diverse range of real-world scenarios. This comprehensive composition is strategically designed to rigorously test the generalization capabilities of our model.

Insight: Figure 3 shows that training with previous public datasets yields lower accuracy on ours. Additionally, removing high-quality deepfake samples from our training set significantly also reduces accuracy, highlighting their importance.”

## 4 Method

In this section, we present our selective adversarial training approach, F-SAT. We also present a taxonomy of the most common corruptions and attacks in audio processing, which can be used for robust evaluation in realistic settings. Additionally, we discuss the implementation of Rand-Augmentation for audio to further enhance the robustness of our detection system.

hodologies.

### 4.1 F-SAT: Frequency-Selective Adversarial Training

Let ${\bf x}$ be a waveform input audio, and ${\bf y}$ be its ground-truth category label, To perform classification, neural networks commonly learn to predict the category $\hat{{\bf y}}=\mathcal{F}_{\theta}({\bf x})$ by optimizing the cross-entropy $H(\hat{{\bf y}},{\bf y})$ between the predictions and the ground truth. We use RawNet3 (Jung et al., ) that can process waveform audio input. The network parameters $\theta$ are estimated by minimizing the expected value of the objective:

$$ $\mathcal{L}_{c}(\mathbf{x},\mathbf{y})=H(F_{\theta}(\mathbf{x}),\mathbf{y}),$ $$

Time domain Attack:
Adversarial attacks on audio can be directly added to the waveform.
Let the additive perturbations be $\bm{\delta}$. For attacks in the time domain, we directly add $\bm{\delta}$ to the waveform ${\bf x}$. Due to the huge amount of freedom of the attack vector $\bm{\delta}$, the added attack vectors are often high frequency. We formulate this attack as:
${\bf x}^{\prime}={\bf x}+\bm{\delta}$.

Frequency Domain Attack:
Attacks can also be applied in the frequency domain, accessed through a reversible Fourier transformation of the waveform. We transform the waveform ${\bf x}$ to the frequency domain ${\bf X}$ using the Short-Time Fourier Transform (STFT). The attack modifies ${\bf X}$ by adding $\bm{\delta}$, yielding ${\bf X}^{\prime}={\bf X}+\bm{\delta}$. The Inverse Short-Time Fourier Transform (ISTFT) then reverts ${\bf X}^{\prime}$ back to the time domain, creating the manipulated audio waveform ${\bf x}^{\prime}$.

Figure: Figure 4: F-SAT Pipeline
Refer to caption: x4.png

Frequency-Selective Attack:
Compared to time-domain attacks, frequency-domain attacks provide an inductive bias that enables the crafting of more controlled perturbations, such as those within a restricted bandwidth. Building on this, we introduce a Frequency-Selective Attack that targets specific frequency ranges within the magnitude component of a waveform. We define this attack as

$$ ${\bf x}^{\prime}=s({\bf x},\bm{\delta},f_{l},f_{u})$ $$

where $f_{l}$ and $f_{u}$ represent the lower and upper frequency boundaries of the range where the attack is to be applied. The procedure can be described as follows:

Starting with STFT result ${\bf X}$, we first compute its magnitude component ${\bf X}_{\rho}$ and phase component ${\bf X}_{\phi}$. To map the frequency range to the index range of the STFT spectrum, we calculate the lower and upper boundary indices $r_{l}$ and $r_{u}$ of the FFT spectrum using formulas:

$$ $r_{l}=\left\lfloor\frac{f_{l}\times n_{\text{fft}}}{sr}\right\rfloor,\quad r_{ u}=\left\lceil\frac{f_{u}\times n_{\text{fft}}}{sr}\right\rceil,$ $$

where $n_{\text{fft}}$ is the number of FFT points, and $sr$ is the sampling rate of the original signal ${\bf x}$.
We then define a mask function $M$ using a diagonal matrix $\mathbf{D}$ such that

$$ $\mathbf{D}_{kk}=\begin{cases}1&\text{if }r_{l}\leq k\leq r_{u}\\ 0&\text{otherwise}\end{cases}$ $$

This matrix $\mathbf{D}$ selectively targets the desired frequency components within the specified range $[r_{l},r_{u}]$ for perturbation. The selective perturbation is then defined as:

$$ $\bm{\delta_{s}}=M(\bm{\delta},r_{l},r_{u})=\mathbf{D}\cdot\bm{\delta}.$ $$

By adding this perturbation $\bm{\delta_{s}}$ to the magnitude component of the spectrogram, the perturbed spectrogram can be represented as:

$$ ${\bf X}^{\prime}=({\bf X}_{\rho}+\bm{\delta_{s}})\cdot e^{j{\bf X}_{\phi}}=({ \bf X}_{\rho}+\mathbf{D}\cdot\bm{\delta})\cdot e^{j{\bf X}_{\phi}}$ $$

Finally, we employ the ISTFT to convert the attacked spectrogram ${\bf X}^{\prime}$ back into the time-domain signal ${\bf x}^{\prime}$, thus generating the attacked audio waveform ${\bf x}^{\prime}$.

Moreover, attackers design these worst-case perturbations to disrupt a trained network $\mathcal{F}_{\theta}$, aiming to maximize misclassifications by optimizing the objective:

$$ ${\bf x}^{\prime}=\arg\max_{{\bf x}^{\prime}}\mathcal{L}_{c}({\bf x}^{\prime}, \mathbf{y}),\quad\text{s.t.}\quad\|{\bf x}^{\prime}-{\bf x}\|_{q}<\epsilon,$ $$

where the perturbation $\bm{\delta}={\bf x}^{\prime}-{\bf x}$ is constrained by the $q$ norm to be less than $\epsilon$, ensuring minimal deviation.

F-SAT:
Based on the Frequency-Selective Attack, we propose F-SAT, an adversarial training method which optimizes perturbations in the frequency domain by targeting the magnitude component within specific frequency ranges.

$$ ${\bf x}^{\prime}_{n+1}=\Pi_{{\bf x}+S}\left(x^{\prime}_{n}+\alpha\cdot M\bigg{ (}\text{sgn}\big{(}\nabla_{{\bf x}^{\prime}}\mathcal{L}(s(\mathbf{x},\bm{ \delta},f_{l},f_{u}),y)\big{)},f_{u},f_{l}\bigg{)}\right),$ $$

where $\Pi_{\mathbf{x}+S}$ represents the projection onto the allowable perturbation set $S$, defined by the condition $\|\mathbf{x}^{\prime}-\mathbf{x}\|_{p}\leq\epsilon$. The parameter $\alpha$ denotes the step size of the update, and $\nabla_{\mathbf{x}^{\prime}}\mathcal{L}$ is the gradient of the loss function with respect to the perturbed input $\mathbf{x}^{\prime}$.

As illustrated in Figure 4, after $K$ iterations (where $n\geq K$), we feed the most perturbed sample back into the detection model and calculate the cross-entropy loss between it and the ground truth label:

$$ $\mathcal{L}_{robust}(\mathbf{x^{\prime}},\mathbf{y})=H(F_{\theta}(\mathbf{x^{ \prime}}),\mathbf{y}),$ $$

Additionally, to maintain a balance between accuracy on original and attacked samples, we train the model using both the original and perturbed samples. We introduce a parameter $\bm{\gamma}$ to control the trade-off between clean loss and robust loss. The clean loss is defined as:

$$ $\mathcal{L}_{clean}(\mathbf{x},\mathbf{y})=H(F_{\theta}(\mathbf{x}),\mathbf{y}),$ $$

The total loss $\mathcal{L}_{total}$ is then computed as a weighted sum of clean and robust losses.

$$ $\mathcal{L}_{total}=\mathcal{L}_{clean}+\bm{\gamma}\cdot\mathcal{L}_{robust}$ $$

Since time domain attacks are often high frequency, by focusing the adversarial training on the high frequency, our approach not only enhances model robustness against frequency-targeted attacks, but also improves defenses against time-domain attacks.

Figure: Figure 5: Overview of various corruption types and adversarial attack strategies affecting audio robustness. The diagram categorizes different forms of corruptions (e.g., noise, filtering, distortion) and adversarial attacks (e.g., white-box, black-box) based on their methods, objectives, and scope of perturbation. This framework outlines the challenges in ensuring the robustness of audio systems against both environmental corruption and intentional adversarial manipulation.
Refer to caption: x5.png

### 4.2 Randaugment for audio

Realistic Corruptions for Deepfake Audio Generation: As shown in Figure 5, we present a taxonomy for analyzing the robustness of deepfake audio detection systems that incorporate the most common corruptions and attacks. Corruptions in audio result from unintentional modifications during recording, processing, or transmission, impacting noise levels and frequency responses. Adversarial attacks, in contrast, are intentional manipulations aimed at deceiving detection systems through subtle changes across various attack methodologies.

RandAugment: Experiments show that the best detection model, RawNet3, experiences a great drop in accuracy when faced with audio corruptions. Inspired by the image-based RandAugment (Cubuk et al., ), which improves model robustness, we adapted this method for audio.
We selectively apply $\mathcal{N}$ transformations from the available options, each assigned a uniform probability. An additional probability $\mathcal{p}$ determines whether each transformation is applied at a given instance, to balance the accuracy between original and corrupted samples. The magnitude of each transformation is controlled within predefined boundaries, with the intensity randomly sampled from this range.

## 5 Experiments

We first compare our approach to existing state-of-the-art methods across three benchmarks and demonstrate improved accuracy. We then assess its robustness against corruption and adversarial attacks. We finally conduct ablation study on enhancements to the detection system’s robustness.

### 5.1 Dataset

All models are trained on our training dataset and then evaluated on various benchmark datasets to test their generalization capabilities.

DeepFakeVox-HQ (test) Our custom test dataset incorporates 14 of the latest and highest quality TTS and VC models to generate fake audio. It also includes fake audio samples directly collected from social media platforms such as YouTube and X, providing a diverse set of real-world scenarios.

ASVspoof2019 (Todisco et al., ) derived from the VCTK base corpus, which includes speech data captured from 107 speakers. It contains three major forms of spoofing attacks, namely synthetic, converted, and replayed speech.

WaveFake (Frank & Schönherr, ) Composed using six neural vocoders, this dataset features fake audio produced by MelGAN, Parallel WaveGAN, MB-MelGAN, FB-MelGAN, HiFi-GAN, and WaveGlow.

### 5.2 Baseline

RawNet2 (Tak et al., ) employs Sinc-Layers (Ravanelli & Bengio, ) to directly extract features from audio waveforms. These layers function as band-pass filters that enhance the detection of spoofed audio content.

RawGAT-ST (Tak et al., ) utilizes spectral and temporal sub-graphs integrated with a graph pooling strategy, effectively processing complex auditory environments.

TE-ResNet (Zhang et al., ) processes synthetic speech detection by first extracting MFCC from input speech. These coefficients are used as features for a CNN that extracts spatial features, followed by a Transformer that analyzes these to detect characteristics of synthetic speech effectively.

RawNet3 (Jung et al., ) begins by using parameterized filterbanks to extract a time-frequency representation from the raw waveform. This is followed by three backbone blocks with residual connections, a structural approach that deviates from ECAPA-TDNN (Desplanques et al., ).

### 5.3 Main Result

Results on Original Data:
We trained all models on DeepFakeVox-HQ and tested them across three benchmarks: our test set, ASVspoof2019, and WaveFake, as shown in Table 2. Our method achieved state-of-the-art results across all three benchmarks. Compared with RawNet3, our method shows improvements of 7.7% points on DeepFakeVox-HQ (test), 8.4% points on ASVspoof2019, and 0.1% points on WaveFake.

Moreover, our baseline is divided into two categories: waveform-processing models, such as RawGAT-ST, RawNet2, and RawNet3, and the spectrum-processing TE-TesNet. Compared to the spectrum-based TE-TesNet, which only performs well on simpler datasets like WaveFake, the waveform models demonstrate superior effectiveness on our more diverse and complex dataset. This indicates that models with raw audio inputs are better equipped to capture detailed features in complex scenarios.

Figure 6 outlines results across various AI-voice synthesis models on our test set. For the fake sources are included in the training, all models demonstrated high accuracy . However, for unforeseen source, our method significantly outperformed others, particularly those from real-world social media platforms like YouTube and Lipsync, by up to 50% points, highlighting its superior generalization capabilities on out-of-distribution data.

Figure: Figure 6: Detailed results across different AI-voice synthesis models on our test set. Real audio is directly sourced from recent YouTube content. Fake audio is divided into sources not included in training and those that are. Models evaluated include RawNet2, RawNet3, and Ours.
Refer to caption: x6.png

Table: Table 2: Comparative performance of our method and baseline models on our test set, ASVspoof2019, and WaveFake. (Note: WaveFake exclusively contains deepfake samples generated from LibriSpeech. We use a portion of LibriSpeech as the real sample for comparison. Results for our test set are calculated using only those deepfake samples whose sources were not covered during training.)

Figure: (a) Performance across various corruptions
Refer to caption: x7.png

Result on Corrupted data: Figure 7(a) shows detailed results for various corruptions. Our methods are depicted in the green region with F-SAT, and in the red region without F-SAT. They outperform all other baseline methods across 24 types of corruptions, with an average absolute increase of 15.3% points. However, as we can observe, aliasing has the most negative impact on accuracy. During aliasing, audio is intentionally downsampled and then resampled back, which not only removes high-frequency features but also introduces distortions characteristic of aliasing. In addition, low-pass filters, time stretching, and noise also significantly impact performance more than other tested corruptions.

Result on Attacked data: Figure 7(b) illustrates detailed results for various adversarial attacks across time and frequency domains. Compared with the baseline model, RawNet3 (blue region) and RawNet3 with random augmentation but without F-SAT (orange region), our methods employing RawNet3 with both random augmentation and F-SAT show enhanced performance against all types of adversarial attacks.

### 5.4 Ablation Study and Analysis

Choice of attack type for F-SAT:
As discussed in Section 4.2, there are three types of adversarial attacks on audio: time domain, frequency domain on magnitude, and frequency domain on phase. All of these can be employed for adversarial training. However, our focus on targeting the magnitude component in attacks is explained in Figure 8(a). Through our evaluation, we found that attacks based on magnitude in the frequency domain are particularly effective at degrading model performance. This is due to the fact that changes in magnitude directly affect the amplitude of audio signals, crucial for maintaining core acoustic features. Such changes alter the sound intensity across frequencies, complicating the model’s task of distinguishing key characteristics between real and fake audio. Conversely, phase attacks have minimal impact on the model’s predictions, as phase alterations primarily influence spatial audio perception and do not significantly affect feature detection.

We further validate our conclusions experimentally, as shown in Table 3. Our findings indicate that F-SAT excels in adversarial training against attacks on both time and phase domains, especially for attacked samples. Surprisingly, incorporating adversarial training focused on the time domain appears to diminish overall accuracy on original data and does not effectively enhance robustness against attacks in either domain. Direct attacks in the time domain may disrupt most features critical for differentiating between real and fake audio, potentially overburdening the model during training.

Additionally, the introduction of RandAug to the RawNet3 baseline model significantly enhances accuracy on corrupted data, suggesting that the model benefits from exposure to varied conditions during training.

Table: Table 3: Ablation study evaluating the impact of our specifically designed RandAug, F-SAT, and its comparative effects against time-domain adversarial training and phase-based adversarial training. (AT is an abbreviation of Adversarial Training)

Figure: (a) Impact of Different Attack Types
Refer to caption: x9.png

Impact of frequency range for F-SAT:
The importance of high-frequency components is underscored in Figure 8(b), which shows the model’s performance under gradient-based adversarial attacks across different frequency ranges. Attacks targeting higher frequencies notably degrade deepfake detection more than lower frequencies, underscoring the vulnerability of high-frequency features. By focusing attacks on high-frequency regions, we preserve low-frequency integrity, simplifying the model’s task of distinguishing deepfake audio features. This approach maintains high accuracy on unattacked data and improves adversarial robustness.

Further analysis on selecting the optimal frequency range for F-SAT is presented in Table 4. Adversarial training within the 4k to 8k frequency range yields the best performance. We observe that increasing the attack frequency range decreases accuracy on original data due to the distortion of more critical features, adding complexity to the model’s learning process.

Moreover, we observe that narrowing the frequency range of F-SAT increases the accuracy for attacked fake data while decreasing it for attacked real data. This supports the findings shown in Figure 2, which highlight a significant gap between the frequency domains where real and fake audio features predominantly exist. Fake audio features are concentrated in higher frequency ranges compared to real audio. In most real-world applications, criminals aim to make fake audio sound real enough to deceive detectors for committing fraud. Therefore, focusing adversarial training on high frequencies effectively enhances the robustness of fake audio.

Table: Table 4: Result for different range selection for F-SAT.

Figure: (a) Impact of Attack Magnitude $\bm{\epsilon}$ on Detection Accuracy
Refer to caption: x11.png

Balance Between Clean and Attacked Data: Figure 9(a) demonstrates that F-SAT achieves optimal performance with an attack magnitude of $\bm{\epsilon}=0.01$, maintaining high accuracy on both attacked and clean data. Figure 9(b) reveals that a ratio of 0.1 between robust and clean loss, ($L_{robust}/L_{clean}$), results in the highest overall accuracy. The results show that excessive focus on attacked data does not consistently boost robustness and may decrease clean data accuracy. Similarly, an undue emphasis on clean data does not reliably enhance accuracy and can weaken robustness. Thus, striking a balance between clean and attacked samples is critical for optimal model performance.

## 6 Conclusion

In our study, we introduce a dataset *DeepFakeVoc-HQ* that addresses diversity and quality issues in prior datasets, and provide a taxonomy to explore common audio corruptions and attacks. We find that leading AI voice detection models depend on vulnerable high-frequency features. This discovery leads us to develop F-SAT, a targeted adversarial training method that focuses on high-frequency components while while preserving the integrity of low-frequency features. Our approach effectively maintains accuracy on unattacked data and enhances robustness against various attacks. These results pioneer robust training for detecting fake audio for the first time, opening up a new direction for identifying such threats.

#### Acknowledgments

This work was supported in part by multiple Google Cyber NYC awards, Columbia SEAS/EVPR Stimulus award, and Columbia SEAS-KFAI Generative AI and Public Discourse Research award.

## Appendix A Appendix

### A.1 explanation of some types of corruptions

Air Absorption:
Air absorption refers to the phenomenon where high-frequency sound waves are more strongly attenuated by air molecules. As sound travels through the air, it loses energy, particularly at higher frequencies, due to air viscosity and thermal conduction.

Room Simulator:
A room simulator is a digital tool that emulates the acoustics of different rooms or spaces. It includes parameters such as room size, wall materials, and shape, which affect how sound reflects and diffuses.

Peaking:
Peaking refers to adjusting a specific range of frequencies around a central frequency. It is commonly used in parametric equalizers.

Aliasing:
Aliasing occurs when high-frequency components are sampled below the Nyquist rate, causing them to appear as lower frequencies. This can be represented using the Nyquist theorem:

$$ $f_{\text{sample}}>2\cdot f_{\text{max}},$ $$

where $f_{\text{sample}}$ is the sampling frequency, and $f_{\text{max}}$ is the maximum frequency of the signal. When this condition is violated, the signal is folded back into the lower frequencies, creating aliasing artifacts.

Bit-Crush:
Bit-crushing reduces the bit depth of an audio signal, causing quantization errors.

Tanh Distortion:
Tanh distortion is a form of soft clipping achieved using the hyperbolic tangent function. The output $y$ is given by:

$$ $y=\tanh(kx),$ $$

where $x$ is the input signal, and $k$ controls the amount of distortion. As $k$ increases, the function approximates a hard clipping effect.

Gain Transition:
Gain transition refers to smoothly adjusting the amplitude over time.

Seven-Band Parametric EQ:
A seven-band parametric EQ provides independent control over seven frequency bands. Each band can be adjusted using three parameters: center frequency ($f_{0}$), gain ($G$), and bandwidth ($Q$). The overall transfer function is a combination of individual band filters:

$$ $H(f)=\prod_{i=1}^{7}H_{i}(f),$ $$

where $H_{i}(f)$ represents the frequency response of the $i$-th band.

Time Mask:
Time masking involves temporarily silencing or removing a segment of audio.

### A.2 RandAugment for Audio

Figure: Figure 10: Python code for RandAugment for audio
Refer to caption: x13.png

Details of RandAugment for Audio are shown in Figure 10.

### A.3 Comprehensive Analysis of all Attack Types

Table 5 presents the detailed adversarial attack results for RawNet3 and our method under various conditions. It includes both white-box and black-box approaches and examines the impacts of attacks in both the time and frequency domains, detailing the different attack hyperparameters used.

Table: Table 5: Adversarial Attack Results on RawNet3 and Its Variants under Various Conditions. For the attack scenarios, we include both white-box and black-box approaches: $A$ represents tests with the same model and identical weights, while $A^{\prime}$ indicates the same model but with different weights, and $B$ denotes tests on a completely different model. For attacks in the time domain, we use $\bm{\epsilon}=10^{-4}$ and $\bm{\alpha}=4\cdot 10^{-5}$. For attacks in the frequency domain, the parameters are $\bm{\epsilon}=10^{-4}$ and $\bm{\alpha}=4\cdot 10^{-4}$.

### A.4 generalization to specturm-based model

We further investigate whether our method can be generalized to spectrum-based models, despite its relatively inferior performance compared to models utilizing raw waveform inputs. We selected a prior baseline model, TE-ResNet, for evaluation. Unlike models trained on raw waveforms, TE-ResNet exhibits a different vulnerability pattern, with its susceptible features primarily concentrated in the lower frequency ranges. To address this, we applied targeted F-SAT on the 0-4kHz frequency range. The results, as shown in Table 6, indicate that incorporating RandAug into TE-ResNet improves model accuracy on the original data by 6.9%, while the additional F-SAT boosts overall robustness under attack scenarios, enhancing Attack Overall accuracy by 7.7%. These findings suggest that our method can be adapted to spectrum-based models, yielding enhanced performance and robustness.

Table: Table 6: Results of the TE-ResNet model on our dataset, enhanced with RandAug and F-SAT. This table demonstrates how our method adapts to models with spectrum input, showcasing improvements and robustness.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What frequency range is the spec-magnitude attack being applied over in Figure 8a?
1. Do you have any _formal_ explanation for why the time-domain attack is less successful than the spec-magnitude attack? To me this seems counter-intuitive because the STFT is a linear and (mostly) invertible function so, from the perspective of the optimization, it should not matter if the attack was computed in the frequency domain or the time domain. I would be very interested in seeing more explanation for why the time-domain attack is unable to reach the solution acheived by the frequency-domain attack. Please also provide the detailed settings for all the adversarial attacks (time, frequency and phase domain) used in AT, F-SAT and during evaluation.
1.In principle, a frequency selective adversarial attack could be constructed entirely in the time domain by applying a band-pass filter to the adversarial perturbation after each optimization step (i.e. include the BP filter as part of the projection operation). This might be less computationally intensive than the proposed approach. Can you provide some discussion on why the proposed approach was favored?
1. Why is the performance of the model trained on DeepFakeVox-HQ so low on the In-the-wild dataset (see Figure 3)
soundness: 2
strengths: 1. Paper is generally well-written and easy to read but some important details are missing
1. DeepFakeVox-HQ is a novel dataset containing data from prior datasets as well as novel deepfakes generated from SOTA speech synthesis models. I appreciate that the authors have curated a test set containing deepfake generation methods not covered in the training set _as well as deepfakes gathered from the internet_.  I encourage the authors to consider uploading the dataset to a platform like Huggingface Hub.
1. The proposed randaugment data augmentation method is effective at improving deepfake detection for RawNet3 models and is likely to be widely adopted if the source code is easy to use (I looked at the README in the attached supplemental material but did not find any instructions for the augmentation).
1. The proposed adversarial training method improves deepfake detection accuracy on clean and adversarially perturbed recordings (though I have some reservations regarding the experimental setup).
summary: The paper attempts to improve deepfake detection by (1) proposing a large training and evaluation dataset called DeepFakeVox-HQ containing diverse synthetic and real speech recordings, (2) proposing a data augmentation method similar to randaugment for deepfake detectors and (3) proposing a frequency-selective adversarial training (F-SAT) method to make deepfake detectors more robust to adversarial attacks. 

DeepFakeVox-HQ is a large dataset containing real and synthetic speech from existing datasets, speech generated by SOTA speech synthesis models as well as deepfakes found in-the-wild (on social media, etc.). Results show that models trained on DeepFakeVox-HQ generally perform better on existing deepfake while the models trained on the existing datasets have weak performance on DeepFakeVox-HQ, which indicates that DeepFakeVox-HQ includes information that prior works do not provide. DeepFakeVox-HQ will likely be a useful resource in deepfake research. 

The proposed RandAugment scheme for deepfake detection utilizes a large bank of audio augmentations during training and yields significant improvements in deepfake detection accuracy.

The key contribution of F-SAT is to add adversarial perturbations to only certain frequency bands, which apparently results in lesser degradation of accuracy on un-perturbed data while providing greater robustness than standard adversarial training.
weaknesses: 1. Some important details about the proposed approaches are not mentioned in the paper. 
   
   1. The value of $\epsilon$ and $p$ (or $q$) used in adversarial training methods should be mentioned in the main body of the paper. Currently, it is mentioned in the caption of a table in the appendix
   1. The settings used for adversarial attacks during AT, F-SAT and evaluation need to be mentioned.
   1. The parameters of the augmentations used in randaugment need to be mentioned at least in the appendix.
   1. Detailed composition of DeepFakeVox-HQ needs to be mentioned including 

      1. the method used for generating deepfakes (particularly noisy deepfakes), 
      1. the number of audios from each deepfake generation system, 
      1. the demographic distribution of the real and fake speakers,
      1. the number of utterances used from each of the datasets from prior works, 
      1. quantitative measurement of synthetic speech quality using metrics like DNSMOS, or NORESQA.

1. The choice of accuracy as a metric seems to be inappropriate for a binary classification task. I would suggest using F1-score and equal error rate as the metrics. Moreover, reading tables and plots with two accuracy metrics for accuracy is a little confusing.
1. Conducting adversarial attacks in the frequency domain and reverting to the temporal domain is not novel and has been done before [2].
1. There is no comparison with other adversarial defenses for audio models. Many of the defenses created for speech and speaker recognition will also apply to the deepfake detection scenario. One method that is quite simple is [3]
1. The common practice is to use signal-to-noise ratio (SNR) as the bound for adversarial attacks in the audio domain [1] instead of $\ell_p$ bounds. I would highly recommend the authors use SNR as well. It is fairly straightforward to convert SNR to $\ell_2$ bounds and vice-versa. The main advantage of using SNR is that one has an idea how _perceptible_ the adversarial attack is.

1. Clarity issues:
   1. The caption Figure 9 needs to state that the results are of F-SAT
   1. Add results for 0-8K in Figure 8b 


[1] Carlini, Nicholas, and David Wagner. "Audio adversarial examples: Targeted attacks on speech-to-text." 2018 IEEE security and privacy workshops (SPW). IEEE, 2018. 

[2] Koerich, Karl Michel, et al. "Cross-representation transferability of adversarial attacks: From spectrograms to audio waveforms." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

[3] Olivier, Raphael, Bhiksha Raj, and Muhammad Shah. "High-frequency adversarial defense for speech and audio." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.Given that F-SAT focuses on high-frequency perturbations, have the authors considered whether these perturbations might be perceptible to human listeners? 




2. Were all baseline models subjected to similar adversarial training procedures as the proposed F-SAT model? Consistency in adversarial training across baseline models is essential to ensure a fair comparison of robustness improvements. If not, would the authors consider including adversarially trained baselines in future comparisons?

3 . How sensitive is F-SAT to the choice of hyperparameters, particularly the frequency ranges and perturbation magnitudes used for adversarial training?
soundness: 3
strengths: 1.DeepFakeVox-HQ stands out as a substantial addition to the field, with over 1.3 million samples, including 270,000 high-quality deepfake samples from 14 sources. This dataset addresses the limitations of existing datasets in diversity and scale, making it a valuable resource for benchmarking future detection models. Releasing this dataset would have a broad impact on the community.

2.The F-SAT method is an important innovation, targeting high-frequency features that are critical for detection but vulnerable to adversarial manipulation. This frequency-focused adversarial training enhances model robustness without compromising accuracy on clean data, addressing a key gap in existing deepfake detection methods.

3.Comprehensive Experimental Evaluation:  
   The experimental design is extensive, evaluating performance across standard benchmarks (ASVspoof2019 and WaveFake) as well as the authors' own test dataset. F-SAT demonstrates clear improvements in robustness across multiple corruption and adversarial attack scenarios. The addition of an ablation study further supports the effectiveness of the proposed method.

4. Extending RandAugment from image processing to audio is an inventive adaptation that helps improve model robustness on both clean and corrupted audio. This demonstrates the authors' resourcefulness in leveraging existing techniques and could be beneficial for future work in audio data augmentation.
summary: This paper addresses the challenge of deepfake audio detection, presenting two major contributions: (1) the creation of DeepFakeVox-HQ, the largest and most diverse public dataset for deepfake audio detection, which enables realistic testing conditions and exposes limitations in existing models, and (2) the introduction of Frequency-Selective Adversarial Training (F-SAT), a novel approach that improves detection robustness by focusing on high-frequency audio components. The work is well-written and logically structured, making complex concepts accessible, and holds significant potential for advancing the robustness and reliability of deepfake audio detection models.
weaknesses: 1. The paper does not specify whether baseline models were subjected to adversarial training. If only the F-SAT model received this enhancement, it could bias the results. Including adversarially-trained versions of baseline models using contemporary adversarial methods would provide a fairer comparison and highlight F-SAT’s unique advantages.

2. While F-SAT’s focus on high-frequency components is intriguing, the rationale behind the reliance on high frequencies for detecting deepfake audio could be further elaborated. 

3.Adversarial training, especially in the frequency domain with iterative updates, can be computationally demanding. Assessing F-SAT's efficiency, particularly compared to baseline models, would improve the paper's practicality.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: My main concern is about the generalization and robustness issues listed above.
I will consider changing my score based on the author's responses and other reviewers.
soundness: 4
strengths: Three main contributions involved in this work include (1) a carefully organized dataset, (2) a deepfake detection method, and (3) the ability against adversarial attacks (with the setting focusing on high-frequency signals).
In general, the contributions of this work are multi-fold.
summary: Glad to review the paper.
This paper proposes a novel method, F-SAT for deepfake audio detection.
The topic of this work is promising, and the paper is easy to follow.
I believe this work has reference values to domain-related researchers.
weaknesses: My major concern is whether the contributions (or advantages) of this work are over-claimed.
Regarding the dataset, although it is well organized and processed, the samples are generated using existing approaches, thus, "the largest" is not a significant contribution.
Regarding generalization, as in Table 2, the significantly superior results of the proposed method are achieved on the self-organized dataset, DeepFake Vox-HQ. However, as the author introduced in Section 3, there are overlapped synthesis methods between training and testing data in this group of results (as in Figure 6). Thus the results in DeepFake Vox-HQ can not indicate out-of-distribution generalization.
Regarding enhancing robustness, in the last paragraph of the related section, the referenced solutions were published in 2019,2018 and 2018, I am not sure whether any recent works focus on the adversarial issue, that should be discussed or compared.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: - Which model was used to create the plot on the left side in Figure 2?
- How many hours of speech does the dataset encompass exactly? How long are the samples?
- Are the samples aligned? Do all models synthesize speech using the same input sentences?
- Is it possible to add a data sheet that outlines the exact sources and utterance lengths per source?
- Are the WaveFake test samples also part of the DeefFakeVox-HQ test set?
- WaveFake contains Japanese language JSUT samples.
    - Are these part of the dataset?
    - Should the Caption of Table 1 make this explicit? Since WaveFake is listed as 
     English-language data set, I assume JSUT is not considered a part of WaveFake in this paper.
    - Do the Utterance numbers in table one exclude JSUT?
    - If yes, should this be mentioned somewhere else?

- Is it possible to include leading works from the audio classification world, like the Audio Spectrogram Transformer (AST)[1], in the evaluations? Related work [2] found it to perform well on the WaveFake-dataset. It would be interesting if it also outperforms other methods on DeepFakeVox-HQ.

- The Wavefake paper [3] trains with binary settings with fake audio from a single source and measures generalization. Training on which source network led to the numbers in Table 2? Are the numbers comparable to the related work?

- Which software libraries have been used to implement this project?

- Which hyperparameters underpin the network training?

Related work:
[1] AST: Audio Spectrogram Transformer, https://arxiv.org/pdf/2104.01778,
[2] Towards generalizing deep-audio fake detection networks, https://arxiv.org/pdf/2305.13033,
[3] WaveFake: A Data Set to Facilitate Audio Deepfake Detection, https://arxiv.org/abs/2111.02813
soundness: 3
strengths: - The paper tackles a significant problem.
- The related work is well-researched and described.
- The adversarial attack perspective is interesting.
- Authors ensure their results are up to date, combining existing datasets with samples from commercial models.
summary: The paper "I CAN HEAR YOU: SELECTIVE ROBUST TRAINING FOR DEEPFAKE AUDIO DETECTION"
introduces the DeepFakeVox-HQ data set, which contains audio from 14 sources.
In addition to the dataset, the authors introduce Frequency-Selective Adversarial Training (F-SAT), a training method that focuses on the high-frequency part of the spectrum. In addition to FSAT, this paper evaluates robustness concerning various input perturbations.
weaknesses: - Traditional compression algorithms like MP3 remove high-frequency content; according to line 84, FSAT focuses on this part of the spectrum.
- If I understand correctly, compression is not part of the corruption set, as shown in Figure 7. Including compression would have been important for real-world applicability.
- Data-set details like the length in hours or training hyperparameters like the learning rate are missing.

Meta-review: This paper makes significant advancements in deepfake detection through three key contributions: (1) Introduction of DeepFakeVox-HQ: A comprehensive dataset comprising high-quality synthetic and real speech recordings, designed to benefit the research community. (2) Development of an enhanced data augmentation method: Tailored to improve the performance of deepfake detectors. (3) Proposal of frequency-selective adversarial training to enhance detector robustness against adversarial attacks. The reviewers have recognized the paper as well-written, with the dataset being a valuable resource for the community and the proposed methods effectively improving detection accuracy. However, the final version should address the reviewers’ suggestions to enhance clarity and reproducibility. Key revisions include: (1) Adversarial training details: Adding the setup specifics for adversarial training and incorporating the Equal Error Rate (EER) as an evaluation metric. (2) Claims revision: Refining the claims of the paper’s contributions, as indicated by Reviewer erKc.
(3) Additional details: Providing more information in response to Reviewer NjHP's feedback. Overall, this work is meaningful to the community but requires these revisions to further ensure clarity and reproducibility.

Decision: Accept (Poster)