I Can Hear You: Selective Robust Training for Deepfake Audio Detection

Authors: Zirui Zhang, Wei Hao, Aroon Sankoh, William Lin, Emanuel Mendiola-Ortiz, Junfeng Yang, Chengzhi Mao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: deepfake audio detection, audio augmentations, frequency-selective adversarial training

Abstract:
Recent advances in AI-generated voices have intensified the challenge of detecting deepfake audio, posing risks for scams and the spread of disinformation. To tackle this issue, we establish the largest public voice dataset to date, named DeepFakeVox-HQ, comprising 1.3 million samples, including 270,000 high-quality deepfake samples from 14 diverse sources. Despite previously reported high accuracy, existing deepfake voice detectors struggle with our diversely collected dataset, and their detection success rates drop even further under realistic corruptions and adversarial attacks. We conduct a holistic investigation into factors that enhance model robustness and show that incorporating a diversified set of voice augmentations is beneficial. Moreover, we find that the best detection models often rely on high-frequency features, which are imperceptible to humans and can be easily manipulated by an attacker. To address this, we propose the F-SAT: Frequency-Selective Adversarial Training method focusing on high-frequency components. Empirical results demonstrate that using our training dataset boosts baseline model performance (without robust training) by 33%, and our robust training further improves accuracy by 7.7% on clean samples and by 29.3% on corrupted and attacked samples, over the state-of-the-art RawNet3 model.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729862159460

thread_type:
official_review

<thread_object>
creator: reviewer_njhp

<content>
summary:
The paper "I CAN HEAR YOU: SELECTIVE ROBUST TRAINING FOR DEEPFAKE AUDIO DETECTION"
introduces the DeepFakeVox-HQ data set, which contains audio from 14 sources.
In addition to the dataset, the authors introduce Frequency-Selective Adversarial Training (F-SAT), a training method that focuses on the high-frequency part of the spectrum. In addition to FSAT, this paper evaluates robustness concerning various input perturbations.

strengths:
- The paper tackles a significant problem.
- The related work is well-researched and described.
- The adversarial attack perspective is interesting.
- Authors ensure their results are up to date, combining existing datasets with samples from commercial models.

weaknesses:
- Traditional compression algorithms like MP3 remove high-frequency content; according to line 84, FSAT focuses on this part of the spectrum.
- If I understand correctly, compression is not part of the corruption set, as shown in Figure 7. Including compression would have been important for real-world applicability.
- Data-set details like the length in hours or training hyperparameters like the learning rate are missing.

questions:
- Which model was used to create the plot on the left side in Figure 2?
- How many hours of speech does the dataset encompass exactly? How long are the samples?
- Are the samples aligned? Do all models synthesize speech using the same input sentences?
- Is it possible to add a data sheet that outlines the exact sources and utterance lengths per source?
- Are the WaveFake test samples also part of the DeefFakeVox-HQ test set?
- WaveFake contains Japanese language JSUT samples.
    - Are these part of the dataset?
    - Should the Caption of Table 1 make this explicit? Since WaveFake is listed as 
     English-language data set, I assume JSUT is not considered a part of WaveFake in this paper.
    - Do the Utterance numbers in table one exclude JSUT?
    - If yes, should this be mentioned somewhere else?

- Is it possible to include leading works from the audio classification world, like the Audio Spectrogram Transformer (AST)[1], in the evaluations? Related work [2] found it to perform well on the WaveFake-dataset. It would be interesting if it also outperforms other methods on DeepFakeVox-HQ.

- The Wavefake paper [3] trains with binary settings with fake audio from a single source and measures generalization. Training on which source network led to the numbers in Table 2? Are the numbers comparable to the related work?

- Which software libraries have been used to implement this project?

- Which hyperparameters underpin the network training?

Related work:
[1] AST: Audio Spectrogram Transformer, https://arxiv.org/pdf/2104.01778,
[2] Towards generalizing deep-audio fake detection networks, https://arxiv.org/pdf/2305.13033,
[3] WaveFake: A Data Set to Facilitate Audio Deepfake Detection, https://arxiv.org/abs/2111.02813
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556828630

thread_type:
official_review

<thread_object>
creator: reviewer_erkc

<content>
summary:
Glad to review the paper.
This paper proposes a novel method, F-SAT for deepfake audio detection.
The topic of this work is promising, and the paper is easy to follow.
I believe this work has reference values to domain-related researchers.

strengths:
Three main contributions involved in this work include (1) a carefully organized dataset, (2) a deepfake detection method, and (3) the ability against adversarial attacks (with the setting focusing on high-frequency signals).
In general, the contributions of this work are multi-fold.

weaknesses:
My major concern is whether the contributions (or advantages) of this work are over-claimed.
Regarding the dataset, although it is well organized and processed, the samples are generated using existing approaches, thus, "the largest" is not a significant contribution.
Regarding generalization, as in Table 2, the significantly superior results of the proposed method are achieved on the self-organized dataset, DeepFake Vox-HQ. However, as the author introduced in Section 3, there are overlapped synthesis methods between training and testing data in this group of results (as in Figure 6). Thus the results in DeepFake Vox-HQ can not indicate out-of-distribution generalization.
Regarding enhancing robustness, in the last paragraph of the related section, the referenced solutions were published in 2019,2018 and 2018, I am not sure whether any recent works focus on the adversarial issue, that should be discussed or compared.

questions:
My main concern is about the generalization and robustness issues listed above.
I will consider changing my score based on the author's responses and other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566031243

thread_type:
official_review

<thread_object>
creator: reviewer_ijuy

<content>
summary:
This paper addresses the challenge of deepfake audio detection, presenting two major contributions: (1) the creation of DeepFakeVox-HQ, the largest and most diverse public dataset for deepfake audio detection, which enables realistic testing conditions and exposes limitations in existing models, and (2) the introduction of Frequency-Selective Adversarial Training (F-SAT), a novel approach that improves detection robustness by focusing on high-frequency audio components. The work is well-written and logically structured, making complex concepts accessible, and holds significant potential for advancing the robustness and reliability of deepfake audio detection models.

strengths:
1.DeepFakeVox-HQ stands out as a substantial addition to the field, with over 1.3 million samples, including 270,000 high-quality deepfake samples from 14 sources. This dataset addresses the limitations of existing datasets in diversity and scale, making it a valuable resource for benchmarking future detection models. Releasing this dataset would have a broad impact on the community.

2.The F-SAT method is an important innovation, targeting high-frequency features that are critical for detection but vulnerable to adversarial manipulation. This frequency-focused adversarial training enhances model robustness without compromising accuracy on clean data, addressing a key gap in existing deepfake detection methods.

3.Comprehensive Experimental Evaluation:  
   The experimental design is extensive, evaluating performance across standard benchmarks (ASVspoof2019 and WaveFake) as well as the authors' own test dataset. F-SAT demonstrates clear improvements in robustness across multiple corruption and adversarial attack scenarios. The addition of an ablation study further supports the effectiveness of the proposed method.

4. Extending RandAugment from image processing to audio is an inventive adaptation that helps improve model robustness on both clean and corrupted audio. This demonstrates the authors' resourcefulness in leveraging existing techniques and could be beneficial for future work in audio data augmentation.

weaknesses:
1. The paper does not specify whether baseline models were subjected to adversarial training. If only the F-SAT model received this enhancement, it could bias the results. Including adversarially-trained versions of baseline models using contemporary adversarial methods would provide a fairer comparison and highlight F-SAT’s unique advantages.

2. While F-SAT’s focus on high-frequency components is intriguing, the rationale behind the reliance on high frequencies for detecting deepfake audio could be further elaborated. 

3.Adversarial training, especially in the frequency domain with iterative updates, can be computationally demanding. Assessing F-SAT's efficiency, particularly compared to baseline models, would improve the paper's practicality.

questions:
1.Given that F-SAT focuses on high-frequency perturbations, have the authors considered whether these perturbations might be perceptible to human listeners? 




2. Were all baseline models subjected to similar adversarial training procedures as the proposed F-SAT model? Consistency in adversarial training across baseline models is essential to ensure a fair comparison of robustness improvements. If not, would the authors consider including adversarially trained baselines in future comparisons?

3 . How sensitive is F-SAT to the choice of hyperparameters, particularly the frequency ranges and perturbation magnitudes used for adversarial training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675115549

thread_type:
official_review

<thread_object>
creator: reviewer_eght

<content>
summary:
The paper attempts to improve deepfake detection by (1) proposing a large training and evaluation dataset called DeepFakeVox-HQ containing diverse synthetic and real speech recordings, (2) proposing a data augmentation method similar to randaugment for deepfake detectors and (3) proposing a frequency-selective adversarial training (F-SAT) method to make deepfake detectors more robust to adversarial attacks. 

DeepFakeVox-HQ is a large dataset containing real and synthetic speech from existing datasets, speech generated by SOTA speech synthesis models as well as deepfakes found in-the-wild (on social media, etc.). Results show that models trained on DeepFakeVox-HQ generally perform better on existing deepfake while the models trained on the existing datasets have weak performance on DeepFakeVox-HQ, which indicates that DeepFakeVox-HQ includes information that prior works do not provide. DeepFakeVox-HQ will likely be a useful resource in deepfake research. 

The proposed RandAugment scheme for deepfake detection utilizes a large bank of audio augmentations during training and yields significant improvements in deepfake detection accuracy.

The key contribution of F-SAT is to add adversarial perturbations to only certain frequency bands, which apparently results in lesser degradation of accuracy on un-perturbed data while providing greater robustness than standard adversarial training.

strengths:
1. Paper is generally well-written and easy to read but some important details are missing
1. DeepFakeVox-HQ is a novel dataset containing data from prior datasets as well as novel deepfakes generated from SOTA speech synthesis models. I appreciate that the authors have curated a test set containing deepfake generation methods not covered in the training set _as well as deepfakes gathered from the internet_.  I encourage the authors to consider uploading the dataset to a platform like Huggingface Hub.
1. The proposed randaugment data augmentation method is effective at improving deepfake detection for RawNet3 models and is likely to be widely adopted if the source code is easy to use (I looked at the README in the attached supplemental material but did not find any instructions for the augmentation).
1. The proposed adversarial training method improves deepfake detection accuracy on clean and adversarially perturbed recordings (though I have some reservations regarding the experimental setup).

weaknesses:
1. Some important details about the proposed approaches are not mentioned in the paper. 
   
   1. The value of $\epsilon$ and $p$ (or $q$) used in adversarial training methods should be mentioned in the main body of the paper. Currently, it is mentioned in the caption of a table in the appendix
   1. The settings used for adversarial attacks during AT, F-SAT and evaluation need to be mentioned.
   1. The parameters of the augmentations used in randaugment need to be mentioned at least in the appendix.
   1. Detailed composition of DeepFakeVox-HQ needs to be mentioned including 

      1. the method used for generating deepfakes (particularly noisy deepfakes), 
      1. the number of audios from each deepfake generation system, 
      1. the demographic distribution of the real and fake speakers,
      1. the number of utterances used from each of the datasets from prior works, 
      1. quantitative measurement of synthetic speech quality using metrics like DNSMOS, or NORESQA.

1. The choice of accuracy as a metric seems to be inappropriate for a binary classification task. I would suggest using F1-score and equal error rate as the metrics. Moreover, reading tables and plots with two accuracy metrics for accuracy is a little confusing.
1. Conducting adversarial attacks in the frequency domain and reverting to the temporal domain is not novel and has been done before [2].
1. There is no comparison with other adversarial defenses for audio models. Many of the defenses created for speech and speaker recognition will also apply to the deepfake detection scenario. One method that is quite simple is [3]
1. The common practice is to use signal-to-noise ratio (SNR) as the bound for adversarial attacks in the audio domain [1] instead of $\ell_p$ bounds. I would highly recommend the authors use SNR as well. It is fairly straightforward to convert SNR to $\ell_2$ bounds and vice-versa. The main advantage of using SNR is that one has an idea how _perceptible_ the adversarial attack is.

1. Clarity issues:
   1. The caption Figure 9 needs to state that the results are of F-SAT
   1. Add results for 0-8K in Figure 8b 


[1] Carlini, Nicholas, and David Wagner. "Audio adversarial examples: Targeted attacks on speech-to-text." 2018 IEEE security and privacy workshops (SPW). IEEE, 2018. 

[2] Koerich, Karl Michel, et al. "Cross-representation transferability of adversarial attacks: From spectrograms to audio waveforms." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

[3] Olivier, Raphael, Bhiksha Raj, and Muhammad Shah. "High-frequency adversarial defense for speech and audio." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

questions:
1. What frequency range is the spec-magnitude attack being applied over in Figure 8a?
1. Do you have any _formal_ explanation for why the time-domain attack is less successful than the spec-magnitude attack? To me this seems counter-intuitive because the STFT is a linear and (mostly) invertible function so, from the perspective of the optimization, it should not matter if the attack was computed in the frequency domain or the time domain. I would be very interested in seeing more explanation for why the time-domain attack is unable to reach the solution acheived by the frequency-domain attack. Please also provide the detailed settings for all the adversarial attacks (time, frequency and phase domain) used in AT, F-SAT and during evaluation.
1.In principle, a frequency selective adversarial attack could be constructed entirely in the time domain by applying a band-pass filter to the adversarial perturbation after each optimization step (i.e. include the BP filter as part of the projection operation). This might be less computationally intensive than the proposed approach. Can you provide some discussion on why the proposed approach was favored?
1. Why is the performance of the model trained on DeepFakeVox-HQ so low on the In-the-wild dataset (see Figure 3)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732512073310

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We ran the experiments asked by the reviewer and updated the paper. Edits are in blue.

comment:
We sincerely thank all reviewers for their thoughtful and insightful feedback. We are encouraged by the recognition of the significance of our dataset and the effectiveness of our robust training methods. We ran all the experiments asked by the reviewers and integrated them to strengthen our paper. We indicate our paper edits in blue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733365738110

thread_type:
public_comment

<thread_object>
creator: ~andrew_c._cullen1

<content>
comment:
Public comment re Ethics

As someone who is not reviewing this paper, I'd just like to flag up that the authors have been sourcing data from social media and YouTube, without appearing to acknowledge that this may be a) against the terms of service of these platforms and b) may constitute human involved research. Both of these factors may be considered as research practices that should require ethical oversight, and, at the very least, demonstration of having been issued appropriate licenses by copyright holders. These issues are exacerbated when one of the contributions of this paper is to produce a dataset that may be distributed, which is almost certainly against the terms of any licenses granted to the authors. 

The Youtube terms of service explicitly state that users are not allowed to 
- "access, reproduce, download, distribute, transmit, broadcast, display, sell, license, alter, modify or otherwise use any part of the Service or any Content except: (a) as expressly authorized by the Service; or (b) with prior written permission from YouTube and, if applicable, the respective rights holders;" 
- "access the Service using any automated means (such as robots, botnets or scrapers) except (a) in the case of public search engines, in accordance with YouTube’s robots.txt file; or (b) with YouTube’s prior written permission;" 
- "use the Service to view or listen to Content other than for personal, non-commercial use (for example, you may not publicly screen videos or stream music from the Service); or"

See: https://www.youtube.com/static?template=terms&gl=AU

All three of those above provisions could be seen to be violated by the authors scraping data from YouTube without a license.

Thanks
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper makes significant advancements in deepfake detection through three key contributions: (1) Introduction of DeepFakeVox-HQ: A comprehensive dataset comprising high-quality synthetic and real speech recordings, designed to benefit the research community. (2) Development of an enhanced data augmentation method: Tailored to improve the performance of deepfake detectors. (3) Proposal of frequency-selective adversarial training to enhance detector robustness against adversarial attacks. The reviewers have recognized the paper as well-written, with the dataset being a valuable resource for the community and the proposed methods effectively improving detection accuracy. However, the final version should address the reviewers’ suggestions to enhance clarity and reproducibility. Key revisions include: (1) Adversarial training details: Adding the setup specifics for adversarial training and incorporating the Equal Error Rate (EER) as an evaluation metric. (2) Claims revision: Refining the claims of the paper’s contributions, as indicated by Reviewer erKc.
(3) Additional details: Providing more information in response to Reviewer NjHP's feedback. Overall, this work is meaningful to the community but requires these revisions to further ensure clarity and reproducibility.