SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection
Authors: Yi Zhu
Surya Koppisetti
Trang Tran
Gaurav Bharaj
Venue: NeurIPS 2024 poster (2024)
Keywords: Deepfake detection
anti-spoofing
synthesized speech
generalization
explainability
self-supervised learning
Primary area: speech_and_audio

Abstract: Audio deepfake detection (ADD) is crucial to combat the misuse of speech synthesized by generative AI models. Existing ADD models suffer from generalization issues to unseen attacks, with a large performance discrepancy between in-domain and out-of-domain data. Moreover, the black-box nature of existing models limits their use in real-world scenarios, where explanations are required for model decisions. To alleviate these issues, we introduce a new ADD model that explicitly uses the Style-LInguistics Mismatch (SLIM) in fake speech to separate them from real speech. SLIM first employs self-supervised pretraining on only real samples to learn the style-linguistics dependency in the real class. The learned features are then used in complement with standard pretrained acoustic features (e.g., Wav2vec) to learn a classifier on the real and fake classes. When the feature encoders are frozen, SLIM outperforms benchmark methods on out-of-domain datasets while achieving competitive results on in-domain data. The features learned by SLIM allow us to quantify the (mis)match between style and linguistic content in a sample, hence facilitating an explanation of the model decision.

SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection
Yi Zhu Surya Koppisetti
Trang Tran Gaurav Bharaj
Reality Defender Inc.
surya, wrw.gaaaaaaaaeeerraaaaaaaaeeeeee ai
Abstract
Audio deepfake detection (ADD) is crucial to combat the misuse of speech synthe- sized by generative Al models. Existing ADD models suffer from generalization issues unseen attacks, with large performance discrepancy between in-domain and out-of-domain data. Moreover, the black-box nature of existing models limits their use real-workh scenarios, where explanations are required for model de- cisions. To alleviate these issues, we introduce a new ADD model that explicitly uses the Inguustiss Mismatch (SLIM) in fake speech to separate them from real speech. SLIM first employs self-supervised pretraining on only real samples learn the dyle-linguisties dependency the real class. The learned features are then used in complement with standard pretrained acoustic features (e.g.. Wav2vec) learn a classifier on the real and fake classes. When the feature encoders are frozen, SL.M outperforms benchmark methods on out-of-domain datasets while achieving competitive results on in-domain data. The features learned by SL.MM allow us to quantify the (mis)match between style and linguistic content in a sample, hence facilitating an explanation of the model decision.
Introduction
The growing interest in generative models has led to an expansion of publicly available tools that can
closely mimic the voice of a real person [73]. Text-to-speech (TTS) voice conversion (VC) systems can now be used to synthesize a fake voice from only a few seconds of real speech recordings [72].
When these generation tools used by bad actors, the generated fake human voices (hereinafter referred to as audio deepfakes) can pose serious dangers. Examples include impersonation of celebrities/family members for robocalls [29), illegal access voice-guarded bank accounts [14], or forgery of evidence in court [56]. Reliable audio deepfake detection (ADD) tools are therefore urgently needed.
State-of-the-art (SOTA) detection systems 81 993 employ self-supervised learning (SSL) encoders
as the frontend feature extractors, and append classification backends to map the high- nigh-dimensional feature representations to binary real/fake decision [35
93 8T] Common SSL encoders for this task the Wav2vec [7]. WavLM (II), and HuBert (25 among others. These models are usually trained
in suply-super manner, with fake samples generated using off-the-shelf TTS/VC tools [34, [93]
12 81 52 92 28). However, current ADD systems known to underperform
deepfakes crafted To tackle this issue, some works have focused extracting more robust features from the input representation 22 71 92]. Additional
by unseen generative models (i.e., unseen attacks) 400 51 67 93
improvements have been reported by finetuning the SSL frontend during downstream supervised training 77 8T 4 and by increasing the diversity of labeled samples via data augmentation or
82 83]. While shown to be effective for in-domain deepfakes, frontend finetuning increases the of training drastically.
continual training on vocoded data [70] 87
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Additionally, outputs from existing ADD systems are hard to explain, i.e., it is unclear to a typical user why an ADD makes certain prediction, which leads to lack of trust 993 For practical applications, it is crucial to understand what information the model is relying on to make decisions, and under which cireumstances the model would fail to successfully detect deepfakes.
group
of works uses explainable AI (xAI) methods 4 to interpret model decisions 32 (6), but they mainly rely on post-hoc visualizations such as saliency maps, which are known to be sensitive to training set-ups [90] and can therefore be inconsistent. Other models focus on specific vocal attributes, such as breath [33], or vocal tract [0] to derive explanations. However, most of these interpretable attributes only account for subset of scepfakeerelated characteristics, hence resulting in a large gap in detection performance compared to SOTA methods [93] 34]. We note that while piderpettbilityxexpliiaabllll is often ambiguous (39,. in this work we mean the model's ability to provide reasons for a certain prediction, e.g., a sample is likely fake because its style and linguistics representations are more misaligned than those of real samples (as will be shown in Section 4.2)
[93]
In this study, we propose a generalizable ADD model that explicitly explores the style-linguistiss mismatch in deepfakes to separate them from real audios, and thereby facilitates an explanation on the model decision. We hypothesize that in real speech, a certain dependency exists between the linguistics information embedded in the verbal content and the style information embedded in the vocal attributes, such as speaker identity and emotion. To synthesize a deepfake audio, both TTS and VC systems artificially combine the verbal content with the vocal attributes of target speaker, and thereby introduce an artificial Our two-stage framework explicitly studies the Lyyle-Linguistiss Mismatch (SLIM) in the fake class to separate it from the real class. During Stage 1, the style-linguistie dependency in the real class is learned by contrasting the style and linguistic subspace representations and generating set of dependency features from each subspace. The learned pairs of style and linguistics features are expected be more correlated for real speech than for deepfake speech. In Stage 2, we employ supervised training, where we fuse the learned dependency features from Stage I with the original style and linguistic representations and train light-weight projection head to classify the input representations as real or fake.
dependency that would differ from the real speech.
Our main contributions are summarized as follows:
We propose SLIM model, which leverages Style and LInguistics Mismatch deepfakes to achieve better generalization to unseen attacks than existing models, and has interpretation capability.
2. SLIM outperforms existing SOTA methods on out-of-domail datasets (In-the the-wild, MLAAD) while being competitive on in-domain datasets (ASVspoof 2019, 2021). This is achieved without increasing the amount of labeled data or the added cost from end-to-end finetuning.
3. Unlike black-box box ADD models, the style-linguistics features learned by SLIM can be used to interpret model decisions. We present analyses to show how the interpretation can be performed on a group level as well as on individual speech samples.
2 Related works
2.1 Audio deepfake detection
Kate-of-thee-an ADD systems mainly rely on fully-supervise training, where the model architectures comprise of one or more speech SSL frontends and backend classifier 193 2 45). Guo et al. 23 developed multi-fusion attentive classifier to process the output from a WavLM frontend; Yang et al. (92) fused outputs from multiple SSL. frontends and reported improvements over using single frontend. However, existing ADD systems experience severe degradation in performance when tested on unseen data [51 67). which questions their applicability and trustworthiness for real-world scenar- ios. To address this issue, multiple works have explored methods to improve model generalizability. With added training cost, improvements have been reported when frontends are finetuned alongside the backend classifiers during downstream training [71 [1]. Further improvements were achieved with data augmentations such RawBoost [70] 71] and neural vocoding [82]. More recent works also show that distilled student models can generalize better than large teacher models [3] 83 Still,
large discrepancies between in-domain and out-of-domain performance are common 93 34
In addition to generalization, existing ADD models also fall short on interpretability. Several studies have shown that current SOTA models may be focusing artifacts introduced in the frequency
2

---

domain during voice synthesis and/or the artifacts in non-speech segments [67 50 40, 96 While line of work proposed to extract speech-related features, such as breath [33] vocal tract, and articulatory movement [10], the overall detection performance was inferior to SSL -SLee methods. Other works resort to xAI methods - for model interpretation, such as SHAP 21,, GradCAM (2)
and Deep Taylor [36]. However, these post-hoc analysis approaches are known to be sensitive to training set-ups [90] and therefore not viable for practical use. Both generalization and interpretability remain challenging issues for current ADD systems.
2.2 Style-linguisties modelling
One standard approach for modelling speech is to decompose it into two subspaces, style and linguistics. The former refers short and long-term paralinguistic attributes, such as speaker identity, emotion, accent, and health state [65]. The latter corresponds to the verbal content of speech [31]. For representing style information, early works relied on handcrafted features, such as GeMAPS 19 18. Later studies showed improved performance by representations learned end-to- end by deep neural networks (DNN), such as the x-vector [68] and ECAPA- TDNN embeddings [[6]. Similarly, the linguistic representations follow similar trend where DNN-based embeddings, such as Whisper [60]. outperform handcrafted features for content-relan rellnted tasks [17.. More recent studies have shown that style and linguistics information can be efficiently encoded together in the SSL representations 7 11 25 To investigate how speech information is encoded in DNNs, a group of works conducted layer- analysis and showed that early to middle layers carry more style related attributes, such as speaker identity [ emotion [63] and articulatory movement [3] while later
layers encode linguistics attributes, such phonetic information and semanties 55 66
Despite these approaches, unclear if completely disentangling style and linguistics information in speech is possible. Studies have shown that a certain dependency exists between these two subspaces: the link between emotional states and word choices (38). the relation between prosody and language understanding (15), and the impact of age sentence coherence [58]. Effectively modeling both the independent and dependent aspects of style and linguistics in speech still remains a challenge.
3 Method
3.1 Motivation
For the majority of generative speech models, the style and línguistic subspaces are assumed to be independent of each other [73 27 78 48). For example, VC systems change the voice of an
utterance by replacing the source speaker's embeddings with those of the target speaker [48] (8). assuming that these embeddings contain no linguistics information. Similarly, modern TTS systems rely on independently Iearned representations to model different speech aspects (c.g.. text, speaker, emotion) to synthesize expressive speech (8 [6 77). Because of this disentanglement assumption, a mismatch likely exists between the style and linguistics information in TTS/VC speech that differentiates it from real speech. To study this hypothesis, we conduct proof-of-concept experiment on a sample subset of ASVSpoof2019 75 Following previous research [61 30 54], we use canonical correlation analysis (CCA) to derive subspace where the linear projections of the style and linguistics embeddings are maximally correlated for the real class. We choose the last layer output of pretrained warrveee-large-leee -o--rr---l-llllllllll----l (22) for linguistics representation, and the pretrained ECAPA HAPA--DDNM embeddings [16] for style representation.
Table 1: Mean and standard deviation of Pearson correlation coefficients (r) calculated between style and linguistics embeddings for real and TTS/VC samples across 5 unseen speakers. Significant difference (as per Welch's t-test) is seen between real speech and all types of generated speech.
Class
Real
A01 (TTS) A02 (TTS) A03 (TTS) A04 (TTS) A05 (VC) A06 (VC)
.38...025
.202+.033
.2174.020
.2444.024
.2534.021
.2141.026
.252+.020
We randomly select 100 real speech samples from ASVspoof2019 75 training set to fit 20-dim CCA features for both linguistics and style representations. We then apply the CCA projection to
3

---

between these projected style and linguistics vectors to quantity the subpace similarities. Details of the tested TTS/VC systems can be found in Appendix A. Table Table[sshows these results: A higher is seen for the real samples, whereas significantly lower correlations are observed for both TTS and VC generated samples. Moreover, -amples on average show lower (0.228) than VC-samples (0.236), indicating that VC-samples are closer to real speech in terms of style-inguisttiss dependency. This could explain why VC samples were found to be more challenging to detect than TTS samples the ASVspoof2019 challenge [40]. While our findings demonstrate the usefulness of CCA for validating the subspace mismatch, its limitations, such that it only explores the linear composites of the variables [85], might make sub-optimal to be used independently for deepfake detection. mismatch and 200 audios from 5 unseen speakers and 6 TTS/VC systems, and compute the correlation values
We therefore develop detection framework that explicitly studies the scales to larger amount of data.
3.2 Formulation of SLIM
Our two-stage Inngussis Mismatch (SLIM) learning framework is outlined in Figure The first stage operates on the real class only and employs self-supervised learning to build style and linguistic representations and their dependencies for real speech. In the second stage, classifier is fit onto the learned representations via supervised training over deepfake datasets with binary (real/fake) labels.
Stage-1
Lstyle
Multi-source bonafide samples
Pretrained style encoder
Compress
Leross
Stage-1 I objective
Pretrained linguistic encoder
Frezze
Compress
Finetune
Linguistics
Stage-2
Pretrained style encoder
ASP +MLP
Labelled doepfake & bonafide samples
Compress
Head
0
Compress
170
LICE Stage-2 objective
Pretrained linguistic encoder
ASP +MLP
Figure 1: SLIM - A two-stage training framework for ADD. Stage extracts style and linguistics representations from frozen SSL encoders, compresses them, and aims to minimize the distance between the compressed representations (Leross), as well as the intra-subspace redundancy (L and Liinguistica). The Stage features and the original subspace representations (pretrained SSL embeddings) are combined in Stage 2 to learn a classifier via supervised training.
3.2.1 Stage 1: One-class self-supervised contrastive training
The goal of the first stage is to learn pairs of dependency features from style and linguistics subspaces, which are expected to be highly correlated for real samples and minimally correlated for deepfakes. Since only real samples are needed, we incorporate other open-source speech datasets to diversify the style variations. Given a speech sample, we first extract the style and linguisties representations separately using pretrained networks. Since recent SSL models achieve superior performance

---

multiple speech downstream tasks compared to conventional speech representations (e.g., ECAPA- TDNN)
91 57 we select group of SSL models finetuned for paralinguisties and In addition, it has been shown that early to middle model layers carry paralinguistics information, while later layers encode linguistics content [54] 5 37 66]; we conducted thorough analyses to examine the cross-correlation between pretrained SSL model layers (Appendix (A.2) and chose layer 0-10's output from Wav2vec-XLoo XLSR fine- tuned for speech emotion recognition to represent style, and layer 14-21's s output from Wav2vec-XL59 finetuned for automatic speech recognition, to represent linguistics information.
25
linguistics tasks as candidate encoders [20
57 6
Both style (Xs) and linguistics (X2) embeddings are three- liree-dimensional tensors E IR KxFT where K denotes the transformer layer index, denotes the feature size, and denotes the number of time steps. These subspace embeddings are sent into compression modules C()), which average the transformer layer outputs and reduce the feature size from 1024 to 256 (more details Appendix (A.4). We refer to the output from the compression modules as dependency features: S C(Xs) for style and f, c( for linguistics, and their temporally averaged versions are denoted , and L.. These dependency features are learned by minimizing the self-contrastive loss Loens defined as:
Leon
Leross + ALintra:
Lintra = Lstyle + Llingusstics
1 T
Leross
S| t=0
L
Lintra |S,S] - 11 [, -1
(2)
Leross denotes the cubss-subsppcc loss; Lintra the intra-subspace loss, defined in terms of Estyle and Llinguisties (Figure D À E [0, 1] is hyperparameter that weighs the two loss terms, 7 the number of time steps; and (.) P is the Frobenius norm. The Cerose term reduces distance between the compressed style and linguistic embeddings, while the Lintra term reduces redundancy within the (temporally averaged) style and linguistic features by pushing off-diagonal elements to zero. The learned dependency features from Stage can be used to quantify whether mismatch exists between the style and linguistics of an audio. We further demonstrate this in Section 4.2
3.2.2 Stage 2: Supervised training
The second stage SLIM follows a standard supervised training scheme, where the dependency features and subspace representations are concatenated and fed into classification head to predict a binary real/fake outcome. As shown in Figure [], the subspace SSL. encoders and compression modules are obtained from Stage and are all frozen during Stage 2. Since the dependency features are specifically designed to capture the style-linguistics mismatch alone, we complement them with the original embeddings in order to capture other artifacts that can help separate real samples from the fake class. The original embedding's 's dimensions are reduced from 1024 to 256 through an attentive statistics pooling (ASP) layer and multi-layer perceptron (MLP) network. The projected subspace embeddings when concatenated with dependency features result in 1024-dim vectors. The classification head consists two fully-come connected layers and a dropout layer. Binary cross- cross-entropy loss is used to jointly train the ASP and MLP modules alongside the classification head.
4 Experiments
Based on the preliminary results from Section 3.1 we systematically assess the in-domain and cross-domain detection performance of SLIM using multiple datasets, and demonstrate how such framework would benefit the interpretation of model decisions.
4.1 Experimental set-up
Stage training. Unlike benchmark models which are trained end-to-end in supervised manner, our model relies on two-stage training where each stage requires different training data to avoid information leakage. Since only real samples are needed in Stage 1, we take advantage of open-source speech datasets by aggregating subsets from Common Voice 3 and RAVDESS [41 as training data and use small portion of real samples from the ASVspoof2019 LA train for validation. Both Common Voice and RAVDESS cover a variety of speaker traits. The former is a crowdsourced dataset collected online from numerous speakers with uncontrolled acoustic environments, while the latter is
5

---

an emotional speech corpus with large variations in prosodic patterns. Such data variety enables our model learn a wider range of style- stlle-iiguisstes combinations.
Stage 2 training and evaluation. For a fair comparison with existing works, we adopt the standard train-test partition, where only the ASVspoof logical access (LA) training and development sets are used for training and validation. For evaluation, we use the test split from ASVspoof2019 LA (75] and ASVspoof2021 DF [40]. ASVspoof2019 LA and ASVspoof2021 DF have been used as standard datasets for evaluating deepfake detection models, where real speech recordings originate from the VCTK and VCC datasets [89 42 94 and the spoofed ones are generated with a variety of TTS and VC systems. Compared to ASVspoof2019 LA, ASVspoof 2021 DF contains more than 100 different types of generated speech in the evaluation set with variety of different compression codecs. This provides a more challenging setting for testing generalization to unseen attacks and robustness to compression. In addition, we assess our model's generalizability on two more recent datasets: fa-the-wilce [5]] and the English subset from MLAAD v3 [53] In-the-wil wwide consists of on audio clips collected from Snglish-speaking celebrities and politicians, featuring more realistic and spontaneous speech samples. The English subset of MLAAD (hereinafter referred to as MLAAD- is a recent dataset with spoofed samples generated using state-of satee-of-hhe--ee the-art open-source TTS and VC systems (more details in Appendix A.3)
Metries. Equal error rate (EER) is a standard metric for evaluating deepfake detection systems [75 40] It refers the point in the detection error tradeoff curve where the false acceptance rate equals the false rejection rate; lower EER suggests better performance. We also report F1-beta scores (3=1) to account for the class imbalance; higher F1 scores suggest better performance.
Benchmarks. We consider several SOTA models to benchmark against and broadly categorize them as follows, based on the training cost: (i) methods which freeze fcature extraction frontends and finetune only the backend classifiers, and (ii) methods which finetune frontends together with the classifiers during supervised training. As benchmarks representing the former case, we consider (W2V-LLGF) 87). Saa22vecXXLSS+-LCO
(W2V-LCNN) (87), six dif- ferent models that share a similar backend classifier SLIM (W2V/WLM/HUB- ASP), model that fuses different SSL representations (SSL-fusion) [92], as well as three methods that do not rely on large SSL encoders, namely, LCNN 122 RawNet2 (69 and PS3DT [8]] For the end-to-end
www.2vec-XLSR+LLL.C
fine-tuning benchmarks, we consider the model in 44) with a backend classifier similar to SLIM's (W2V-ASP-ft), and the model in 770 with RawBoost augmentation and AASIST backend (W2V- AASIST). Using frozen frontends, five variants of SLIM are considered, where the input at Stage 2 is: (i) only the style embedding, (ii) only the linguisties embedding, (ii) the combination of style and linguistics embeddings, (iv) only the style-linguistics dependency features, and (v) the fusion of style and linguistic embeddings and their dependency features. We emphasize that the original SLIM framework does not involve any finetuning of frontends, since the finetuning may change the disentanglement of style and linguistics embeddings and thus hamper model explainability. However, to compare with finetuned benchmarks, we include a variant of SLIM that finetunes all modules during Stage 2, noting that this would compromise the feature interpretation.
Implementation details. We implement our models using the SpeechBrain toolkit 62
v1.0.0.
The hyperparameters used for Stage and Stage 2 training are provided in Appendix A.7 When setting up our customized benchmark models, we followed consistent training recipes where only the model architectures were changed and the same data augmentation method was used. Each round of evaluation was repeated three times with different random seeds, and the mean values are reported. Unlike some previous works which limit the duration of test speech, we only applied amplitude normalization during evaluation and kept the full duration of all samples.
4.2 Experiment results
Detection performance. Table summarizes the detection performance of all models and compares the number of trainable parameters. We discuss the models with frozen frontend here, and compare the models with finetuned frontend in Section. 4.3 ASVspoof2019 eval set contains 19 types of attacks, out of which 6 are seen during training. This makes it the simplest of the four test datasets. We see that majority of the models achieve near-perfect performance, with several including SLIM reporting EER below 1%. As expected, degradation is seen when models are tested

---

on ASVspoof2021, where the majority of attacks are unseen. Both W2V-LCNN and SLIM are top- oop-performers, with no significant difference between the two. With the out-of-domain datasets [In-the-wil sild and MLAAD-EN), more severe degradation is observed, where the majority report EERs over 20%. SLIM, however, outperforms the others with EER of 12.9% and 13.5% on In-the-wild and MLAAD-EN, respectively. It should be noted that although ASVspoof 2021 is often used standard dataset to evaluate model generalizability to unseen attacks [40), part the real samples in ASVspoof2021 originate from the same dataset (the VCTK corpus (89 as the ASVspoof2019 training data 75 12 69
As result, the real samples from ASVspoof2019 and ASVspoof2021 share similar distribution, whereas the In-the-will and MLAAD-EN samples share nearly overlap with ASVspoof (further discussion in Appendix A.3). Generalization to In-the the- wild and MLAAD-EN EN is therefore more challenging than to ASVspoof2021. The large gains reported by SLIM demonstrates how the style-linguistics mismatch helps with generalization to unseen data.
87 SU.
In Table 2 we also demonstrate the benefits of introducing Stage by considering features from SLIM variants as inputs to Stage 2: dependency features, the style and linguisties embeddings (Enesty and Encring), well as their combination. The architecture of the classification head is kept the same, except for the number of neurons in the input layer. The dependency features outperform the rest on the two out-of-domain datasets, while the subspace embeddings perform better ASVspoof Simply concatenating the style and linguistics embeddings does not yield significant improvements when compared to benchmark models. This suggests that the style- linguliin dependency may not be fully captured by supervised training methods without explicit guidance.
Table 2:
Detection performance on different deepfake datasets. Experiments were repeated three times with different random seeds, and average metric values are reported. #Param refers to the number of trainable parameters (in millions). For SLIM, we sum up parameters trained at both stages. A few models do not make their code open-source, we therefore include the metrics reported in their papers and skip parameter calculation (N/A). Lowest EERs are bolded per category.
Model ASVspoof19 ASVspoof21 In-the-wild MLAAD-EN #Param (million)
Category
EER
F1
EER
F1f
EER FIf EER,
F1
LCNN
12
3.7
.834
25.5
.197
65.6
.373
37.2
.654
RawNet2
3.0
.875
22.3
.213
37.8
.602
33.9
.676
PS3DT
88
4.5
29.7
N/A
W2V-ASP
3.3
.858
19.6
.233
30.2
.705
29.1
.715
WLM-ASP
0.3
.983
9.0
.426
25.4
.751
30.3
.709
HUB-ASP
0.5
.975
15.4
.289
29.9
.718
31.0
.702
W2V-LLGF 81
2.3
.936
9.4
.402
25.1
.756
27.8
.731
10
Frozen frontend (Section. 4.2
W2V-LCNN 87
0.6
8.1
24.5
N/A
W2V+WLM
1.8
.916
22.5
.203
30.3
.704
27.0
.739
W2V+HUB
0.9
.956
14.2
.310
27.9
.737
27.6
.732
WLM+HUB
0.8
.963
16.7
.269
29.2
.724
28.5
.720
SSL-Fusion 92
0.3
.981
8.9
.419
24.2
.765
26.5
.739
10
SLIM variants (ours)
Enc,ty
6.7
.740
8.6
.438
29.2
.724
25.4
.756
Encting
5.9
.764
9.3
.407
30.4
.708
25.0
.760
EnCstyle+
3.5
.834
9.0
.429
25.1
.757
23.9
.772
10
Dependency
2.8
.897
20.5
.234
25.8
.750
19.8
.811
Full
0.6
.969
8.3
.451
12.9
.895
13.5
.865
11
Finetuned frontend (Section. 4.3
W2V-ASP 44)
0.3
.984
4.5
.646
18.6
.836
19.2
.817
317
W2V-AASIST
71
0.2
.991
3.6
.707
17.5
.847
14.5
.856
317
SLIM (ours) 0.2
.989
4.4
.651
12.5
.898
10.7
.892
253
Style-linguistics mismatch of deepfakes. Figure shows the distribution cosine distances between the style and linguistics dependency features for the real and fake classes; larger distances indicate a higher mismatch. Since the distance values approximately follow Gaussian distribution with unequal variances, we further conduct Welch's t-test [ to examine the statistical significance

---

of the difference between real and fake samples. For all three datasets, the average cosine distance is found be significantly lower for real speech than for deepfake samples (p 1e "). This further corroborates our hypothesis that higher style-linguistics mismatch exists for fakes. On the other hand, the distance distributions of real and fake samples still share a large overlap, indicating that dependency features alone are not sufficient for perfectly discriminating between the two classes.
Class
Bonafide
-1
Deepfake
-2
Cosine distance
-3
-4-
-5-
-6-
-7-
ASVspoof2021 DF eval
In-the-wild Dataset
MLAAD-EN
Figure 2: Cosine distance (log scale) calculated between the style and linguistics dependency features for ASVspoof2021 DF eval, In-the-wte wild, and MLAAD-EN AN. Whiskers from top to bottom represent the 75% quartile, median, and 25% quartile of the distribution.
Analysis of style-linguistics dependency features. Table 2 demonstrates that style-linguisties dependency features can provide better generalizability than the subspace embeddings (Table SLMM variants, rows 1-4). To examine these results, we first aggregate ASVspoof2021. In-the-wild, and MLAAD and project the dependency features well as the concatenated subspace embeddings to 2-dim space using t-SNE for visualization (Figure B3. Since we use frozen frontends, the embeddings input Stage 2 training are not affected by backpropagation. Ideal embeddings would exhibit maximal separation between the real and fake classes, while showing minimal shift within each class for different dataset distributions. discrimination between real and fakes (3(c) 3(d) than the concatenated subspace embeddings (3(a and (bb).. and also a smaller shift between datasets: fake and real samples from the same dataset (color) clusters have less overlap in distribution in the plots.
Figure 3, we see that the dependency features show larger
60
50
50
50
25
25
-25
-25
-25
-50
50 -50
50
-50
50
-50
50
(b)
(c)
(d)
Figure 3: Projected embeddings using t-SNE for style-linguistic representations: (a) subspace embeddings real class, (b) subspace embeddings- fake class, (c) dependency features real class, (d) dependency features fake class. Data distributions are visualized on the upper and right side of the embedding plots. Red: 20Vsp
Green: In-the-wild; Blue: MLAAD-EN
Interpretation of model decisions. Next, we perform a qualitative evaluation of the model deci- sions. F the mel-spectrograms of four samples selected from In-the-wild. These four demonstrate typical acoustic characteristics that represent larger group of recordings: (1) top-left a fake sample with audible artifacts high-frequency region; (2) top-right is a fake sample with unnaturally long pauses heard before and after the phrase "but not"; (3) bottom left is a real sample
8

---

with an atypical speech style where the word pronunciations are elongated; (4) bottom right is real speech recorded in a noisy condition. We find that among the top- pe-reforrming systems shown in Table 2 only SLJM classified all four samples correctly (both frozen and fine-tuned versions; with all features), while others mostly failed on (2) and (4). Findings here suggest that SLJM provides guidance when abnormalities in style and linguistics occur. Such guidance can be complemented via post-hoc analysis tools such as human evaluations or saliency maps [4] for further interpretation.
Additionally, we note that the decisions made by dependency features and the original subspace representations are complementary each other. Samples in the right column are correctly identified as fake by the dependency features but missed by the original subspace representations, and vice versa (left column missed by dependency features). These results corroborate with the nature of the two feature types. The dependency features are learned by modelling the general linguiiigssss relationship seen in real speech, therefore samples with mismatched liyguiiin pattern are likely to be flagged as "not real." The original style and linguisties embeddings, on the other hand, are sensitive to signal artifacts, which could be the deepfake imperfections generated during speech synthesis [67]. the amount of background noise and device artifacts. By combining the two features, SLIM captures a variety of abnormalities and achieves improved classification.
High-frequency artifacts
Abnormal rhythm
The cstuliihment protected isell" but not
the citions of our country.
FAKE
50
50
50
100
150
200
750
300
20
40
60
100
120
140
-50
-40
-2
0
20
-50
-40
-20
20
Atypical style patterns: elongated words
Noisy condition
REAL
50
50
20 40 D0
80
30 20 20 60 50
-40 -20
20
-40 -20 20
Figure 4: Mel-spectrograms of select samples from In-the-wild. SLIM classifies all four correctly, and when reporting fakes, provides guidance on abnormalities in style and/or linguistics. Also, the dependency and subspace features in SLIM are complementary to each other. Left: samples missed by dependency features but correctly identified by the style and linguistic features; right: vice versa.
4.3 Ablation studies
Effects of finetuning SSL frontend. From Table 2 we see that the frontend finetuning helps to further decrease the EER for SLIM. The finetuned version of SLIM performs better than the rest on In-the-wild and MLAAD-EN EN, while providing comparable performance on ofVspoof2019 and 2SVspo0f2021. However, it should be noted that the interpretation of tiyeuiiinusssiics mismatch becomes difficult after finetuning, since the two subspace representations may no longer be maximally disentangled.
Effects of classification backend. In Stage 2, subspace representations are sent into ASP+MLP layers, which output 256-dim embeddings fuse with the dependency features. Previous works have shown that different backend architectures may lead to a significant difference in the detec- tion performance [8]. With the input fixed (dependency features and subspace embeddings), we find that removing the ASP and MIP layers degrades EER across the four datasets (Table 4 Ap- n..). while using the LCNN 87 or L.LGF []] backends improves EER on ASVspoof2019 and 22Vs,, but not on In- willd ld and MLAAD-EN EN.
Misclassifications made by SLIM. We studied the misclassifications made by SLIM through a com- bination of qualitative analysis methods, such as listening tests, visualizations of speestrrrrog,,sss noise-level estimation, and prediction of number of speakers. In general, we found that shorter and low-quality samples, e.g., those with estimated Mean Opinion Score (MOS) lower than 2 (using NISQA 47), as well as recordings with multiple speakers were more likely to be misclassified. While the former is general challenge faced by all detection methods, the latter can be explained by

---

the design of SLIM, where the pretraining stage does not involve any multi-speakke samples. In multi-speaker scenario (e.g. conversational speech), speech patterns from more than one speaker could lead to higher misalignment between linguistic content and speaker attributes, resulting in false alarms.
Limitations
Since framework explicitly focuses style-linguistics mismatch, it is possible that real speech samples with atypical style-linguistics dependency (c.g., dysarthric speech [59] or children speech) may be misclassified. With the rapidly growing number of speech datasets, to conduct self-supervised pretraining on datasets with more diverse speech styles (e.g.. (95]). Also, while we focused English samples due the scarcity of multilingual deepfake datasets the time of writing, the proposed framework can be scaled to more languages with recent deepfake datasets, such MLAAD-v4. Finally, although SLIM can benefit from frontend finetuning and more advanced backends, doing so would affect the feature interpretation and will require modifications to the training approach. We plan to explore these directions in the future.
countermeasure
Conclusion
We present SLIM, new audio deepfake detection framework that explicitly models the style- linguisties mismatch in speech to detect deepfakes. Without requiring a large amount of labeled data or the added cost of end-to-end finetuning pretrained encoders, SLIM outperforms existing benchmarks on out-of-domain datasets, while being competitive on in-domain datasets. The learned Alyle-linguistics dependency features are complementary to the individual pretrained style and linguistics subspace representations, and facilitate result interpretation by guiding our attention to where style-linguistics misalignment occurs, which can be further analyzed by variety of qualitative studies.
10

---

References
[1] Nor Aishah Ahad and Sharipah Soaad Syed Yahaya, Sensitivity analysis of welch'ssttestt In AIP Conference proceedings, volume 1605, pages 888-893 American Institute Physics, 2014.
[2] Zaynab Almutairi and Hebah Elgibreen. A review of modern audio deepfake detection methods: Challenges and future directions. Algorithms, 15(55)155,, 2022.
[3] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multiinguu speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages
4218 4228, 2020.
[4] Alejandro Barredo Arricta, Natalia Díaz Mazz-Rorriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, laxonomies, opportunities and challenges toward responsible ai. Information fuxion, 58:82-115, 2020.
[5] Takanori Ashihara, Mare Delcroix. Takafumi Moriya, Kohei Matsuura, Taichi Asami, and Yusuke Ijima. What do supervid speech and speaker models learn? new findings from cross model layer- wise analysis. arXiv preprint arXiv:240I. 17632, 2024.
[6] Arun Babu, Changhan Wang. Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et XIs-r: sulf--upe cross- coss-linguul speech representation learning at scale. arXiv preprint (eX::::2111092222 2021.
Alexei Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12460, 2020.
[8] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298-1312. PMLR, 2022.
[9] Susan Baldwin. Compute canada: advancing computational research. In Journal of Physics: Conference Series, volume 341, page 012001. 1OP Publishing. 2012.
[10] Logan Blue, Kevin Warren, Hadi Abdullah, Cassidy Gibson, Luis Vargas, Jessica O' D'Dell, Kevin Butler, and Patrick Traynor. Who are you (i really wanna know)? detecting audio [DeepFakes] through vocal tract reconstruction. In 31 USENIX Security Symposium (USENIX Security 22), pages 2691 2698-2708, 2022.
[11] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka. Xiong Xiao, al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1555-1566 2022.
[12] Akash Chintha, Bao Thai, Saniat Javid Sohrawardi, Kartavya Bhatt, Andrea Hickerson, Matthew Wright,, and Raymond Ptucha. Recurrent convolutional structures for audio spoof and video deepfake detection. IEFE Journal of Selected Topics in Signal Processing, 1424-1102 2020.
[13] Cheol Jun Cho, Peter Wu, AbdeIrahman Mohamed, and Gopala K Anumanchipalli. Evidence of vocal tract articulation in self-supervised learning of speech. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[14] Joseph Cox. How broke into a bank account with an ai-generated voice. htttpe///ww..ve www.::wwwwwwcccc.ooooo...... en/ article/dy7axx wwxx//hoowi1
hov- i- broke- into-i--o----- bankk-------- waapka-ccoott-tu with- an- ai aaa-aa-ggneere -oiie 2023.
Accessed: 2024-04-30.
[15] Anne Cutler. Delphine Dahan, and Wilma Van Donselaar. Prosody in the comprehension of spoken language: A literature review. Language and speech, 40(2):141-201, 1997.
[16] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.07 200200000....... 2020.
[17] Amandeep Singh Dhanjal and Williamjeet Singh. A comprehensive survey on automatic speech recognition using neural networks. Multimedia Tools and Applications, 83(8):23367-23413 2024.
[18] Florian Eyben, Martin Wöllmer, and Bjöm Schuller. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 2010.
11

---

[19] Florian Eyben, Klaus R Scherer, Bjöm W Schuller. Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps. Petri Laukka, Shrikanth Narayanan, et al. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE transactions on affective computing. 7(2):190- 100-222-22, 2015.
[20] Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. Exploring wav2vec 2.0 on speaker verification and language identification. Interspeech 2021. 2021.
[21] Wanying Ge, Jose Patino, Massimiliano Todisco, and Nicholas Evans. Explaining deep learning models for spoofing and deepfake detection with shapley additive explanations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 6387-6391. IEEE, 2022.
[22] Jonatas Grosman. Fine-tuned XLSR-33 large model for speech recognition in English. https:// huggingface. bugginnface..c//joaateee wwt./oonnattagrssaaaaaaeeeaeeeee 2wwreeeeeeee--eeeeeee large- x1sr- 53- english 2021.
[23] Yinlin Guo, Haofan Huang. Xi Chen, He Zhao, and Yuehai Wang. Audio deepfake detection with self-supervised wavlm and multi-fusion attentive classifier. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12702 12706. IEEE, 2024.
[24] Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conversion from non-parallel corpora using variational auto-encoder. In 2016 Asia-Paciipe Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pages 1-6. IEEE, 2016.
[25] Wei-Ning Hsu, Benjamin Bolte. Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self supervised speech representation learning by masked prediction of hidden units. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3466 2021.
[26] Jee-weon Jung, Hee-Soo Heo, Hemlata Tak. Hye-jin Shim, Joon Son Chung, Bong-Jin Lee. Ha-Jin Yu, and Nicholas Evans. Aasist: Audio apti-spoofing using integrated spectro -eeepoal graph attention networks. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6367-6371 IEEE, 2022.
[27] Navdeep Kaur and Parminder Singh. Conventional and contemporary approaches used in text speech synthesis: A review. Artificial Intelligence Review, 56(7):5837-58800 2023.
[28] Awais Khan, Khalid Mahmood Malik, and Shah Nawaz. Frame uttuerreeece convergence: A spectra- temporal approach for unitied spooting detection. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10761-10765. IEFF, 2024.
[29] Kate Knibbs. Researchers say the deepfake biden robocall was likely made with tools from ai startup elevenlabs. https://www wwred. http://sttttrr/beeeeooooooaa-lla -ew.aall--deeetrreeeeeeeeeeeaaa 2024. Accessed: 2024-04-30.
[30] Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 5519-3529, PMLR, 2019.
[31] William& A Kretzschmar, The linguistics of speech. Cambridge University Press, 2009.
[32] Il- Youu Kwak, Sungsu Kwag. Junhee Lee, Youngbae Jeon, Jeonghwan Hwang, Hyo-Jung Choi, Jong- Hoon Yang, Yul Han, Jun Ho Huh, Choong-Hoon Lee, et al. Voice spoofing detection through residual network, max feature map, and depthwise separable convolution. JEEE Access, 2023.
[33] Seth Layton, Thiago De Andrade, Daniel Olszewski, Kevin Warren, Carrie Gates, Kevin Butler, and Patrick Traynor. Every breath you don't take: Deepfake speech detection using breath. arXiv preprint arXiv:2404. 15143, 2024.
[34] Menglu Li, Yasaman Ahmadiadli, and Xiao-Ping Zhang, Audio anti-spoofing detection; A survey, arXiv preprint ar Xiv:24004. 13914, 2024.
[35] Yuang Li, Min Zhang, Mengxin Ren, Miaomiao Ma, Daimeng Wei, and Hao Yang. Cross-domann audio deepfake detection: Dataset and analysis. arXiv preprint xXiv:2404.04900 2024.
[36] Suk-Young Young Lim. Dong-Kyu Chae, and Sang Chul Lee. Detecting deepfake voice using explainable deep learning techniques. Applied Sciences, 12(8):3926, 2022.
[37] Guan- Lin, Lue- Feng, Wei- Wei- Huang, Yuan Tseng, Tzu-Han Lin, Chen-An Li, Hung-yi yi Lee, and Nigel G Ward. On the utility of self supervised models for prosody- related tasks. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1111 JEEE, 2023.
12

---

[38] Kristen A Lindquist, Jennifer K MacCormack, and Holly Shablack. The role of language in emotion:
Predictions from psychological constructionism. Frontiers psychology, 6: 6:121301, 2015.
[39] Zachary C. Lipton. The mythos of model interpretability: In machine learning, the concept of in- terpretability is both important and slippery. Queue, 16(3):31-57, jun 2018. ISSN 1542-7730. doi: 10. 111144/3236388.241114
URI httph:t/////oii http:://ddoooor/////// org/10. 1145/3236386..... 3241340
[40] Xuechen Liu, Xin Wang. Md Sahidullah, Jose Patino, Héctor Delgado, Tomi Kinnunen, Massimiliano Todisco, Junichi Yamagishi. Nicholas Evans, Andreas Nautsch, et al. Asvspoof 2021: Towards spoofed and deepfake speech detection in the wild. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[41] Steven Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions north american english. PloS one, 1355)10196331, 2018.
[42] Jaime Lorenzo-Treee Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnunen, and Zhenhua Ling. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. In The Speaker and Language Recognition Workshop, pages 195-202. ISCA, 2018.
[43] Jingze Lu, Yuxiang Zhang, Wenchao Wang, Zengqiang Shang, and Pengyuan Zhang. One-class knowledge distillation for spoofing speech detection. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11251-11255. IEEE, 2024.
[44] Juan M. Martín- Darrin--Doassaa and Aitor Álvarez. The vicomtech audio deeptake detection system based on wav2vec2 the 2022 add challenge. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9241 9245, 2022. 9747768.
doi: (0.11009CCASSPP492222222
[45] Momina Masood, Mariam Nawaz, Khalid Mahmood Malik, Ali Javed, Aun Irtaza, and Hafiz Malik. Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward. Applied intelligence, 53(4):3974 4026, 2023.
[46] Driss Matrouf, J-F Bonastre, and Corinne Fredouille. Effect of speech transformation on impostor acceptance. In 2006 JEEE international conference on acoustics speech and signal processing proceedings. volume 1, pages I-I. IEEE, 2006.
[47] Gabriel Mittag. Babak Naderi, Assmaa Chehadi, and Sebastian Moller. Nisqa: A deep oan-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. Interspeech 2021, 2021.
[48] Seyed Hamidreza Mohammadi and Alexander Kain, An overview of voice conversion systems. Speech Communication, 88:65-82, 2017.
[49] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7): 1877-1884, 2016.
[50] Nicolas Müller, Franziska Dieckmann, Pavel Czempin, Roman Canals, Konstantin Böttinger, and Jennifer Williams. Speech is silver, silence is golden: What do asvspoof-trained models really Iearn? 2021 Edition of the Automutic Speaker Verification and Spoofing Countermeasures Challenge, 2021.
[51] Nicolas Müller, Pavel Czempin, Franziska Diekmann, Adam Froghyar, and Konstantin Böttinger. Does audio deepfake detection generalize? Interspeech 2022, 2022.
[52] Nicolas M Müller, Philip Sperl, and Konstantin Böttinger. Complex- valued neural networks for voice anti-spoofing. arXiv preprint ar 233: /8000, 2023.
[53] Nicolas M Müller. Piotr Kawa, Wei Herng Choong. Edresson Casanova, Eren Gölge, Thorsten Müller, Piotr Syga, Philip Sperl, and Konstantin Böttinger. Mlaad: The multi-language audio anti-spoofing dataset. arXiv preprint arXiv:2401.095 2024.
[54] Ankita Pasad, Ju-Chich Chou, and Karen Livescu. Layer-wise analysis of a self-supervised speech representation model. In 202/ IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 914-921. IEEE, 2021.
[55] Ankita Pasad, Bowen Shi, and Karen Livescu. Comparative layer-wise wise analysis of self self-superiied speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5, IEEE, 2023.
13

---

[56] Terry Pender. Ai threatens courts with fake evidence. uw prof says. hettps:/www.jddee com/ legalnews/ ai- 05.
2023. Accessed: 2024-05-
[57] Leonardo Pepino. Pablo Riera, and Luciana Ferrer. Emotion recognition from speech using wav2vec 2.0 embeddings. Interspeech 2021, 2021.
[58] Natalie Pereira, Ana Paula Bresolin Gonçalves, Mariana Goulart, Marina Amarante Tarrasconi, Renata Kochhann, and Rochele Paz Fonseca. Age-related differences in conversational discourse abilities a comparative study. Dementia & Neuroprychologia, 13:53-77 71, 2019.
[59] Zhaopeng Qian, Kejing Xiao, and Chongchong Yu. survey of technologies for automatic dysarthric speech recognition. EURASIP Journal on Audio, Speech, and Music Processing, 2023(I):48, 2023.
[60] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL luttps://arxiv http://axxiv..og/bbbaaaaa org/abs/
2212. 04356)
[61] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Singular vector canonical
[62] Mirco Ravanelli, Titouan Parcollet. Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. Speechbrain: A general-purpose speech toolkit. arXiv preprint arXiv:2106. 04624, 2021.
[63] Alexandra Saliba, Yuanchao Li. Ramon Sanabria, and Catherine Lai. Layer-wise analysis of self acoustic word embeddings: A study on speech emotion recognition. arXiv preprint arXiv:2402.02617, 2024.
[64] Mare Schröder, Marcela Charfuelan, Sathish Pammi, and Ingmar Steiner. Open source voice creation toolkit for the mary tts platform. In 12th Annual Conference of the International Speech Communication Association Inttrppe 2011, pages 3253-3256. ISCA, 2011.
[65] Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian MüLler, and Shrikamth Narayanan. Paralinguistics in speech and language-stateee statc-of-thh-art and the challenge. Computer Speech & Language, 27(1):4339, 2013.
[66] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah. What all do audio transformer models hear? prohing acoustic representations for language delivery and its structure. arXiv preprint arXiv:2 101.00877 2021.
[67] Tsu-Hsien Shih, Chin- Yuan Ych, and Ming-Syan Chen. Does audio deepfake detection rely artifacts? In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 12450. IEEE, 2024.
[68] David Snyder, Daniel Pomeea, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5329-5333. IEEE, 2018.
[69] Hemlata Tak, Jec- Bee-Weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, and Nicholas Evans. End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection. In ASVSPOOF 2021, Automatic Speaker Verification and Spoofing Countermeasures Challenge, pages 1-8, ISCA, 2021.
[70] Hemlata Tak, Madhu Kamble, Jose Patino, Massimiliano Todisco, and Nicholas Evans. Rawboost: A raw data boosting and augmentation method applied to automatic speaker verification anti- anti-spoofing In ICASSP 2022-20 2022-2022100 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6382-6386. IEEE, 2022.
[71] Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee-weon Jung, Junichi Yamagishi, and Nicholas Evans. Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation. In The Speaker and Language Recognition Workshop (Odyssey 2022). ISCA, 2022.
[72] Xu Tan. Neural text-to-speech syntheis.. Springer Nature, 2023.
[73] Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. survey on neural speech synthesis. arXiv preprint arXiv:2106.1 155611 2021.
14

---

[74] Naftali Tishby. Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprini physicx/0004057. 2000.
[75] Massimiliano Todisco, Xin Wang, Ville Vestman, Md Sahidullah, Héctor Delgado, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Tomi Kinnunen, and Kong Aik Lee. Asvspoof 2019: Future horizons in spoofed and fake audio detection. arXiv preprint arXiv:190..0544 2019.
[76] Markus Toman and Michael Pucher. An open source speech synthesis frontend for hts. In Text, Speech, and Dialogue: 18th International Conference, TSD 2015, Pilsen, Czech Republic, September 14-17, 2015, Proceedings 18, pages 291 291-298. Springer, 2015.
[77] Andreas Triantayyllpoolos and Bjöm Schuller. Expressivity and speech synthesis. arXiv preprint vrXiv:2404.1936. 2024.
[78] Andreas Triantafyllopoulss. Bjöm Schuller, Gökçe lymen, Metin Sezgin, Xiangheng He, Zijiang Yang. Panagiotis Tzirakis, Shuo Liu, Silvan Mertes, Elisabeth André, et al. An overview of affective speech synthesis and conversion in the deep learning era. Proceedings of the IEEE, 2023.
[79] Nik Vaessen and David A Van Leeuwen. Fine-tuning wav2vec2 for speaker recognition. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7967- IEEE, 2022.
[80] Aaron Van Den Oord. Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukeuoglu, et al. Wavenet: A generative model for raw audio. 12, 2016.
arXiv preprint
[81] Xin Wang and Junichi Yamagishi. Investigating self superviid front ends for speech spoofing counter- measures, arXiv preprint arXiv:2111.07725. 2021.
[82] Xin Wang and Junichi Yamagishi. Spoofed training data for speech spoofing countermeasure can be efficiently created using neural vocoders. In ICASSP 2023-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[83] Xin Wang and Junichi Yamagishi. Can large-scale vocoded spoofed data improve speech spoofing countermeasure with self-supervised front end? In ICASSP 2024-2024 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10311- 103315.
(5. IEEE, 2024.
[84] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba. A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding. arXiv preprint arXiv:21 I 1.02735, 2021.
[85] David Weenink. Canonical correlation analysis. In Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam, volume 25, pages 19.99. University of Amsterdam Amsterdam, 2003.
[86] Zhizheng Wa, Oliver Watts, and Simon King. Merlin: An open source neural network speech synthesis system. In 9th ISCA Speech Synthesis Workshop, pages 202-207, 2016.
[87] Yuankun Xie, Haonan Cheng, Yutian Wang, and Long Ye. Learning self-supervised iivariatn feature representation for generalized audio deepfake detection. In Proc. INTERSPEECH, volume 2023, pages 3308-28122 12, 2023.
[88] Amit Kumar Singh Yadav, Ziyue Xiang, Kratika Bhagtani, Paolo Bestagini, Stefano Tubaro, and Edward J Delp. Ps3dt: Synthetic speech detection using patched spectrogram transformer. In 2023 International Conference on Machine Learning and Applications (ICMLA), pages 496-503. IEEE, 2023.
[89] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English Multi- speaker Corpus for CSTR Voice Cloning Toolkit, 2019.
[90] Masahiro Yanagawa and Junya Sato. Seeing is not always believing: Discrepancies in saliency maps. Radiology: Artificial Intelligence, 6(1)x2230488 2023.
[91] Shu-wen Yang, Heng-Jui Chang, Zili Huang, Andy T Liu, Cheng-I Lai, Haibin Wu, Jiatong Shi, Xuankai Chang, Hsiang- Shiang-Sheng Tsai, Wen-Chin Huang, et al. A large-scale evaluation of specch foundation models. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[92] Yujie Yang, Haochen Qin, Hang Zhou, Chengeheng Wang, Tianyu Guo, Kai Han, and Yunhe Wang. robust audio deepfake detection system via multi-view feature. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13131-13135, TEEE, 2024.
15

---

[93] Jiangyan Yi. Chenglong Wang. Jianhua Tao, Xiaohui Zhang, Chu Yuan Zhang. and Yan Zhao. Audio deepfake detection: A survey. arXiv preprint arXiv:2308. 14970. 2023.
[94] Zhao Yi, Wen-Chin Huang, Xiaohai Tian, Junichi Yamagishi, Rohan Kumar Das, Tomi Kinnunen, Zhen- Hua Ling, and Tomoki Toda. Voice conversion challenge 2020- -nnttn--inggeeee semi parallel and cross- lingual voice conversion- Joint Workshop for Blizzard Challenge and Voice Conversion Challenge 2020. ISCA, 2020.
[95] Chang Zeng, Xin Wang, Xiaoxiao Miao, Erica Cooper, and Junichi Yamagishi. Improving generalization ability of countermeasures for new mismatch scenario by combining multiple advanced regularization terms. arXiv preprint arXiv:2305. 2023.
[96] Yuxiang Zhang, Zhuo Li, Jingze Lu. Hua Hua, Wenchao Wang, and Pengyuan Zhang. The impact of silence on speech anti anti-spoofing /EEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
16

---

A Appendix
A.1 Details of generative models tested in the CCA analysis
Details on the systems tested in Table. I Jare as follows. A01 is a TTS model that uses Flite 76 that converts text into a sequence of linguistic features. A Hidden Markov Model (HMM) is then used to estimate the duration of phones. The acoustic features are predicted by an NN-based acoustic model, including MFCCs, FO, and voicing flags, which are imput to the WaveNet vocoder [80] for waveform synthesis. A02 and A03 follows a similar
pipeline A01, where A02
uses the WORLD vocoder 449 and A03 relies on recipes from the Merlin toolbox [86]. A04 is waveform concatenation TTS model based on the MaryTTS platform 644, major difference between A04 and the previous three models is that A04 preserves the short-term acoustic features of the natural speech, which may make it more difficult to separate from real speech. A05 and 06 are VC models. A05 encodes the spectral features into and decode them with a desired target speaker representation [24). A06 relies on the source-filter modelling speech where the conversion can be achieved by replacing the filters of the input signals by that of the target speaker [46]. The non-speech segments are kept intact during conversion.
embeddings using a VAE
A.2 Layer-wise analysis of pretrained SSL models
As mentioned in Section, 3.2.T we use the Wav2vec-XLSs model finetuned for emotion recognition (Wav2vec-SER) and speech recognition (Wav2vec-ASR) tasks to extract the style and linguistics representations, respectively.
The style representation is based on the pretrained model obtained from httptt::/tutgiiiggtteeee and the linguistics representation is based on the pretrained model obtained from https: httt:::://uggiiiie htgguuugggggg aceeinenge..... co/ x1w--xssr553-- jonatasgrosman/ wwwpaaagrromaa/vavvveee222 http://aavveceeaaaaeee -aa.oe-----re---------- englisshhh To obtain a maximal disentanglement between the two subspace representations, we calculate Spearman's 's rank correlation coefficient values between different layers from the two models to examine the layer-wise similarity. These correlation values and our final layer selection are demonstrated in Figure 5 Based on existing works, which showed how linguistics and paralinguisties information propagate through layers, we choose layer 0-10 from Wav2vec-SHe -SRR backbone to represent style information, and layer 14-21 from Wav2vec-ASR backbone to represent linguistics information. The correlation values between these two groups are shown close to O, indicating a better disentanglement.
Layer 0-10 chosen from W2V-SER
Correlation region
W2V-SER
1.0
14
0.8
16
0.6
W2V-ASR
0.4
10
12
0.2
14
16
W2V-SER
W2V-ASR
Layer 14-21 chosen from W2V-ASR
Figure 5: Spearman correlation coefficients calculated across all layers from two pretrained Wav2vec- XLSR backbones. Blue highlights layers 0-10 from Wav2vec-SER represent style information. Red highlights layers 14-21 from Wav2vec-ASB to represent linguistics information. The correlation values between the selected layers can be read from the overlapping region.
17

---

A.3 Dataset details
Table describes the details of datasets used for Stage and Stage training and evaluation. Figure shows the projected WavLM embeddings for real and fake samples from the four employed datasets using t-SNE. We choose WavLM since the top-perti perporrorning model in the single-encoder category (Table 2 For both classes, overlap can be seen between ASVspoof2019 and ASVspoof2021 samples, while samples from In In-the-wild and MLAAD-EN can be separated nearly perfectly. This corroborates with the results reported in Table 2 where all employed ADD systems trained on ASVspoof2019 perform better on ASVspoof than In-the-wile and MLAAD-EN.
Table 3: Summary of datasets used for Stage and Stage 2 training and evaluation.
Stage datasets
Name
Split #Sample #Real #Fake #Attacks Speech type
Environment
Common Voice Train
3k
3k
Scripted Scripted
Crowdsourced
RAVDESS
Train
3k
3k
Studio
19 LA train
Valid
500
500
Scripted
Studio
Stage 2 datasets
Name
Split #Sample
#Real
#Fake
#Attacks Speech type Environment
19 LA train
Train 25380
2580
22800
Scripted
Studio
19 LA dev
Valid 24884
2548
22336
Scripted
Studio
19 LA eval
Test 71237
7355
63882
17
Scripted
Studio
21 DF eval
Test 611829
22617
589212
100+
Scripted
Studio
In in-the-wild
Test 31779
11816
19963
N/A
Spontaneous
In-the-wild
MLAAD-EN
Test 37998
18999
18999
25
Scripted
Studio
Real
Fake
wn-l
Insthe-mild
ASVspoo2019
ASVepooE021
AS 2016
MLAAD
MI ALA
Figure 6: Projected WavLM embeddings for real and fake classes from the four employed datasets. Left: real class embeddings. Right: fake class embeddings.
A.4 Details of the compression module
Figure the architecture of the compression module. The input is first passed through pooling layer to obtain an average of different SSL layer outputs. Since the goal of compression modules is to project the original style/linguisties embeddings to subspace where the compressed embeddings can be maximally correlated, we use bottleneck layers to remove the redundant information that is not shared across the two subspaces. Similar to the design of an autoencoder 774,, the bottleneck layer first compresses the feature dimension from 1024-dim to 256-dim, then recovers it back to 1024-dim. In practice, we found using only one bottleneck layer enough to obtain meaningful compressed representations. A projection head is applied at the end to reduce the final output dimension to 256.
18

---

Xpr
{256, T)
MILP
Head
LeakyRel.U
Dropout
(1024, T}
MLPP
(256,
Dropout
Bottle neck
N
LeakyReLU
BatchNomm
(256, T)
ML.
{1024, T
Pooling
X XKT
{23, 1024, T)
Figure 7: Architecture of the compression module with input and output dimensions. Input XK, F.F represents the original subspace representation encoded by the SSL frontend, where denotes the transformer layer index, F denotes the feature size, and T denotes the number of time steps.
A.5 PyTorch implementation of the Stage 1 training objective
Algorithm| [ [hoows PyTorch-styl implementation of the Stage 1 training objective, which minimizes the cross-subspace distance loss and an intra-subspace redundancy loss. The subspace embeddings are first normalized across the whole batch before passing into the loss calculations. We experimented with two types of distance for the cross- cross-subspace loss: Euclidean and Cosine distance. While no significant difference is found when comparing the performance achieved by the two, the former provides slightly better results, hence is adopted the final distance measure.
Algorithm 1: PyTorch-style code for the Stage loss function
Input:
Eatyle Fllinguiatic
Output: Lstagel
// Normalize embeddings from both subspaces
batch_ size Sayle-Shape[0]
2 Xstllenorm = torch nn. www.aatcooonmmmm..mmmmmmmmmm
wealoormmmmxxyyyaa 1dxxyl affine=False) batch batch size
Xlinguistic_noue torch. nn. Bot.Bathhooodddeee ((Xinnusstt affine=False) / batch size
// Computation of cross-subspace loss
tinah. hin.t..ioooo.(m(((( (Xstle_ Xlinguisti_ ord='fro)
torch.pow(D,P)
// Computation of intra-subspace redundancy loss
6 Vlinguistic torch. mm(Xinguxxiiic Xtinguistic, norm-1, TT, Xlinguistic, norm)
8 Clinguistic = pow hingii/iinnniiettoooc.eeeeee
Clingissic
caaccliinll..oomme (Vinguissic Sotepuiitt-torc.eeee(((
2)
Vatlee torch.m h...m/mmmmm. Xatyle_ norm- T, Xstyls_nom)
10 Cayle = torch.linalg. mall..norm(( Vstyle- worrcc.eyeee@age.sssappeep
Cayle torch, Borch.pow(Crylee 2)
// Final loss term
12 Langel = D + (Cstyle + Ciinguistt)))
A.6 Performance comparison of different backend classifiers
Table 4 shows the performance obtained when the ASP+MLP layers are swapped with other layer choices.
19

---

Table Performance comparison of different backend classifiers used in SLIM. Frontend encoders are frozen.
EER
SLIM backend
ASVapoof2019
ASVspoof2021 In-the-wild MLAAD-EN
Original (ASP+MLP)
0.6
8.3
12.9
13.5
None
0,9
9.1
13.1
13.7
LLGF
0.4
7.5
13.5
13.0
LCNN
0.3
7.9
12.8
13.9
A.7 Hyperparameters and computation resources
Table 5 describes the optimal hyperparameters and architecture details of SLIM used for Stage and Stage training. The hyperparameter names of the data augmentation modules can be found in SpeechBrain vl.0.0. Experiments were conducted on the Compute Canada cluster [9] with total of four NVIDIA V100 GPUs (32GB RAM).
20

---

Table 5: Hyperparameters and architecture details of SLIM.
Parameter
SLIM
Stage 1 Optimization
Batch size
16
Epochs
50
GPUs
4
Audio length
5s
Optimizer
AdamW
LRscheduler
Linear
Starting LR
.005
End LR
.0001
Strly-stop patience
epochs
.007
Training time
3h
SSL frontend
Style encoder
Way2vec-XLn XLSR-SER
Style layers
0-10
Linguistic encoder
Wav2vec-XI.SR-AN
Linguistic layers
14-21
Compression module
Bottleneck layers
BN dropout
0.1
FC dropout
0.1
Compression output dim
256
Stage 2 Optimization
Batch size
2
Epochs
10
GPUs
4
Audio length
5s
Optimizer
AdamW W Linear
LRscheduler
Starting LR
.0001
End LR
.00001
Early-stop patience
3 epochs 10h
Training time
Classifier
FC dropout
0.25
Stage 2 data augmentation
Num augmentations
Concat with original
True
Augment prob
Augment choices
Noise, Reverb, SpecAug
SNR_his
15dB
SNR_low
OdB
Reverb
RIR noise
Drop_freq low
Dreq_Freq_at high
I
prop_freq_count low
Drop_freq__omlee high
3
frow_freq_width
.05
Drop_chunk_comme low
5
mrop_chunk_lenghh
1000
Drop_chunnk high
2000
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of our study listed at the end of Section and summarized in the abstract. The experimental results in Section support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations our work in Section 5 and in parts of Section 4.3
In Section 2
also present gaps in the literature and the resulting limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
22

---

Answer: [Yes]
Justification: We take an experimental approach to validate our hypothesis on style- linguisties mismatch in deepfake audios. Details on the motivation and the experiments are presented in Section 3.1 The training objectives of our framework are detailed in
Section 3.2.1 Equations
and 2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Our proposed framework is described in detail in Section 3 We also provide details regarding model architecture and its implementation (Appendix A.4 A.5), pseu- docode of the training objective (Appendix (.5), and hyperparameters used for obtaining the reported performance (Appendix A.7).
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility.
23

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All datasets employed in this study are publicly available (Appendix A.3). While the training script and model weights are not explicitly released, we provided sufficient details in Section
S and Appendix A faithfully reproduce and verify our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
htapii//guiiee//oCeeuubimiiie inolliccc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co... ciaipss.c/puulcc//uuuuu http://gggiidee/ddSSuubbbbiii swi.suubiiiiiiooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test procedures detailed in Section 4.1 The data splits provided Appendix A.3 The hyperparameters are listed Appendix A.7
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted statistical significance tests for our experiments. The details can be found in Section and
and Section 4.2
Guidelines:

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources described in Appendix A.7
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Authors have reviewed and respect NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification: As described in Section [ our proposed framework is designed to detect deepfake audios, hence our work can be used for positive societal impacts. We have also presented limitations in Sections 4.3 and 6 to scope-limit the scenarios in which our framework can be used.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The employed datasets, code, and model backbones all properly cited The URLs for obtaining pretrained models are also provided (Ap-
(Sections 3 4.1
4.Tb.
pendix A.2
Guidelines:
26

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "h5qKHYpboV", "reviewer": "Reviewer_DiH3", "summary": "This paper proposes a new method for audio deepfake detection by first employing self-supervised pre-training on real samples only and then used to do real/fake classification. The proposed method achieves SOTA performance in both within-domain and cross-domain scenarios.", "questions": "See the Weaknesses part.", "limitations": "I did not see any obvious limitations for this work. It is a fairly good paper but not very impressive to me.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed technique is sound and reasonable. Learning the correlation between style and linguistics for detecting deepfakes makes sense to me.\n\n2. The comprehensive ablation studies in Table 2 further verify the effectiveness of the proposed method.\n\n3. The analysis (Interpretation of model decisions) and visualization (Mel-spectrograms) are reasonable.", "weaknesses": "1. The idea of capturing the mismatch between style and linguistics is promising, but it's unclear how this mismatch correlates with deepfake samples. More intuitive illustrations and examples are needed to better validate this motivation.\n\n2. In Figure 2 (training framework for ADD), the source of supervision for style and linguistics is not apparent. How do you ensure that each encoder learns the corresponding features? Additionally, how do you achieve perfect disentanglement between the style and linguistics encoders?\n\n3. It appears that the latest dataset used is ASVspoof2021, which is quite old. Why not incorporate more recent and advanced deepfake datasets for evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing our manuscript and for acknowledging our contribution to the field. We have provided a point-by-point response as follows:\n\n- “The idea of capturing the mismatch between style and linguistics is promising, but it's unclear how this mismatch correlates with deepfake samples.”\n\nAs the idea of style-linguistics mismatch has not been systematically investigated before, we first referred to existing linguistic studies that show concrete examples of how these two aspects are correlated in real speech (Section 2.2, Page 3, line 111-113), e.g., the impact of emotional status on word choices. Given that main-stream TTS and VC methods model these two aspects independently, such subtle correlation in real speech may be missing in deepfakes. For example, VC systems swap the original voice identity with a new one, without considering if the new voice would match in style with the word choices. To verify that the hypothesized mismatch does exist in deepfakes, we then provided preliminary results of CCA analysis (Section 3.1, Page 3-4, line 117-142; Table 1) that show a significantly higher correlation value of the two aspects in real speech and a lower correlation in deepfake speech, which also aligns with the distance between the dependency features learned by SLIM (Page 8, Figure 2). Although it is challenging to exhaustively list all mismatch cases, we provided a spectrogram illustration in Figure 4 (top right), which demonstrates a deepfake sample identified by SLIM that shows abnormal rhythm of pauses when uttering certain words. We agree that more examples could benefit the understanding of the mismatch, which requires a more systematic and detailed investigation. We plan to pursue this for future analysis.\n\n- ”In Figure 2 (training framework for ADD), the source of supervision for style and linguistics is not apparent. How do you ensure that each encoder learns the corresponding features? Additionally, how do you achieve perfect disentanglement between the style and linguistics encoders?”\n\nWe acknowledge that a perfect disentanglement of the two aspects is a challenging task. However, based on existing works on how information propagates through self-supervised learning (SSL) model layers (reference listed in Section 2.2, page 3, line 106-109), it is possible to obtain two representations which have maximal information about one of the aspects while retaining minimal information of the other. In our work, we tried to limit the entanglement by choosing and freezing the pretrained embeddings fine-tuned for tasks that are likely independent of each other, i.e. ASR for linguistics and SER for style. To ensure a satisfactory disentanglement of our adopted representations, we performed a correlation analysis in Appendix 1, Figure 5, where the average correlation value of the two representations is close to 0. These results help to ensure that the two input representations are maximally disentangled (if not perfectly). Due to page limit, we were not able to integrate these analysis results into the main text.\n\n- “It appears that the latest dataset used is ASVspoof2021, which is quite old. Why not incorporate more recent and advanced deepfake datasets for evaluation?”\n\nWe employed four test datasets, out of which both the In-the-wild and MLAAD are newer datasets than ASVspoof2021. The MLAAD was the latest one at the time of writing, of which the latest version was released in April 2024. While the ASVspoof2021 is not the latest dataset, it does have the advantage of covering a variety of attacks, which is summarized in Appendix 2, Table 3 (Page 16). The ASVspoof 2021 dataset also includes different types of lossy codecs with varied bitrates, typically used for media storage, facilitating the evaluation of model robustness to codecs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "APPQFI5pSV", "reviewer": "Reviewer_4aqZ", "summary": "\"SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection\" describes a motivation and systematic approach to  disentangling different components of speaking characteristics, in order to perform audio deepfake detection. This paper demonstrates a working 2-stage training pipeline, numerous ablations, and metrics over several English language focused datasets. In addition, qualitative demonstrations of the space of features learned and used by the model form a study which backs the philosophical approach of the paper with regard to disentangling stylistic characteristics of real speakers, in order to have a generalized defense against audio deepfakes.", "questions": "What are the systems tested in Table 1? Either by name, or citation? What is the source of the speakers? If these have PII, a description of the speakers broad categorizations is sufficient. If pulled from an existing dataset, speaker ids would be good. As it stands this table is largely uninformative, without any material information beyond a general design motivation for follow-on work (since CCA shows some behavior differences between methods).\n\nGiven the importance of both sample rate and noise in audio, it would be very useful to test this approach under those forms of degradation - e.g. does the method scale down to data of narrow bandwidth, at low samplerate or under the presence of additive noise / background sound (such as music, crowd noise, applause, and so on). The prosodic example may hold under reasonable conditions, but how many detections are relying on prosodic features versus simpler acoustic artifacts? Figure 2. hints at this to some extent, but some explicit description and study would be useful.\n\nGenerally the data examples shown are extremely noisy, and the synthesis methods are not particularly high quality. Testing on both clean audio, and higher quality synthesis, as well as under controlled degradations could raise my score. After all it is plausible an attacker may use telephony as a transmission channel - especially if the degradations imposed by the channel give the attacker a further advantage.\n\nAs a general direction - it may be useful to directly answer some of the questions posed by the titles of the citations in this paper e.g. \"Does audio deepfake detection generalize?\" - the claim here being \"yes\", but demonstrations being limited to existing datasets rather than further tests with recently developed technologies / APIs and so on. \"Does deepfake detection rely on artifacts?\" - the claim here is also (somewhat) \"yes\", which hurts the counterclaim of being generalized to some degree, unless these artifacts are general across a broad swath of methods, which would be a surprising finding given existing demonstrations.\n\nThe primary concern in order to raise my score would be a more proper scoping of the generalization claims, and the domain claims around this method given limitations of the testing datasets. The conclusion also discusses a fair bit about qualitative analyses which are largely relegated to the appendix, so there is further mismatch between the chosen title and the final claim.\n\nLarger and more diverse datasets (multi-lingual being one option, more unusual speaker styles would be another), or more particularly use of a variety of recent, high performing methods would raise my score if the writing is mostly unchanged. Some of these methods may only be available by API, which is unfortunate but perhaps necessary - additionally TortoiseTTS and spinoffs should have specific, stronger synthesis exemplars than those demonstrated, especially under the assumption an attacker may be doing manual selection given a corpus of intermediate generations to choose the best final result.", "limitations": "The authors have addressed some limitations of their work, however this review is partly hinging on the gap between claims, and the effective limitations and demonstrated results. More writing on the limitations, and particularly potential harms of deploying unbalanced \"defense\" methods in terms of accessibility would be beneficial.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall approach taken here is useful, the problem is relevant, and the overall experiments cover a portion of the necessary ground to match the claims. \n\nQualitative studies both in the main body of the paper, and the appendix are generally interesting and it is worth considering if these experiments can be fit into the unified paper \"flow\" for this publication, especially given final conclusion writing. Overall ablation study and description of methods are well done, and form a high quality \"core\" to the paper. Architecture figures and ablation tables are both outstanding in terms of clearly communicating approach, and variations in the results. Use of open source toolkits, and sharing of hyperparameters should help reproducibility.", "weaknesses": "The portion of the title \"... Generalized Audio Deepfake Detection\" claims a robust and general method for \"audio deepfake detection\". However, there are a broad range of prosodic styles from people with impaired speech, new learners of languages, children developing their ability to communicate and so on.\n\nFor a paper about generalized deepfake detection, where such detection keys particularly on the prosody of at least one example, I would like to see a larger study and example of voices in-the-wild, as opposed to the current examples which seem to be reasonably fluent speakers and performers, who may have \"trained\" speech patterns by and large. This is despite the use of the \"in-the-wild\" dataset, which doesn't seem truly wild in terms of robustness testing. Defense approaches should have some example study and discussion of False Positives beyond pure metrics (though the metrics discussions here are well done) - particularly when out-of-domain speaking patterns may have heavy overlap with \"deepfake\" data and the features used for classification, demonstrated as part of the paper. The examples in Figure 4. along with accompanied writing are a start down this road but not sufficient.\n\nMozilla Common Voice has a decent amount of this type of truly-in-the-wild speech for some qualitative study, and there are existing papers which use the same dataset for few-and-zero-shot TTS and voice conversion. The \"in-the-wild\" dataset here seems to largely focus on imitative TTS and voice conversion, and their \"real\" counterparts, which would generally point to celebrities, politicians and other public figures who (very likely) do not have the types of speech patterns mentioned previously. Though dysarthric speech is mentioned briefly in the limitations section, the issues which crop up from study of dysarthric speech are also found to some extent in many \"typical\" speakers as well, in more subtle ways so directly addressing this with some examples would strengthen the core claim of the paper in regards to \"generalized detection\".\n\nMLAAD is multilingual, but some details of the dataset construction lead to limitations in its testing (outside the scope of this paper, beyond the continued critique that broader datasets and synthetic generation methods are needed to test generalization). However, here only the EN subset appears to be used - which again reduces the claims from the title since it means the bulk of testing is on English locales. This is not a problem in terms of the experiments, but the writing and claims of the paper should be limited around this fact. Additionally, these are speech deepfakes not the broader category of \"audio\" per-se, so maybe some further adjustment is warranted, though other papers in this subarea tend to use \"audio deepfake\" to describe speech deepfakes.\n\nAs it stands, the examples shown do not convince me that the \"attacks\" used here are sufficiently high-quality to claim a generalized defense, though the developed method seems to perform well on the datasets used, and the overall scientific study (though limited) is well done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time spent reviewing our work and for sharing your detailed comments which helped us to revise our work. A point-by-point response to the posted questions can be seen below:\n\n- “What are the systems tested in Table 1?...”\n\nAs the samples referred to in Table 1 are part of the ASV2019 training data, the detailed PII is not available. We agree that detailed information of the generative models will be useful. Considering that each deepfake attack in ASV2019 has its own synthesis pipeline, and the relevant details are already summarized in the ASV2019 summary paper [1] , we will add a reference to the appropriate section and guide readers to refer to the summary paper for details.\n\n- “..., it would be very useful to test this approach under those forms of degradation.”\n\nBased on the first part of the question on robustness to degraded speech, we performed an evaluation of SLIM on 12 different codecs within the more recent ASVspoof5 dataset (released in July 2024, two months after our initial submission to NeurIPS). The results are summarized in Table 1 in the global rebuttal PDF. Note that ASVSpoof5 includes the Opus codec, which is used in telephony systems. In addition, we point out that the ASVspoof 2021, which is one of the test datasets, includes different types of lossy codecs with varied bitrates, typically used for media storage.\n\nRegarding the second half of the question on prosodic features vs simpler acoustic artifacts, we agree that separating the two categories of speech samples would definitely help in identifying the true source of deepfake. However, when we performed listening tests and spectrogram visualizations on the deepfake samples, there were many cases where a sample could manifest a combination of artifacts and style-linguistics mismatch pattern. It was therefore challenging to divide samples into two distinct categories and perform testing separately. To gain some insight, we performed an ablation on SLIM (Table 2 - SLIM variants), where we experimented using only the SSL features (rows 1-3 under `SLIM variants’; corresponding to the artifact cases), only the dependency features (row 4; corresponding to the mismatch cases), and the combination of the two (row 5; corresponding to leveraging both sources). The difference in performance can be used to gauge the question of the actual source of the deepfake samples. Our results show that the mismatch cases could be a smaller portion in ASV2021, since the dependency-alone performance was much worse than using SSL features.  For the In-the-wild and MLAAD datasets, the performance of dependency features is on par with, if not better than, SSL features, demonstrating that these two datasets have more mismatch cases. \n\n- ” Testing on both clean audio, and higher quality synthesis, as well as under controlled degradations could raise my score.”\n\nWe evaluated our model on the ASVspoof5 dataset released in July 2024, where the most recent generative models were used together with different codec degradations. We provide a breakdown of the model performance with regard to different types of unseen attacks, as well as unseen codecs. These results can be found in Table 1 and Table 2 in the rebuttal PDF.\n\n- ”The primary concern in order to raise my score would be a more proper scoping of the generalization claims,...”\n\nWe agree that our current system only operates on English data and cannot yet handle all prosodic styles (as briefly discussed in Limitations). In the introduction (Page 1, line 30), we specified that the current SOTA methods lack generalizability to unseen attacks, which we aim to tackle in this study. Following your suggestion, we will revise the manuscript in various parts, including the abstract, introduction, and limitations, to scope-limit our use of the term generalization to “unseen attacks”. \n\nAudio deepfake detection (ADD) in the ADD community indeed most commonly refers to speech deepfakes, we will clarify this in our paper. Regarding multilinguality, our current approach leverages pretrained embeddings that were trained on tasks from English data (e.g. emotion recognition), so we are currently limited by the availability of language-specific high quality pre-trained embeddings. \n\nRegarding evaluation on prosody styles, we point that our test sets cover a decent variety of speakers (e.g., 58 celebrities in the In-the-wild, and 100+ speakers in total for all test datasets). Performance across diverse test sets indicates our model’s ability to do well on a variety of speaking styles. However, we acknowledge that an extensive evaluation on special prosodic styles such as pathological speech, children speech, or new learners of a language, has not been included in our study due to small sample size, limited variability of speech content, and noise issues (e.g., [2], [3]). We will acknowledge this limitation in the paper. \n\n- ”Larger and more diverse datasets (multi-lingual being one option, more unusual speaker styles would be another), or more particularly use of a variety of recent, high performing methods would raise my score if the writing is mostly unchanged.”\n\nWe performed evaluation of SLIM on the ASVspoof5 data, where some of the recent main-stream platforms (e.g., tortoiseTTS) are used for generating the deepfake data. The results are reported in Table 1 and Table 2 in the rebuttal PDF.\n \nReference:\n\n[1] X. Wang, et. al., “ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech,” Elsevier Computer Speech & Language Journal, vol. 64, 2020. \n\n[2] Coppock, Harry, et al. \"COVID-19 detection from audio: seven grains of salt.\" The Lancet Digital Health 3.9 (2021): e537-e538.\n\n[3] Schu, Guilherme, et. al., \"On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches.\" IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZI7sAC3b9P", "reviewer": "Reviewer_chQg", "summary": "The paper suggests a novel method for detecting synthesized speech. Namely, the framework introduced in the paper allows the detection of a statistically significant mismatch between the style (i. e. paralinguistic attributes) and linguistic characteristics of synthesized speech samples, which helps to differentiate them from the real speech samples. The framework is based on frozen pre-trained SSL encoders and relatively small learnable parts that make the experiments computationally feasible for a small cluster.\n\nThe authors compare their method with several baselines, and it outperforms SoTa model in a cross-domain setup.", "questions": "Questions:\n\n- Is the confidence of your detector connected with the severity of the artifacts in the synthesized speech samples? Do the mel spectrograms of the most confidently correctly detected synthesized samples contain the most clearly visible artifacts?\n- Which types of TTS models are easier to detect, and which are harder to detect by your method?\n- You provided examples of correctly classified speech samples. What about incorrectly classified ones? Can you share an explanation or guess what are the peculiarities of these \"complicated\" samples that prevent them from being correctly classified by your method?\n- Do you plan to upload your code to Git Hub?\n\nSuggestions for the current paper:\n- I suggest adding the visual analysis of mel spectrograms of the **incorrectly** classified speech samples to the paper.\n\nSuggestions for future research:\n- Your current research makes a step forward toward more explainable synthesized speech detection. Another step in the same direction was made in Topological Data Analysis for Speech Processing by Tulchinskii et al. (2023). They have shown that synthetic and real speech samples can be separated using the barcodes of the attention maps of the HuBERT model. The sum of the bars in the barcode for real speech is bigger than the sum of bars in the barcode for the synthesized speech - at least on some attention heads of the HuBERT. It would be interesting to investigate in more detail, how the topological characteristics (i. e. barcodes or other characteristics) of the speech embeddings are connected with the linguistic and paralinguistic properties of the speech.", "limitations": "The authors addressed the limitations adequately.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of the proposed method is non-trivial and allows us to better understand the differences between real and synthesized speech.\n- The proposed method outperforms SoTa on out-of-domain data and is on par with SoTa on in-domain data.\n- The evaluation is good, and the method is compared with a number of very decent baselines.\n- The paper is well-written and easy to follow.", "weaknesses": "- The Analysis section is somewhat limited (especially the \"Interpretation of model decisions\" part). The authors claim that the success of their method is connected with very particular artifacts in synthesized speech; however, this point is weakly supported since only four small cherry-picked correctly classified examples were provided (see the \"Questions\" part).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time spent reviewing our manuscript and for finding our work innovative. A point-by-point response to your questions can be seen as follows.\n\n- ”Is the confidence of your detector connected with the severity of the artifacts in the synthesized speech samples? Do the mel spectrograms of the most confidently correctly detected synthesized samples contain the most clearly visible artifacts?”\n\nWe did notice that the low-quality samples (i.e., those with NISQA MOS < 2) were labeled as deepfakes with high confidence by SLIM. These samples were correctly classified with high confidence by the baseline models compared in Table 2. For the ones that did not contain a significant amount of noise, we did not observe significant correlation between the detector confidence scores and the severity of artifacts. We also did not see clearly separable patterns in the mel-spectrograms when comparing the most confident correctly detected samples with less confident ones. While mel-spectrograms are useful as a supplementary tool for studying samples, in our experience they don’t fully reveal all the deepfake artifacts that are important for a model to make a decision. \n\nWe would like to also point out that the actual source of the deepfakes remains an open question. This is one of the reasons why recent works have begun focusing on \"interpretation-driven detection,\" e.g. formant analysis to report deepfakes. In our case, the `interpretation’ is incorporated into our model design, where the distance between pairs of style-linguistics dependency features can be directly used to quantify the mismatch (Figure 2, Page 8). We also show that the dependency features are complementary to the features that focus on the deepfake artifacts. Such complementarity can be seen from Table 2 - SLIM variants (Page 7), where models using only SSL features perform better on ASVspoof2021, whereas the models using dependency features outperform on MLAAD. Fusion of the two resulted in the best performance.\n\n- “Which types of TTS models are easier to detect, and which are harder to detect by your method?”\n\nAs some details of the TTS models used in employed datasets are not known, we performed an analysis on the recent ASVspoof5 dataset and presented a breakdown of model performance for different attacks and codecs (Table 1 and 2 in the rebuttal PDF). In general, degradation is seen when codecs with lower bit rates are applied. We found systems with zero-shot capability are harder to detect than other methods (e.g., YourTTS).\n\n- ”You provided examples of correctly classified speech samples. What about incorrectly classified ones? Can you share an explanation or guess what are the peculiarities of these \"complicated\" samples that prevent them from being correctly classified by your method?”\n\nWe will add incorrectly classified samples to the paper. In general, we observed that severely degraded samples (e.g. audio too short/noisy) were commonly misclassified. This could be due to the design of our model, which by nature may require longer duration to capture the style-linguistics mismatch. The observation here also aligns with the difference seen between style/linguistics-only features and dependency features in Table 2, where the former performs better on ASVspoof2021 and the latter performs better on MLAAD.\n\n- ”Do you plan to upload your code to GitHub?”\n\nSince we are currently filing for IP, we do not currently plan to release the training code. However, we provided details in the paper to facilitate easy implementation of our model. The Appendix includes a detailed description of the model architecture (Appendix A.3), training hyperparameters (Appendix A.6), the list of pre-training datasets (Appendix A.2), and a Pytorch-style pseudocode of the training objective (Appendix A.4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YyO4lIB9zJ", "reviewer": "Reviewer_1UM9", "summary": "This paper proposed a new approach named Style-Linguistic Mismatch (SLIM) for generalizable audio deepfake detection. The authors claimed that a certain dependency between linguistic information and style information can generalize well for audio anti-spoofing tasks. Additionally, the proposed method can also explain the final decision of the deep learning model. In order to prove the claim, a proof-of-concept experiment was conducted to show that real audio has a higher correlation coefficient between linguistics and style in Table 1. Based on this, the authors proposed a two-stage learning framework. In the first stage, the dependency is captured by two compression modules based on SSL features. In the second stage, a simple projection head is trained on the combination of the extracted dependency with the SSL features. \n\nThe authors conducted sufficient experiments to demonstrate the generalization ability of the proposed method. Two in-domain evaluation datasets and two out-domain evaluation datasets were used for this point. Besides, the performance beats several SOTA baseline methods on these datasets, enhancing the reliability of this work. The authors also provided enough analysis and visualization to prove their hypothesis. \n\nOverall, it is a good paper for generalizable audio deepfake detection task.", "questions": "I am curious whether the author applied their method to other unseen attacking scenarios. For example, in [1], the author proposed a new unseen scenario, where some audio genres in the evaluation dataset are not presented in the training dataset. I think if the author can provide some results on this scenario, it would be more convincing for their proposed method.\n\n[1] Zeng, Chang, et al. \"Improving Generalization Ability of Countermeasures for New Mismatch Scenario by Combining Multiple Advanced Regularization Terms.\" arXiv preprint arXiv:2305.10940 (2023).", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Explicitly explored the dependency between style and linguistics for audio deepfake detection\n2. Pay attention to interpretable and generalizable audio deepfake detection simultaneously, which is novel to the community.\n3. Good performance was shown in the experiments, especially significantly improving the performance on the out-domain datasets, which is quite crucial for anti-spoofing tasks.", "weaknesses": "1. The first training stage adopts an idea from an abnormal detection task, in which only real audio (normal data) was used for training. However, the real normal data is of a much larger order of magnitude than that used in the experiment. These corner cases should be considered in the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time spent reviewing our manuscript and pointing us to another dataset that we believe would be a great fit for further evaluation of the proposed method. Below is our point-by-point response:\n\n- “magnitude of stage 1 training data”\n\nWe acknowledge that our current approach is limited to available real data. As listed in Section 2, the datasets for Stage 1 are currently limited to CommonVoice and RAVDESS. As a follow-up for more robustness, we plan to leverage larger pre-training datasets, for example, those used in learning self-supervised speech representations. We will clarify this in the Limitations section. \n\n- “applying SLIM to applied other unseen attacking scenarios”\n\nThank you for the reference, this is indeed an interesting setup and we will extend our experiments accordingly in the future. However, our current scope of “unseen” attacks is limited to English datasets and VC/TTS systems in public evaluation datasets. (Please also see our response to Reviewer 4aqZ regarding our problem scope.) Since the recommended dataset is in Chinese language, our model will need to be retrained on Chinese data to perform the evaluation. In the future, we plan to extend our investigation to include more recent TTS/VC methods as well as more varied genres and languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful comments and suggestions; we appreciate all reviewers’ positive feedback on our fundamental approach motivated by the style-linguistics mismatch modeling for deepfake speech detection, our experiments, and our overall paper presentation. 

The reviewers’ common concerns are mainly on the scope limit of our model/claims and the extension to more diverse and multilingual data. We clarify that our current scope is limited to speech deepfake detection in English. The proposed model performs well on a variety of unseen deepfake attacks and varied types of signal degradation. We agree that it is important to extend our study to multiple languages as well as to more diverse speaking styles. Both these directions are currently on our research roadmap.

There were similar questions related to robustness to compression codecs (Reviewer 4aqZ), detailed analysis on performance obtained under different / more recent generative models (Reviewer chQg and Reviewer 4aqZ), and how the model performs on more challenging datasets / datasets with more genres (Reviewer 4aqZ and Reviewer 1UM9). At the time of our initial submission, MLAAD was the most recent open-source dataset (version 3 released in April 2024), which we employed as one of the evaluated datasets. However, it did not incorporate different codecs. While ASVspoof2021 indeed includes codecs, the types of generative models were not state-of-the-art. Following the questions brought up by the reviewers, we performed an extra round of evaluation of SLIM on the ASVspoof5 dataset (released in July 2024, after our initial submission to NeurIPS) which has 10+ types of codecs and more recent TTS and VC systems. We report a breakdown of the evaluation results in the attached PDF file. Restricted by the constraints on training data in the ASVspoof5 challenge, we were not able to use the same Wav2vec-XLSR as backbones, and substituted them with WavLM-Base backbones. Apart from a few resulting changes in the training hyperparameters, the overall training strategy remains very much the same. To respect the anonymity rule, we confirm that the results in the PDF file were analyzed and created only for the rebuttal phase and they do not overlap with any public information. 

The following results can be found in the attached PDF file:
- Breakdown of SLIM’s performance under clean and 12 different codec conditions
- Breakdown of SLIM’s performance for 16 different unseen attacks

In general, we see that the proposed model generalizes well across different codecs and attack types in ASVSpoof5.

Meta-review: Frequently lengthy detailed reviews, and discussion about the content of a paper is due to drilling down into the weaknesses of the work, and disagreements as to the significance of the limitations between author and reviewer.  However, when lengthy discussions are accompanied by Accept recommendations, these are strong evidence of reviewer enthusiasm.

The broad and deep advances of generative models motivate comparable attention given to deepfake detection.   Much of the discussion and limitations of the work come from what extensions can be done, specifically multilinguality, and how this approach could be evaluated on more realistic attacks.  The reviewers (and this meta-reviewer) find that the authors have made a sufficient and significant effort to evaluate against the highest quality public test sets.  Of course, this is a dynamic environment with new attacks being developed contemporaneously with defense techniques.

The central idea of using the style-linguistic signal to identify deepfakes is novel and effective.  It does not solve every problem or identify every deepfake, but it is an worthwhile contribution to the topic, well described and evaluated.
Decision: Accept (poster)