Utilizing Speaker Profiles for Impersonation Audio Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:48:25

Abstract: Fake audio detection is an emerging active topic. A growing number of literatures have aimed to detect fake utterance, which are mostly generated by Text-to-speech (TTS) or voice conversion (VC). However, countermeasures against impersonation remains an underexplored area. Impersonation is a fake type that involves an imitator replicating specific traits and speech style of a target speaker. Unlike TTS and VC, which often leave digital traces or signal artifacts, impersonation involves live  human beings producing entirely natural speech, rendering the detection of impersonation audio a challenging task. Thus, we propose a novel method that integrates speaker profiles into the process of impersonation audio detection. Speaker profiles are inherent characteristics that are challenging for impersonators to mimic accurately, such as speaker's age, job. We aim to leverage these features to extract discriminative information for detecting impersonation audio. Moreover, there is no large impersonated speech corpora available for quantitative study of impersonation impacts. To address this gap, we further design the first large-scale, diverse-speaker Chinese impersonation dataset, named ImPersonation Audio Detection (IPAD), to advance the community's research on impersonation audio detection. We evaluate several existing fake audio detection methods on our proposed dataset IPAD, demonstrating its necessity and the challenges. Additionally, our findings reveal that incorporating speaker profiles can significantly enhance the model's performance in detecting impersonation audio.

Main paper: 
 introduction: 
 Over the past few years, speech synthesis and voice conversion technologies have made great improvement, enabling the generation of high-fidelity and human-like speech [29, 44] . However, the misuse of these technologies can facilitate the spread of misleading information and contribute to cybercrimes such as fraud and extortion [42] . Given the devastating consequences of fake audio, fake audio detection has become an urgent and essential task that needs to be addressed.In recent years, a growing number of scholars have made attempts to detect fake audio. Most previous fake audio detection research has primarily focused on four kinds of fake types: textto-speech, voice conversion, replay and partially fake [42] . Textto-speech (TTS) [34] is a technique that generates intelligible and natural speech from any given text via deep learning based models. Voice conversion (VC) [34] aims to alter the timbre and prosody of a given speaker's speech to that another speaker, while keeping the content of the speech remains the same. These two spoofing techniques are widely used in a series of competitions, such as the ASVspoof [31] and ADD challenge [40, 41] . Replay attack [17] is referred to as a form of replaying pre-recorded genuine utterances of a target speaker to an automatic speaker verification (ASV) system. Partially fake [39] focuses on only changing several words in an utterance. where fake segment is generated by manipulating the original utterances with genuine or synthesized audio clips. Despite the considerable attention given to these spoofing techniques [9] , countermeasures against impersonation remain relatively underexplored [25, 34] .Impersonation [9, 25, 34] entails an imitator mimicking specific traits associated with the prosody, pitch, dialect,lexical and speech style of a particular target speaker. This form of fake audio is generated by real human beings and poses a significant threat to speaker verification systems, as criminals could potentially use impersonation to gain unauthorized access [25] . Also, Wu et al. [34] points out that, despite its higher cost, impersonation audio is more effective at evading detection due to its naturalness, making it a challenging fake type to detection, when compared to TTS and VC.Unlike the four previously mentioned spoofing attacks, which typically leave traces via the physical characteristics of recording and playback devices, or through artifacts introduced by signal processing in synthesis or conversion systems, impersonation audio is entirely natural speech produced by actual human beings [25, 34] . This makes the detection of impersonation audio a challenging task. Thus we propose an innovative method that integrates the speaker profiles into the detection of impersonation audio. Speaker profiles refer to inherent attributes such as the speaker's age, hometown, job and so on. We aim to leverage these inherent characteristics that are challenging for impersonators to imitate accurately for impersonation audio detection. As speakers from the same hometown typically share accents and those with the same job often use a similar lexicon, a graph-based approach is ideal for modeling the interconnected relationships between these attributes [22, 27] . Accordingly, we introduce a speaker profile extractor that employs a mutual information-based graph embedding method [22, 27] to gather speaker profile information. Subsequently, the features enriched with the speaker profiles are integrated with the features derived from the front-end feature extractor module through a fusion module. Finally, the fusion module's output is fed to the back-end classifier, which generates the high-level representation aiming at distinguishing between impersonated utterances and genuine ones. 1 : Characteristics of representative datasets on fake audio detection. # Utts, # Hours and # Spks represent number of utterances, hours and speakers, respectively. We will make our dataset publicly available once our paper is accepted.However, a significant obstacle is the absence of large-scale impersonated speech datasets, limiting quantitative analysis of impersonation effects, primarily due to the challenges associated with acquiring high-quality impersonation data, which is both scarce and expensive. An impersonation dataset is designed by [18] , focusing on investigating the vulnerability of speaker verification. However, only two inexperienced impersonators are involved to mimic utterances from YOHO corpus [4] . Hautamäki et al. [9] constructs a small Finnish impersonation dataset in 2013. All these prior datasets suffer limitations such as few speakers and short durations. Addressing these gaps, this paper presents a diversescenarios, diverse-speaker impersonation dataset, named ImPersonation Audio Detection (IPAD), to benefit the community's research. As Sahidullah et al. [25] indicate that professional impersonators result in higher deception rates than amateurs, we specifically curated our dataset with audio from skilled impersonators, making our IPAD dataset more pratical. Moreover, we are committed to ensuring a balanced distribution of speakers across different age groups and genders.The main contributions of this paper are as follows:• We propose a novel method that integrates speaker profiles into the detection of impersonation audio. To this end, we utilize a graph-based approach to extract speaker profile information. Additionally, our proposed method does not require labeled speaker profiles during the test period. • We present the first diverse-scenarios, diverse-speaker impersonation dataset, named ImPersonation Audio Detection (IPAD), to promote the community's research on impersonation audio detection. The impersonation dataset will be publicly available.and demonstrated our speaker profiles integrated method can achieve impressive results. 
related work 2.1 fake audio detection methods: 
 In recently years, many detection methods have been introduced to discriminate fake audio files from real speech, mainly focusing on the pipeline detector and end-to-end detector solutions [42] .The feature extraction, which aims to learn discriminative features via capturing audio fake artifacts from speech signals, is the key module of the pipeline detector. The features used in previous can be roughly divided into two categories [42] : handcrafted features and deep features. Linear frequency cepstral coefficients (LFCC) is a commonly used handcrafted features that uses linear filerbanks, capturing more spetral details in the high frequency region. LFCC in conjunction with Gaussian Mixture Models (GMM) and Light Convolutional Neural Networks (LCNN), have been adopted as the baseline models for ASVspoof 2021 [31] and ADD challenge. [40, 41] . Nevertheless, handcrafted features are flawed by biases due to limitation of handmade representations [43] . Deep features, derived from deep neural networks, have been proposed to address these limitations. Pre-trained self-supervised speech models, such as Wav2vec [3] , Hubert [11] and WavLM [5] , are the most widely used ones [30] . Wang and Yamagishi [30] investigate the performance of spoof speech detection using embedding features extracted from different self-pretrained models. The back-end classifier, tasked with learning high-level feature representations from the front-end input features, is indispensable in the fake audio detection. One of the extensively used classifiers is Light CNN (LCNN) [33] , as it is an effective model employed as the baseline model in a series of competitions, such as ASVspoof 2017 [17] , ASVspoof 2019 [21] and ADD 2022 [40] .End-to-End Models are deep neural networks that integrate feature extraction and classification in an end-to-end manner have shown competitive performance in fake audio detection. Notable models include RawNet2 [14] and its derivatives, RawNet3 [15] and TO-RawNet [28] ; the Differentiable Architecture Search (DARTS) influenced Raw PC-DARTS [8] ; Transformer-based Rawformer [37] ; the Graph Neural Network-based AASIST [13] and its orthogonal regularization variant, Orth-AASIST [28] ;.However, previous models have primarily targeted fake types such as TTS and VC, which often leave digital traces or signal artifacts. In contrast, impersonation audio is entirely natural speech produced by real humans, making it challenging for these methods to effectively detect. Filling this gap, in this paper, we propose a novel method that integrates speaker profiles that are inherent characteristics that are challenging for impersonators to mimic accurately into the detection of impersonation audio.The advancement of fake audio detection techniques significantly depends on well-established datasets, which encompass various fake types and diverse acoustic conditions. Table 1 summarizes the characteristics of representation datasets in the field of fake audio detection along with our proposed dataset. Most earlier spoofed datasets were primarily developed to bolster defenses against spoofing attacks in ASV systems. Moreover, the spoofing types are not diverse. Some spoofing datasets focus exclusively on a single type of TTS method [19] or a specific VC Method [1] . To alleviate this issue, Wu et al. [35] design a standard public spoofing dataset SAS which consists of various TTS and VC methods. The SAS dataset is used to support ASVspoof 2015 [36] , which aims to detect the spoofed speech. Replay is considered as a low cost and challenging attack included in the ASVspoof 2017 challenge [17] . The ASVspoof 2019 [21] and 2021 datasets [31] both consist of replay, TTS and VC attacks.In recent years, a few attempts have been made to design datasets mainly for fake audio detection systems. In 2020, Reimao and Tzerpos [23] developed a publicly available dataset FoR containing synthetic utterances, which are generated with open-source TTS tools. In 2021, Frank and Schönherr [7] developed a fake audio dataset named WaveFake, which contains two speaker's fake utterances synthesised by the latest TTS models. However, these datasets have not covered some real-life challenging situations. The datasets in ADD 2022 challenge [40] are designed to fill the gap. The fake utterances in LF dataset are generated using the latest state-of-the-art TTS and VC models, which contain diversified noise interference. The fake utterances in PF dataset are chosen from the HAD dataset [39] designed by, which are generated by manipulating the original genuine utterances with real or synthesized audio segments.Few previous studies have been dedicated to the construction of voice imitation datasets. In 2004, an impersonation database is developed by [18] , which is used for investigating the vulnerability of speaker verification. Two novice impersonators were tasked with mimicking voices from the YOHO corpus. They listened to and subsequently imitated 40 training utterances from selected speakers. In 2013, a small Finnish impersonation dataset was designed by [9] . Our dataset IPAD markedly distinguishes itself from previous efforts by incorporating a significantly larger number of speakers. Additionally, our audio is extracted from videos downloaded from entertainment programs on web media, enhancing the practicality of our IPAD dataset. 
method: 
 In this section we provide details of the developed methods to detect impersonation audio. An overview of our approach is illustrated in Figure 1 .Speaker profiles include different attributes, such as speaker's age, hometown. For each attribute, we will learn a attribute encoder to extract attribute-specific information.Speakers with the same attribute value often exhibit similar characteristics. For instance, speakers from the same hometown typically share similar accents. Thus, we employ a graph-based approach, in which we model each audio in the training set as a node, to effectively model the interconnected relationships between speaker attribute.We first give a introduce to the problem statement. Suppose we are provided with a set of node features, in our method, i.e. audio features, X = {x 1 , x 2 , . . . , x 𝑁 }, where 𝑁 is the number of nodes in the graph and x 𝑖 ∈ R 𝑓 encodes the feature of node 𝑖. We are also provided with relational information between these nodes in the form of an adjacency matrix, A ∈ R 𝑁 ×𝑁 . We assume that 𝐴 𝑖 𝑗 = 1 if there exists an edge between node 𝑖 and node 𝑗, for example, node 𝑖 and node 𝑗 have the same job in job attribute encoder. We draw inspiration from [22, 27] , where learn the encoder relaying on maximizing local mutual information between global summary vector and local node representations. More precisely, we learn a low-dimensional representation for each node x 𝑖 , i.e., h 𝑖 ∈ R 𝑑 , such that the average mutual information between the global summary vector s ∈ R 𝑑 , and local node representations {h 1 , h 2 , . . . , h 𝑁 } is maximized.To this end, we first introduce a attribute encoder E, consisting two linear layer with ReLU activation. Then we can generate the local node representation matrix H following Eq. ( 1 EQUATIONwhere Â = A +𝑤I N , I N is the 𝑁 × 𝑁 identity matrix. Dii = 𝑗 Âij , E is the trainable network, and 𝜎 is the ReLU non-linearity function.Our approach adjusts the impact of self-connections by introducing a weight parameter 𝑤 ∈ R. A higher 𝑤 value increases the node's self-relevance in its embedding, consequently lessening the influence of adjacent nodes.Then we can calculate the graph-level summary representation s by employing a readout function R.EQUATION)where 𝜎 is the logistic sigmoid non-linearity function, and h i represents the 𝑖-th row of the node embedding matrix H.We follow [22, 27] , introducing a scoring network D that discriminates the true samples. i.e., (h i , s) from (h ′ j , s) as a proxy for maximizing the local mutual information. D ((h i , s) represents the probability scores assigned to the patch-summary pair. Here negative representation h ′ j is obtained by row-wise shuffling, i.e. X → X ′ . The corruption function used here is designed to encourage the representations to properly encode structural similarities of different nodes. Then we calculate the H ′ following Eq.( 1). The scoring network scores patch-summary pairs by applying a simple bilinear transformation function:EQUATION)where 𝜎 is the logistic sigmoid non-linearity function, and 𝑀 is the trainable scoring matrix.Finally, we can update parameters of E, R and D by optimizing the following attribute specific cross entropy loss L 𝐺 .EQUATIONIn this subsection, we will describe the framework of our proposed speaker profiles integrated detection method in detail.For each attribute, we first train an attribute encoder to extract attribute specific information following the method described in subsection 3.1. The input audio features are obtained by passing the audio through a Embedding layer. Suppose we have 𝑠 types of attributes, we can obtain 𝑠 attribute encoders {E (1) , E (2) , . . . , E (𝑠 ) }. There encoders contain relevant information regarding corresponding speaker profile.Suppose we are provided a mini-batch of audio, W = {w 1 , w 2 , . . . , w 𝑛 }. We first employ the speaker profile extractor to extract speaker profile information. Specifically, for each w 𝑖 , we adopt an Embedding layer 𝐸, followed by 𝑠 learned attribute encoders to obtain the speaker profiles incorporated representations, then we take an average of these representations:EQUATIONHere 𝐴𝑣𝑔 represents the averaging operation. The v 𝑖 calculated by Eq.( 5) is the output of the speaker profile extractor for w 𝑖 . A key consequence is that the produced representation v 𝑖 contains speaker profile information, such as speaker's age, hometown, job. Then we can obtain V = {v 1 , v 2 , . . . , v 𝑛 }.Next, we can derive the deep features Q by employing the frontend feature extractor F 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑜𝑟 (wav2vec2 [3] for example).EQUATIONSubsequently, We amalgamate representation V from the speaker profile extractor and Q from the front-end feature extractor via a feature fusion module F 𝑓 𝑢𝑠𝑖𝑜𝑛 .K = {k 1 , k 2 , . . . , k 𝑛 } = F 𝑓 𝑢𝑠𝑖𝑜𝑛 (V, Q) (7)Here K represents the integrated representation. This is obtained by the frame-level concatenation of corresponding features from V and Q. In detail, if q 𝑖 is the feature with dimensions (𝑡, 𝑓 ), where 𝑡 represents the number of time frames and 𝑓 represents the embedding size of the frond-end feature extractor. v 𝑖 is the feature with dimensions (𝑙, ), where 𝑙 denotes the output size of the speaker profile extractor, then 𝑣 1 is replicated 𝑚 times to align with the dimensions of 𝑞 1 . As a result, the fused feature 𝑘 1 will have dimensions (𝑡, 𝑓 + 𝑙).Ultimately, we engage a back-end classifier F 𝑐𝑙𝑎𝑠𝑠𝑖 𝑓 𝑖𝑒𝑟 , Light CNN (LCNN) [33] for example, to detect fake audio. Suppose the labels for the mini-batch of audio W are Y = {y 1 , y 2 , . . . , y 𝑛 }. We can formulate the classification loss L 𝐶𝐿𝑆 as follows:EQUATION)where ŷ𝑖 is the model's predicted probability that the 𝑖-th audio is a bona fide one. Here the prediction is obtained by feeding K to F 𝑐𝑙𝑎𝑠𝑠𝑖 𝑓 𝑖𝑒𝑟 . Then we can update the parameters of F 𝑐𝑙𝑎𝑠𝑠𝑖 𝑓 𝑖𝑒𝑟 by optimizing L 𝐶𝐿𝑆 .Additionally, during the test phase, since we have already trained the attribute encoder for each speaker profile, our method does not require labeled speaker profiles for prediction. 
dataset 4.1 dataset collection policy: 
 We construct our impersonation dataset IPAD through five steps. The audio is extracted from videos downloaded from web media, resulting in the "in the wild" characteristics of our IPAD dataset.Step 1: Download Videos. In order to build our dataset from scratch, we collect videos from several variety entertainment programs that contain segments featuring impersonators mimicking others. The programs include The Sound suena 6 , Fun with Liza and Gods 7 , Copycat Singers 8 . In total, we have collected 168.34 hours of video for subsequent audio slicing in the imitation dataset.Step 2: Manual Labeling. We recruit nine annotators to label our dataset. For each video, they are tasked with identifying segments where the impersonator is speaking as themselves and segments where the impersonator is mimicking others, marking the start and end times with precision to the second for subsequent audio segmentation. For the first condition, annotations required include the speaker's name, hometown, age, job, gender, and the scenario. In instances of imitation, annotations needed to cover the impersonator's name, hometown, age, job, gender, the scenario of the imitation, as well as the name of the person being imitated. For the hometown, we require annotations to be specific to the province level. Regarding the scenario, they were categorized into dubbing, 6 https://zh.wikipedia.org/wiki/%E7%99%BE%E5%8F%98%E5%A4%A7%E5%92%96%E7% A7%80 7 https://zh.wikipedia.org/wiki/%E8%8D%83%E5%8A%A0%E7%A6%8F%E7\%A5%BF% E5%A3%BD 8 https://baike.baidu.com/item/%E5%A4%A9%E4%B8%8B%E6%97%A0%E5%8F%8C/ 19885272 conversational speech, and singing. If a segment did not fit into these three categories, it was labeled as 'other'. For virtually all videos, two annotators were assigned to provide labels. Subsequently, a reviewer would reconcile any discrepancies between the annotations, making necessary adjustments. This process was instituted to ensure the quality of the dataset.Step 3: Extract Audio from Video. we leverage FFmpeg9 to extract and convert specific audio segments from videos into mono wav files with a 16,000 Hz sampling rate. This process ensures the standardization of our audio dataset for consistent analysis.Step 4: Audio Segmentation. Following the extraction of the audio, we employed a Voice Activity Detection (VAD) tool [38] to eliminate segments of silence. For audio clips exceeding 10 seconds in duration, the VAD model 10 [38] was utilized to determine the start and end points of valid speech within the input audio, ultimately discarding non-speech parts in the audio.Step 5: Train, Dev, Test and Unseen Test Split. After acquiring the complete set of audio, we split the audio from dubbing and conversational speech scenarios into train, dev, and test sets. The allocation is based on the number of utterances per speaker, with the stipulation that the speakers across the train, dev, and test sets must be mutually exclusive to avoid any overlap. In detail, to ensure the dataset's train, dev, and test splits maintain balance in terms of age and gender, we divide speakers into four age groups: 20-35, 35-50, over 50, and unknown. For each age group and gender, speakers were allocated to train, dev, and test in a 3:2:5 ratio by utterance count. Consequently, the number of speakers designated for the train, dev, and test sets are 73, 43, and 95, respectively. Additionally, audio from singing and the other scenarios are segregated into an unseen test (unseen) set. Therefore, we can not only detect fake utterances on the test set, but also to evaluate the generalization of fake audio detection models on unseen scenarios.There are four sets in our impersonation dataset IPAD: train, dev, test and unseen test (unseen). Key statistics for different subsets of the impersonation dataset, categorized into "Real" and "Fake" with a further cumulative "Total", are summarized in Table 3 .As our IPAD dataset is partitioned into train, dev, and test subsets based on the dubbing and conversational speech scenarios, with singing and other scenarios being treated as unseen set. In Tables 4, we have meticulously compiled the number of utterances, the count of speakers, and the distribution of speaker profiles -ages, jobs, and hometowns for male and female within each specific scenario for the various subsets. 
experiments: 
 In this section, we first introduce our evaluation metric in Sec. 5.1. In the remaining subsections, we primarily address the following three questions:• Can models trained on existing fake audio detection dataset ASVspoof2019 LA reliably detect impersonation audio in Sec. 5.2 ? • How do existing models perform on the impersonation audio dataset IPAD in Sec. 5.3 ? • Does integrating speaker profiles improve performance on the IPAD dataset in Sec. 5.4?Equal error rate (EER) is used as the evaluation metric for the detection tasks. Previously, EER is used as the evaluaion metrics for fake audio detection tasks in the ASVspoof [31] and ADD challanges [40, 41] . Let 𝑃 𝑓 𝑎 (𝜃 ) and 𝑃 𝑚𝑖𝑠𝑠 (𝜃 ) denote the false alarm and miss rates at threshold 𝜃 respectively.#{fake trials with score > 𝜃 } #{total fake trials} (9)EQUATIONThe functions 𝑃 𝑓 𝑎 (𝜃 ) and 𝑃 𝑚𝑖𝑠𝑠 (𝜃 ) monotonically decrease and increase, respectively, as a function of 𝜃 . The EER corresponds to the threshold 𝜃 𝐸𝐸𝑅 at which the two detection error rates are equal, i.e., EER = 𝑃 𝑓 𝑎 (𝜃 𝐸𝐸𝑅 ) = 𝑃 𝑚𝑖𝑠𝑠 (𝜃 𝐸𝐸𝑅 ). A lower EER value indicates a model with better performance.ASVspoof2019 LA dataset 5.2.1 Experimental Setup. We evaluate the discriminative performance of different combination of frond-end features and back-end classifiers, trained with ASVspoof2019 LA [21] on our IPAD dataset. We choose the ASVspoof2019 LA dataset [21] because it is the most commonly used dataset in fake audio detection research. Our objective is to evaluate whether models trained on this dataset can effectively handle impersonation-type spoofing attacks.The handcrafted features analyzed include linear frequency cepstral coefficients (LFCC), mel-frequency cepstral coefficients (MFCC), inverted MFCC (IMFCC) and constant-Q cepstral coefficients (CQCC). LFCC is obtained using linear triangular filters. while MFCC originates from mel-scale triangular filters, designed with a denser distribution in lower frequencies to mimic the human ear's perception. IMFCC employs triangular filters arranged linearly across an inverted-mel scale, thereby giving higher emphasis to the highfrequency areas. CQCC is obtained from the discrete cosine transform of the log power magnitude spectrum derived by constant-Q transform. For all these features, we apply a 50ms window size with a 20ms shift and extract features with 60 dimensions. The self-supervised feature includes Wav2Vec 2.0 [3] which combines contrastive learning with masking and HuBERT that uses quantized MFCC features as targets learned with classic k-mean. We leverage the "wav2vec2-base 11 " and "hubert-base 12 " checkpoint from Huggingface's Transformer library [32] .We choose Light CNN (LCNN) [33] , Squeeze-and-Excitation network (SENet) [12] , Xception [24] and ResNet [10] as our back-end classifiers due to their popularity and effectiveness. Short introductions of these classifiers are provided below.• LCNN [33] consisting of convolutional and max-pooling layers with Max-FeatureMap (MFM) activation is extensively used as the baseline model of the ASVspoof [31] and ADD [40, 41] competitions. • SENet [12] dynamically adjusts channel-wise features by explicitly modeling the interdependencies between channels. • Xception [24] , which is employed as a baseline model in[16], utilizes depth-wise separable convolutions to effectively capture both cross-channel and spatial correlations. • ResNet [10] is introduces as a classifier for fake audio detection in [2] , employing a residual mapping.We also evaluate the performances of four widely used endto-end competitive models: RawNet2 [14] , Raw PC-DARTS [8] , Rawformer [37] and AASIST [13] on our proposed dataset. The four end-to-end models are trained on ASVspoof. Brief descriptions are provided below.• RawNet2 [14] operates directly on raw audio via time-domain convolution. Tak et al. [26] applied it for anti-spoofing, securing second against A17 attacks in ASVspoof 2019. • Raw PC-DARTS [8] utilizes an automatic approach, which not only operates directly upon the raw speech signal but also jointly optimizes of both the network architecture and network parameters. • Rawformer [37] integrates convolution layer and transformer to model local and global artefacts and relationship directly on raw audio. • AASIST [13] , which employs a heterogeneous stacking graph attention layer to model artifacts across temporal and spectral segments.. We report the EER (%) for front-end features combined with classifiers and end-to-end models, all trained on ASVspoo2019 LA, across ASVspoof2019 LA test set, IPAD's test set, and unseen set, as detailed in Table 5 and 7 .Results from Table 5 and 7 reveal that models trained on the ASVspoof2019 LA and tested on IPAD dataset exhibit markedly high EER (%), hovering around 50. This is not surprising, as ASVspoof2019 is mainly tailoring for identifying machine-generated audio and real audio. In contrast, our IPAD dataset comprises solely of humanproduced audio, resulting in failure detection of impersonation audio when models are trained on ASVspoof2019 LA. This indicate that models trained on the ASVspoof2019 LA struggle to detect impersonated audio, suggesting that impersonation as an attack type can significantly increase the success rate of spoofing attacks.5.3.1 Experimental Setup. The front-end features combined with classifiers and end-to-end models are the same as those evaluated in Sec. 5.2, but trained on the IPAD's train set.. For handcrafted features combined with classifiers, from Table 6 , we observe that, with the exception of the LFCC+LCNN combination on the test set and the LFCC+ResNet on the unseen set, the LFCC feature generally outperforms other handcrafted features, suggesting its effectiveness in impersonation audio detection.As presented in Tables 6, our findings reveal that self-supervised features outperform handcrafted features on both the test and unseen sets. Specifically, averaged over four different back-ends, wav2vec-base exhibits an average performance of 23.51 and 28.59 on IPAD's test and unseen sets, respectively. However, the best handcrafted feature demonstrates performance of 25.46 and 29.53 on IPAD's test and unseen sets. This indicates that pretrained models are more adept at capturing information pertinent to impersonation audio detection compared to handcrafted features. For end-to-end models, Table 8 reveals that among the evaluated models, AASIST achieves the best performance on both the test and unseen set with the lowest EER (%) at 23.73 and 30.25, respectively, demonstrating superior performance in impersonation audio detection and great generalization capabilities in unseen conditions.In this subsection, we evaluate the performance of existing models on the IPAD dataset. The results indicate that there is still significant room for improvement.profiles integrated method 5.4.1 Experimental Setup. As indicated in Sec 5.3, self-supervised features outperform handcrafted features in the detection of impersonation audio, thus we employ self-supervised features in our speaker profiles integrated method. The self-supervised models considered include Wav2Vec 2.0 [3] , HuBERT [11] and WavLM [5] . WavLM [5] , largely paralleling HuBERT, introduces advancements in spoken content and speaker identity by integrating a gated relative position bias and enriching training data with an utterance mixing approach. We utilize the self-supervised models as the frond-end feature extractor and instance the Embedding layer of the speaker profile extractor as the convolutional waveform encoder of the corresponding frond-end feature extractor. The output size of the speaker profile extractor in Sec. 3.2 is 128 in our experiments.For self-supervised pre-trained models, we leverage the pretrained checkpoint from Huggingface's Transformer library [32] . Below are the models used in our experiment: "wav2vec2-base 13 ", "wav2vec2-large 14 ", "hubert-base 15 ", "hubert-large 16 ", "wavlm-base" 17 and "wavlm-large" 18 . For the back-end classifier, we opted for the widely utilized LCNN [33] .. We report the model's performance on the IPAD dataset with ("w/") and without ("w/o") speaker profile 13 https://huggingface.co/facebook/wav2vec2-base 14 https://huggingface.co/facebook/wav2vec2-large 15 information. Results for test and unseen sets of IPAD are detailed in Table 9 .We observe that wavlm-large consistently yields the best results on both the test and unseen sets, when no speaker profiles are integrated, indicating its robust and useful audio feature extraction capabilities. Surprisingly, we find that the performance of wav2veclarge is inferior to that of wav2vec-small, and similarly, hubert-large underperforms hubert-small. We speculate that this may be due to the fact that the models were only pretrained on genuine audio, and simply increasing model size does not enhance the model's ability to detect impersonation audio. Furthermore, when comparing base models, wav2vec-small emerges with the optimal performance. We find that the incorporation of speaker profiles can significantly enhances the detection of impersonation audio. wavlmlarge with speaker profile information integrated achieved the best EER(%) of 21.97 and 23.96 on IPAD's test and unseen sets, respectively. The inclusion of speaker profiles has led to notable improvements for all six pretrained models on both IPAD's test and unseen sets. On average, the EER(%) decreased by 1.26 on the test set and by 3.17 on the unseen set. This suggests that the utilization of speaker profiles enable models to better leverage information such as the speaker's job and age in the detection of imitated audio. Additionally, the more pronounced improvement on the unseen set indicates that the introduction of speaker profiles bolsters the model's generalization capabilities in out-of-domain situations. 
conclusions: 
 In this work, we investigate countermeasures against impersonation. Different from spoofing attacks like TTS and VC, which leave physical or digital traces, impersonation involves live human beings producing entirely natural speech. We propose a novel strategy that utilize speaker profiles for impersonation audio detection. Moreover, we propose the ImPersonation Audio Detection (IPAD) dataset to promote the community's research on impersonation audio detection, filling the gap that there is no large-scale impersonated speech corpora available. To provide baselines for future practitioners, we train several existing models on our IPAD dataset. Finally, we demonstrate that incorporating speaker profiles into the process of impersonation audio detection can achieve notable improvements. Future work includes constructing an English-language impersonation dataset and exploring how to better utilize speaker profiles from other modalities for impersonation audio detection.  
 
 Appendix: https://zh.wikipedia.org/wiki/%E5%A3%B0%E4%B8%B4%E5%85%B6%E5%A2%83https://zh.wikipedia.org/wiki/%E6%88%91%E6%98%AF%E7%89%B9%E4%BC%98% E5%A3%B0https://baike.baidu.com/item/%E5%A3%B0%E6%BC%94%E7%9A%84%E5%8A%9B% E9%87%8F/57929334https://zh.wikipedia.org/wiki/%E5%BC%80%E9%97%A8%E5%A4%A7%E5%90%89https://baike.baidu.com/item/%E6%AC%A2%E4%B9%90%E6%80%BB%E5%8A%A8% E5%91%98/11011573https://ffmpeg.org/https://modelscope.cn/models/iic/speech_fsmn_vad_zh-cn-16k-commonpytorch/summaryhttps://huggingface.co/facebook/wav2vec2-basehttps://huggingface.co/facebook/hubert-base-ls960 


Decision: The paper addresses the problem of fake audio detection and proposes a method that integrates speaker profiles into the process of impersonation audio detection. It also introduces a large-scale, diverse-speaker Chinese impersonation dataset, named ImPersonation Audio Detection (IPAD). Presents the performance of selected existing fake audio detection methods on the proposed dataset IPAD. 

The proposed method employs speaker profile (such as speaker's age and job) in audio detection and utilizes a graph-based approach to extract it; which is a limited novelty as textual (semantic) information is used with similar approaches in audio-related tasks in recent work. On the other hand, overall, constructing a new large-scale dataset and evaluating it with selected methods put a value on it. After considering the paper, the reviewer's comments, and the rebuttal I recommend 'accept (poster)' for the paper.

The reviewers highlight the following strengths and limitations:

Strengths:
1. The paper proposes a novel approach to incorporate speaker profile information to enhance the detection of impersonated audio. 
2. The experiments and results are significant enough to prove the correctness of the authors’ hypothesis and to demonstrate the usefulness of IPAD Dataset. 
3. The paper is well-written and clear.

Limitations:
1. A description of the ASVspoof2019 dataset is missing.
2. The paper lacks the results of 4 end-to-end system performances when speaker profiles are integrated.

Reviews: review: paper_summary: paper_summary: Thank you to all the authors for their efforts and for the opportunity to review the manuscript. The paper presents an approach for impersonation audio detection that uses integrated speaker profiles such as their job, location, gender, etc. The authors also develop and plan to release the ImPersonation Audio Detection (IPAD) dataset for fostering further research in the area. Comprehensive experiments demonstrate the validity of the hypothesis.
strength_weakness: strength_weakness: 1. The paper proposes a novel approach to incorporate speaker profile information to enhance detection of impersonated audio. 
2. The introduction, besides minor typos is well-written. The authors carefully distinguish generated audio from natural impersonated speech describing the motivation and their contributions clearly. 
3. Figure 1 is very illustrative and useful for demonstrating the training process. However, the font size is too small. If font size can be increased, it would be more presentable.
4. The experiments and results are significant enough to prove the correctness of authors’ hypothesis and to demonstrate the usefulness of IPAD Dataset. However, improvements can be made by experimenting with performance on cross datasets. For example, how do all the baselines trained on IPAD Dataset perform when evaluated on old impersonation datasets such as Hautamäki et al. [9] and Lau et al. [18]. This could be provided as an additional experiment in the supplementary material.
limitations: limitations: 1. Section 2.1 is well-written but includes too many details about fake audio detection (synthetic) but not much detail about impersonated or mimicked audio (real) which is the main focus of the paper. Some efforts have explored methods in this direction and should be referenced to provide readers with comprehensive information. Some examples that could be included in references (not comprehensive or exhaustive list):
    1. https://ieeexplore.ieee.org/document/9215407 
    2. https://www.cs.joensuu.fi/odyssey2014/program/pdfs/54.pdf 
    3. https://link.springer.com/article/10.1007/s10772-012-9163-3
2. “The corruption function used here is designed to encourage the representations to properly encode structural similarities of different nodes.” Could you explain in slightly greater detail how is the corruption function helping the proposed method? In text, it is clear how it is implemented but not clear how it helps enhance performance. Perhaps the explanation could include some experimental validation.
3. What is the value of s used in experiments? Is it same as the number of speaker attributes, i.e. name, hometown, age, job, gender, and the scenario?

Besides, there are some minor grammatical errors and typos which could be removed by proof-reading:

1. Abbreviations should preferably be used the first time the phrase is mentioned. For example, TTS should be defined in line 65 instead of line 66. Similarly for VC. This is very minor but will improve the writing of the manuscript. 
2. “Partially fake focuses on only changing several words in an utterance. where fake segment is generated by manipulating the original utterances with genuine or synthesized audio clips.” If this is a full sentence, there should be a ‘,’ instead of a ‘.’
3. In line 83, “dialect,lexical”, there is a missing space.
4. Line 73 - full form should also be mentioned for ADD.
5. Line 87, “Wu et al.” should be treated as a plural subject instead of singular. 
6. Line 137 - “Hautamäki et al. [9] constructs a small Finnish impersonation dataset in 2013.” “constructs” should be “constructed”.
7. Line 145 - “pratical” should be spelt correctly as “practical”.
8. Line 173 - “previous can be roughly divided into two categories”, there is a “work” missing after “previous”.
9. Consistency in full-form of abbreviations, for example, in line 187, Linear frequency cepstral coefficients, either all first letters should be uppercase throughout the manuscript or this should be made consistent with other full-forms.
10. “In 2004, an impersonation database is” should be past tense with the use of “was”. 
11. Line 330 - “We first give a introduce to the problem statement.” “a introduce” should be “an introduction”

There are several more which I could not list here. A proof-read and grammar check is strongly recommended.

Overall, the proposed method is novel and significant analysis is present in the paper. However, major improvements need to be made in writing, such as correct grammar and use of correct scientific terminology as described above.
paper_summary: paper_summary: In this paper, the authors proposed a new speech signal classification problem known as impersonation detection, where a classifier is asked to determine whether a speech signal is recorded from a human subject or a human object that is impersonating someone else. 

Due to the lack of existing datasets, the authors proposed a dataset for this purpose known as IPAD. 

The authors proposed a method for impersonation detection which is trained on IPAD. The proposed method is not only based on low-level analysis of the speech signal but also high level analysis such as the deducing the speaker's gender, age, origin, etc. 

The proposed method achieved higher performance than other methods that are trained for synthetic speech detection tasks.
strength_weakness: strength_weakness: - The authors proposed the new IPAD dataset 
- The authors proposed a new method for the IPAD dataset, where the speaker attribute is fused with audio features extract from waveform for analysis
- The proposed method achieved better performance on the IPAD dataset compared to existing methods designed for synthetic speech detection
limitations: limitations: - The performance gap of the proposed method and methods trained on ASVspoof2019 is not very big. Why do methods designed for ASVspoof2019 work on impersonation detection problem?  The methods designed for ASVspoof2019 tend to operate on very short speech signals, and it is believed that they rely on low-level features to make decisions. If they also work for IPAD dataset, I think there can be two possibilities:
  1. Low-level features alone can be used for impersonation detection. However, since all signals in IPAD are human speech signals, it may be difficult to classify them based on low-level features only.
  2. The diversity of IPAD dataset is limited. That is, all methods trained on IPAD tend to overfit the dataset. The performance evaluation results are not reliable.

- In IPAD dataset, the authors only include signals from experienced impersonators. It would be interesting for authors to report the performance of the the proposed method and baseline methods on:
  - Synthesized impersonation speech
  - Impersonation speech from less skilled impersonators
paper_summary: paper_summary: this manuscript proposes a novel method that integrates speaker profiles into the process of impersonation audio detection. Speaker profiles are inherent characteristics that are challenging for impersonators to mimic accurately, such as the speaker's age, job. the authors aim to leverage these features to extract discriminative information for detecting impersonation audio. Moreover, there is no large impersonated speech corpora available for quantitative study of impersonation impacts. To address this gap, the authors further design the first large-scale, diverse-speaker Chinese impersonation dataset, named ImPersonation Audio Detection (IPAD), to advance the community's research on impersonation audio detection. the authors evaluate several existing fake audio detection methods on proposed dataset IPAD, demonstrating its necessity and the challenges. Additionally, the findings reveal that incorporating speaker profiles can significantly enhance the model's performance in detecting impersonation audio.
strength_weakness: strength_weakness: 1：authors propose a novel method that integrates speaker profiles into the detection of impersonation audio. To this end, they utilize a graph-based approach to extract speaker profile information. Additionally, the proposed method does not require labeled speaker profiles during the test period.


2：authors present the first diverse-scenarios, diverse-speaker impersonation dataset, named ImPersonation Audio Detection (IPAD), to promote the community's research on impersonation audio detection.
limitations: limitations: 1:I am concerned about the fairness of this experiment. Taking Table 5 as an example, the compared dataset contains real samples and deepfake samples, while the proposed dataset is filled with real samples, which inevitably leads to the failure of existing detection methods.

2. This manuscript lacks relevant work on non-personalized audio detection, and we hope to supplement the relevant knowledge background.

3:I am concerned about the practicality of this scenario. In most cases, the detector only has one voice clip and no prior knowledge of the speaker. In addition, the speaker's healthy condition, age, and recording time can all have impact on the speaker's timbre, and these data are difficult to track.
paper_summary: paper_summary: Fake audio detection is an emerging active topic. A growing number of literatures have aimed to detect fake utterance,which are mostly generated by Text-to-speech(TTS) or voicecon version(VC).However, countermeasures against impersonation remain an underexplored area.This paper presents the first diverse-scenarios, diverse-speaker impersonation dataset, named ImPersonation Audio Detection (IPAD), to promote the community’s research on impersonation audio detection. Also, propose a novel method that integrates speaker profiles into the detection of impersonation audio. I believe this paper has significant practical implications for the future development of audio detection. Therefore, I am inclined to accept this paper.
strength_weakness: strength_weakness: 1. The paper is well-written and clear.
2. It highlights the lack of research on impersonation issues in existing audio spoofing detection and constructs a dataset based on impersonation.
3. It introduces a graph-based method to extract speaker profiles, enhancing the model's performance.
limitations: limitations: 1. My first question is any voice that exhibits impersonation considered "fake," or does it specifically need to mimic a target label already in the dataset?
2. I recommand the authors conduct a small experiment to verify whether the impersonation voices in the dataset can be misclassified by neural networks, similar to TTS and VC audio.
3. The paper lacks a demo that could showcase some audio samples from the current database.
4. I would like the authors to present the results of four end-to-end models in Section 5.4.
5. The main results of the experiments focus on the ASVspoof2019 dataset, which needs to be described in Table 1.