dataesr/scientific-paragraphs-categorization huggingface dataset:
The second dataset is a reference set based on the IEEE Recommended Practices for Speech Quality Measurements, as described in  [47] , extensively used in speech analysis testing of speaker verification. It sets out seventy-two lists of ten phrases described as the 1965 Revised List of Phonetically Balanced Sentences, otherwise known as the 'Harvard Sentences'. These are widely used in telecommunications, speech, and acoustics research, where standardised and repeatable speech sequences are needed. TABLE 3. Description of the datasets employed throughout the various sets of experiments. The number of utterances for each speaker is balanced across each dataset. For example, in dataset one, training set, there are 800 utterances; given that the number of speakers is 8, this means 100 utterances per speaker.
This is valid for every other set. For the classification tasks, gender has been taken into account. Hence, the speakers have been divided between male and female voices. The considered methodology aims to detect the energy concentration of the formant structure, which heavily differs amongst these two categories. Each dataset is further described within the text. The procedure applied to extract a subset of the ASVspoof 2019 challenge dataset is presented in VII-C. In both datasets, two real-language sources were used from a female (speaker 1) and a male (speaker 2); for the synthetic speech, five correspondent sources (T1, T2, T3, T4, T5 described in table 10) were employed for the female case and one source (T1) for the male one. The synthetic speech voices of all TTS algorithms were selected to have an English accent.
The start and end of each sample were trimmed to remove any non-speech segments and decimated to a set of 60k total samples. Regarding the IMFs extraction procedure, each set of 60k samples for one sentence was then windowed into non-overlapping collections of 5,000 samples and passed to the EMD sifting procedure. Afterwards, the features presented in Table  1  were extracted. We note that in some cases, we found that for high-frequency instantaneous frequency features, it would be advantageous also to apply a median filter (we used a window of 2ms). In the first dataset, the total number of recorded sentences was 960, equally proportioned samples of the same sentences across all voice recordings, with 80% randomly selected for training and the rest for testing. In the second dataset, we use the first sentence from each of the seventy-two lists of the Harvard Sentences to construct the training dataset.
The testing dataset was given by the second sentence from each of the seventy-two lists of the Harvard Sentences. This led to 1,152 utterances split equally between training and testing sets. The third dataset corresponds to a subset of the ASVspoof 2019 challenge database described in  [55]. Details on the extracted sets of sentences are given in subsection VII-C. We underline the importance of the settings provided by this dataset: they will tackle text-independent and speaker-independent verification systems (TI-SI-SV) to test our novel methodology in the most general environment encountered in ASV challenges. Table  4  presents the set of experiments considered. With experiment one, we firstly present a discussion showing how the EMD-MFCC approach provides more powerful discrimination in detecting individual vocal tracts required in ASV systems compared to other sets of EMD features (IMFs, IFs, Spline Coefficients, etc.
). We also provide the benchmark model comparison of the traditional MFCC extraction on the raw speech signals presented in table 6. Furthermore, given the wide variety of features often employed in Speaker Verification or Speaker Recognition tasks (see  [56] ), we also propose additional benchmark features applied both on the raw data and on the IMFs. Table  5  provides a detailed description of such features, with the used configuration and references required for further understanding. Results of these features run on the IMFs are provided in table 7. We discuss results for the individual speech features and then introduce the EMD-MFCC-MKL framework. describing  the three experiments conducted. Note that, in experiment one, both dataset one and dataset two are employed. Note that all the proposed sets of features have been extracted on dataset one and are widely discussed.
For the second dataset, the MFCCs on the raw data and the EMD-MFCCs for the female voice only were considered. In experiment two, both datasets are used, and the MFCCs on the raw data and the IMFs basis functions are employed to assess the discrimination power of the EMD-MFCC-MKL-SVM in detecting different types of TTS algorithms. Experiment three provides results for the EMD-MFCC-MKL-SVM applied to a subset of the ASVspoof 2019 challenge dataset considering both the male and the female cases and multiple TTS algorithms. Details are provided within each section related to the different experiments.
Each of these procedures carries specific pros and cons, highlighted in Figure  12. Note that  [57]  also suggest the hybrid approach as presented in  [65]. We select the best performing features for dataset one and dataset two for the female voice only obtained in Experiment one and repeat a similar exercise by considering the different TTS algorithms presented in table  10. We show the best performance and provide the additional results within the Supplement Materials. Experiment three runs the EMD-MFCC-MKL solution for ASV systems on a selected subset of the ASVspoof 2019 challenge dataset. In this way, a text-independent and speaker-independent environment is tested. Results will be presented for a range of different TTS algorithms and male and female voices. We focus on the best performing cases and present the additional results in the Supplement Materials.
A. EXPERIMENT ONE: BIOMETRIC CYBER RISK MITIGATION VIA SYNTHETIC VS REAL VOICE DISCRIMINATION Throughout these sections, we focus on the female voice examples to present the results. We found they generally presented the more challenging task in TD-SD-SV scenarios given the wider variation in spectral energy in the speech signals and higher non-stationary generally present in the formant structures in the 5kHz to 20kHz range. Note that results for the individual SVMs of the male voice are presented in the Supplement Materials. We start by assessing the ability of the benchmark ASV features in the classification of real and synthetic voices. We extract such features for the female voice versus the synthetic voice generated with TTS algorithm T1 for dataset one. This is done firstly on the raw speech data, and then the features are extracted not on the raw speech but rather on the IMF basis function representations of the speech.
SVM WITH SPEECH BENCHMARK ASV FEATURES We first set up the results for the benchmark comparison, which is based on applying the benchmark ASV features to the raw speech signal. Table  6  presents the results. Note that, for this task, we focus on the female voice discrimination task with TTS algorithm T1 for dataset one. The configuration applied to obtain such coefficients are presented in table  5. We performed one SVM per individual coefficient. We selected M = 12 as is the standard recommendation when utilizing these features in speech analysis. We present results for the radial basis function kernel. Other kernels have been employed and produced similar results. The features are divided into cepstral coefficients or linear prediction coefficients, and they use various filter banks when the former are considered, or different transforms are applied once the linear prediction coefficients are obtained when it comes to the latter ones.
The highest accuracy is achieved by the NGCCs and the PLPs with an accuracy score of 0. 850. Next, we perform an equivalent procedure and extract these features on the IMF basis functions of the correspondent dataset. Results are in table 7. Our proposed methodology relies on this new approach for which, instead of the raw speech, each IMF basis is passed one by one through an individual transformation of the selected features (i. e. MFCCs, LFCCs, BFCCs, etc. ) to form adaptive features for the classification of real and synthetic voice. The standard practice of classification problems for this kind of setting is constructing a vector collecting all the coefficients for the feature of interest or multiple features and then carrying the learning procedure. Since voice is highly biometric, highly non-stationary and adverse environments might arise, such standard procedures tend to create noise in the classification tasks rather than provide discriminant information.
This produces the summary statistics of the IMF bases, the Instantaneous Frequency signals and the spline coefficients that characterize the IMF bases. We summarize these three signals using the summary statistics described in Table  1. These results are in Tables  1  and 2  in the Supplement Materials and demonstrate the out-of-sample classification results for dataset one for Speaker 1 versus synthetic voice attacker and Speaker 2 versus synthetic voice attacker. We performed one SVM training and then out-of-sample testing per feature component, where, for instance, we took each sentence and then took each IMF. After, given each IMF, we extracted summary statistics and then ran the SVM training and out-of-sample testing for various kernel families. This allows building a library of individual features and their performance in the real vs synthetic voice discrimination over the voice recordings database.
This is our newly proposed methodology which utilized EMD-MFCC. Table  6  shows the out-ofsample results for the EMD-MFCCs of the female voice for dataset one, and table 3 in the Supplement Materials presents the correspondent results for the male voice. The results are presented for the radial basis function kernel choice, and the remaining results for other kernel choices are similar in performance, so we omit them to reduce space. The way to interpret these results is as part of a stage of constructing a library of individual feature sets to pass to a multiple-kernel learning solution. FIGURE 10. The panels represent the coefficient functions M k (s) given in Eqn. 17 computed on a sliding window for one sentence of Speaker 1. Note that panel (a) refers to the original signal, and the correspondent quantity is denoted as M(s) with no sub-index.
As a consequence, our final combined kernel should be more representative of the classification problem. Table  8  displays results for Speaker 1 out-of-sample performance for dataset 1. Since we select the best performing EMD-MFCC features amongst several kernels, the header of table 8 is organised as follows: the top row shows which is the basis of interest, i. e. γ 1 (t ), γ 2 (t ), γ 3 (t ), γ K (t ) and γ K +1 (t ). The index following MFCC-gives this information. The second row highlights the best individually performing coefficient and, therefore, the one selected for the MKL formulation. The last row shows which kernel offers the best performance for that feature; for example, for column one of the table, for the first IMF γ 1 (t ) (MFCC-1), the best performing coefficient was the 7-th one Note that the t-SNE algorithm is presented in the Supplement Materials.
the first IMF. Indeed, the selected coefficients for this case corresponds to the 7-th. Different combinations have been tried, and similar excellent performance was observed. These results are far higher than those of the current state-of-the-art reference of MFCC applied to raw speech when also placed in an MKL-SVM framework. This demonstrates the superior performance of the IMF-MFCC feature class when combined with an MKL-SVM classifier framework. Note that each EMD-MFCC-MKL table will follow the structure described in this section. Note that the individual performances related to the MFCCs on the raw speech signals are not presented for other datasets.
HARVARD PHONETICALLY BALANCED SENTENCES EXAMPLE: REAL SPEECH VS SYNTHETIC SPEECH CLASSIFICATION As performed for dataset one, a similar analysis was confirmed by dataset two on the gold standard speech data set given by the Harvard phonetically balanced sentences. Table  9  shows results related to Speaker 1, hence the female discrimination case study (as per the above study one). We provide a summary of the EMD-MFCC features only and MKL-SVM classifier compared to the MFCC on raw speech in an MKL-SVM classifier. In this example, we utilised a Radial basis function kernel, and the feature set was based upon the EMD-MFCCs that best performed in individual feature classifiers in the out-of-sample analysis.
B. EXPERIMENT TWO: OTHER TTS ALGORITHMS In this subsection, we replicate the EMD-MFCC-MKL conducted in experiment one by taking into account different Text-To-Speech (TTS) algorithms, presented in Table  10. Note that we replicate the experiment for the female voice only but both dataset one and dataset two. The first TTS algorithm corresponds to the interface of the Google-Text-to-Speech API interface provided by the Python library gTTS. It relies on WaveNet  [62]  and hence uses a Deep Learning procedure. It offers 120 languages and dialects (see https://cloud. google. com/speech-totext/docs/languages). The second TTS algorithm corresponds to Espeak (online at http://espeak. sourceforge. net/) that instead employs a formant synthesis procedure. It also provides several languages (the complete list is given online).
Afterwards, we use the Python library Pyttsx, a crossplatform text-to-speech wrapper providing access to different TTS tools. Amongst others, we select the Microsoft Speech Engine SAPI5 (online at https://docs. microsoft. com/enus/previous-versions/windows/desktop/ms723627(v=vs. 85)), making use of a concatenative algorithm. The last TTS algorithm is the IBM Watson TTS (whose documentation can be found online at https://cloud. ibm. com/docs/text-tospeech), which also provides access to its API through a Python interface and relies on neural voice technologies, hence making use of Deep Neural Network (DNN). The TTS service is in the IBM Watson Cloud and supports a large number of languages, from which we selected the option of UK English. As in experiment one, we firstly carry out individual feature SVMs, hence one for each mel-frequency cepstral coefficient of the obtained IMFs.
Results concerning these SVMs are provided in the Supplement Materials in tables 4 and 5 for dataset one and tables 6 and 7 for dataset two. We then selected the best performing cepstral coefficients per IMF basis function and then carried out the EMD-MFCC-MKL procedure as presented in experiment one. Results for the IBM TTS algorithm are provided in tables 11 and 12 for dataset one and dataset two, respectively. Results for the remaining algorithms are in Supplement Materials in tables 8, 9, 10 and 11. As in the previous experiments, the best performing MFCCs for the first three IMFs are high-frequency ones confirming our initial claim that most of the discrimination power for female voices should come from these regions of the time-frequency plane. Furthermore, TABLE  7. Out-of-sample results of the SVMs carried with the standard features used in ASV tasks applied to the IMFs.
The features description is given in table  5. Results for these features applied to the raw data are provided in table 6. Note that each value corresponds to the accuracy achieved by the SVM carried with the coefficient given in the row of the IMF basis in the column referring to the feature provided. the achieved accuracy are consistent with experiment one both datasets and all the TTS algorithms, with the EMD-MFCCs outperforming the traditional MFCCs on the raw data in each case study. This highlights that the EMD-MFCC-MKL within a TD-SD-SV system is robust to different types of TTS spoofing attacks. Tables  11  and 12  show that when using the combination of five and four EMD-MFFCs features, a level of accuracy greater than 90% is attained, hence providing the necessary countermeasure for an ASV system.
C. EXPERIMENT THREE: APPLICATION ON THE ASVspoof 2019 CHALLENGE DATASET One of the biggest problems affecting ASV studies is the comparison of various techniques evaluated over different datasets. As a result, since 2015, the research community  [70] -  [73]  has started to release evaluation databases as SAS, ASVspoof 2015, ASVspoof 2017, ASVspoof 2019 challenge, AVspoof, RedDots Replayed databases. Consistently with these purposes, we investigate the new technique combining EMD-MFCCs by employing the ASVspoof 2019 challenge database  [55]. Table  13  (also at https://www. asvspoof. org/asvspoof2019/asvspoof2019_ evaluation_plan. pdf) describes the structure of such a dataset. It subdivides into two different scenarios: logical access (LA) and physical access (PA). The former involves spoofing attacks directly injected into the ASV system.
Such attacks are generated using text-to-speech synthesis (TTS) and voice conversion (VC) technologies. In the PA scenario, speech is assumed to be captured by a microphone in a physical, reverberant space. Hence, replay spoofing attacks are irecordings of bonafide speech assumed to be captured and then represented to the microphone of an ASV system using a replay device. In this work, we only take into account the Logical Access scenario to target the TTS algorithms used within this database. The LA database contains bonafide speech and spoofed speech data obtained using 17 different TTS and VC systems. Figure  13  shows its spoofing attacks structure and the ones we extracted for our experiments. Note that data for the training of TTS and VC systems partly comes from the VCKT database (online at http://dx.
doi. org/10. 7488/ds/1994), but there is no overlap with the data contained in the 2019 database. Among the 17 spoofing voice generation systems, 6 are known attacks, while 11 are unknown. The training and development sets contain known attacks only, while the evaluation set contains 2 known and 11 unknown spoofing attacks. Regarding the 6 known attacks, there are 2 VC systems and 4 TTS systems. Particularly, VC systems use neural-network-based and spectral-filtering-based approaches  [74]. TTS use either waveform concatenation or neural-network-based speech synthesis using a conventional source-filter vocoder  [75]  or a WaveNet-based vocoder  [62]. We extract three of the TTS spoofing attacks for the training and development sets. The generation algorithms for the spoof voices fall into either Deep Learning or Concatenative types and can be compared to our results (see table  10 ).
In particular, the chosen algorithms are decoded in the LA protocol of the ASVspoof 2019 challenge as ''A01'', ''A02'' and ''A04'', respectively. The TTS ''A01'' algorithm is obtained with a neural waveform model, while the TTS ''A02'' algorithm is generated through a vocoder. Lastly, the TTS ''A04'' algorithm is a waveform concatenation. There are 8 male and TABLE 8. Multi kernel learning SVMs results of the synthetic voice generated with TTS T1 versus Speaker 1 for dataset 1. We select the best features according to their performances when individually tested (i. e. through the out-of-sample accuracy). The first line indicates the considered features, which is always an IMF-MFCC. The IMF indices are given in each MFCC component as -1, -2, -3, -K, -K+1. The second line refers to the coefficient number, and the third line to the selected kernel for that feature.
Hence, a text-independent and speaker-independent scenario is the one of interest in this experiment (TI-SI-SV). Furthermore, the number of TABLE 9. Multi kernel learning SVMs results of the synthetic voice generated with TTS T1 versus Speaker 1 for dataset 2. We select the best features according to their performances when individually tested (i. e. through the out-of-sample accuracy). The first line indicates the considered features, which is always an IMF-MFCC. The IMF indices are given in each MFCC component as -1, -2, -3, -K, -K+1. The second line refers to the coefficient number, and the third line to the selected kernel for that feature. The table represents a model selection comparison in which each row corresponds to a different MKL model combining different sets of features. The numbers in each row refer to the η m weights as expressed in Eqn.
This corresponds to 127 utterances for every speaker, 2,540 utterances for every group (i. e. natural, A01, TABLE 12. Multi kernel learning SVMs results of the synthetic voice generated with the IBM TTS algorithm versus Speaker 1 for dataset 2. We select the best features according to their performances when individually tested (i. e. through the out-of-sample accuracy). The first line indicates the considered features, which is always an IMF-MFCC. The IMF indices are given in each MFCC component as -1, -2, -3, -K, -K+1. The second line refers to the coefficient number and the third line to the selected kernel for that feature. The table represents a model selection comparison in which each row corresponds to a different MKL model combining different sets of features. The numbers in each row refer to the η m weights as expressed in Eqn.
26. The highlighted accuracy scores correspond to those combinations of features and kernel models greater than 90%. The first portion of the table demonstrates the EMD-MFCC-MKL solutions, while the second portion is the state-of-the-art reference of the classical MFCC-MKL. A02, A04), with total of 10,160 utterances. Regarding the development subset, we first balanced the number of speakers within each gender and randomly selected the minimum TABLE 14. Summary of the extracted database from the ASVspoof 2019 challenge database to conduct our experiment three. Note that we selected two subsets, i. e. the training and the development. Furthermore, for the spoofed speech, we considered three of the TTS voices only. Note that the datasets is balanced in terms of number of utterances per speaker.
Therefore, each group will have 616 utterances leading to 2464 utterances for the development set. For our experiments, we used the training set to train the individual features required to develop the EMD-MFCC-MKL and the development set for testing such a procedure with the added trait of gender, hence dividing the utterances according to it. Table  14  provides a summary of such a dataset. Each utterances speech recording duration was approximately 1sec to 3sec maximum sampled at 16kHz producing between 25k and 150k samples per spoken utterance. The start and end of each sample were trimmed to remove any non-speech segments and decimated to a set of 40k total samples. The procedure concerning the EMD extraction followed the same applied to the other datasets, i. e. each set of 40k samples for one sentence was then windowed into non-overlapping collections of 5,000 samples and passed to the EMD sifting procedure.
Then, for each IMFs, we extracted M = 12 cepstral coefficients similarly to experiments one and two. We carried one individual SVM per coefficient per IMF for the female and male cases by considering the three different TTS algorithms. Results of the individual features are provided in the Supplement Materials in tables 14 and 15. For both genders, better performances are achieved by the MFCCs of the second or the third IMF basis function detecting lower speech formants and the fundamental frequency. In this context, multiple speakers are trained together through a unique model, and, particularly at high-frequencies of female voices, the non-stationarity of each speaker might be strongly biometric, resulting in out-ofsample accuracy levels of 70% for high cepstral coefficients TABLE 18. Multi kernel learning SVMs results of the male case versus the synthetic voice generated with the A02 TTS algorithm of the ASVspoof challenge dataset.