How Distinguishable Are Vocoder Models? Analyzing Vocoder Fingerprints for Fake Audio
Authors: Chu Yuan Zhang
Jiangyan Yi
Jianhua Tao
Tao Wang
Xinrui Yan
Chenglong Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: In recent years, vocoders powered by deep neural networks (DNNs) have found much success in the task of generating raw waveforms from acoustic features, as the audio generated becomes increasingly realistic. This development however raises a few challenges, especially in the field of forensics, where the attribution of audio to real or generated sources is vital. To our knowledge, our investigation constitutes the first efforts to answer the question on the existence of vocoder fingerprints and to analyze them. In this paper, we present our discoveries in identifying the sources of generated audio waveforms. Our experiments conducted on the multi-speaker LibriTTS dataset show that (1) vocoder models do leave model-specific fingerprints on the audio they generate, and (2) minor differences in vocoder training can result in sufficiently different fingerprints in generated audio as to allow for distinguishing between the two. We believe that these differences are strong evidence that there exist vocoder-specific fingerprints that can be exploited for source identification purposes.

Under review as a conference paper at ICLR 2023
How DISTINGUISHABLE ARE VOCODER MODELS? ANALYZING VOCODER FINGERPRINTS FOR FAKE AUDIO
Anonymous authors Paper under double-bling brind review
ABSTRACT
recent years, vocoders powered by deep neural networks (DNNs) have found much success in the task of generating raw waveforms from acoustic features, the audio generated becomes increasingly realistic. This development however raises a few challenges, especially in the field of forensics, where the attribution audio to real or generated sources vital. To our knowledge. our investiga- tion constitutes the first efforts to answer this question the existence of vocoder fingerprints and to analyze them. In this paper, we present our discoveries in iden- tifying the sources of generated audio waveforms. Our experiments conducted on the multi-speaker LibriTTS dataset show that (1) vocoder models do leave model-specific fingerprints on the audio they generate, and (2) minor differences in vocoder training can result in sufficiently different fingerprints in generated audio as allow for distinguishing between the two. We believe that these differ- ences are strong evidence that there exist vocoder-specific fingerprints that can be exploited for source identification purposes.
INTRODUCTION
Over the past decades, advancements in text-to-speech (TTS) and voice conversion (VC) technolo- gies permit the artificial generation of speech audio. As the last step TTS or VC pipelines, gen- erating raw waveforms from acoustic features, vocoders are important components in these tasks. With the development of deep learning, deep neural networks (DNNss have seen increasing usage in both TTS and VC. Recent advancements in these areas (Wang et al. 2017 Shen et al. 2018
Kalchbrenner et al. 2018 Li et al. 2019 Ren et al. 2022) have led to the capacity generating increasingly realistic and natural audio. While such technological progress is certainly welcomed, does also admittedly come with challenges, especially when one considers the potential misuse and abuse of TTS and VC technologies, either inadvertently maliciously. The risks and challenges associated with these technologies mostly arise in the following areas:
Audio forensies. As TTS technologies generate increasingly realistic audio, they also become in- creasingly prone to being misused, by their very nature, to mislead or even defraud victims who cannot judge the authenticity of audio. The possibility of misuse and abuse evidently gives rise to concerns about security, and public order. Therefore, it is consistent with the public interest to find methods combat such risk of malicious usage by coming up with effective forensic strategies. While modern audio forensic techniques used in challenges such as ASVspoof (Wu et al. 2015) Wang et al., 2020 Yamagishi et al. 202T) and ADD (Yi al. 2022) have achieved effective results in separating fake audios from real ones, they focus more on performing binary classification be- tween real and fake sets. There is nonetheless also an interest in surpassing the constraints of binary real/fake classification and actually pinpoint the source responsible for generating any fake audio.
Intellectual property protection. Mainstream TTS and VC systems such as Shen et al. (018)) Arik al. (2018), and Ren et al. (2022) are built with much effort, as they require careful design and fine-tuning of models and algorithms, as well as extensive training with more data than can reasonably be collected by any person. As such, it is in the interest of researchers and commer- cial software developers alike to ensure that they are properly recognized to hold the ownership and copyright of their intellectual properties, the enforcement of which would not be feasible without the

---

Under review as a conference paper at ICLR 2023
Andu PWW
TD Full Baicar
audie
Legend
Aech vocodes architcttre
Saad seed sead to iniialine model weight prlor smining
Soal: 1900 TD: Full
TD vanin das, the
Janseet used tain the midel.
Buach batch size for traning
TDD SS1 Taniee
ED: Full
Figure I: A t-SNE projection of vocoder fingerprints belonging to models of different architectures, initialization seeds, and hyperparameters, obtained in our experiments. ("PWG" refers to the Parallel WaveGAN architecture (Yamamoto et af. 2020); "Full" refers to the clean subset of LibriTTS (see section SSI consists of 50,000 randomly selected samples from the "Full" set)
ability to detect unauthorized usage of system. While there have been efforts in watermarking given DNN model (Zhang et al. (2018), these efforts require crafted inputs on models that have been specifically tweaked for watermarking during training, instead of relying on properties intrinsic to the vocoder model in performing waveform generation.
Owing the reasons above, important be able identify the source of fake audio. Given the advancement of TTS and VC technologies, and judging by the results obtained in work by other scholars, where generated audios are often judged to be quite natural and realistic, we consider infeasible to manually perform such scrutiny and source identification. Consequently, this calls for way to automatically extract relevant information from audio samples and classify fake audio based on its source of generation, with minimal human intervention.
While the idea source-dependent fingerprints have been explored on GAN-generated images (Yu et al, 2019 Marra et al. 2019), those efforts rely on image properties and are not directly applicable to waveforms generated by vocoders. While there has been some preliminary investigation on the concept of vocoder fingerprints (aan et al. 2022), the effort of the study rests specific level. We seek to find the existence of fingerprints that allow for distinguishing models sharing the same architecture, separating audios generated by different vocoder models, and identi- fying their respective sources of generation.
architecture-
In this paper, we put forth and seek explain the following questions:
1. Do vocoder models leave model-specific fingerprints on the audio they generate?
2. What is the condition for their existence? In other words, how similar can vocoder model instances be before it becomes infeasible to distinguish between the two?
3. How do we present these fingerprints for the purposes of fake audio source identification?
To answer these questions, we perform experiments to evaluate our hypothesis that vocoder models used generate audio leave fingerprint on the audio they generate that is specific the model, as well as to explore how these fingerprints may be visualized and exploited for the aforementioned purposes of audio forensics and intellectual property protection. Our experiments confirm the exis- tence of these fingerprints and show that vocoder fingerprints depend on architecture, training data, initialization seeds, and training hyperparameters, as shown Figure
RELATED WORK
As our work deals with finding vocoder-specific features to aid in fake audio source identification, task that falls the purview of forensics, it behooves us to reiterate some of the work done by other scholars in the areas of vocoders and audio forensics.
2

---

Under review as a conference paper at ICLR 2023
Vocoders refer to signal processing devices that synthesize speech audio waveform from some more compressed form data, usually acoustic features, although certain vocoders take linguistic features inputs (Tan et al. 202I). Apart from some modern end-to-end systems that directly generate waveform from text, vocoders form the last step in TTS and VC. Notable traditional (i.e., non- neural) vocoder implementations include: Griffin-Lim (Grifhn & Lim 1984), STRAIGHT (Kawa- hara 2006), and WORLD (Morise et 2016).
With the development of deep learning and neural networks came a variety of neural vocoders, of which the notable ones include: WaveNet (van den Oord et al. 2016 2018), WaveRNN (Kalchbren- ner al., 2018). LPCNet (Valin Skoglund 2))9aa Given the length of waveforms, autoregres- sive waveform generation takes much inference time; therefore modern neural vocoders have taken a non-autoregressivve approach with generative models, among which are Flo FloWaveNet (Kim et al.) 2019). Parallel WaveGAN (Yamamoto et al. 2020), HiFiGAN (Kong et al. 2020), Multi-band Mel- GAN (Yang et al.) 2020), Style-MeiGAN (Mustafa et al. 2021). These non-autoregressive models are able to achieve faster-than-real-in time inference speeds.
One of the primary aspects of audio forensics, known as "integrity verification", deals with the determination of authenticity of a given audio clip, i.e., if the given audio represents real speech and if it has been tempered with (Maher 2009: Zakariah et al. 2018). Various techniques have been used to analyze details and features contained in the audio, such the Electronic Network Frequency -Nicolalde-Rodriguee et al., 2010), background noise (Malik 2013), recording device (Qamhan al. 2021), energy (Luo al. 2018).
In recent years, there has been significant progress in integrity verification of fake audio. Many chal- lenges have accelerated the progress and research in this area, including the AS A challenges (Wu et al. 2015; Kinnunen et al.] 2017 Wang et al., [2200] Yamagishi et al. (021). In addition these, ADD 2022 (Yi et al.J (022) initiated the first audio deep synthesis detection challenge for more challenging realistic scenarios. In summary. the research on fake audio detection can be broadly divided into two parts: front-end features and back-end classifiers. In the first category, many features (Todisco et al.) 2017: Davis & Mermelstein] 19800 (Sahidullah et al. (2015) have been widely used for detection. As for the back-end classifier, many effective classifier designs (He al.]
2016 Bagherinezhad et al. 2017 Hu al. 2018 Tak et al. [22l]havee contributed to the detection research.
3 METHODOLOGY
3.1 VOCODER FINGERPRINT
To answer the questions we put forth, we study the impact of each of the following aspects of vocoder models:
1. Vocoder architecture: We procure 7 models, each with different architecture, ranging from traditional signal processing vocoders to more modern GAN-based vocoders, and study the separability of the fingerprints left by these models the audio they generate. The architectures are chosen to represent both traditional and neural vocoders. Specifically, WORLD is a traditional signal processing vocoder that achieves promising results; LPCNet utilizes linear prediction, and achieves considerable performance despite the limitations of autoregressive vocoders; the GAN-based vocoders are chosen for their performance ported in their respective papers, achieving faster-than-reat than--eal--imm inference speeds given their nonautoregressive nature.
2. Training data: In the training process, each training sample may differently influence the direction to optimize the objective function, hence result in slightly different weights when the model converges. To study the impact of training data, we train models of the same architecture with the same initialization seed, but each model is trained with a slightly different training set that differ from each other different magnitudes (see Table 4). and analyze the fingerprints left by the models.
3. Initialization seeds: Initialization seeds determine the initial weight values before train- ing begins, and thus can potentially subtly influence the resulting weights as the model converges. To examine the impact of initialization seeds, train 4 models of the same
3

---

Under review as a conference paper at ICLR 2023
Real
Roal asdio
Feature Extractor
Fingerprint Extractor
256-D
Classification Layer
fingerprin
F-2
by
woodeer moddll
Figure 2: The overall fingerprint extraction and classification pipeline.
architecture, with the exact same training set, but each model is initialized with a different seed. We then analyze the fingerprints left by the models the generated audio.
4. Training hyperparameters: We train 5 models of the same architecture, with the same training set and initialization seed, but adjust the dropout rate and batch size used in the training of each model. We then analyze the fingerprints left by the models on the audio. The choice dropout rate and batch size are justified because dropout rate, by "freezing" certain weights, make the vocoder models less "reactive" to incoming training samples, while batch sizes influence the order in which the samples contribute to the tweaking of model parameters.
5. Architecture and parameter differences, combined: We train 3 models of 4 GAN vocoder architectures each, varying seed and training hyperparameters, We then copy- synthesize audio using the vocoders such that the testing audio is generated by vocoders whose parameters are unknown to the classification network even though the architecture is known. We examine the performance of the classification network in this scenario.
3.2 VOCODER MODEL.
Investigating the existence of vocoder fingerprint requires focus vocoder-specifie information and features, and as such, all other variables need to be fixed, with the vocoder model used to generate audio waveforms being the only independent variable in our experiments. To accomplish this, we generate fake audio by performing copy synthesis on real audio samples from the LibriTTS dataset. This is done by extracting acoustic features from these real audio samples and passing the extracted features through trained vocoder models to generate new waveforms, which will then be used for fingerprint extraction classification. The purpose behind this design is eliminate any risk of potential interference that could be induced by acoustic models. Furthermore, since mainstream TTS and VC pipelines like Tacotron 1/2 (Wang et al. 2017 Shen et al. 2018) and FastSpeech (Ren et al. 2022) still rely vocoder models as the final step to generate audio waveforms, think that vocoder differences are the most likely to cause fingerprint differences.
In addition, samples in the real audio corpus exhibit some variation in terms of speech content and speaker characteristics. the effect of which should not be relied on for the purpose source identification; evidently, the text even phonemic content of sentences being spoken little to do with vocoders. To account for and preclude the impact of this naturally generated audio waveforms from each model mirror and correspond each other well real audio waveforms. More specifically, for every real audio sample æ, we generate fake audio samples {&m m ,,....,Ay M) using each of the M models studied the experiment, that groups of real-and-fale audio waveforms from the same acoustic features maximally balance out the effect of the naturally occurring variations intrinsic the real audio corpus.
variance, the
3.3 FINGERPRINT EXTRACTION PIPELINE
As the motivation behind fingerprint extraction and analysis is essentially to facilitate generation source identification of fake audio, the fingerprints themselves should be such that audio samples from a single source - either real recordings or generated by a single vocoder model form dis- tinct, preferably linearly-separable, clusters in the feature space that can then be easily distinguished even with a single culle-comeettd layer (FC layer). As such, we train our fingerprint extraction network essentially part of classification model.
4

---

Under review as a conference paper at ICLR 2023
Table Datasets used in our experiments.
(a) Dataset used in our experiments to train vocoder models: all audio
(b) Real audios used for copy synthesis to generate training data for fingerprint
comes from LibriTTS. This table excerpted from Zen et al. 2019).
Speakers Male
Subset
Hours
Female
Total
Utterances
Subset
Speakers
Utterances
train-clean- 100
53.78
123
124
247
33,236
train
1129
50,000
train-clean-366
191.29
430
474
904
116,500
dev
40
5,000
dev-clean
8.97
20
5,736
test-clean
8.56
19
20
39
4,837
Jest
39
4,500
Total
1208
59,500
Total
262.60
592
638
1,230
160,369
As shown in Figure Each audio sample is first passed through a feature extractor to extract its LFCC features, and the features are then fed into the fingerprint extractor, which is Res2Net network. The network produces vector for every input, and the vector is then passed into FC classification layer to classified into a finite number labels, representing the source of the audio (either real or the model that generated it).
4 EXPERIMENTS AND RESULTS
In this section, we present experiments well the results and observations obtained. We perform four experiments, each of which focuses on one independent variable, while the rest are controlled and kept the same across models. The vocoder models are trained following the setup in their original paper as closely as possible.
For each audio waveform sample, we first extract its linear frequency cepstral coefficient (LFCC) representation, with 20 LFCC frame, using Hamming window with the window length of M = 480 and hop length of 240. The LFCC matrix then truncated or O-filled contain 500 frames. The LFCC representation of the audio sample is then passed to Res2Net network (Gao al., (22T)) and projected onto a feature space with a dimensionality of 256, then passed through single FC layer classifier to decide between the finite number of sources. The entire network (extraction + classification) trained using cross-entropy loss.
4.1 DATASET
In our experiments, we use the LibriTTS dataset (Zen et al. 2019) to train the neural vocoder models as well as to generate the audio corpus for vocoder fingerprint extraction network through copy synthesis. We employ only the clean speech subset of the dataset, as shown Table Ia With the exception of the second experiment described below in section the clean training set of LibriTTS used in its entirety to train the neural vocoder models.
To train our extraction-classifcattii ples from the aforementioned LibrITTS training set, 5,000 samples from the validation set, and 4,500 samples from the testing set, shown in Table [b] to serve the real audio portion of the classifier training set the basis to perform copy synthesis. The fake audio part thus mirrors the real audio portion, with each class essentially mirroring the real audio
network in each experiment, randomly sample 50,000 sam-
4.2 EVALUATION METRIC
As discussed in section 3.3 the performance of fingerprint extraction and analysis should be directly related to the ability to classify the audio samples and identify their source of generation, as such, evaluate fingerprint extraction model in a way similar to classifier evaluations: by using precision, recall, and Fl scores.
4.3 IMPACT OF VOCODER ARCHITECTURES
In our first experiment, we test for the existence ceatttectuur-speccit fingerprints in the generated waveforms. We acquire models of the following 6 architectures:
5

---

Under review as a conference paper at ICLR 2023
Table Results (%) on the impact of vocoder ar- chitectures
4.3
Table 3: Results (%) on the impact of train- ing data (4.4)
Precision
Recall
Fl score
REAL
98.30
97.49
97.89
Precision
Recall
Fl score
PWG
99.42
99.87
99.65
REAL
99.78
99.58
99.68
HIFIGAN
97.73
98.36
98.04
S50
93.14
92.56
92.84
MBLMELGAN
99.62
99.56
99.59
SS1
92.79
95.58
94.17
S_MELGAN
99.73
99.82
99.78
SS2
97.76
94.09
95.89
WORLD
99.80
99.58
99.69
SS3
93.94
95.42
94.68
LPCNET
100.00
99.93
99.97
Average
95.48
95.44
95.45
Average 99.23
99.23
99.23
WORLD (Morise et al. 2016): estimates the fundamental frequency (FO), spectral enve- lope, and aperiodic parameters, well generates speech similar to some input speech based on the estimated features.
LPCNet (Valin & Skoglund, 2019a): a neural vocoder combining linear prediction and recurrent neural networks (RNNs) and generates audio waveforms.
Parallel WaveGAN (PWG: Yamamoto al. 2020 neural vocoder that generates audio from Mel spectrum using a generational adversarial network (GAN),
HiFiGAN (Kong et al. 2020): GAN- based neural vocoder that employs two discrimina- tors to perform multi-scale and multi ppriod discriminators.
Multi-band MelGAN (Yang et al., 2020): a GAN-based neural vocoder with generator that produces sub-band signals from which the full-band signals are constructed.
Style-MelGAN (Mustafa et al. 2021): GAN-based neural vocoder that allows for the synthesis of high-fidelity speech with relatively low computational complexity.
We follow the setup instructions in ParallelWaveGAN and attempt to unify the spectrum input to the different GAN vocoder models, as to remove the influence of the input features the vocoders. The GAN models all take 80-band mel-spectra as inputs, with a frame length of 50 ms and a hop length of 12.5 ms; as a result, these features are pre-extracted and remain the same for all GAN models. The LPCNet vocoderis trained directly the raw waveforms, since it has built-in feature extractors.
then tag the models as WORLD, LPCNET, PWG, IIFIGAN, MBLMELGAN and S_MELGAN, respec- tively. Other than WORLD, which does not require training, each model is trained with the entirety of the LibriTTS clean speech subset described in section 4.1 The models are each trained on a sin- gle Tesla K80 GPU, with their default hyperparameters according their respective papers, except the training batch size of Style-MelGAN was lowered to 16 for the sake of memory usage.
We then train the extraction-classficaai classification network described in section 3.3 to distinguish be- tween audio waveforms from classes: (REAL, WORLD, LPCNET, PWG, HIFIGAN, MB MB_MELGAN, ..MLLGA).. To this end, we sample 50,000 waveform files from each source for training and 5,000 files for validation.
In this experiment, obtained results reported in Table and Figure 4 showing that models with different architectures produce clearly distinguishable audios. The only relative confusion in this experiment is between the original (real) audio and the audio generated with HiFiGAN, all other misclassifications constitute less than 1% of the evaluation set. Furthermore, if we were to evaluate the separability between generated audios from different models, no misclassification amounts to more than 0.5% of all audio samples of its real class. As different vocoder architectures take different approaches to construct audio waveforms from spectrograms or other feature representations, this separability is to be expected.
4.4
IMPACT OF TRAINING DATA
In second experiment, explore the relationship between the training dataset used to train the vocoder model and its fingerprint on generated waveforms. We fix the models to a single GAN architecture: Parallel WaveGAN (Yamamoto al. 2020), and use slightly different training datasets

---

Under review as a conference paper at ICLR 2023
Table Vocoder models with different hyper-
Table 4: Difference (in utterances) of training data used in the second experiment
parameters
Batch size
# Steps
SSO
SS1
SS2
SS3
Dropout
trained
550
100
10,000
BASE
0
6
500,000
101
10,001
IDROPOUT
0.1
6
500,000
852
100
101
10,100
2DROPOUT
0.2
500,000
SS3
10,000
10,001
10,100
4BATCH
0
4
750,000
8ВАТCH
0
8
375,000
Table 7: Results (%) of on the impact of hyperpa-
Table 6: Results (%) of on the impact of initial-
rameters (.6
ization seeds 4.5)
Precision
Recall
Fl score
Precision
Recall
score
99.82
99.84
99.83
REAL
99.76
99.84
99.80
SEED-
94.51
94.91
94.71
BASE
97.38
94.93
96.14
IDROPOUT
97.70
97.07
97.38
SEED-1001
94.49
94.42
94.45
2DROPOUT
99.12
98.09
98.60
SEED-1002
94.37
97.16
95.74
4BATCH
95.66
97.96
96.79
SEED-1003
95.52
92.33
93,90
8ВАТCH
95.08
96.71
95.89
Average
95.74
95.73
95.73
Average
97.45
97.43
97.43
to train them. This variance is controlled by sampling 50,000 waveform files from the LibriTTS clean speech set for training and 5,000 waveforms for testing. We tag this base set SUBSET-O. Three more training subsets of the same sizes are then constructed, each differing from the base set by replacing 1, 100, and 10000 samples, respectively. We use tags {SUBSET-i denote them. The four subsets are used as training sets train 4 Parallel WaveGAN models {ssi S 3). The training processes of the four models do however share the same validation set, since validation data has no bearing on model parameters. We then train the ixtraction-llassffcaa network deseribed
< i 3) to
section 3.3 to distinguish between audio waveforms from classes: {REAL, ssi 0 < < 3}. To this end, we sample 50,000 waveform files from each source for training and 5,000 files for validation.
Table 3 and Figure (see appendix) display the classification results and confusion matrix, respec- tively. The results obtained from the experiment corroborate our hypothesis that fingerprints are vocoder-specific
finer granularity than architecture difference. Furthermore, it is shown that even a minor difference in vocoder model training (as the case in ssO vs. ss1, where the training sets differ by only sample) can result models that produce sufficiently different fingerprints as to allow for distinguishing between audio generated by the two models.
Interestingly, if were between the fingerprints, then it would seem that the magnitude of difference in training data has no meaningful impact on separability. Two models with training sets differing by 1 sample (accounting for 0.002% of the training set) still produced casily distinguishable fingerprints.
classification accuracy the sole metric to judge the separability
4.5 IMPACT OF INIIIALIZATION SEEDS
In our third experiment, we investigate an even smaller variance factor: that of the initialization seeds. To accomplish this, we limit the models the Parallel WaveGAN architecture, just as in the second experiment discussed above; we also use the same training data to train all models, and instead manually vary the seeds used in the initialization of model parameters. Specifically, we seeds (1000, 1001, 1002, 1003} to initialize 4 models that tag {SEED-s), which then trained with the entirety of the LibriTTS clean speech subset. We then train the classifier to distinguish between 5 classes: {REAL, SEED-S $ 50,000 waveform files from each source for training and 5,000 files for validation.
(1000, 1001, 1002, 1003)). We again sample
Table and Figure (see appendix) display the classification results and confusion matrix, respec- tively. These results show that, even when models share the same architecture and are trained with the same data, the variation of initialization seeds can still result in sufficiently different fingerprints

---

Under review as a conference paper at ICLR 2023
for distinguishing between audio generated by the models. Given that the weight parameters of the vocoder models are supposed to converge towards a configuration that minimizes the objective function, our observation from this experiment entails that differences in vocoder weights can result in different fingerprints being produced by the models.
4.6 IMPACT OF TRAINING HYPERPARAMETEE
Our fourth experiment reported in this paper seeks to explore the impact of training hyperparameters that govern the training procedure, while fixing the architecture, training data, and initialization seeds of the vocoder models. We again limit the models to the Parallel WaveGAN architecture, with initialization seed as 1000 for all models, and use the entire clean training set of LibriTTS to train the vocoder models.
For the purposes of this experiment, we are interested specifically in the impact of adjusting the dropout rate and the batch size in training. We copy the from the previous experiment and retag it as the baseline. This baseline does not use dropout, and each batch contains 6 samples. We then train four more models with the following parameters:
1000 model
The last two models are trained for 150% and 75% of the number of steps as the baseline model, respectively, to compensate for the batch size influencing the total amount of data. Any unmentioned hyperparameter kept the same as in the baseline model.
We then train the classifier to distinguish between 6 classes: {REAL, BASE, IDROPOUT, 2DROPOUT, 4BATCH, 8BATCH). As with all other experiments, we again sample 50,000 waveform files from each source for training and 5,000 files for validation.
Table and Figure (see appendix) display the classification results and confusion matrix, respec- tively. These results show that, even when models share the same architecture and are trained with the same data and initialization seeds, the precise manner of training can result in different models producing fingerprints that are different enough for distinguishing between audio generated by the models. It would therefore seem that the vocoder fingerprint is volatile to training.
4.7 COMPARISON OF ARCHITECTURAL AND PARAMETER DIFFERENCES
We conduct our last experiment to investigate the hypothesis that, even though differences in vocoder model parameters are sufficient to produce distinguishable fingerprints, differences architec- tures predominate differences in parameters. In other words, if we have fingerprint extraction- classification model that can detect fingerprints produced by vocoders of different architectures (such as the one presented in Experiment 1), it should still be able to reliably classify between these same architectures even when the vocoders have different parameters.
We train vocoders of 4 architectures: Parallel WaveGAN, HiFiGAN, Style MelGAN and Multiband
MelGAN, denoted as P, H, S, and M, respectively. We train 3 vocoders for each architecture: The first with initialization seed of 1000, the second with an initialization seed of 1001 and the third trained with a batch size of 8. The vocoders are trained with the LibriTTS dataset. The training setups of the models summarized in Table 9
We apply copy synthesis on the WSJO dataset (Garofolo et al. 2007). As the dataset comes in sph files, these files transformed into wwv through sph2pipe resampled to 24 kHz, and randomly split into waaniin-validaaioo--vaaaee -valluatonn sets with a 64%/1669 6% %.720% ratio. The training set is fed into models PO. HO, SO, and MO, and the results are labelled based on the vocoder architecture and used as the training set to train the Res2Net classifier. The validation set is fed into all models listed in Table labelled with the architecture of the vocoder, and used as the validation set for the Res2Net classifier. The evaluation set is fed into the vocoders that were not involved in the training set. The copy-s copy-synthesis setup is summarized in Table
Table 10 and Figure 8 (see appendix) display the classification results and confusion matrix. The results obtained from the experiment corroborate our hypothesis that, while fingerprints are vocoder- specific on a finer granularity than architecture difference (see Experiments 2-4), these differences are also less pronounced than architectural differences. As our experiments reported here deals
8

---

Under review as a conference paper at ICLR 2023
Table Vocoder models used for copy synthesis in our final experiment.
Subset
Parallel WaveGAN
HiRIGAN
Style MelGAN
Multiband MelGAN
training
PO
H0
S0
M0
validation
PO. Pl, P2
HO. HI, H2
SO. S1, S2
MO, MI. M2
evaluation
Pl. P2
HI. H2
S1. S2
MI. M2
Table Vocoder model setups in our final experiment.
Model
Architecture
Seed
Batch
Steps
Table 10:
Results (%) on the com-
PO
Parallel WaveGAN
1000
6
500,000
bined effects of vocoder architectures
PI
Parallel WaveGAN
1001
6
500,000
and training setups
P2
Parallel WaveGAN
1000
8
375,000
HO
HiFiGAN
1000
1,000,000
Precision
Recall
Fl score
H1
HiFiGAN
1001
1,000,000
Real
99.15
100.00
99.57
H2
HiFiGAN
1000
2,000,000 8
P
83.80
96. 16
89.56
SO
Style MelGAN
1000
16
1,000,000
H
89.53
90.74
90.13
S1
Style MelGAN
1001
16
1,000,000
M
90.25
81.64
85.73
S2
Style MelGAN
1000
8
2,000,000
99.27
92.35
95.68
M0
Multiband MelGAN
1000
16
500,000
Average
91.65
91.31
91.31
M1
Multiband MeIGAN
1001
16
500,000
M2
Multiband MeIGAN
1000
8
1,000,000
solely with the acoustic features that depend the specific vocoders being used generate given audio sample, we believe that these results offer much insight; namely that any fine-grained differ- ence resulting from different training setups may be overshadowed by the differences in vocoder architecture, in the process of vocoder fingerprint analysis. The precise relationship between archi- tecture and parameters, however, remains an objective to be investigated our future efforts.
5 VISUALIZATION
While we observe the effectiveness of fingerprints in distinguishing between audio generated by different vocoder models through evaluating the classification performances, it is also possible to directly see the separability of these fingerprints through the t-SNE method (van der Maaten & Hin- ton 2008) that maps from a space with higher dimensionality (in our case, 256) to a 2D surface while making similar points closer and dissimilar points farther on the new surface. Figure 3 (see Appendix) shows the results of t-SNE mapping of the fingerprints extracted in our four experiments. While minor overlaps exist, corresponding the confusions in classification, the mapped represen- tations still clearly show that fingerprints belonging to different vocoder models are distinguishable. Furthermore, the fact that the t-SNE algorithm naturally clusters fingerprints of the same model to- gether indirectly confirms that vocoder fingerprints are model-specific, in that fingerprints of each model forms concentrated distribution around different mean values.
6 CONCLUSION
We have presented our study of extracting vocoder fingerprints as an effort to identify fake audio generation source, Our five experiments verify the existence model-specific vocoder fingerprints and demonstrate that even small differences in the training process of vocoder models can result in their fingerprints being distinct, allowing for fine-grained source identification of fake audio. Fur- thermore, we show that is possible to determine the architecture of vocoder used to generate an audio sample, even if the precise parameters of the said vocoder model has been changed because of a different training setup. We find this result encouraging because it suggests that our proposed us- age of vocoder fingerprint extraction for forensics and copyright protection may be viable. Despite results, we have yet to study the impact of different languages (and therefore different phono- logical feature distributions) and acoustic models can have on the fingerprints generated audios, and exploring these questions will form the focus of our future studies.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sercan Arik, Jitong Chen, Kainan Peng. Wei Ping. and Yanqi Zhou. Neural voice cloning with few samples. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. LCNN: Lookup-Based Convolu- tional Neural Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7120-7129, 2017.
Davide Chicco and Giuseppe Jurman. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1): 6, January 2020. ISSN 1471-2164. doi:
S. Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):357-366, August 1980. ISSN 0096-3518. doi: 10.1 10.1109/TASSP.1990. 1163420.
Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Tom: Res2Net: A New Multi-Scale Backbone Architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):652-666 662, February 2021. ISSN 1939-3539. doi: 10.1 109TTPAM. 29387293.758.
John S. Garofolo, David Graff, Doug Paul, and David Pallett. CSR-I (WSJO) Complete, May 2007.
D. Griffin and Jae Lim. Signal estimation from modified short-time Fourier transform. IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 32(2):236-243, April 1984. ISSN 0096- 3518. doi: 1ALL10/TASSSP..84..16666..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109/CVPR.2266.999
Jie Hu, Li Shen, and Gang Sun. Squeeze-and- -xxcitatio Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient Neural Audio Synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, July 2018.
Hideki Kawahara. STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually iso- morphic decomposition of speech sounds. Acoustical Science and Technology, 27(6):349-33 2006. doi: 10. 10.125//st..7.3449
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. FloWaveNet A Generative Flow for Raw Audio. In International Conference on Machine Learning, pp. 3370-3378. PMLR, May 2019.
Tomi Kinnunen, Md. Sahidullah, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, Junichi Yamagishi, and Kong Aik Lee. The ASVspoof 2017 Challenge: Assessing the Limits of Replay Spoofing Attack Detection. In Interspeech 2017, pp. 2-6. ISCA, August 2017. doi: 10.21437/ Interspeech.2017-11111
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXivv2010.0560 [cs, eess), October 2020.
Naihan Li. Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural Speech Synthesis with Transformer Network. Proceedings of the AAAI Conference on Artificial Intelligence, 33:6706- 6713, July 2019. ISSN 2374-3468, 2159-5399. doi: 10. 1099///aaai 133101.33016766.
10

---

Under review as a conference paper at ICLR 2023
Da Luo, Pawel Korus, and Jiwu Huang. Band Energy Difference for Source Attribution in Audio Forensics. IEEE Transactions on Information Forensics and Security, 13(9):2179-218)) Septem- ber 2018. ISSN 1556-6021. doi: (01.9//IFFS.1112822212...
Robert C. Maher. Audio forensic examination. IEEE Signal Processing Magazine, 26(2):84-94, March 2009, ISSN 1558-0792. doi: 10.1 N0.1009/MSP20008.595 900...30000..
Hafiz Malik. Acoustic Environment Identification and Its Applications to Audio Forensies. IEEE Transactions on Information Forensics and Security, 8(11):1877-183) November 2013. ISSN
1556-6021. doi: 10.1109/TIFS 201. 2013.220888.
Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. Do GANs Leave Artificial Fingerprints? In 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 506-511, San Jose, CA, USA, March 2019. IEEE ISBN 978-1-72811- 198-8. doi: (..110//MIR.2201900000
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. WORLD: A Vocoder-Base High-Qualigy Speech Synthesis System for Real-Time Applications. IEICE Transactions on Information and Systems, 1999.1(7:1877-1884, 2016. ISSN 0916-8532, 1745-1361. 2015EDP7457.
doi: 10.15877transinf.
Ahmed Mustafa, Nicola Pia, and Guillaume Fuchs. StyleMeIGAN; Efficient High- Ligh-Fidelity Ad- versarial Vocoder with Temporal Adaptive Normalization. arXiv:2011.01557 [cs, eess], February 2021.
Daniel Patricio ficolalde-Rodriguus José Antonio Apolinario, and Luiz Wagner Pereira Biscainho. Audio Authenticity: Detecting ENF Discontinuity With High Precision Phase Analysis. IEEE Transactions on Information Forensics and Security, 5(3):534 543, September 2010. ISSN 1556-
6021. doi: 10. 109//099/ 2000TTTFS..200.255227777
Mustafa A. Qamhan, Hamdi Altaheri, Ali Hamid Meftah, Ghulam Muhammad, and Yousef Ajami Alotaibi. Digital Audio Forensics: Microphone and Environment Classification Using Deep Learning. IEEE Access, (662719-62736 2021. ISSN 2169-3536. doi: 10. 10.1109/ACCESS.200 20021. 3073786.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin. Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In International Conference on Learning Representations, March 2022.
Md. Sahidullah, Tomi Kinnunen, and Cemal Hanilçi. A comparison of features for synthetic speech detection. In Interspeech 2015, pp. 2087-2091, ISCA, September 2015. doi: 10.21437/ Socrspeech.2015--22
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgian- nakis, and Yonghui Wu. Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783, April 2018. doi: 10. 100..S0//ASS............ 84611368.
Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-End anti-spoofing with RawNet2. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. doi: 10.1109/CASSP999222.... 201.:2...221...12255
June 2021.
Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A Survey on Neural Speech Synthesis. arXiv:2106.1555 15561 [cs, eess], July 2021.
Massimiliano Todisco, Héctor Delgado, and Nicholas Evans. Constant Q cepstral coefficients: A spoofing countermeasure for automatic speaker verification. Computer Speech & Language, 45: 516-535, September 2017. ISSN 08852308. doi: 10.1016f.cs1.20117.0..00
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving Neural Speech Synthesis through Linear Prediction. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5895, May 2019a. doi: 10.1 11100//ASSS201996696600 8682884

---

Under review as a conference paper at ICLR 2023
Mean-Mare Valin and Jan Skoglund. A Real-Time Wideband Neural Vocoder at 1.6kb/s Using LPC- Net. In Interspeech 2019, pp. 1255.
ISCA, September 2019b. doi: 10.21497/lmtesspeechh.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. a6Xi::1699.032: [cs], September 2016.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg. Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast High-Fidelity Speech Synthesis. In International Conference on Machine Learn- ing. pp. 1916-9926. PMLR, July 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, (86): 2008. ISSN 1533-7928.
Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, Lauri Juvela, Paavo Alku, Yu-Huai Peng. Hsin-Te Te Hwang, Yu Tsao, Hsin-Min Wang, Sébastien Le Maguer, Markus Becker, Fergus Henderson, Rob Clark, Yu Zhang, Quan Wang, Ye Jia, Kai Onuma, Koji Mushika, Takashi Kaneda, Yuan Jiang, Li-Juan Liu, Yi-Chiao Wu, Wen-Chin Huang, Tomoki Toda, Kou Tanaka, Hirokazu Kameoka, Ingmar Steiner, Driss Matrouf, Jean-François Bonastre, Avashna Govender, Srikanth Ronanki, Xuan Zhang. and Zhen- Hua Ling. ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech. Computer Speech & Language, 64: 101114, November 2020. ISSN 08852308. doi: 10. 10.10166j.e3..222010111
Yuxuan Wang, R.J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards End-to-End Speech Synthesis. In Interspeech 2017, pp. 4006-4010. ISCA, August 2017. doi: 10.24437//nerrpeechh20177.
Zhizheng Wu, Tomi Kinnunen, Nicholas Evans, Junichi Yamagishi, Cemal Hanilçi, Md. Sahidullah, and Aleksandr Sizov. spoof 2015: The first automatic speaker verification spoofing and countermeasures challenge. Interspeech 2015, pp. 2037-2041. ISCA, September 2015. doi: 10.21455/Inexxpecchh015511
Junichi Yamagishi. Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and Héctor Delgado. V 2021: Accelerating progress in spoofed and deepfake speech detection. eess], September 2021.
[cs,
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. 1rXiv:1910.11460 [es, eess], February 2020.
Xinrui Yan, Jiangyan Yi, Jianhua Tao, Chenglong Wang, Haoxin Ma. Tao Wang, Shiming Wang. and Ruibo Fu. An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio. arXiv:2208. 09646 [cs, eess], August 2022. doi: 10.1145/552463555625.
Geng Yang, Shan Yang, Kai Liu, Peng Fang. Wei Chen, and Lei Xie. Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechh arXiv:2005.00 106 [cs, eess), November 2020.
Jiangyan Yi, Ruibo Fu, Jianhua Tao, Shuai Nie, Haoxin Ma, Chenglong Wang. Tao Wang, Zhengkun Tian, Ye Bai, Cunhang Fan, Shan Liang, Shiming Wang, Shuai Zhang, Xinrui Yan, Le Xu, Zhengqi Wen, and Haizhou Li. ADD 2022: The first Audio Deep Synthesis Detection Chal- lenge. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9216-9220, May 2022. doi: 10. 1AA10CCAA994992222229996696
12

---

Under review as a conference paper at ICLR 2023
Ning Yu, Larry Davis, and Mario Fritz. Attributing Fake Images to GANs; Learning and Analyzing GAN Fingerprints. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7555-7565, Scoul, Korea (South), October 2019. IEEE. ISBN 978-1-72814-803-88 doi: 10.1 10.11099CCC..2219900775...
Mohammed Zakariah, Muhammad Khurram Khan, and Hafiz Malik. Digital multimedia audio forensics: Pust, present and future. Multimedia Tools and Applications, 77(1):109--104) January
2018. ISSN 1380-7501, 1573-7721. doi: 10. 10.1007/s11042--16-47777
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: Corpus Derived from LibriSpeech for Text-to-Speech. arXiv:1904.0220 [cs, eess], April 2019.
Jillong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph. Stoecklin, Heqing Huang, and Ian Molloy, Protecting Intellectual Property of Deep Neural Networks with Watermarking. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, Pp. 159-172 172, Incheon Republic of Korea, May 2018. ACM. ISBN 978-1-4503-5577-6 doi: 10.1145/3196494.11666
APPENDIX
75
90-
Tpent
-50
-00
35
1
35
in
(a) Different architectures
(b) Different training sets
80
0-
-40
-80
-15
-50
-25
50
15
-50
-50
20
60
Different initialization seeds
(d) Different training hyperparameters
Figure 3: t-SNE representation of fingerprints extracted in our experiments.
13

---

Under review as a conference paper at ICLR 2023
4:307
real
The lobal
mo molgan
Trus 1bb0
551
4301
21
100
5. s_melgan
world
552
Ipcnet
4497
101
4294
rea
PWG
hilgen
melgan
melgan
Iocnet
Prodicted lobel
Figure 5: Confusion matrix on the impact of training data
Figure 4: Confusion matrix on the impact of vocoder architectures
base
4272
11
6
126
The label
True Hebel
seed 1001
99
42:9
53
96
2dropout-
10
50
4414
20
seed 1002
4372
105
4355
Boatch
58
54
4352
www.
1000
2008 1008
Mompos
Abalch
Blotch
Predicted lbel
Predicted labal
Figure 7: Confusion matrix of Exper- iment (varied training hyperparame- ters)
Figure 6: Confusion matrix Experi- ment. (varied initialization seeds)
Jaweggnn
hodd
hiligan
nbumelann
syle_melgan
Figure 8: Confusion matrix of Experiment 5 (varied architecture and training hyperparameters)
14


Reviews:
[{"review_id": "jDSBjqtaSB", "reviewer": "Reviewer_4uuW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the fingerprints of the vocoders. The authors studied whether resynthesized speech can be classified into real speech and a set of vocoders. Specifically, A deep classifier is trained on top of the LFCC feature extracted from the synthesized waveform. In this paper, 7 vocoders are studied in the experiments: WORLD, LPCNet, Parallel WaveGAN, HiFiGAN, Multi-band WaveGAN, and Style-MelGAN.\nThe experiment result shows that different vocoders have their own fingerprint, and the fingerprint is sensitive to training hyperparameters and the vocoder architecture.\n", "summary_of_the_review": "Overall speaking, this paper mainly provided the experiment results without further investigation and explanation, which might not be inspiring enough to motivate the community. And therefore, the paper is not ready to publish without deeper investigation.", "strength_and_weaknesses": "### Strength:\nThe research topic of this paper is recently emerging and interesting.\n\nUnlike standard fake speech detection, which is mainly a binary classification problem, this paper further studies whether we can tell which vocoder is used for fake speech synthesis.\n\n### Weaknesses:\nThe authors mainly showed the experiment result without further investigation. For example, why the vocoder training is so sensitive to the training data, hyperparameter, and random seed? (Even differences in one data point or the random seed lead to significant differences) \n\nThere’s not much insight into the usage scenario of the proposed fingerprint extraction and classification pipeline. How can the fingerprint of the vocoders be used in a real-world scenario? How to use the fingerprint of the vocoders to protect the copyright? And how can the fingerprint of the vocoders provide benefits to fake speech detection?\n\nOnly “Copy synthesis” is studied in this paper. The paper mentioned vocoders are used in speech synthesis technology, such as TTS and VC, and therefore the fingerprint of the vocoder is important to fake speech detection. However, whether the fingerprint extraction is indeed affected by the TTS or VC pipeline is not studied. \n\nThis paper only studies the classification in the in-domain scenarios. That is, the vocoders are given, and the training data is limited to LibriTTS. However, in the real-world scenario, one cannot know and list all the possible vocoders. As a result, a simple classification experiment might not be enough. The authors can consider including the “unknown” category or computing the similarity between each vocoder.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0GsEIJqH0e", "reviewer": "Reviewer_S3gk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to verify the existence of vocoder fingerprints by training a classifier to identify the sources of generated audio waveforms. The authors analyzed the distinguishability of different vocoders from four aspects: (1) vocoders with different architectures, (2) vocoders trained on different datasets, (3) vocoders with different initial weights, and (4) vocoders with different batch sizes and dropout rates. The experimental results show that with different architectures or even small differences in the training process, the fingerprints of different vocoders are distinct. All different vocoders are distinguishable under the four scenarios mentioned above.", "summary_of_the_review": "Overall, given the lack of novelty, further analysis on factors that cause distinct fingerprints, and experimental details, the contribution, significance, and reproducibility of this work are limited.", "strength_and_weaknesses": "**Strength:**\n\nThis work analyzed the distinguishability of different vocoders from four aspects and showed the existence of distinct vocoder fingerprints in these four scenarios.\n\n**Weaknesses:**\n\nThe novelty and significance of this work are limited.\n\nDiscovering fingerprints of vocoders with different architectures has been studied by Yan et al., and the novelty of this work is extending the analysis to other aspects, such as training data, initial weights, and batch sizes.\n\nAlthough the experimental results show the existence of distinct vocoder fingerprints under different scenarios, the lack of further analysis and discussion makes the results less significant and explainable. To better support the results, the author should provide more insight, explain factors that cause distinct fingerprints, or show how the classifier distinguishes different vocoders.\n\nXinrui Yan, Jiangyan Yi, Jianhua Tao, Chenglong Wang, Haoxin Ma, Tao Wang, Shiming Wang, and Ruibo Fu. An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio. arXiv:2208.09646 [cs, eess], August 2022. doi: 10.1145/3552466.3556525.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-pDBhzRlK3-", "reviewer": "Reviewer_k7xs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the ability to identify the source model that generated the audio files. The authors explore generating samples from seven models different models. Specifically, the authors trained a classification network to distinguish between the samples. Results suggest that such classifiers can almost perfectly distinguish between the different vocoders. \nThe authors additionally provide an analysis of different training configurations such as batch_size, seed, dropout rate, and dataset split. Authors found that all such changes result in samples that are distinguishable from each other via a classification network. ", "summary_of_the_review": "As recently a large number of neural vocoders were proposed, the idea of analyzing them, specifically from the point of view of vocoder fingerprints is important and interesting research. \nHowever, I still have some issues/questions: \n1) did the authors analyze the same recording recorded from two different microphones? for instance, using WSJ0? as this research analyses the acoustic features of the generated audio, I would expect to see the same trend also for different mics.\n2) the authors said that such findings allow source identification of fake audio, however, I'm not sure how. The authors show that even the smallest change in the training pipeline causes a completely different set of samples, so how can we know if given samples were generated from model-1 using different seeds or from model-2? Generally speaking, although the findings are interesting, it is not clear how to interpret that. Can we remove such behavior? Do we want/need to fix it? Can we identify if a sample was generated from model-1 also when considering different variations of the model? I believe all these questions should be discussed here in the paper. \n3) Res2Net model was proposed for images, did you re-train the classifier on audio? If so, the visualizations in Figure 1 and Figure 7 are not very surprising, or am I missing something? These features are used for a linear classifier, which according to the results works pretty well. So it makes sense that these features are linearly separable. \n\nI'm willing to increase my score in case I missed something or if the authors will address the concerns raised above. ", "strength_and_weaknesses": "**Strengths:** \n1) this is an important topic and the analysis is interesting.\n2) the paper is clearly written and easy to follow.\n\n**Weaknesses:**\nalthough the analysis is interesting: \n1) some important and basic baselines are missing.\n2) some of the claims in the paper should be adjusted / better motivated (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject