Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the fingerprints of the vocoders. The authors studied whether resynthesized speech can be classified into real speech and a set of vocoders. Specifically, A deep classifier is trained on top of the LFCC feature extracted from the synthesized waveform. In this paper, 7 vocoders are studied in the experiments: WORLD, LPCNet, Parallel WaveGAN, HiFiGAN, Multi-band WaveGAN, and Style-MelGAN.
The experiment result shows that different vocoders have their own fingerprint, and the fingerprint is sensitive to training hyperparameters and the vocoder architecture.

strength_and_weaknesses: ### Strength:
The research topic of this paper is recently emerging and interesting.

Unlike standard fake speech detection, which is mainly a binary classification problem, this paper further studies whether we can tell which vocoder is used for fake speech synthesis.

### Weaknesses:
The authors mainly showed the experiment result without further investigation. For example, why the vocoder training is so sensitive to the training data, hyperparameter, and random seed? (Even differences in one data point or the random seed lead to significant differences) 

There’s not much insight into the usage scenario of the proposed fingerprint extraction and classification pipeline. How can the fingerprint of the vocoders be used in a real-world scenario? How to use the fingerprint of the vocoders to protect the copyright? And how can the fingerprint of the vocoders provide benefits to fake speech detection?

Only “Copy synthesis” is studied in this paper. The paper mentioned vocoders are used in speech synthesis technology, such as TTS and VC, and therefore the fingerprint of the vocoder is important to fake speech detection. However, whether the fingerprint extraction is indeed affected by the TTS or VC pipeline is not studied. 

This paper only studies the classification in the in-domain scenarios. That is, the vocoders are given, and the training data is limited to LibriTTS. However, in the real-world scenario, one cannot know and list all the possible vocoders. As a result, a simple classification experiment might not be enough. The authors can consider including the “unknown” category or computing the similarity between each vocoder.

clarity,_quality,_novelty_and_reproducibility: This paper lacks related works and an explanation of why those works are related to this paper. Also, no related works are mentioned to support the observation from the experiment results. 

The novelty is limited. Compared to the previous work (Yan et al. 2022), this paper further extends the experiments to study the effect of hyperparameters and training data. However, the methodology is similar to the previous work, and not much deeper insight is provided in this paper.

No source code and detailed experiment setup are provided. Therefore, the experiment result might not be easy to reproduce.

summary_of_the_review: Overall speaking, this paper mainly provided the experiment results without further investigation and explanation, which might not be inspiring enough to motivate the community. And therefore, the paper is not ready to publish without deeper investigation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to verify the existence of vocoder fingerprints by training a classifier to identify the sources of generated audio waveforms. The authors analyzed the distinguishability of different vocoders from four aspects: (1) vocoders with different architectures, (2) vocoders trained on different datasets, (3) vocoders with different initial weights, and (4) vocoders with different batch sizes and dropout rates. The experimental results show that with different architectures or even small differences in the training process, the fingerprints of different vocoders are distinct. All different vocoders are distinguishable under the four scenarios mentioned above.
strength_and_weaknesses: **Strength:**

This work analyzed the distinguishability of different vocoders from four aspects and showed the existence of distinct vocoder fingerprints in these four scenarios.

**Weaknesses:**

The novelty and significance of this work are limited.

Discovering fingerprints of vocoders with different architectures has been studied by Yan et al., and the novelty of this work is extending the analysis to other aspects, such as training data, initial weights, and batch sizes.

Although the experimental results show the existence of distinct vocoder fingerprints under different scenarios, the lack of further analysis and discussion makes the results less significant and explainable. To better support the results, the author should provide more insight, explain factors that cause distinct fingerprints, or show how the classifier distinguishes different vocoders.

Xinrui Yan, Jiangyan Yi, Jianhua Tao, Chenglong Wang, Haoxin Ma, Tao Wang, Shiming Wang, and Ruibo Fu. An Initial Investigation for Detecting Vocoder Fingerprints of Fake Audio. arXiv:2208.09646 [cs, eess], August 2022. doi: 10.1145/3552466.3556525.

clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality:**

This paper is organized well and presented clearly.

**Novelty:**

As mentioned above, the novelty of this work is limited.

**Reproducibility:**

Some details for implementing and training the vocoders are missing. Note that if some vocoder is not well converged and well optimized, the classifier can easily identify it according to specific distortion in generated utterances. Below are some examples of missing details.

- What were the acoustic features used for the vocoders? What were the parameters to extract the features?

- The original LPCNet restores waveforms based on self-defined acoustic features (BFCC and others). It is said that the LPCNet in this work used Mel spectrums. Why?

- How were the vocoders trained? Did the training configurations (e.g., steps, batch size) the same as their original papers? Or were they set the same for training different vocoders?

In Section 4.2, for the fifth architecture, does it mean "Multi-band WaveGAN" or "Multi-band MelGAN"?

summary_of_the_review: Overall, given the lack of novelty, further analysis on factors that cause distinct fingerprints, and experimental details, the contribution, significance, and reproducibility of this work are limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper analyzes the ability to identify the source model that generated the audio files. The authors explore generating samples from seven models different models. Specifically, the authors trained a classification network to distinguish between the samples. Results suggest that such classifiers can almost perfectly distinguish between the different vocoders. 
The authors additionally provide an analysis of different training configurations such as batch_size, seed, dropout rate, and dataset split. Authors found that all such changes result in samples that are distinguishable from each other via a classification network. 
strength_and_weaknesses: **Strengths:** 
1) this is an important topic and the analysis is interesting.
2) the paper is clearly written and easy to follow.

**Weaknesses:**
although the analysis is interesting: 
1) some important and basic baselines are missing.
2) some of the claims in the paper should be adjusted / better motivated (see below).
clarity,_quality,_novelty_and_reproducibility: **Clarity:** the paper is clearly written.

**Novelty:** the idea and analysis are novel.

**Reproducibility:** the authors provide all the details to reproduce the results.
summary_of_the_review: As recently a large number of neural vocoders were proposed, the idea of analyzing them, specifically from the point of view of vocoder fingerprints is important and interesting research. 
However, I still have some issues/questions: 
1) did the authors analyze the same recording recorded from two different microphones? for instance, using WSJ0? as this research analyses the acoustic features of the generated audio, I would expect to see the same trend also for different mics.
2) the authors said that such findings allow source identification of fake audio, however, I'm not sure how. The authors show that even the smallest change in the training pipeline causes a completely different set of samples, so how can we know if given samples were generated from model-1 using different seeds or from model-2? Generally speaking, although the findings are interesting, it is not clear how to interpret that. Can we remove such behavior? Do we want/need to fix it? Can we identify if a sample was generated from model-1 also when considering different variations of the model? I believe all these questions should be discussed here in the paper. 
3) Res2Net model was proposed for images, did you re-train the classifier on audio? If so, the visualizations in Figure 1 and Figure 7 are not very surprising, or am I missing something? These features are used for a linear classifier, which according to the results works pretty well. So it makes sense that these features are linearly separable. 

I'm willing to increase my score in case I missed something or if the authors will address the concerns raised above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to verify the existence of vocoder fingerprints by training a classifier to identify the sources of generated audio waveforms. The authors analyzed the distinguishability of different vocoders from architectures, datasets, initial weights, and batch sizes and dropout rates. The experimental results show that the fingerprints of different vocoders are distinct. All different vocoders are distinguishable under the scenarios mentioned above. But after the rebuttal, the authors further add an experiment to show that the vocoder architecture differences may overshadow fine-grained differences resulting from different training setups in the vocoder fingerprint analysis process.

Discovering fingerprints of vocoders with different architectures has been studied by previous work, and the novelty of this work is extending the analysis to the effect of hyperparameters and training data. However, the methodology is similar to the previous work, and not much deeper insight is provided in this paper.
justification_for_why_not_higher_score: The paper lacks novelty and new insight, so there is no good reason to accept.
Decision: Reject