E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
Authors: Zhisheng Zhang
Derui Wang
Yifan Mi
Zhiyong Wu
JieGao
Yuxin Cao
Kai Ye
Jason Xue
Jie Hao
Venue: NeurIPS 2025 poster ()
Keywords: Voice Protection; Adversarial Examples; Speech Synthesis; Data Poisoning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a proactive defensive framework against malicious production LLM-based speech synthesis to protect our voice information.

Abstract: Recent advancements in speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world applications. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to address the production large language model (LLM)-based speech synthesis. While previous studies have considered the protection for fine-tuning synthesizers, they assume manually annotated transcripts. Given the labor intensity of manual annotation, end-to-end (E2E) systems leveraging automatic speech recognition (ASR) to generate transcripts are becoming increasingly prevalent, e.g., voice cloning via commercial APIs. Therefore, this E2E speech synthesis also requires new security mechanisms. To tackle these challenges, we propose E2E-VGuard, a proactive defense framework for two emerging threats: (1) production LLM-based speech synthesis, and (2) the novel attack arising from ASR-driven E2E scenarios. Specifically, we employ the encoder ensemble with a feature extractor to protect timbre, while ASR-targeted adversarial examples disrupt pronunciation. Moreover, we incorporate the psychoacoustic model to ensure perturbative imperceptibility. For a comprehensive evaluation, we test 16 open-source synthesizers and 3 commercial APIs across Chinese and English datasets, confirming E2E-VGuard's effectiveness in timbre and pronunciation protection. Real-world deployment validation is also conducted. Our code and demo page are available at https://wxzyd123.github.io/e2e-vguard/.

E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
Zhisheng Zhang Derui Wang?, Yifan Mi?, Zhiyong Wu'*,
Jie Gao', Yuxin Cao², Kai Ye Minhui Xue Jie Hao
1 Shenzhen International Graduate School, Tsinghua University CSIRO's Data61
2 Beijing University of Posts and Telecommunications
Responsible Al Research (RAIR) Centre, The University of Adelaide 5 National University of Singapore 6 'The University of Hong Kong Corresponding authors
Abstract
Recent advancements in speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world appli- cations. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to address the production large language model (LLM)-based speech synthesis. While previous studies have considered the protection for fine-tuning synthesizers, they assume manually anno- tated transcripts. Given the labor intensity of manual annotation, end-to-end (F2E) systems leveraging automatic speech recognition (ASR) to generate transcripts are becoming increasingly prevalent, e.g. voice cloning via commercial APIs. Therefore, this E2E speech synthesis also requires new security mechanisms. To tackle these challenges, we propose E2E-VGuard, a proactive defense framework for two emerging threats: (1) production LLM-based speech synthesis, and (2) the novel attack arising from ASR-driven E2E scenarios. Specifically, we employ the encoder ensemble with feature extractor to protect timbre, while ASRR-targete adversarial examples disrupt pronunciation. Moreover, we incorporate the psy- choacoustic model to ensure perturbative imperceptibility. For comprehensive evaluation, we test 16 open-source synthesizers and 3 commercial APIs across Chinese and English datasets, confirming #2FF-GGuard's effectiveness in timbre and pronunciation protection. Real-world deployment validation is also conducted. Our code and demo page are available at https://wzzdd123. github.io/e2e-vguaa.
1 Introduction
High-quality synthetic speech based deepfake techniques [1, 2] has been applied in daily scenarios, such as video dubbing, and vehicle-mounted voice assistants. Large language models (LLMs) [3, 4] have furthered the development of speech synthesis, i.e., text-to-speech (TTS). Current TTS models enhance the synthesis performance by integrating LLMS as a core component for paralanguage features, achieving human-level results. The most advanced technique can be based foundation model [5]. TTS models can be divided into two categories, i.e., zero-shot [1] and fine- tuning-based [6]. Zero-shot models utilize reference audio as the prompt clone the voice. In contrast, fine-tuning-ban based models require a few minutes of speech samples to replicate the target speaker better. The advancements in speech synthesis, on the one hand, bring huge convenience; on the other hand, they pose a potential security threat in the hands of pirate users. These pirate users may conduct illegal speech synthesis for illegal purposes, such as telccommunication fraud. Therefore, the prevention approach against unauthorized synthesis is of vital importance.
audio
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Existing Defenses. Existing protective methods against voice cloning focus on two main types: (I) Defense based on adversarial examples (AEs), e.g., AntiFake [7], and Attack VC [8]. (2) Defense based on unlearnable examples (UEs), e.g., POP [9], and SafeSpeech [10]. AFs-based protection generates audio AEs through TT'S models' encoders to prevent zero-shot voice cloning. contrast, UEs-based protection utilizes universal objective to generate model- perturbation, which disrupts the training phase of TTS models to achieve tine-tuning-bet baaed speech synthesis.
Limitations and Challenges. Prior studies have some limitations when considering broader scenarios: (1) Industrial-level and LLM-based TTS. LLMs have advanced the previous deep neural network (NN)-baser speech synthesis. Common approaches employ speech tokenizer to encode the imput waveform into discrete tokens, which are fed into LLMs. The decoded outputs from LLMs then guide the generation module, such as the flow-matching module [1]. The key distinction lies in decoding audio signals into discrete tokens rather than continuous embeddings, a direction rarely explored in this protection of LLMM-based TTS research. Moreover, voice replication products have emerged in the industry, where voice cloning via API constitutes the focus of this paper. (2) Und-To-EEn Scenario. The assumption in prior studies is that the text corresponding to the audio has been provided. A more realistic scenario is that, for a customized dataset, the text needs to be obtained by the training party itself. For example, the text transcripts are transcribed through an automatic speech recognition (ASR) system, rather than relying on manually annotated open-source datasets. In fact, commercial APIs typically accept only audio input and rely
ASR their backend.
Why is End-to-End Wine-Tuning Important? End-to-end fine-tuning aims to ensure that both input and output data are exclusively of audio type and integrates an ASR system into the training process for text recognition. Based on this, two research questions (RQ) should be considered. RQI: Why fine-tuning? On the one hand, some models, such as VITS [6], only support fine-tuning without zero- shot capabilities. On the other hand, fine-tuning can achieve better synthetic performance than relying solely on a single sample in zero-shot scenarios. RQ2: Why end-to-end fine-tuning? Adversaries often collect audio data from public social platforms like YouTube and Bilibili, where the audio does not come with corresponding text data. Manually annotating text is typically time-consuming and labor-intensive. Therefore, leveraging an ASR system based on deep learning techniques is a more practical choice due to its efficiency and high recognition accuracy. In the real world, the industrial product for speech synthesis trains a new speaker via an API connection with only audio input. The supplemental ASR system is utilized for automatic recognition in their service.
Our Solution and Contributions. To counter the LLM-based and end-to-end scenarios, we propose E2E-VGuard, a proactive defense framework that disrupts both timbre and pronunciation. For the timbre, we introduce untargeted and targeted speaker protection based on the proposed feature loss, which utilizes ensembled encoders and feature extractor to obtain audio features for LLM-based TTS, resulting in dissimilar synthetic speeches to achieve identification protection. For the pronunciation, E2E-VGuard generates the audio AEs to fool the ASR system with incorrectly recognized text and disrupt model's learning process of the text and pronumciation. Moreover, to realize imperceptibility, introduce the psychoacoustic model [11] to add the perturbation within specific frequency domain, reducing the detection by the human ears. For a comprehensive evaluation, we conduct experiments on both English and Chinese datasets, verifying their effectiveness and transferability across 16 open-source, 3 commercial models, and 7 ASR systems. E2E- VGuard is robust against sophisticated data augmentation and perturbation removal techniques. Moreover, we have validated the E2E- VGuard's robustness in the real world. Our contributions can be summarized as follows:
We introduce more realistic and challenging scenario of end-to-end fine-tuning-base based speech synthesis, and we propose a proactive framework, E2E-V -Guard, to protect individual information.
We consider defensive waveform disruption from the perspectives of timbre and pronunciation. For the timbre disruption, we propose feature objective based on the encoder ensemble and feature extractor. For the pronunciation disruption, we utilize AEs against ASR systems to fool TTS models with incorrect text and impact pronunciation.
We utilize the psychoacoustic model with &n-ormm to enhance the perturbation imperceptibility for better human audible perception.
We evaluate the effectiveness, transferability, and robustness of VGurrd through compre- hensive experiments across diverse settings: 19 TTS models (including 16 open-source and commercial), 7 ASR systems, and 3 English and Chinese datasets.
2

---

2 Related Work
Speech Synthesis. Modern speech synthesis can be primarily categorized into two types: DNN-basee and LLM-based architectures. The former mainly focuses on building a synthesizer [6. 12], while the latter integrates the LLM with a synthesizer and encodes audio into discretized tokens [1]. The LLM captures the prosody and semantic features, while the synthesizer captures timbre and environmental information [1]. Recently, audio foundation models have advanced rapidly. During the pre-training phase, models typically acquire basic question-an guestion-answering answering capabilities. In the post-training phase, they gain downstream tasks, e.g., TTS, through supervised fine-tuning (SFT). This emerging TTS approach based on speech foundation models is also the focus of this paper, e.g, Step-Audio [5].
Voice Protection. Defenses against synthetic speech can be categorized into proactive and passive defenses [7, 13, 14]. We primarily focus on proactive defense techniques, which aim to reduce speaker similarity in synthesized audio at the data source. For example. Huang al. [8] and Yu et al. [7] utilized adversarial examples to disrupt voice cloning. Recently, Zhang el al. [9, 10] proposed unlearnable samples to degrade the quality of speech synthesis systems, thereby defending against tie-ttnnig-Bae -based voice synthesis. In addition, we note recent advances in passive defense techniques, such as the robust deepfake audio detection method proposed by Zhang et al. [15], which aims to mitigate the risks posed by synthetic audio. However, existing approaches remain ineffective against more advanced TTS models and end-to-end scenarios at the data level.
3
D2E-VGuare Design
3.1 Threat Model
In this section, we analyze the necessity of end-to-end fine-tuning through two examples.
ByteDance's API.' Taking ByteDance's 's voice cloning product as an example, third-party users upload audio to the company's server infrastructure via the API. First, the input audio is transcribed into text using an ASR system. Then, both the transcrihed text and audio information are fed into the TTS model for voice training, enabling synthesis of target text using the trained voice timbre. This commercial API-based speech synthesis approach also fits into the end-to-end scenario.
Open-sourced WebUI Operation. GPT-SoVITS [2] is open-source zero-shot voice cloning model and supports few-sample fine-tuning to improve voice similarity. The project provides a WebUI-based fine-tuning workflow: the corresponding text is first obtained utilizing an ASR system, i.e., Whisper by OpenAl [16]. Subsequently, fine-tuming is performed based on the text and audio, which also satisfies the end-to-end scenario requirements.
From these two examples, we can observe that end-to-end speech synthesis is prevalent in real- world applications. Moreover, ASR-based workflows are gradually becoming mainstream, which typically run in model or server backends and are thus not directly accessible to end-users.
3.2 Problem Formulation
From the perspectives of timbre and promunciation, we implement 22E-VGuard for voice anti-cloning. At the timbre level, previous work has focused targeted [8, 7] and untargeted [9] attacks, i.e., whether to select a specific target speaker for timbre perturbation. We consider two approaches to timbre protection with a broader defensive selection: targeted and untargeted. For pronunciation protection, adversarial targeted attacks are employed against ASR systems, ensuring that the ASR system transcribes into a specific target text. This because we aim for the transcribed text to be meaningful rather than gibberish, thereby reducing the adversary's 's detection of textual alterations. The workflow is shown Figure 1, and the objective function of 2uuard can be expressed as:
L(x') = (xsr(x') + a Liea + B Epso((')
s.t. || - xlppS and x' E [-1,1]",
where (.sr(.) represents the loss of the ASR system, Leea (-) denotes the speech feature loss, and (-sy(-) is the perceptual optimization function for embedded perturbations, i.e., the psychoacoustic
3

---

Sytthan Languige
Text
Eroader
Pertertaten
Protected
System
Lorge
Lengage
Synthasbar
Spaech
Takenierr
Figure 1: The workflow of E2E-VG and the end-to-end speech synthesis pipeline with LLM.
model. a and 3 are weight coefficients for multi-task balance. Moreover, x and x' represent the original and protected audio, respectively, with features normalized to fall within the range [-1, 1]. The perturbation := x' x is bounded by an f--norm constraint e. is the waveform length.
3.3 Timbre Prevention
TTS models typically generate high-quality timbre-similar audio using reference speech samples. We implement timbre-level voice protection achieve anti-cloning voice protection and ensure cloned audio dissimilarity from the original speakers. Previous research [7, 8] primarily employs encoders for timbre extraction and optimization create timbre divergence between original and target audio. Building upon the timbre encoder ensemble [7], we further incorporate acoustic features by the MFCC [17] extractor to better perturb speaker identity features. In designing E2E-VGuard, we propose two protection methods: targeted and untargeted timbre protection.
Untargeted Timbre Protection. Untargeted protection maximizes feature distance between original audio æ and protected audio x', rendering synthesized audio unrecognizable as the original speaker. Zero-shot TTS models typically employ an encoder to extract cloning-relevan features, e.g., timbre in Voice [1] and style in Style'TTS2 [12]. combining them with pre-trained articulation patterns for speech synthesis. Building on prior findings demonstrating enhanced transferability through encoder ensemble [7, 18], we extract timbre and acoustic features through multiple encoders from target TTS models to improve the protective generalizability against unseen models. Moreover, to counter LLMM-based TTS, we consider to protect at the audio's 's original features by changing the discrete tokens obtained by the audio tokenizer for the LLM component with the MFCC extractor to protect articulation and prosodic patterns. The objective function can be formulated as:
feea Σ CS(E,()), + (S( 1=1
where & the number of selected encoders, CS(-.) is cosine similarity with lower values indicating reduced similarity [18], E()) is the timbre encoder, and (-) denotes the MFCC extractor.
Targeted Timbre Protection. Targeted protection steers original audio features toward designated target speaker 2,, causing TTS models to synthesize target-like audio. For target selection, we construct speaker database following AntiFake [7], choosing the most dissimilar speaker by feature distance for each protected audio. This systematically identifies the most dissimilar speaker in feature space, contrasting with conventional random opposite-gendes sampling. The optimization function is:
& Lfea(x) = Σ (SSE,())), CS(M
Both methods effectively protect the speaker's identity. Untargeted protection enables broader adversarial sample exploration due to undefined optimization targets. Targeted protection leverages carefully selected feature-divergent samples for enhanced timbre disruption.
3.4 Pronunciation Prevention
Fine-tuning TTS model requires pairs of text and audio data to achieve alignment between and pronunciation. For instance, VITS utilizes the monotonic alignment search algorithm to search for

---

the correspondence between time frames and characters. Text data can be obtained through manual annotation or automatic recognition by an ASR system. The former consumes a lot of manpower, time, and costs, therefore, it is more common to employ an ASR system to recognize text information. Previous work [19, 20, 21] has shown that ASR systems are relatively vulnerable and susceptible to adversarial examples that interfere with recognition accuracy. Based on this finding, we consider utilizing adversarial examples to disrupt the ASR system's 's recognition, causing the protected audio different text. Incorrect text-audio pairs will disrupt the pre-trained model's s learning of the pronunciation, preventing the synthesis of audio with the corresponding pronunciation based on the desired text, thereby effectively protecting personal unauthorized audio data.
to be recognized
Adversarial attacks on ASR systems can also be divided into targeted and untargeted types, differing in whether a specific target text is provided when optimizing. The generated text by the untargeted attack is incoherent, while targeted attacks ensure the readability of recognized text by specifying the text, effectively reducing the adversary's s awareness of the anomalous recognition text. Therefore, we choose targeted attacks against ASR systems. The selection of the target text affects the effectiveness of adversarial examples. For instance, long audio paired with short target text may result in the latter part of the recognized text retaining the original correct text, necessitating further optimization considerations. Therefore, we need to consider the selected text and its length. In targeted attacks on timbre in Section 3.3, since the chosen audio already contains specific pronunciation information different from the original audio, and the application of MFCC also benefits the ASR system's recognition of the target audio text [19] in the optimization of adversarial examples, we select the target audio's text the target text. For untargeted attacks on timbre, we select text of the same length as the audio for different audio, which is more beneficial for adversarial attacks against ASR systems. In summary, the perturbation of pronunciation information can be represented as:
(asr( = F (ASR(z²), ye)
where F is the objective function of the ASR system, such as the connectionist temporal classification (CTC) [22] loss for Wav2vec2 [23]. A-R(-) computes the input audio obtain outputs, such the probability distribution of recognized words. Additionally, Yr denotes the targeted text.
By optimizing Eq. (4), the adversary utilizes the ASR system and obtains incorrect text, thereby interfering with eext-rrouuncition alignment, making the synthesized audio unintelligible.
3.5 Psychoacoustic Model
The perturbation embedded in specific region will be masked, making it imperceptible for the human ear hear the sound in that region, which is known as the masking effect. Leveraging this characteristic, we optimize the perception of the embedded perturbation utilizing the psychoacoustic model to enhance the naturalness and imperceptibility. The masking effect can be divided into temporal masking and frequency masking. Following the settings of V-Cloak [24], we employ the frequency masking part ensure perturbations are imperceptible to the human ear.
We set the original audio as the masker so that the perturbation (maskee) remains below the masking threshold. Let F represent the total number of frequencies and fg represent the masking threshold of the original audio æ, with each element indicating the maximum acceptable perturbation at frequency f. Assume pa represents the log-magnitude power spectral density (PSD) of audio x. Therefore, the objective function of the psychoacoustic model can be expressed as:
Cxsy(=')
1 F
Σ max (0.Pe--(()) -
(5)
where max(0, -) ensures the value is nen-negative.
Additionally, we impose constraints on the perturbation through lz. as Duan et al. [25] found that & performs optimally in correlation with human perception within the É, norm. Therefore, we further reduce the perceptibility of the embedded perturbations by introducing L2, formulated as:
L2(x') = || x||2-
(6)
To ensure that the protected audio does not exceed the range it should belong to, after obtaining the final protected audio, we map its range back to between -1 and to guarantee it is normal audio waveform. In conclusion, the algorithm of E2E- -Guar has been provided in Appendix B.
5

---

Experiments and Analyses
Experiment Organization. In this section, we introduce our experimental evaluation. We first provide our experimental settings in Section 4.1. Then, we evaluate the effectiveness of end-to-end fine-tuning-basee speech synthesis in Section 4.2, zero-shot scenarios in Section 4.3, and commercial API test in Section 4.4. Moreover, explore the effect of each component in Section 4.5 and test the robustness of -Guard Section 4.6. In the Appendix, there is also an inevitable evaluation. We validate the effect across multilingual and multi-speaker settings in Appendix FF and various ASR systems in Appendix F. Finally, we conduct a human survey of the effectiveness and perception in Appendix G. All of our experiments ethical considerations about human study and commercial test are provided in the Appendix, and some limitations and discussions of the E2E-VGuard have been discussed Appendix A.
conducted
NVIDIA 4090 GPU. Moreover, the
4.1 Experimental Settings
In this section, we introduce the experimental settings utilized in experiments.
Synthesizers. We select total of 16 TTS models for evaluation. Section 4.2 includes 6 models: GPT-SoVITS (GSV) [2]. CosyVoice Voice [1], Llasa- 1B [26], LlBsa-BB|[[ [26), StyleTTS2 [12], and VITS [6],
used for end-to-end fine-tuning tests. Section 4.3 includes 7 models: Index-TTS [27], FereRedTTS- IS [28]. SeppAAuii-TTTS [5], Paark-TTS [29], XTTS [30], FishSpeech [31], and Dia-1.6B [32], for zero-shot validation. Moreover, we test models based on in-context learning rather than speaker encoder for feature extraction in Section 4.3: VALLE-X [33), E2-TTS [34), and F5-TTS [35]. Section 4.4 involves commercial APIs: ByteDance, Alibaba, and MiniMax. Notably, Step-Audio-TTl developed through post-training of speech foundation model. Further details about open-source models' architectures and sources have been provided in the Appendix D.I.
Encoders. In Section 3.3, six encoders serve as feature extractors: posterior encoders from VITS and GSV, MFCC features, WavLM [36], CAM++ [37] from Voice, and the style encoder from StyleTTS2. Among these, MFCC represents traditional acoustic features, WavLM is speaker verification system, while the remaining four are timbre or style encoders from TTS systems. This multi-encoder framework improves E2E-VGuard's cross-model transferability in timbre preservation.
ASR Systems. Adversaries may employ different ASR systems recognize text. We conduct model- specific adversarial attacks against ASR systems and can effectively induce misclassification. Seven ASR models are selected, namely Wav2vec2 [23]. Whisper (base, small, medium, and large) [16], Conformer [38], and CitriNet [39]. Appendix D.2 shows different structures in detail.
Datasets. We selected both
and multi multi-speaker datasets in English and Chinese to verify E2E- VGuard's protection performance across different scenarios, employing LibriTTS [40] for English single-speaker evaluation following [10], CMU ARCTIC [41] for English multi-speaker testing, and THCHS30 [42] for Chinese multi-speaker assessment. For each dataset, we have randomly allocated 80% for training and 20% for testing. If the model requires validation set, utilize 10% of the training set as the validation set.
Metrics. The strength and perception metrics are considered.
Word Error Rate (WER) [9]. It represents the speech intelligibility, Higher WER reflects lower speech quality. We utilize a pre-trained ASR model, OpenAl's Whisper with medium size [16].
Speaker Similarity (SIM) [10]. SIM measures the speaker similarity of two speeches. Lower SIM reflects lower similarity between original and synthetic speeches. We employ ECAPA-TDNN [43] extract speaker embeddings and compute SIM values.
Signal-to-Noise Ratio (SNR) [10]. SNR reflects the ratio of the embedded perturbation.
Perceptual Evaluation of Speech Quality (PESQ) [18]: PESQ is an objective perceptual score of the speech quality, ranging from -0.5 to 4.5.
Mean Opinion Score (MOS) [9]. MOS is obtained through human interaction as subjective metric, which measures the human perception of speeches, ranging from 0 to 5.
Hyperparameter Settings. For fine-tuning, we keep the conventional settings with training details in Appendix D.I. Morcover, the hyperparameters in Eq. (1) are set to balance the effectiveness and

---

Table 1: Effectiveness and Perception Results of End-
tetttttnnnnn-aaa Speech Synthesis.
The best and second- best protective results are highlighted with bold and underlined. respectively.
Method
GSV (2)
Cuuy Voice [I]
Limu-18 [26]
Liuea [26]
StyleTS52 [12]
VITS [6]
Imperceptibiityy
WER:1
S(M())
WER(1]
SB3(]]
WER(T)
S(M[1)
WER(T)
S0M[L]
WER(7)
SIML4
WER(1)
S0MG])
SNR(") PESQIT)
7,499
AlsacVC
0,636
0.588
15.201
0.563
0.674
0.000
-2.456
3,890
AutiFike
26.846
0.149
0.33
22.391
0.250
15.500
0.254
0.253
41.491
12.539
1.759
POP (9)
0.677
4,452
01755
7.283
0.683
5,002
0.675
1
07385
15.281
18.425
1.318
POP4ESP
40.308
0.208
10:312
0.259
273355
0.280
36.619
0.287
7.320
0.258
55.411
0.349
11:345
1/571
SufeSpeech [19]
44.777
0.339
9,596
0.459
9.367
0.288
16.970
0.269
5.215
0.366
105.524
0.150
7.647
1,412
EZE- VGuard (UT)
66.471
0.123
21.566
0.081
74.956
0.155
$0.221
0.134
45.856
0.082
05.740
0.106
18.523
1.54p
22E-VGuard (T)
94,812
0.284
72.143
0.375
63.945
0.442
$9.510
0.310
54.732
0.229
125.299
0.245
20.470
2.324
Table 2: Protective performance across industrial-leved and LLM-based models of E2E- under sero-shot end-to-end speech synthesis.
Mathool
[ndes-17S[[2]]
FIS- (25)
Step- 11o--Addo---11115
Spartc-TTS [29]
XT1S-2 [10]
FichSpeech [31]
Dio- (22)
WTF(())
SIM[_)
(FF2(1)
SIMI5
WER(())
WFRR(T)
cleat
3.547
0.614
1.382
0.805
2.505
0.575
1:341
01.00
1.256
01.555
1.548
0.699
4.506
0.351
AnniFike (7)
3,685
0.147
3,152
0.276
2.305
0.206
4,314
0.243
7.135
0.193
5.495
0.178
5.471
0.263
PEIP (9]
1.2.0
01362
4.491
0.366
5.665
0.112
10.667
0.180
73859
0.338
5.905
0.179
6.366
0:115
ScteSpeachi [100
5.532
0.244
0.098
0.539
3,764
0.334
23.866
0.144
13.522
0.230
21.335
0.397
59,601
0.250
EZE-VGuurd (U7T)
4.474
6.528
5.156
0.008
5.761
9.746
53.200
VGaaV
imperceptibilityy of each component. We determine hyperparameters through experiments evaluating both loss values and component effectiveness, ultimately selecting a = 500 and B = x 10 Additionally, the in Eq. (1) is 8/255, and we optimize perturbation for 500 iterations.
4.2 Effectiveness on End-To-Enu Fine- Fine-Tuning Scenarios
To assess the effectiveness and transferability, we utilize E2E- VGuard to protect the LibriTTS dataset with the untargeted and targeted mode in Section 3.3 and set the target ASR system as Wav2vec2 [23].
Fine-tuning on Protected Dataset. After protecting the LibriTTS dataset, users can upload publicly social platforms. Adversaries may require these samples unauthorizedly and utilize advanced synthesizers for fine-tuning-bast speech synthesis.
Speech Synthesis and Evaluation. TTS models possess the capabilities of speech synthesis after fine- tuning. Fine-tuning-bass models can generate speeches with speaker ID and synthesized text, while zero-shot models require reference audio and synthesized text for feature extraction and cloning. Table shows the experimental results across different TTS models after fine mne-ttnnng on datasets protected by different strategies. We can find that our protected E2E-V VGuard can achieve outstanding protective strength than baselines in terms of timbre (SIM) and pronunciation (WER). For fine-tuning- based models, the E2E- VGuard achieves average increase of 19.775% (targeted, T) in WER compared to the best baseline values while reducing SIM by an average of 0.043 (UT), indicating lower speech intelliigbillity and similarity. The protection effect improves more significantly zero-shot models, with WER increasing by an average of 32.841% % (UT) and 50.060% (T) and SIM decreasing by 0.119 (UT). It demonstrates that audio protected by the E2E-VGuard effectively safeguards private information, prevents high-quality speech synthesis, and exhibits strong transferability across models.
Perception Analyses. The embedded perturbation should not interfere with the normal utilization of speeches. Table presents the simulated perception metries of different baselines. The SNR values are higher than all baselines, representing the lowest noise ratio and quality disruption.
4.3 Effectiveness on End-To-End Zero-shot Scenarios
In this section, we conduct zero-shot end-to-end speech synthesis on seven industrial-level and LLM-based TTS models. The reference audio transcripts are automatically obtained through an ASR system. Table 2 presents the test results, where our 22Ea-G (UT) achieves SOTA performance in voice timbre preservation across all models than baselines. Regarding pronunciation prevention, the average WER values of E22E-VGuar are 21.603% for UT and 23.604% for T, respectively, outperforming AntiFake's 's 4.932% and SafeSpeech's 19.255%. This indicates the synthesized audio demonstrates both dissimilar timbre characteristics and reduced pronunciation clarity.
7

---

Table 4: The ablation study of the E2E- EuB-d.
Table 3: Evaluation on ICL-based TTS models.
Methed
15-T18
12-TIN
(A1
w/e
Elfediveacs
Imperopuiiiit
SIMI3
WER(1)
SMM..
WERID)
WER801
SIMIUS
WUIUI'
AutiFsll (7)
4.003
8.24
02282
01655
4.004
0.474
0.399
06.40
18.15
0.51
0.20
12,142
101 400
03333
66.445
01488
13,M5
1.111
V.Gaarr (T)
79783
0.319
61991
0.373
56.001
0.170
0128
19.059
134
73775
95,100
0108
06-411
18.523
1344
20
0.8
20
0.8
WEE
WEn
3.5
SIM
35
SIM
0.5
WER
0.0
0.0
ByteDance
MiniMax
ByteDance Allbaba
MiniMax
Models (clean)
Models (E2ENGuarc)
(a) Clean mode.
(b) E2E-VGuard mode.
Figure Evaluation of protective performance on commercial APIs.
The experimental results from Section 4.2 and Section 4.3 demonstrate that our proposed E2E- VGuard effectively protects the latest open-source industrial-leve' and LLM- --adedaddddd models from timbre and pronunciation perspectives. The method achieves current SOTA performance levels in safeguarding personal information security, regardless of whether tine-tuning-bee
or zero-shot speech synthesis.
Evaluation on ICL.-based TTS models. Many existing TTS models extract timbre representations of target speakers through speaker encoder, and E2E-VGuard also utilizes this approach by integrating multiple speaker encoders to achieve timbre-level prevention. Additionally, some models obtain information such as timbre features through in-context learning (ICL) rather than using a speaker encoder. Table 3 presents our experimental results on three ICL-based models. The results demon- strate that E2E- maintains SOTA voice protection performance on ICL-based models and exhibits strong transferability. This effectiveness stems from our speaker encoder ensemble technique, which successfully hides or modifies the timbre information of the original speaker. Consequently, the timbre prevention of the proposed
does not rely on the specific speaker encoder.
4.4 Evaluation via Commercial APIs
Voice cloning through commercial APIs is relatively convenient, requiring only audio data input to train the target speaker. Once the server generates the trained speaker ID, speech synthesis can be readily implemented. This approach eliminates the need for local model deployment while achieving high-quality synthesis, the underlying models are black-box systems that demand robust defense mechanisms. We select three common commercial products supporting voice replication, including ByteDance, Alibaba, and MiniMax (represented by company names), for evaluation.
As shown in Figure 2, our experimental results reveal that compared with unprotected audio, E2E- VGuard reduces the average SIM score from 0.689 to 0.203 while increasing WER values. This demonstrates VGuard's'''s strong transferability, effectively safeguarding voiceprint information even in black-box scenarios involving commercial APIs. These performance improvements across similarity and pronunciation metrics confirm its practical effectiveness for real-world deployment.
4.5
Ablation Study
In the ablation study, we explore the functiinnaltty of our proposed optimization objectives and the hyperparameter selection for the trade-off of the perturbative performance and imperceptibilityy
Component Analyses. In this part, we explore the role of each component in Eq. (1). We separately investigate the impacts of Casr. Leea- and Epsy & L2 by removing each component from E2E- VGuard and evaluating the resulting protection effectiveness. Table 4 presents the results of this ablation study. We observe that removing the perceptual optimization module, i.e., Lysy & L2, achieves better protection but significantly increases perceptual disruption to the original audio due to the lack of perceptual alignment of noise, resulting in low SNR of 12.942. To directly examine

---

the effects of Lasr and fea. we optimize each term individually. When Lee is removed and only Casr retained, the SIM value on the GSV model is relatively high at 0.409, indicating significant leakage of voiceprint information. Conversely, when Lasr is removed and only L is retained, the disruption of text-proo prxtuunnnniiiionnn alignment diminishes, with the WER on the GSV model dropping to 49.059%. When all three components are present, they collectively provide effective protection at both the pronunciation and timbre levels while maintaining better imperceptibility of perturbations.
Hyperparameter Selection. In the Eq. (I), we employ hyperparameters a and 3 to balance the effectiveness of E2E- VGuard'ss' protection and the imperceptibility of the embedded, with values empirically set to 500 and 5 x 10 3 [24]. respectively. The value of a = 500 is chosen to amplify the loss for speaker identity protection, aligning its optimization scale approximately with that of Las. 3 is used to trade off the protection effectiveness and the perception quality of the perturbation. Specifically, a larger 3 (such as 5 x 10-2) yields better perception quality but weaker protection, e.g. the speaker similarity degrading, whereas smaller 3 (such as 5 x 10-1) enhances protection at the cost of reduced perception quality e.g., SNR lower than 15. Through empirical evaluation with various 3 values, we find that 3 = 5 10- satisfies an approximate trade-off between protection and quality of perception.
4.6 Robustness Test
In real-world scenarios, strong adversaries can find the obtained dataset with specific modifications, and adversarial techniques may be employed to improve synthesis quality. In this part, following [9, 10), we conduct the robustness test against perturbation removal and advanced data augmentation techniques. Moreover, we evaluate the effectiveness and robustness of E2E-VGuard in the real world.
Perturbation Removal. High-quality speech synthesis often requires high-quality input audio without audible perturbation [10]. Therefore, adversaries may utilize perturbation removal techniques to improve the quality of training samples and weaken unknown strategies users adopt before uploading. We refer to the use of two efficient denoising techniques [10], spectral gating (SG), and a DNN-based model named denoiser [44] to denoise each protected audio sample. This experiment is conducted on two models, i.e., VITS and GSV. For the SG method, the WER and SIM on the VITS model are 51.005% and 0.224, respectively, while on the GSV model, the WER and SIM are 31. 318958 and 0,251, respectively. The denoiser can nearly remove the audible background noise. We test the protective performance utilizing the denoiser for denoising. On the GSV model, the WER and SIM are 23. 23.10% and 0.243, respectively. The WER and SIM on the GSV model are 34.10% and 0. 0.261, respectively. This shows that even after removing the audible noise, E2E- VGuard can still protect speaker privacy, especially at the level of the speaker identity, with an average SIM value of only 0.238 and 0.252 across these two models using the SG and denoiser, respectively. The WER on the VITS model exceeds 50% after denoising using the SG method, effectively disrupting pronunciation.
Data Augmentation. Data augmentation used to alter the specific structures of embedded pertur- bations, thereby reducing effects. We consider three categories of data augmentation techniques:
Adversarial Defender [45]: Hussain el al. [45] discovered that in the audio field, certain adversarial defense methods can effectively disrupt adversarial audio examples, e.g., proposed E2F- E2F-VVGuard These adversarial techniques include: Down-sampling and Up-sampling (RS), Mel-spectrogram Extraction and Inversion (Mel), coantization-eeuunntiin
(Q-D), and Filtering. Audio Processor [9]: We consider speech-processing techniques to simulate real-world operations, following Zhang al. [9], including Speed Adjustment (Speed), adding Gaussian noise (Gaussian),
Time Masking (TiM), Pitch Shifting (PS), MP3 compression, and Tanh Distortion (Tanh).
Filters [9]: Filtering techniques are commonly used alter perturbations. We consider three types of filter techniques: Band-Pass. Filter (BPF), Low-Pass Filter (LPF), and High-Pass Filter (HPF).
Table 5 shows the results of data augmentations. We observe that the Mel technique significantly improves the intelligibility of synthesized audio, with WER values decreasing by 39. 39.194% and 19.825% on the VITS and GSV models, respectively. However, the SIM values remain high. Although data augmentation can disrupt the perturbation and reduce its protective effect, the embedded perturbation persists, and transforming the audio inherently degrades its quality, e.g., Mel, and Filters.
Real- World Robustness. After users upload audio protected by E2F-VGuard, adversaries may utilize various types of microphones to record and collect the played audio in the real world for voice cloning. To verify the robustness of E2E-VGuart in tver-the-air scenarios, we employed different

---

Table Results under data augmentation and defensive methods. The underline indicates the most significant decreases in protection compared to training without augmentation ("w/ o" in the Table).
Model
Metric
w/o
Adversaiial Defonder (45)
Andio Procesoo [9]
Filters [9]
Resample
Mel
Q-D
Filerrin
Speed
Gouseian
TM
PS
MP3
Tanb
BPP
HPF
VITS
WER
SIMC1
96.735
0.113
94.827
0.115
55,633
0.122
103.247
0.082
93.552
0.128
$4.101
0.050
84-381
0.063
95.654
0.058
91.796
0.099
$2,308
0.125
95,441
0.143
97.258
0.136
95.553
0.118
136.5
0.065
GSV
WER
SIMC1
69.148
0.074
55,085
0.95
35.210
0.158
75.011
0.117
57.497
0.147
77,642
0.044
44.232
0.128
77.365
0,109
68.607
0.049
41,908
0.154
71.446
0.126
83,050
40.46
$2882
0.135
48.1366
0.229
0.5
80-
60
20-
0.0
bonign
Lenovo viro
Lenove (onve=Vauiil Kiuck
Kaoli+ VNV
(a) Real-world environment.
(b) Results.
Figure 3: Robust test in the real world. (a) shows the experimental environment. (b) The experimental
results. "Lenovo+VIVO" represents the "speaker-microphone""
speakers play the audio the speaker and different microphones record the audio as adversaries. We conduct experiments in a quiet environment with background noise averaging 22 dBA (measured by taking the average over 10 seconds). We apply the built-in speaker of a Lenovo Laptop to play the audio and record the audio using VIVO and XiaoMi phones placed approximately one meter away from the speaker simulate the adversaries as shown in Figure 3a. In each test, we play 10 audio samples and ensure that the speaker's s loudness averages 46 dBA. Finally, for the recorded audio, employ the GSV model for fine-tuning and cloning due to its excellent few-shot cloning capability [2]. The original synthesized results without perturbation after recording are 16.837% and 0.485 of WER and SIM values with high speaker consistency, The average WER and SIM are 72.615% and 0.068, respectively, in Figure 3b, indicating excellent protection of voice timbre and pronunciation in real-world scenarios. Additionally, when we replace the speaker with mobile device, XiaoMi phone, and use the VIVO phone as the recording device, the protective effect remains high. This experiment demonstrates the robustness of E2E- 32E-VGuart VGuard the real world, as the over-the-air transmission acts form of data augmentation [18), and Section 4.6 illustrates the effectiveness of E2E- VGuard in handling data augmentation.
5
Conclusion
This paper focuses on the current mainstream industrial-Ievel and LLM-based TTS models. Consider- ing the more practical scenario of end-to-end speech synthesis, we propose a protection technique, VGuard, that effectively safeguards audio content from both timbre and pronunciation per- spectives. We conduct extensive experiments various speech synthesis models and multilingual datasets for evaluation. I Limitations and future work, e.g., time efficiency and the E2F- VGuard's reliance, are discussed in the appendix.
Acknowledgment
We sincerely appreciate anonymous reviewers for their insightful and valuable feedback. Zhisheng Zhang, Yifan Mi, and Jie Hao are supported in part by the National Natural Science Foundation of China under Grant No. U21B2020 and the Fundamental Research Funds for the Central Universities under Grant No. 2024ZCJH05. Zhisheng Zhang, Jie Gao, and Zhiyong Wu are supported in part by the National Natural Science Foundation of China under Grant No. 620761 44 and the Shenzhen Science and Technology Program under Grant No. JCYJ2022081 1 0101810110000 4030.
10

---

References
[1] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv: 240.000077700 .55407, 2024.
[2] RVC-Boss. Gpt-sovits. http:e//giiibb..co///ooeesse
Com/RVC-Boss/GPT. SoVITS, 2024.
[3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[4] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[5] Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint wrXiv:2502.11940 2025.
[6] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530-5540. PMLR, 2021.
[7] Zhiyuan Yu, Shixuan Zhai, and Ning Zhang. Antifake: Using adversarial audio prevent unauthorized speech synthesis. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 460-474, 2023.
[8] Chien-yu Huang, Yist Lin, Hung-yi Lee, and Lin-shan Lee. Defending your voice: Adversarial attack on voice conversion. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 552-559. IEEE, 2021.
[9] Zhisheng Zhang, Qianyi Yang, Derui Wang, Pengyang Huang, Yuxin Cao, Kai Ye, and Jie Hao. Mitigating unauthorized speech synthesis for voice protection. In Proceedings of the 1st ACM Workshop on Large Al Systems and Models with Privacy and Safety Analysis, 2024.
[10] Zhisheng Zhang, Derui Wang, Qianyi Yang, Pengyang Huang, Junhan Pu, Yuxin Cao, Kai Ye, Jie Hao, and Yixian Yang. Safespeech: Robust and universal voice protection against malicious speech synthesis. In 34th USENIX Security Symposium (USENIX Security 25), Seattle, WA, USA, 2025.
[11] Eberhard Zwicker and Hugo Fastl. Psychoacoustics: Facts and models, volume 22. Springer Science & Business Media, 2013.
[12] Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts
2: Towards human- text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36, 2024.
[13] Zhisheng Zhang and Pengyang Huang. Hiddenspeaker: Generate imperceptible unlearnable audios for speaker verification system. In IJCNN. IEEE, 2024.
[14] Jie Gao, Haiyun Li, Zhisheng Zhang, and Zhiyong Wu. Black-box adversarial defense against voice conversion using latent space perturbation. ICASSP. IEEE, 2025.
[15] Zirui Zhang, Wei Hao, Aroon Sankoh, William Lin, Emanuel Mendiola-Orie Junfeng Yang, and Chengzhi Mao. can hear you: Selective robust training for deepfake audio detection. In ICLR, 2025.
[16] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. International conference on machine learning, pages 28492-28518, PMLR, 2023.
[17] Sigurdur Sigurdsson, Kaare Brandt Petersen, and Tue Lehn-Schipler. Mel frequency cepstral coefficients: An evaluation of robustness of mp3 encoded music. ISMIR, pages 286-289, 2006.
11

---

[18] Guangke Chen, Yedi Zhang, Fu Song, Ting Wang. Xiaoning Du, and Yang Liu. A proactive and dual prevention mechanism against illegal song covers empowered by singing voice conversion. In Symposium on Network and Distributed System Security (NDSS), 2025.
[19] Zheng Fang, Tao Wang. Lingchen Zhao, Shenyi Zhang, Bowen Li, Yunjie Ge, Qi Li, Chao Shen, and Qian Wang. Zero-query adversarial attack on black-box automatic speech recog- nition systems. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 630-644, 2024.
[20] Yunjie Ge, Lingchen Zhao, Qian Wang. Yiheng Duan, and Minxin Du Advddos: Zero-query adversarial attacks against commercial speech recognition systems. IEEE Transactions on Information Forensics and Security, 18:3647-3661, 2023.
[21] Weifei Jin, Yuxin Cao, Junjie Su, Derui Wang, Yedi Zhang, Minhui Xue, Jie Hao, Jin Song Dong, and Yixian Yang. Whispering under the eaves: Protecting user privacy against commercial and Ilm-powered automatic speech recognition systems. In USENIX Security, 2025.
[22] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[23] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for supervide learning of speech representations. Advances neural information processing systems, 33:12449-12460. 2020.
[24] Jiangyi Deng, Fei Teng. Yanjiao Chen, Xiaofu Chen, Zhaohui Wang, and Wenyuan Xu. {V- Cloak]: Intelligibility-. naturalness-& (Timbre-Preservingg (Real-Time) voice anonymization. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5181-5198, 2023.
[25] Rui Duan, Zhe Qu, Shangqing Zhao, Leah Ding, Yao Liu, and Zhuo Lu. Perception-awan attack: Creating adversarial music via reverse-engineering human perception. Proceedings of the 2022 ACM SIGSAC conference computer and communications security, pages 905-919, 2022.
[26] Zhen Ye, Xinfa Zhu Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.08 2502.04222..... 04800 2025.
[27] Wei Deng, Siyi Zhou, Jingchen Shu, Jinchao Wang, and Lu Wang. Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system. arXiv preprint arXiv:2502.05512 2025.
[28] Hao-Han Guo, Kun Xie, Yi-Chen Wu, and Feng-Long Xie. Fireredtts- An upgraded streamable foundation text-to-speech system. arXiv preprint wrXiv:2503.20499 2025.
[29] Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-uts: An efficient llmebased text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710 2025.
[30] Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Tulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. Xits: massively multilingual zero-shot text-to-specch model. In Interspeech 2024, pages 4978-4982, 2024.
[31] Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis. arXiv preprint arXiv:2411.011555 2024.
[32] Nari Labs. Dia.
hawsaaarrraaaaaaaaaa 2025.
[33] Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03996 2023.
12

---

[34] Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung- Hsien Tsai, Zhen Xiao, Hemin Yang. Zirun Zhu, Min Tang, Xu Tan. et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 682-689 IEFE, 2024.
[35] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint wrXiv:24/0006888 2024.
[36] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):1505-1518. 2022.
[37] Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. Cam++: fast and efficient network for speaker verification using context-aware masking. In Interspeech 2023, pages 5301-5305, 2023.
[38] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-aaggennee transformer for speech recognition. In INTERSPEECI, 2020.
[39] Somshubra Majumdar, Jagadeesh Balam, Oleksii Hrinchuk, Vitaly Lavrukhin, Vahid Noroozi, and Boris Ginsburg, Citrinet: Closing the gap between non-autoregressive and autoregressive end-to-end models for automatic speech recognition. arXiv preprint arXiv:2104.0172), 2021.
[40] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint 1rXiv:1904.02882, 2019.
[41] John Kominek and Alan W Black. The emu arctic speech databases. In SSW, pages 223-224, 2004.
[42] Dong Wang and Xuewei Zhang. Thchs-30: free chinese speech corpus. arXiv preprint arXiv:1512.01882, 2015.
[43] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. In INTERSPEECH, 2020.
[44] Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Interspeech, 2020.
[45] Shehzeen Hussain, Paarth Neekhara, Shlomo Dubnov, Julian McAuley, and Farinaz Koushanfar. Waveguard: Understanding and mitigating audio adversarial examples. 30th USENIX security symposium (USENIX Security 21), pages 2273-2290, 2021.
[46] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[47] Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
[48] Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. istftne:: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6207-6211, IEFF, 2022.
[49] Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In Proceedings of the International Conference on Learning Representations, 2024.
13

---

[50] Sang-gil gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In ICLR. 2023.
[51] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint WrXiv:2412.15115, 2024.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 2023.
[53] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contribution of this paper has been described in the method and experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss some limitations originating from our experiments in Appendix A, egg., subjective biases when conducting human survey.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper contains no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail the design of the method in Section 3, with details including the hyperparameters used, the encoder selected, and the fine-tuning employed described in Section 4.1 and Appendix D to ensure that the experiments can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets we used are open-source, including LibriTTS [40], CMU ARC- TIC [41], and THCHS30 [42], and do not contain private data. Our source code link is available http://ggiibbb.comm/wxxyy222233/ -uuard.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ mublic/guider wttss//oooSuSsiiooonnn.o oonooooo)yy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wtcc/publccgggiiiiee gwwges/iie////// httpc////giiiee//ddeuuuuiiiiii wwweeeubbmsssiooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The training and test details have been provided in Section 4.1 and Appendix D, e.g., devices, data splits, hyperparameters training and perturbation generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have considered the possible bias of human experiments in the subjective experiments, so we introduce 95% confidence intervals to enhance the credibility of the conclusions, and the specific results and descriptions are in the Appendix G, Table 10 and Table 11.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We introduce the computing resources of all the experiments in Section 4.1 and the time of speech protection in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttpp:///euriiii wwt.rriis.ccppuulpee wwwpblic/EhhssSuideeee
Answer: [Yes]
Justification:
We have considered and followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our experiments do not involve private data, all experiments run locally, the only calls to commercial APIs we have made in the Appendix A do not affect the company in any way, and all deepfake audio will not be used for any other social activities.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We introduce a protective defensive framework to better protect our personal information, and do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited them appropriately in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The human perception a vital perspective of our work, and we introduce the details of participants, including the recruitment and filtering process, the demographic group (e.g., ages), and the compensation (volunteer).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: This paper has received approvals from the local Human Ethics Research, and detailed ethical considerations about the human study have been discussed in Appendix A.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not involve LLMs any important, original, or non- -ta-dtandard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: : http://scccc/ooeeeene//e/e/////e /LLM) for what should or should not be described.
21

---

A Discussions and Limitations
Ethical Considerations. To verify the subjective perception of 1522-VGuee in the protected and synthesized audio for human ears, we conduct human subjective testing experiments in Appendix G. These experiments have received approval from the local Human Ethics Research Committee. In the questionnaire, all recruited volunteers are anonymous and consented to their answers being used only for academic research. We do not collect any information beyond the content of the questionnaire, and we maintain strict confidentiality regarding the responses. All synthesized audio is uniformly discarded after the completion of the subjective survey to ensure no security risks through leakage. Additionally, the experiments described in Section 4.4 involve testing with commercial APIs. Before using the company's 's products, we have completed real-name authentication and documented the process. Our experimental testing is conducted internally and will not affect the company's 's operations or generate unintended usage impacts.
Broader Impacts. This paper primarily focuses on proposing proactive defense technique. Our intention is positive, aiming to protect individuals' voices from infringement. We will open-source the E2E- VGuard, grant users the right to use it, and sign relevant disclaimer clauses to ensure that their usage behaviors are not related to the designer and publisher of the -uurdd According to the licensing agreement, this will not affect the normal and positive usage of speech synthesis technology. Additionally, our testing using commercial APIs is conducted solely for local user testing and will not impact any company's products services. This paper will not result in any negative socictal impacts.
Subjective Bias. The conclusions of the subjective experiments are derived from human responses, which can be influenced by the answering circumstances, potentially leading to subjective bias. To reduce the bias in subjective experiments, we calculate the 95% confidence interval of the MOS values following [9] and recruit sufficient number of volunteers to enhance the reliability of the conclusions. We also implement certain filtering measures to eliminate low-quality responses. According to the conclusions of the subjective experiments, human subjective perception is generally aligned with objective evaluations, indicating that we have minimized the interference of subjective bias on the experimental results.
Limitations of the Target ASR System. In designing E2E-VGuard, we focus on protecting against a specific ASR system, aiming to cause errors in the text recognized by the target ASR. This breaks the alignment between text and pronunciation in the TTS model. The reason we do not pursue universal approach is that previous methods [19] based on universal attacks typically rely on clustering multiple ASRs, which consumes more time and computing resources. We aim to simplify our system as much as possible to improve the efficiency of the protection. Therefore, we conduct targeted protection using the targeted ASR and then transfer the results to other models. The experiments in Appendix E also demonstrate that our method remains effective when employing different targeted ASR systems.
Time Overhead and Acceleration Strategies. As the defender, we consider scenarios where users upload audio to the internet after E2E-VGuard protects audio samples. We test the average time for E2E-VGuard to protect audio on the LibriTTS dataset using a device equipped with one NVIDIA 4090 GPU with 24 GB of memory. On average, E2E- VGuard takes 97.982 seconds and 111.495 seconds to protect audio in untargeted and targeted settings, respectively. This protection time is on the same level as the baselines, which are 44.871 seconds for Attack [8] and 203.248 seconds for AntiFake [7]. The shorter optimization time may allow users to protect the target audio more quickly, The additional time overhead for targeted protection with E2E- VGuard, compared to untargeted protection, mainly comes from selecting a target speaker from the speaker database. Moreover, some accelcration methods can reduce the time overhead, e.g., data batching and multi-GPU parallelization.
Adaptive Adversaries. In the Section 4.6, we evaluate the robustness of E2E- GGuard from three aspects. The experiments demonstrate that E2E-VGuart can resist denoising techniques and remains effective against the adversarial example defense techniques proposed by Hussain et al. [45] in the audio domain. Additionally, we test various audio compression and filtering techniques and simulate the adversary's 's acquisition of audio using different speakers and microphones in real- world scenarios. The VGuard can still effectively protect the audio. The reason for E2E- VGuard's strong robustness lies in its application of various feature encoders to capture information from the latent space of the audio, enabling perturbations to be better embedded and thus resistant to being disrupted by denoising techniques and others.
22

---

Encoder Ensemble. TTS models, especially zero-shot ones, typically design speaker encoder to extract the timbre embedding of the reference audio. For specific TTS model, one can perform an adversarial attack on its speaker encoder to mislead the extraction of the target timbre, thereby protecting the original speaker's 's timbre. In practice, as defenders, we cannot know what type of TTS model the adversary might employ, so our designed proactive defense framework, E2E- VGuard, should possess transfcrability. Previous research has shown that clustering encoders from different models can achieve outstanding transferability. Based on this, we utilize an encoder ensemble approach and combine with a feature encoder to better extract and protect the reference speaker's timbre. The encoders we selected are highly representative and can cover mainstream generative architectures and backbones, such as VAE from VITS [6], diffusion model from StyleTTS2 [12]. and flow matching from CosyVoice [1].
Eliminating ASR System. In the scenario described in this paper, we have developed an end-to-end fine-tuning method based on an ASR system that does not require manually labeled text. We propose VGuad for timbre and pronunciation protection. However, assuming the adversary has enough human resources to obtain text through manual labeling, the effectiveness of E2E- E2E-VGuart VGuard remains a concern. We conduct experiments on the GSV model, using both untargeted and targeted audio protection. For the reference text in fine-tuning, we provide the correct text instead of the text obtained through ASR transcription. The experimental results show that the WER and SIM are 39.659%, 0.161 (T) and 73.784%, 0.278 (UT). This indicates that using clean text can still achieve effective protection at the timbre level. and pronunciation will continue to be affected. This demonstrates that the perturbations added by E2E- -Gurrd can interfere with the TTS model's 's learning of pronunciation information. Therefore, E2E- VGuard can still provide some protective effect even when manually labeled correct text is used for fine-tuning.
"Imperceptibility" consideration. In the scenario of our paper, the embedded perturbations should be "harmless" to the original audio, meaning that the original text content remains unaltered and the normal usability of the protected audio is unaffected. "Usability" represents whether the audio can be utilized normally in our daily lives. Rather than requiring perceptual indistinguishabilit between the protected and original audio. We have verified through both objective (Section 4.2) and subjective experiments (Appendix G) that the perturbations we generated do not cause huge disruptions to the original audio. Moreover, from the robustness perspective, assuming strong adver- saries can distinguish embedded perturbations, they can utilize adversarial techniques to improve the performance of the synthesized speech, causing privacy leakage. However, the robustness validated in Section 4.6 ensures that the adversary cannot effectively remove the embedded perturbation, thereby enhancing protection efficacy against speech synthesis. Therefore, even if the adversary perceives the perturbations, the robustness of 12E-VGuard ensures that privacy data is not completely leaked.
B
Algorithm
Algorithm provides a detailed illustration of each step that E2E-VGuard utilizes to protect audio. The imput data includes the audio to be protected x. long text Y, the target ASR system, and the optimization numbers mar_ pee____chhh The output data is the protected audio x'. Initially, the function init_perturbate
employed to randomly set the initial value of 8, ensuring it stays within e), Subsequently, perturbation optimization is performed for mar_epoch steps. In each step, C to C3 are calculated separately, and their weighted sum yields the objective function value C. For calculating C2 E2E--GGardd offers two methods, with differing target texts for each case. If untargeted protection is applicd, the target text is a segment randomly extracted from the given long text Y, matching the length of the original text y + ASR(z). For targeted protection, the target text corresponds to the transcription of the target audio zt. Using C, gradient information can be computed to optimize 6, thereby generating the protected audio z'.
Regarding Perturbation Generation. Following the classical Projected Gradient Descent (PGD) [46] algorithm in the adversarial attack domain, we compute the gradient of the loss func- tion for variable x to derive the perturbation: = (V. where sign(-) denotes the sign function and L represents the loss function. Subsequently, d is projected onto the e-ball constraint, ie., = Clamp( - si sign(V, (V,L), -6,)), e). Using 6, the protected audio is updated at each step as x' Clamp(x + 8, - 1, 1), as shown in Algorithm 1.
23

---

Algorithm 1: E2E-VGuard.
Inputs:
imput audio text diet Y. ASR system ASR(-). optimization numbers mar _eoccl.
Parameters:
perturbation boundary e, weight coefficients Eq. (1) and B. Output: protected audio x'.
1 +
2 x' < x #;;;
for - 1 to mar_ epoch do
C1 e F (ASR(z"), M:);
if Untargeted__ 86m then
Me [AAS((0)
7
C2
((E CSMM))))))
elsc
Ft select_ solect_target_ taggettsppsaspkerri
Y: ASR(z));
C2 Σ (S(E,(()) (C)
end
Cs - Σ max (0.pv- (z())) + - A=1
12
C C ta. C2 3
8 + Clamp(lssssiiaeapaeeeelae sign V.C,, -c,6);
x' t
end
Table 6: The detailed comparison of related works and E2E- VGuard
Method
Target
Type
Waveforns
Phrase
Trunsferability
Impercptibility
Robustness
Pronmncaation
AmackVC [8]
Vaice Protection of Mentificssion
AEs
X
Inference
X
Ex constraint
X
Antiaake
Encoder Ensemble
Frequncc Pesally
x
SafeSpeech [10]
Voice Protection of Synthesis Quality and Mentifictiion
UEs
tin-ttuning
Pivotal Objective
STOI and STFT loss
Eyp
12E.NGuard
E2E Zero-shot Encoder Ensemble Fine-tuning
AEs
with Feature Extractor
Psychoacoustic Model
(1) Waveform: whether the pertarbation added on original woveform. (2) Transferability: the appliad apgeonch to enhance perturiation' transferability.
(3) Robastness: whether the robustness has been validated.
C Comparison with Related Work
To provide a clearer comparison of the distinctions and advantages between E2E-VGuard and prior works, we present Table 6. This table compares aspects including algorithmic design objectives, types of data protection, whether perturbation is applied on the waveform, targeted speech synthesis types (phrases), transferability. techniques for enhancing imperceptibillity robustness verification. and consideration pronunciation-lewe protection.
-Guard effectively safeguards end-to-end speech synthesis systems, covering both zero-shot and miee-tuning-base scenarios. integrates an MFCC extractor based on an encoder ensemble to conceal speaker identity at the completed audio feature level. Specifically, E2E- VGuard demonstrates strong adaptability to LLM-based speech synthesis models. Moreover, it employs a psychoacoustic model to minimize human perception of injected noise. In summary, E2E-VGual achieves more effective, robust, and perceptually superior audio protection algorithm.
D Details of Experimental Information
In this section, we illustrate the detailed information of selected synthesizers and ASR systems.

---

Table 7: The detailed information and comparison of selected synthesizers.
Vocye (1]
Liasa-1B (26)
[26]
StyleTTS2 [12]
Index-TIS (27)
FireRedT -S- (28)
Type
line tuming
zerv shot
zcro-shot
zerursbot
zcro-shol
zetorshot
zerorshol
Industrial?
X
LLME
X
Backhome
VAE
GPT2 [3]
Transformer
Lluma3- 1B
Liama) 8D [4]
diffusion model
GPT2
semantic LM
LM
HiB-GAN (47)
Hiñ-GAN
HIRCIAN ISTFFNer (48)
ISTPTNer (48)
HinGIN
(49)
Big BigYGAN2 [50]
semantic decoder
2021
2024
2024 Pall (20)
2025
2025
2023
2025
2025
Fime-tune
Foll (100/ 200)
Fell (50 & 25)
LoRA
LoRA (2)
Full (50)
Sep-AAaio-TTS [S]
Spurk-TS (29)
XTTS-v2 (30)
FishSpeech [31]
Dia-L6B [32]
FS-TTS [35]
E2-TTS (34]
VALLE-X [33]
Type
zero-shot
2cra-thoe
22r0-shor
zero-shot
2ern-thor
zeno-thot
zero-shot
2ero-thot
Industrial?
X
I.MM2
X
X
matiching Trunsformer BigVGAN (50)
Step-Andia [5]
Quen2.550. (51]
GPT2 [3]
Llamaa [52]
Transfomer
[NT
Codee
Vocoder
HARGAN
docoder
HiRGAN
Firefly-GAN (31)
DAC decoder
Vacos (49)
Codec decoder
RT
2025
2025
2024
2024
2025
2024
2024
2023
Finc- tune-tune
Ty Type: whether model perfoem romo-shot TTS. ISFFT Shori-Time Fourier Transform L.M: whether I.M composent comployed. Fine-tune: fine-tuning type epochs.
D.1 Details of Synthesizers
To provide a more comprehensive comparison the models we adopted, we create Table 7, which outlines the following aspects: model type, industrial origin, whether the model is LLM-based, backbone architecture, vocoder used to convert latent variables into perceptible waveforms, release time (RT), and parameter settings employed in the fine-tuning process described in Section 4.2.
In terms of model types, select VITS, classic and backbone model requiring fine-tuning, along with other mainstream zero-shot models. Among the models, eight originate from the industry, and most (12 out of 16) are LLM-based. The LLMs utilized include Qwen2.5, Llama 3, Llama, Step-Audio, GPT2, Transformer based model, and a Neural Audio Codec. These language models assist the synthesizer better learning rhythm, prosody, and semantic features. The "Fine-tune column in the table indicates the implementation details used for validating the end-to-end fine-tuning scenario Section 4.2: "Full" denotes full- hull-parameter training, while "LoRA" represents using an auxiliary Low-Rank Adaptation (LoRA) adapter to learn input features. Notably, full-parameter fine-tuning of Llasa-8B demands substantial computational resources, whereas LoRA maintains low computational resource requirements while remaining effective. The second row of numbers indicates training epochs, where "100 200" in the table represents training 100 iterations for the single-speaker dataset and 200 for the multi-speaker datasets. "50 & 25" means training 50 epochs for GPT and 25 epochs for SoVITS in the GPT-SoVITS model.
D.2 Details of ASR Systems
In real-world scenarios, adversaries may employ different ASR systems recognize textual informa- tion from audio. We briefly introduce the ASR systems considered in the experiments Section 4.1, and in this section, present Table 8 to provide detailed comparisons of the ASR systems used in the experiments described in the Appendix E. This includes differences in the acoustic models, loss function types, and recognition performance measured by the WER metric.
From Table 8, we observe that the selected models incorporate two common backbone architectures. i.e., Transformer and CNN, and employ diverse loss function types. We specifically include the Whisper model [16]. a multilingual ASR system known for its high recognition accuracy. The largest variant, large-v3, achieves a WER of only 2.7%, making it the best-performing model among those selected. These seven ASR systems across four categories effectively represent current mainstream technologies in the field of automatic speech recognition.
25

---

Table The detailed information and comparison of selected ASR systems.
Models
Acoustic Model
Loss Type
WER in test-clean (%)
Transformer & CNN
Wav2vec2 [23]
CTC
3.4
5.0 (basc) 3.4 (small)
Whisper [16]
Transformer
Cross Entropy
2.9 (medium) 2.7 (large)
Conformer [38]
Transformer
CTC
3.7
CitriNet [39]
CNN
CTC
4.4
(1) CNN:
convolutional neural network. (2) WER test-clean; WER
value LibriSpeech test-clean dataset.
Table The protective effectiveness and imperceptibility targeting adaptive ASR models.
Effectiveness
Imperceptibllity
Model
VITS
GSV
WER(()
SIM(.)
WER(
SIM())
SNR(1)
PESQ(T)
Whisper-base [16]
99.598
0.088
19.362
Whisper-smal small [16]
93.996
0.164
103.518
0.138
19.000
2.077
Whisper mediumm [16]
0.173
Whispee-largo-v3 [16]
84.618
0.163
66.534
0.164
19.245
2.019
Conformer
105.145
0.126
81.753
0.180
12.835
L6388
CirnNNrr [39]
89.520
0.137
59.921
0.302
12.948
1.629
E Adaptive ASR Systems
In Section 4.2, Section 4.3, and Section 4.6, we have evaluated the protected effectiveness against the Wav2vec2 model. In the real world, adversaries can employ more types of ASR systems. Therefore, e2a-r should effective when utilizing different ASR models for text recognition. this section, we test the protective performance across six other ASR models.
Table 9 shows the protection effectiveness and imperceptibility of models. The results demonstrate that our method provides strong protection for various ASR models, with WER values consistently above 50%, indicating low intelligibility of synthesized audio, and SIM values below the threshold of 0.25 [43], indicating low similarity to the original audio. Whisper-large v3, known for superior text recognition accuracy, ease of utilization, and multilingual capabilities, is widely adopted in the industry. -Gurr also effectively protects against Whisper- large-v3. achieving average WER and SIM values of 75.576% and 0.163, respectively, across two models. Additionally, the imperceptibilitt of perturbations generated for different models outperforms strong baselines ie., AntiFake, POP+ESP, and SafeSpeech.
across different ASR
F Multilingual and Multi-Speaker Evaluation
In Section 4.2 and Appendix E, we evaluate the effectiveness and transferability the proposed E2E- VGuard on a single-speaker English dataset, LibriTTS. However, adversaries may encounter diverse speech samples, including multi-speaker and multilingual scenarios. address this, we further validate our method CMU ARCTIC, a multi-speaker dataset, and THCHS30, a multi-speaker Mandarin datasct targeting the Wav2vec2 model. Specifically, we fine-tune VITS and GSV models on CMU ARCTIC while using the GSV model for fine-tuning THCHS30.
Figure 4a illustrates the experimental results on the multi-speaker dataset ("-o" represents fine-tuning on the original dataset, and "-p" denotes fine-tuning on the protected dataset in the figure). We can find that both models are capable of synthesizing high-quality audio with corresponding speaker timbres clean samples. After fine-tuning the audio protected by E2E-VGuard, the WER and SIM averaged 103.021 and 144, respectively, indicating that E2E- VGuard can effectively prevent pronunciation and timbre information in multi-speaker end-to-end fine-tuning scenario. Figure 4b demonstrates the fine-tuning effect of GSV on the THCHS30 dataset, where the recognition ASR system uses multilingual recognition model, whisper-base the target for adversarial attacks.
26

---

150
150
0.8
WER
0.8
SIM
WER
0.6
WER
0.4
0.4 05
50-
50
0.2
0.0
G5V-
GSV-p
GSV-o
GSV-p THCHS30
CMU ARCTIC
(a) Multi-speakkr dataset.
(b) Mandarin dataset.
Figure 4: Test results of Multi-speaker and Mandarin datasets.
Table 11: Human perceptual evaluation the qual- ity and intelligibility of the synthesized speeches by different training samples.
Table 10: The subjective evaluation of the ground truth (GT) and protected dataset.
MOS(()
Intelligibilitt(()
MOS(1)
clean
4.842 + 0.123
100.000%
Aake
2.055 0.309
99.074%
GT
4.788 0.157
POP+ESP
0.851 0.364
89.814%
F2H- VGuard (UT) 3.522 0.218
F2F-VGuarrd (U/T) 2.615 0.331
50.925%
C2E-VGuard (T)
3.060 0.373
8.333%
G Human Study
In our previous experiments, we have validated the effectiveness of E2E-VGuard and its perception through objective evaluation metries. However, we also need to verify the subjective perception of audio by human ears, synthesized audio in the real world needs to interact with humans. Therefore, this experiment conducts a subjective evaluation to validate human perception and discrimination of synthesized audio, well as the perception of protected audio.
Recruitment Process. This subjective survey has been approved by the Human Ethics Research Committee at the first author's 's institution. We create the questionnaire through Credamo and recruit 36 volunteers participate in the survey, which is comparable similar studies, such as AntiFake of 24 participants. All volunteers are over 18 years old and possess good English skills. Their average response time is 200.194 seconds.
Filtering, We prohibit volunteers under 18 from participating in the questionnaire. Within the questionnaire, include two simple random arithmetic questions, and incorrect answers result in rejection. We also filter out all non-serious responses, e.g., the same answers across all questions, or excessively short response times.
Questionnaire Setup. We establish two sections for subjective testing for the synthesized audio and protected audio, with total of 22 samples.
Task 1: Study on Protected Speech. In the subjective test of protected audio, we select audio samples protected by E2u-VGuard to test naturalness and similarity to the original audio. In order to improve the confidence level of the subjective experiment and reduce the potential bias, we caleulate the MOS by taking into account the 95% confidence intervals, which can be found in the previous research [10]. Results in Table 10 show that the MOS 3.522+0.218 suggests that the embedded perturbations do not significantly reduce normal audio usability, and the distortions are acceptable to human cars when the MOS value surpasses 3.0 [7].
Task 2: Study on Synthetic Speech. In this part, we select 3 synthesized audio samples trained on original samples, maseline-protectee methods, and E2E- PrEE-VGuard-Wa protted- samples. Table 11 presents the experimental results for Task 2, revealing that compared to synthesis from the original audio. For synthetic audio evaluation, we utilize audio quality (MOS) and pronunciation intelligibility. Audio quality assesses noise levels and perceptual quality, calculated consistently with Task 1. The ESP method exhibits the worst synthesis quality because its perturbation addition process causes the most severe distortion to the original audio, rendering it unusable. Pronunciation intelligibility
27

---

involves presenting participants with both audio and its corresponding prompt text to judge whether the audio content matches the given text. The table results show that E2E-VGuard effectively disrupts the model's original pronunciation patterns: only 50.925% (UT) and 8.333% (T) of participants perceived correct pronunciation alignment with the text, while most participants identified mismatches, demonstrating significant improvement over previous baselines.
Through human study, we observe two key findings: (1) Human auditory perception aligns with objective metrics from prior experiments; (2) Experimental results confirm that E2E-VGuard's noise injection not only bypasses human auditory detection but also substantially reduces the probability of participants being deceived by deepfake audio.
28


Reviews:
[{"review_id": "NvG6vssxmN", "reviewer": "Reviewer_9Dhx", "summary": "The paper proposes a defense framework against voice cloning threats, especially focusing on modern LLM-based end-to-end speech synthesis systems. Existing defenses struggle with LLM integration and end-to-end workflows that use automatic speech recognition (ASR) for text transcription. The core idea is to be disrupting both timbre (through encoder ensemble and MFCC features) and pronunciation (via ASR-targeted adversarial examples), while using psychoacoustic models to keep perturbations imperceptible.\n\nExperiments on 16 TTS models (13 open-source, 3 commercial), 7 ASR systems, and Chinese/English datasets show E2E-VGuard significantly reduces speaker similarity (SIM) and increases word error rate (WER), outperforming baselines. It also demonstrates robustness against perturbation removal and real-world over-the-air attacks.", "questions": "1. How does E2E-VGuard perform on long audio sequences (e.g., >10 minutes)? Pronunciation disruption might weaken in longer contexts due to ASR cumulative errors.\n\n2. Can this framework be optimized for real-time deployment? For example, via lightweight encoders or parallel processing, especially for mobile applications.\n\n3. Have you tested E2E-VGuard in multi-speaker conversations? Defending against voice cloning in dialogues presents unique challenges (e.g., speaker diarization).", "limitations": "1. Computational efficiency: the time required to protect audio samples using E2E-VGuard is relatively high, which may limit its applicability in real-time scenarios. More work needs to optimize the computational efficiency of the framework.\n\n2. ASR system dependency: the effectiveness of E2E-VGuard is demonstrated against specific ASR systems, while the results are promising, the framework's performance may vary when applied to different ASR systems. A more comprehensive evaluation across a wider range of ASR systems would be beneficial.\n\n3. Language scope: this work is primarily evaluated on Chinese and English, effectiveness in low-resource languages remains untested.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. The problem addressed in the paper is highly relevant in the context of increasing security risks associated with advanced speech synthesis technology. The paper presents a novel approach to protecting speech synthesis systems by combining multiple techniques to disrupt both timbre and pronunciation. Combines encoder ensembles, adversarial ASR attacks, and psychoacoustics in voice protection for the first time. The use of psychoacoustic models to ensure imperceptibility of perturbations is particularly innovative.\n\n2. The paper provides a thorough evaluation of the proposed framework. The authors conduct extensive experiments across various models and datasets, demonstrating the effectiveness of E2E-VGuard in protecting against unauthorized speech synthesis.\n\n3. The paper is clearly explains the design and implementation of E2E-VGuard. The methodology is detailed, and the results are presented very clear.\n\nWeaknesses：\n\n1. The paper mentions that E2E-VGuard takes a significant amount of time  (e.g., ~100 seconds per audio) to protect audio samples, which could be a limitation in real-time applications.\n\n2. The effectiveness of E2E-VGuard is evaluated against specific ASR systems, and while the results are promising, the paper could benefit from a broader evaluation across a wider range of ASR systems to demonstrate more universal applicability.\n\n3. Claims of imperceptibility rely on SNR/PESQ metrics, lacks perceptual tests beyond MOS (e.g., ABX listening tests).", "was_revised": true, "final_justification": "All my concerns are properly addressed.", "rebuttal": "We sincerely appreciate your constructive and helpful comments. We will do our best to address your concerns and revise the paper accordingly.\n\n**W1/Q2/L1: E2E-VGuard requires a relatively long time to protect audio (~100s per audio), which limits real-time protection. Can this framework be optimized for real-time deployment?**\n\nR: Thanks for your suggestion regarding time optimization. We will elaborate on the time requirements of our application scenarios discussed in the paper and subsequently optimize the computational time of our framework based on your valuable feedback.\n\nFirst, our paper focuses on the **offline scenario** where users protect audio before uploading to social media (Lines 272–273). This offline scenario holds minimal real-time requirements and high runtime tolerance [1*], allowing us to prioritize more effective protection against these LLM-based and E2E threats. Compared to other baselines, our computational time is **comparable or even superior**, *e.g.*, AntiFake [7] requires 203.248 seconds and SongBsAb [1\\*] needs 287 seconds per audio. Moreover, E2E-VGuard has **low hardware demands**, requiring only 4–5 GB of memory and enabling deployment on **consumer laptops** (e.g., those equipped with RTX 3060 or 4060 GPUs). \n\nFurthermore, with additional computational resources, the time overhead can be further reduced through techniques like parallel and multi-GPU processing. In the paper, we design E2E-VGuard as a single-audio processing software for low-resource GPU deployment as AntiFake [7]. To enhance efficiency, we optimize it to accept batch inputs. Building on this, we optimize E2E-VGuard for multi-GPU parallelization, significantly accelerating audio protection. We experiment to demonstrate this. The initial framework takes 104.62 seconds to process a 5-second audio sample on a single 4090 GPU, and we optimize it from two aspects:\n\n- [Batch Process] With 8 samples per batch on one GPU, the runtime is reduced to 59.88 seconds per audio. \n- [Multi-GPU Parallel Process] Using two RTX 4090 GPUs (processing 4 samples per GPU in parallel), the runtime drops to **30.17 seconds per audio**.\n\nTherefore, with greater computational resources, the time overhead might potentially and ideally be reduced to near-linear. Moreover, for a single audio, we can clip it into segments for batch and parallel processing to significantly reduce the time overhead, a benefit that equally holds for multiple audio files.\n\nFor mobile deployment, we can construct a client-server architecture [10]: E2E-VGuard can operate on a backend as the trustworthy server with enough computational resources. Mobile devices transmit audio to the trustworthy server and receive protected audio within an acceptable response time, enabling practical mobile applications.\n\n**W2/L2: The paper could benefit from a broader evaluation across a wider range of ASR systems to demonstrate more universal applicability.**\n\nR: First, in Appendix E, we validate the effectiveness of commonly used ASR systems, including six models: Whisper with four sizes, Citrinet, and Conformer. These ASR systems exhibit diverse structures and are representative, as shown in Table 7. The results in Table 8 demonstrate the effectiveness and scalability of E2E-VGuard across different ASR models.\n\nFurthermore, in Section 4.4, we conduct experiments on commercial black-box models, for which the ASR systems are entirely unknown. Our architecture is not specifically optimized for these systems, but can achieve a certain level of protection, relying on the transferability from the specific white-box ASR system.\n\nFinally, in the \"Eliminating ASR System\" section of Appendix A, we discuss that our method remains highly effective even when the transcribed reference text is correct. Therefore, the effectiveness might not be largely reduced across different ASR systems. This is because our optimization from the audio content perspective partially influences the features of the original audio at the feature level (the original audible textual content remains unaltered).\n\n**W3: Claims of imperceptibility rely on SNR/PESQ metrics, lacks perceptual tests beyond MOS (e.g., ABX listening tests).**\n\nR: The MOS aligns with human auditory perception, thereby reflecting the usability of the protected audio.\n\nWe employ both objective and subjective metrics to evaluate the perturbative perception. SNR is selected as it can effectively reflect the magnitude of embedded perturbations, where it serves as a core metric for noise perception [2*]. PESQ represents the audio perception objectively. The MOS values reflect human auditory perception, thereby validating the usability of the protected audio. The subjective experiments in Appendix G confirm that the embedded noise remains acceptable because the MOS surpasses 3 [7].\n\nABX listening tests primarily assess the distinction between protected and original audio, while we aim to ensure that embedded perturbations do not compromise the original audio's usability in daily life. The MOS values confirm that our protected audio maintains usability without impairing normal utility. Furthermore, even if adversaries can distinguish protected audio from unprotected audio, the experiments in Section 4.6 verify the robustness of the E2E-VGuard to ensure adversaries cannot effectively denoise or synthesize the audio.\n\n**Q1: How does E2E-VGuard perform on long audio sequences (e.g., >10 minutes)? Pronunciation disruption might weaken in longer contexts due to ASR cumulative errors.**\n\nR: For the long audio, we can segment the audio into shorter clips (approximately 3-8 seconds). Since TTS models, whether fine-tuning or zero-shot, cannot effectively process long audio inputs, we leverage the built-in audio segmentation tool provided by the GSV model [2]. Our framework has already demonstrated protection capabilities for extended audio and large-scale datasets, such as the CMU ARCTIC corpus (total duration: 1 hour 20 minutes) and the THCHS30 dataset with 1 hour duration. Segmenting long audio into shorter clips could also mitigate ASR cumulative errors, as illustrated by our experimental results with shorter inputs, *e.g.*, Table 1 and Table 2. Regarding time overhead, the approach in response to W1 employs batch and parallel processing, greatly reducing the computational burden associated with long audio processing. \n\n**Q3: Have you tested E2E-VGuard in multi-speaker conversations? Defending against voice cloning in dialogues presents unique challenges (e.g., speaker diarization).**\n\nR: E2E-VGuard can also achieve protection for multi-speaker conversations. First, we select an English dialogue involving two speakers and utilize a speaker diarization model [3\\*] to extract separate audio segments for each speaker, which are then employed for voice cloning in an unprotected scenario. We obtain a SIM score of 0.703 using the CosyVoice for TTS. Protection for this dialogue can be conducted from two perspectives: **holistic protection** and **separated protection**. Holistic protection refers to protecting the entire original audio directly without any segmentation, while separated protection involves independently protecting each speaker’s segment as identified by the speaker diarization model. After voice cloning, the SIM score under holistic protection drops to **0.188**, and under separated protection, it drops to **0.185**, both indicating effective protection.\n\nThis performance stems from two main reasons. Firstly, as demonstrated in Appendix F, our method is effective for multi-speaker scenarios. As long as the speaker diarization model can accurately separate the speakers, our approach achieves effective protection. Therefore, the primary challenge in protecting multi-speaker conversations lies in the accuracy of the speaker diarization task itself. Secondly, based on prior research [22], optimizing both content and identity can effectively conceal speaker information, enabling anonymization across different speakers. From this perspective, our method provides strong protection for diverse speakers, allowing the identities of participants in a dialogue to be hidden. Therefore, E2E-VGuard can achieve holistic protection without identity segmentations.\n\n**L3: Language scope: this work is primarily evaluated on Chinese and English, effectiveness in low-resource languages remains untested.**\n\nR: We acknowledge this suggestion for the language expansion. E2E-VGuard can also achieve protection in Hindi and Polish. If the ASR system supports a particular language, E2E-VGuard can possess the ability to achieve protection because we conduct optimization on ASR systems. We choose Chinese and English primarily due to their widespread usage and because many TTS models support only these two languages, *e.g.*, Index-TTS and Spark-TTS.\n\nWe conduct language-level evaluations on Hindi and Polish. For the TTS model, we select XTTS-v2 (as it supports these languages), and for the ASR model, we employ Whisper for text recognition. In the case of Hindi, the SIM score for clean sample cloning is 0.593, which drops to only **0.284** after protection, while the WER increases by **88.462%**. For Polish, the SIM score decreases from 0.567 to **0.192**. This evaluation confirms the scalability of E2E-VGuard across different languages.\n\n[1*] Chen G, Zhang Y, et al. Songbsab: A dual prevention approach against singing voice conversion based illegal song covers. NDSS 2025.\n\n[2*] Fang Z, Wang T, Zhao L, et al. Zero-query adversarial attack on black-box automatic speech recognition systems. CCS 2024.\n\n[3*] https://huggingface.co/pyannote/speaker-diarization-3.1\n\nNote: [N*] represents the new reference and [N] denotes the reference in the paper."}, {"review_id": "ScmneYeE2H", "reviewer": "Reviewer_wFwk", "summary": "This paper presents E2E-VGuard, a proactive audio protection framework against unauthorized voice cloning attacks, especially targeting production-level, LLM-based, and end-to-end (E2E) TTS pipelines. The approach combines: 1. an encoder ensemble for timbre disruption, 2. adversarial attacks on ASR for pronunciation confusion, and 3. psychoacoustic masking for imperceptible perturbation. The method is validated across various open-source and commercial TTS APIs, under both fine-tuning and zero-shot settings, with extensive robustness analyses.", "questions": "- Shouldn’t it be necessary to assess whether the perturbation added to the protected audio is truly imperceptible and whether the original content of the audio is preserved? What is the WER (Word Error Rate) of the protected audios?\n- Recently, many zero-shot TTS models that utilize in-context learning (ICL) rather than extracting speaker embeddings have been introduced, such as VALL-E[1], F5-TTS, E2-TTS, and VoiceBox. Have you evaluated your method on these LLM and ICL-based zero-shot TTS models? If so, what are the results?\n\n[1] Wang, Chengyi, et al. \"Neural codec language models are zero-shot text to speech synthesizers.\" arXiv preprint arXiv:2301.02111 (2023).\n[2] Chen, Yushen, et al. \"F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.\" arXiv preprint arXiv:2410.06885 (2024).\n[3] Eskimez, Sefik Emre, et al. \"E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts.\" 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024.\n[4] Le, Matthew, et al. \"Voicebox: Text-guided multilingual universal speech generation at scale.\" Advances in neural information processing systems 36 (2023): 14005-14034.", "limitations": "While E2E-VGuard presents promising results against a range of current TTS models and demonstrates robust protection under both fine-tuning and zero-shot scenarios, several important limitations remain. First, the proposed defense assumes that attackers will use ASR systems to generate transcripts; however, a motivated adversary could simply perform manual transcription of the protected audio, effectively bypassing the pronunciation perturbation and weakening the defense. Second, the analysis of the protected audio itself is insufficient. In particular, the paper does not report the WER (Word Error Rate) or perceptual metrics of the protected audio, making it difficult to assess how much of the original content is preserved and to what extent the perturbation is truly imperceptible to humans or ASR systems. Thorough evaluation and reporting of these metrics would significantly strengthen the claims regarding the imperceptibility and practical usability of the approach. Finally, the current evaluation mainly focuses on speaker embedding-based systems, while recent advances in zero-shot TTS—including models like VALL-E [1], F5-TTS [2], E2-TTS [3], and VoiceBox [4]—increasingly utilize large language models and in-context learning (ICL) instead of explicit speaker embeddings. It remains unclear whether E2E-VGuard is effective against such ICL-based systems, as their mechanisms for speaker adaptation may not rely on features targeted by the proposed method. Explicit experiments and analyses on these next-generation models are necessary to confirm the broader applicability and long-term effectiveness of the defense.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\nThe paper is well-written with clear motivation and sufficient background. The problem formulation and the E2E scenario definition are easy to follow. The paper provides empirical evaluation, including both open and commercial TTS models and a variety of attacks and augmentations. This increases  its practical relevance and transferability. The proposed method, E2E-VGuard, outperforms previous proposed methods and achieves state-of-the art performance on the task. Plus, the experiments on the real-world scenario and commercial API adds credibility to robustness.\n\n### Weaknesses\nThe defense assumes attackers will use ASR. However, motivated attackers could manually transcribe audio, circumventing the pronunciation perturbation. Plus, it is required to deeply analysis the protected audio. For instance, the WER of the protected audio is not provided, which makes it difficult to judge to what extent the perturbation is truly imperceptible to humans or ASR systems. A thorough analysis and reporting of the WER on the protected audio itself would significantly strengthen the claim regarding the imperceptibility of the perturbation, as it directly reflects whether the original content is preserved for benign users.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's insightful and valuable comments. We will endeavor to address your concerns as much as possible.\n\n**W1: Motivated attackers could manually transcribe audio, circumventing the pronunciation perturbation.**\n\nR: Thanks for this suggestion. Providing correct text during speech synthesis still cannot circumvent E2E-VGuard, as we utilize the ASR module combined with MFCC features to conceal audio information at the phonetic level, and audio's content **features** have been altered (though the original audio's audible content remains unchanged).\n\n Regarding the correctness of the reference text, we have considered this point in our paper. On one hand, in the E2E speech synthesis scenarios, the adversary may not necessarily be able to manually transcribe the text, such as when utilizing black-box commercial API models, where the reference text depends on the unknown internal ASR system of the black-box model. On the other hand, we have also validated the scenario where the adversary can manually transcribe the text, *i.e.*, when the provided reference text is accurate. We test this case in the \"Eliminating ASR System\" section of Appendix A (lines 600–611). Specifically, on the LibriTTS dataset, we evaluate the voice cloning performance of the GSV model utilizing correct reference text, achieving WER and SIM scores of **73.784%** and **0.278 (UT)**, as well as **39.659%** and **0.161 (T)**, respectively. The WER remains still higher compared to unprotected speech samples.\n\n**W2: It is required to deeply analyze the protected audio. For instance, the WER of the protected audio is not provided, which makes it difficult to judge to what extent the perturbation is truly imperceptible to humans or ASR systems.**\n\nR: We appreciate the insightful comment regarding the perceptual quality of our protected audio. On the LibriTTS dataset, the protected audio achieves a WER value of **13.862%**, while AntiFake-protected samples yield a WER value of 31.408%. This demonstrates that our attack on ASR systems does not greatly compromise the content integrity of the original audio.\n\nRegarding the perturbation perception, we analyze the effectiveness of our framework compared to prior methods through both objective and subjective evaluations. For objective metrics, Section 4.2 quantifies minimal audio degradation through SNR (>20 dB in the targeted mode) and PESQ metrics. For subjective evaluation, Appendix G presents human listening perception where our method achieves a MOS value exceeding 3.0, indicating that most listeners regard the embedded perturbation as imperceptible or acceptable [7].\n\nThe strength of our perception optimization stems from two key aspects: (1) The perception optimization algorithm based on the psychoacoustic model in Section 3.5 can hide perturbations to areas inaudible to human ears. (2) We embed perturbations on waveforms instead of the audio embeddings in the latent space [1\\*]. By linearly adding perturbations directly onto the waveform, E2E-VGuard introduces much less audio distortion compared to embedding-based protection methods like AttackVC (which produces negative SNR values) [8] and a black-box optimization [1\\*] based on the autoencoder. This fundamental design choice inherently preserves better audio fidelity.\n\n**Q1: Shouldn’t it be necessary to assess whether the perturbation added to the protected audio is truly imperceptible and whether the original content of the audio is preserved? What is the WER (Word Error Rate) of the protected audios?**\n\nR: We should evaluate whether the embedded perturbations affect the normal usability of the audio and whether the original content is preserved. The WER of the protected audio is 13.862%, and we can conduct a further in-depth analysis of this issue.\n\nIn the scenario of our paper, the embedded perturbations should be \"harmless\" to the original audio, meaning that the original text content remains unaltered and the normal usability of the protected audio is unaffected. \"Usability\" represents whether the audio can be utilized normally in our daily lives. Rather than requiring perceptual indistinguishability between the protected and the original audio. We have verified through both objective (Section 4.2) and subjective (Appendix G) experiments that the perturbations we generated do not cause large changes to the original audio. \n\nMoreover, from the robustness perspective, assuming strong adversaries can distinguish embedded perturbations, they can utilize adversarial techniques to improve the performance of the synthesized speech, causing privacy leakage. However, the robustness validated in Section 4.6 ensures that the adversary cannot effectively remove the embedded perturbation, thereby enhancing protection efficacy against speech synthesis. Therefore, even if the adversary perceives the perturbations, the robustness of E2E-VGuard ensures that privacy data is not completely leaked. \n\nOn the other hand, calculating the WER for the original audio relies on the ASR system. When considering the E2E scenario, E2E-VGuard performs an adversarial attack on the ASR system to protect audio, which has potential transferability to other ASR models, thereby affecting recognition results. The WER calculation model is not the same as perturbation generation, and our WER is only ~10%, which appears relatively low. Moreover, the audible content does not undergo great changes. In our response to W2, we also explain the reasons for the superior perceptual quality of our method.\n\nWe will present these further considerations in the revised version.\n\n**Q3: Effectiveness Evaluation of TTS models based on in-context learning (ICL), such as VALL-X, F5-TTS, E2-TTS, and VoiceBox.**\n\nR: We sincerely appreciate this valuable suggestion regarding model scope expansion. In this experiment, we evaluate the effectiveness across three ICL-based TTS models: VALLE-X, F5-TTS, and E2-TTS (note that VoiceBox is excluded due to the lack of an official open-source code). The results are shown in Table R-1. \n\nTable R-1: Experimental results on ICL-based models.\n\n| Method          | VALLE-X     |           | F5-TTS     |           | E2-TTS     |           |\n| --------------- | ----------- | --------- | ---------- | --------- | ---------- | --------- |\n|                 | WER         | SIM       | WER        | SIM       | WER        | SIM       |\n| clean           | 14.450      | 0.519     | 4.268      | 0.676     | 5.401      | 0.678     |\n| AntiFake        | 96.469      | 0.249     | 4.303      | 0.282     | 4.004      | 0.269     |\n| E2E-VGuard (UT) | **129.483** | **0.175** | 10.776     | **0.053** | 7.064      | **0.138** |\n| E2E-VGuard (T)  | 88.707      | 0.176     | **70.034** | 0.319     | **84.913** | 0.372     |\n\nThis effectiveness stems from our speaker encoder ensemble technique, which successfully hides or modifies the timbre information of the original speaker. Consequently, the timbre prevention of the proposed E2E-VGuard does not rely on the specific speaker encoder. As evidenced in Table 1, we applied no specialized processing, *e.g.*, extracting their encoders for model-specific attacks, to Llasa-1/8B and TTS models in Table 2. Instead, the high transferability of our method enables effective performance across diverse models.\n\nThese additional experiments for model scope expansion will be added in the revised version based on your insightful comments.\n\n**Limitations**\n\nR: Thanks for these proposed suggestions. Regarding the impact of ASR systems in E2E scenarios, perceptual analyses, content consistency metrics, and ICL-based TTS models, we have provided detailed explanations in the preceding sections. We hope these clarifications can address your concerns.\n\n[1*] Gao J, Li H, Zhang Z, et al. Black-box adversarial defense against voice conversion using latent space perturbation. ICASSP 2025.\n\nNote: [N*] represents the new reference and [N] denotes the reference in the paper."}, {"review_id": "LJj1cKJqaK", "reviewer": "Reviewer_jyho", "summary": "In this paper, the authors propose the E2E-VGuard method, which aims to prevent a user’s speech from being used as a prompt or training data for unknown speech synthesis systems by perturbing both timbre and textual information, while keeping perceptual differences minimal. They design various loss functions to optimize the perturbation process and conduct comprehensive experiments to evaluate the effectiveness of their method, including real-world scenarios. This topic is gaining significant attention as concerns over user data privacy grow, and as speech synthesis systems become nearly indistinguishable from real human speech. Therefore, I strongly believe that this work makes a valuable contribution to the safety and security of modern speech synthesis systems and can significantly benefit the research community.", "questions": "- In Table 2, the WER results for each zero-shot E2E TTS system vary significantly—some systems achieve WERs below 10, while others exceed 50. What is the reason behind this discrepancy? Is it due to the differing characteristics of the zero-shot E2E TTS systems? If so, what characteristics might explain this variation, in the authors’ opinion? This variation also appears to relate to the robustness of the proposed system and warrants further discussion.\n- Since the method adopts several numbers of different objective functions, I wonder how did authors adjust the loss weight effectively. In the paper, the $\\alpha$ and $\\beta$ are set as 500 and $5 * 10^{-3}$, respectively, which are drastically different between.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- First of all, the topic of this paper is highly important and is currently receiving significant attention. Therefore, I believe this work has the potential to make a meaningful contribution and impact on society.\n- The authors effectively harmonize several objective functions necessary to perturb the input speech signal.\n- Additionally, the authors conduct a very comprehensive and well-designed evaluation. They effectively simulate real-world scenarios by incorporating several state-of-the-art TTS systems, including commercial APIs. They also test the robustness of the perturbed signal under conditions that could realistically occur in practice. Finally, they simulate the case where speech is physically transmitted and recorded as a different signal, and evaluate robustness in that context as well. I believe this comprehensive and practical evaluation can serve as a valuable reference for future research on similar topics.\n### Weaknesses\n- The biggest weakness is the lack of clarity in the proposed method. It is unclear what the perturbation method entails. The authors mention optimizing various objective functions and perturbing the original waveform’s content and timbre information. However, they do not specify what kind of network is being optimized. Are they optimizing the perturbation signal $\\delta$? If so, how is $\\delta$ generated?\n- Following this, the authors optimize a network for only 500 iterations, which is significantly fewer than what is typical for neural network training. I could not find the batch size used in the training process, but considering the size of the training dataset, 500 iterations may not even be enough to cover the entire dataset for a single epoch. Is this because the network is optimized for a specific speech signal each time? In other words, does the perturbation network need to be optimized individually for every single target waveform?\n- In the evaluation, the authors present results for untargeted (UT) and targeted (T) E2E-VGuard systems. I understand that UT and T refer to perturbations in timbre; however, I was unable to clearly understand the distinction in text-wise perturbation between UT and T. Based on my understanding, the authors chose to attack the ASR system with a specific target text, as indicated in line 184 (“Therefore, we choose targeted attacks against ASR systems”). This suggests that a specific target text was set, and the original signal was perturbed to match that target content. However, when I listened to the speech samples on the demo page, both the T and UT attacks seemed to retain the same text content as the original signal. Did I understand this correctly? I believe the authors should provide a more detailed explanation of their T and UT attack settings in the experimental section.\n- In Section 4.6 (Perturbation Removal Experiment), the authors adopt a spectral gating denoising technique. After denoising, the performance appears to drop significantly—for example, the WER decreases from 95.7% to 51.0%. This suggests that with a more effective denoising method, the effectiveness of the proposed approach could deteriorate even further. Therefore, the authors should evaluate the impact of denoising on random noise signals as well, or test with various denoising techniques, to demonstrate that the proposed system is robust against different types of denoising methods.", "was_revised": true, "final_justification": "The authors’ rebuttal addresses the concerns I raised. Considering the quality of the paper and the potential impact this work could have on speech privacy protection research, I would like to increase my score from 3 to 4.", "rebuttal": "We sincerely appreciate your valuable feedback. We are doing our best to address your concerns.\n\n**W1: They do not specify what kind of network is being optimized. Are they optimizing the perturbation signal $\\delta$? If so, how is $\\delta$ generated?**\n\nR: First and foremost, it is important to clarify that **E2E-VGuard does not involve the design or training of a perturbation network**. Perturbation generation relies on the derivation of the loss function, which we will illustrate in detail.\n\nIn adversarial attacks, a critical step lies in designing an effective optimization objective, based on which perturbations can be generated. Specifically, following the classical Projected Gradient Descent (PGD) [1*] algorithm in the adversarial attack domain, we compute the gradient of the loss function for variable $x$ to derive the perturbation: **$\\delta = -\\texttt{sign}(\\nabla_x L)$**, where $\\texttt{sign}(\\cdot)$ denotes the sign function and $L$ represents the loss function. Subsequently, $\\delta$ is projected onto the $\\epsilon$-ball constraint, *i.e.*, $\\delta = \\texttt{Clamp}(-\\texttt{sign}(\\nabla_x L), -\\epsilon, \\epsilon)$. Using $\\delta$, the protected audio is updated at each step as $x' = \\texttt{Clamp}(x + \\delta, -1, 1)$, as shown in Lines 13–14 of Algorithm 1. \n\nReferring to previous methodology descriptions [7-10], our E2E-VGuard design primarily focuses on defining the optimization objective, with no network training involved in this process. The encoder and ASR models utilized in the loss function are introduced in Section 4.1. These components utilize pre-trained checkpoints to obtain speaker embeddings and content features, respectively. Moreover, these models' gradients are disabled to ensure that the gradient information of $ L$ depends solely on the optimization variable $x$, enabling more effective optimization.\n\nWe will illustrate it more clearly in the revised version.\n\n**W2: 500 epochs are too few for training the network and no batch sizes are given. Does the perturbation network need to be optimized individually for every single target waveform?**\n\nR: Similar to W1, E2E-VGuard is not involved in the design and training of the perturbation network.\n\nIn our paper, the 500 iterations refer to optimization steps in the PGD algorithm, not epochs for training a neural network. These 500 iterations allow the loss value to converge effectively.  This number aligns with related literature, *e.g.*, SafeSpeech employs 200 iterations [10] while AntiFake utilizes 1000 [7]. Since we do not perform perturbation network training but instead optimize each speech sample individually, the concept of batch size is not introduced. Our framework is implemented as a software solution [7], which takes a single audio input and outputs the protected audio. Therefore, for each sample to be protected, we optimize the optimization objective to generate the corresponding perturbation. Parallel and batch processing can also significantly reduce the time overhead of the E2E-VGuard.\n\n**W3: Distinction between untargeted (UT) and targeted (T) attacks in text-wise perturbation.**\n\nR: Targeted and untargeted attacks represent two common methods in adversarial attacks [1\\*]. Both timbre and pronunciation can be conducted in either targeted or untargeted attacks. The distinction between untargeted and targeted timbre protection lies in whether a specific target speech is selected as the optimization objective (in Section 3.3). For attacks against ASR systems, targeted attacks are generally preferred, as untargeted attacks (*i.e.*, without a designated target text) produce random output text, which may raise adversaries' detection (lines 181–184). Therefore, we consider targeted attacks on ASR systems. It is essential to select an appropriate target text (lines 184–187). The choice of target text is also correlated with the timbre protection mode.\n\nFor targeted timbre protection (T), where the target speech is selected, we aim to align the protected audio with the target speech in feature space, including both timbre and textual content. Therefore, we set the ASR attack's target text to match the target speech's text, thereby enhancing the ASR attack effect and pronunciation protection (lines 187–191). For untargeted timbre protection (UT), we randomly select a text of the same length as the original speech for the ASR attack (lines 191–193). For example:\n\n> Original Text: His voice was changed as he spoke next.\n>\n> Target Text (T): Please open the door. (Text from the target speech)\n>\n> Target Text (UT): Her voice shifted as she began to talk. (Same-length text)\n\nOur pronunciation prevention operates at the feature level and ensures the original audio's usability without large distortion. In the demo audio:\n\n- For protected audio, both untargeted and targeted protection retain the original textual content and speaker identity as much as possible auditorily.\n- For synthesized audio, the targeted (T) protection effectively disrupts the speech pronunciation, generating incoherent output to achieve protection.\n\n**W4: The authors should evaluate the impact of denoising on random noise signals as well, or test with various denoising techniques.**\n\nR: Thanks for the valuable suggestion. We conduct new tests with stronger denoising techniques. We utilize an industrial-level DNN-based denoising model [2*], which can remove nearly all noise audible to human ears. The test results show that after denoising (which removes the audible perturbation, hence *the comparison against clean audio*), the protection is still effective (GSV: WER 23.1%, SIM 0.243; VITS: WER 34.1%, SIM 0.261).\n\nThe notably low SIM values persist even after denoising because while such techniques eliminate audible noise, they concurrently cause partial loss of critical original audio information, *e.g.*, timbre and phoneme features [10]. Furthermore, under the more advanced diffusion-based denoising method [3*], the SIM values remain low at 0.233 for GSV and 0.277 for VITS, demonstrating E2E-VGuard's robustness against advanced denoising techniques.\n\n**Q1: In Table 2, the WER results for each zero-shot E2E TTS system vary significantly. What is the reason behind this discrepancy?**\n\nR: This discrepancy arises from inherent features of TTS models, as our evaluation encompasses a diverse range of distinct TTS models, *i.e.*, E2E and non-E2E scenarios. On one hand, we aim to evaluate speech synthesis in E2E scenarios (requiring reference text input). On the other hand, we also seek to evaluate emerging LLM-based TTS models. For certain TTS models like XTTS-v2, which operates without reference text input during zero-shot speech synthesis, the reference audio is solely used to extract speaker embeddings, while pronunciation information is entirely produced by the model's pre-trained knowledge and remains unaffected by errors in reference text. However, most models achieve WER values superior to baselines in Table 2.\n\nIn E2E scenarios, our approach delivers substantially stronger voice protection than baselines, as evidenced by fine-tuning-based E2E TTS models in Table 1 and zero-shot-based E2E TTS models, *e.g.*, Step-Audio-TTS, FireRedTTS-1S, partially presented in Table 2. \n\nWe have included non-E2E scenarios in our evaluation to comprehensively assess both production LLM-based TTS models, *e.g.*, XTTS-v2, while demonstrating the effectiveness in preserving timbre protection across diverse architectures.\n\n**Q2: Since the method adopts several different objective functions, I wonder how did authors adjust the loss weight effectively.**\n\nR: We adjust the loss weight based on initial magnitude balance and experimental assessment. Eq. (1) incorporates $ L_{asr}$, $ L_{fea}$, and $ L_{psy}$, whose initial magnitudes during optimization are $\\sim 10^3$, $\\sim 5$, and $\\sim 10^5$, respectively. The parameter $\\alpha$ is set to 500 to amplify timbre protection, thereby aligning its optimization scale with that of $ L_{asr}$. Moreover, $\\beta$ serves to balance the perturbation perception and protection strength, and its selection is validated through experimental results. Specifically, we conduct tests using the CosyVoice model for voice cloning on small samples, selecting an appropriate $\\beta$ value based on the balance between these two factors, as shown in Table R-1. It can be observed that when $\\beta$ is larger ($5 \\times 10^{-2}$), although SNR values are high, there is minimal protection for timbre and pronunciation, and a lower ratio of the perturbation could also diminish the robustness [10]. Conversely, when $\\beta$ is smaller ($5 \\times 10^{-4}$), protection strength does not improve obviously, and the SNR drops to 13.982, resulting in poor perceptual quality. Therefore, we select a more appropriate $\\beta$ value of $5 \\times 10^{-3}$, which achieves a better balance among protection effectiveness, perceptual quality, and robustness.\n\nTable R-1: Impact of the hyperparameter.\n\n| $\\beta$            | WER    | SIM   | SNR    |\n| ------------------ | ------ | ----- | ------ |\n| clean              | 7.407  | 0.905 | -      |\n| $5 \\times 10^{-4}$ | 14.815 | 0.069 | 13.982 |\n| $5 \\times 10^{-3}$ | 18.519 | 0.074 | 18.182 |\n| $5 \\times 10^{-2}$ | 7.407  | 0.241 | 24.096 |\n\n**Paper Formatting Suggestion**\n\nWe are grateful for your suggestions on the details of our paper, and we will follow the comments to enhance the presentation.\n\nThese additional experiments and clarifications will be included in the revised version.\n\n[1*] Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks. ICLR 2018.\n\n[2*] https://github.com/facebookresearch/denoiser.\n\n[3*] Wu S, et al. Defending against adversarial audio via diffusion model. ICLR 2023.\n\nNote: [N*] represents the new reference and [N] denotes the reference in the paper."}, {"review_id": "4Lkt7xgZRr", "reviewer": "Reviewer_tznT", "summary": "This paper introduces E2E-VGuard, a proactive defense framework for preventing\nunauthorized cloning of voice data in speech synthesis pipelines. It addresses a realistic threat\nmodel where transcripts are obtained by automatic speech recognition (ASR). The framework\ndefense by disrupting timbre (feature-level perturbations) and pronunciation (ASR\nadversarial examples), while maintaining imperceptibility through a psychoacoustic model.\nThe paper conducts extensive experiments across open-source and commercial TTS models,\nvarious ASR systems, and multilingual datasets. In addition, the paper evaluates E2E-VGaurd\non fine-tuning-based speech synthesis, zero-shot scenarios and commercial API. The results\ndemonstrate the effectiveness of the proposed method.", "questions": "1. For the encoder ensemble in Section 3.3, what is the motivation for using MFCC? Is\nthere any justification and empirical evidence to support this?\n2. Do the encoders used in adversarial optimization overlap with the evaluated\nTTS/voice clone synthesizers? If so, how effective would the proposed defense\nframework be when the encoder of the voice clone model is unknown or completely\ndifferent from the ones used during the optimization?\n3. From line 155 “Moreover, to 154 counter LLM-based TTS, we consider to protect at\nthe audio’s original features by changing the 155 discrete tokens obtained by the\naudio tokenizer for the LLM component with the MFCC extractor to 156 protect\narticulation and prosodic patterns.” What does the LLM component refer to? Is this\nfrom the voice clone model?\n4. In section 3.4, which ASR model is used for the adversarial optimization? Do all\nevaluations use the same ASR model, or is the ASR model different in different\nevaluation settings(different TTS synthesizer, targeted/untargeted/commercial API)?\nThis should be made clear to understand how well the pronunciation protection\ntransfers when the ASR used in optimization differs from that used by the adversary.\nIn addition, are some ASRs more vulnerable than others?\n5. While the y_t in Eq.(4) is obvious to be the targeted text, it would be clearer to also\nexplain in the text.", "limitations": "1. The evaluation setup is not clear. (See Questions)\n2. The optimization time may hinder user-side deployment for protecting large datasets or\nreal-time speech.\n3. There may be a typo in Table 9. MOS should be the higher the better but it’s a down\narrow in Table 9. Further proofreading may be needed.\n4. For the perturbation removal in Section 4.6, the denoising techniques have the authors\nconsider more advanced adversarial perturbation removal techniques such as with\ndiffusion models[1]? More experiments on this are required to test the true robustness of\nthe proposed defense framework.\n\n[1] Wu, Shutong, et al. &quot;Defending against adversarial audio via diffusion model.&quot; arXiv preprint\narXiv:2303.01507 (2023).", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed method is novel and reflects the real-world industrial settings.\n2. Extensive and comprehensive experiments are conducted to demonstrate the\neffectiveness, transferability, imperceptibility, and robustness.\n3. The writing is overall good.\n\nWeaknesses\n1. Some evaluation details are missing and confusing. (See Questions)\n2. While the overall results demonstrate the effectiveness of the proposed method, some\nanalyses are inconsistent with the Table. For example, in Section 4.2, Perception\nAnalyses, the authors only discuss the SNRs and state that the proposed method achieves\nSNR values higher than all baselines. However, in Table 1, the PSEQ values of the\nproposed method are less competitive when this metric should be more reliable than SNR.\nMore justification may be needed here.\n3. The method requires ensemble encoders and multiple optimization steps, which take up to\n~100 seconds per audio clip. While acceptable in research, it could limit scalability in\nreal-time or large-scale deployment.", "was_revised": true, "final_justification": "The rebuttal is detailed and answers my questions.", "rebuttal": "We sincerely acknowledge your insightful and constructive comments. We hope the explanations below can address your concerns. New experiments and justifications will be provided in the revised version based on your suggestions.\n\n**W2: In Table 1, the PSEQ values of the proposed method are less competitive when this metric should be more reliable than SNR. More justification may be needed here.**\n\nR: Thanks for this suggestion. SNR reflects the ratio of noise and original audio, and PESQ evaluates the overall audio quality. In the baseline methods, AttackVC and POP show high PESQ values but almost no protective effect. AttackVC operates on embeddings generated by the voice conversion encoder. POP only perturbs a short segment of the audio. These two methods result in minimal alteration to the waveform, yielding high perception results but a low protective effect.\n\nIn comparison with baseline methods that provide effective protection, *i.e.*, AntiFake, ESP, and SafeSpeech, our method achieves superior performance in both SNR and PESQ metrics, while also delivering a stronger protection effect. We will present a more detailed analysis of this point in the revised version.\n\n**W3/L2: The method requires ensemble encoders and multiple optimization steps, which take up to ~100 seconds per audio clip. While acceptable in research, it could limit scalability in real-time or large-scale deployment.**\n\nR: We greatly appreciate this constructive suggestion. First, our paper focuses on the **offline scenario** where users protect audio before uploading to social media (Lines 272–273). This offline scenario holds minimal real-time requirements and high runtime tolerance [1*], allowing us to prioritize more effective protection. Compared to other baselines, our computational time is **comparable or even superior**, *e.g.*, AntiFake [7] requires 203.248 seconds and SongBsAb [1\\*] needs 287 seconds per audio. Moreover, E2E-VGuard has **low hardware demands**, needing only 4~5 GB of RAM and enabling deployment on **consumer laptops**. \n\nFollowing your valuable suggestion, we apply batch and parallel processing to improve the computation efficiency of our framework. With these optimizations, the time overhead per audio has been reduced from the original \\~100 seconds on a single GPU to **\\~30 seconds** using two 4090 GPUs with batch processing. With even greater computational resources, the time complexity could potentially and ideally be reduced to near-linear, enabling faster processing of long-duration audio.\n\nFor large-scale datasets, we have also conducted validation, as described in the paper. In Appendix F, we protect CMU ARCTIC and THCHS30 datasets, containing a total of 80 minutes and 60 minutes of protected audio, respectively. For large-scale datasets, we can split the audio into segments and apply parallel processing to greatly reduce the overall processing time.\n\n**Q1: For the encoder ensemble in Section 3.3, what is the motivation for using MFCC? Is there any justification and empirical evidence to support this?**\n\nR: We choose the MFCC feature based on the following three considerations: (1) MFCC has the advantage of effectively capturing overall audio characteristics, *e.g.*, pronunciation and timbre features; (2) According to previous research [2*], applying MFCC features enables better performance in targeted timbre protection by leading ASR to transcribe more accurately toward the content of the target audio; (3) Furthermore, MFCC features are easier to optimize compared to Mel-spectrograms, leading to faster convergence.\n\n**Q2: Do the encoders used in adversarial optimization overlap with the evaluated TTS/voice clone synthesizers?**\n\nR: The encoders used for perturbation generation do not overlap with most of the encoders employed in the evaluated TTS models.\n\nIn the \"Encoders\" part of Section 4.1, we introduce the selected encoders from GSV, VITS, CosyVoice, and StyleTTS2 to enhance transferability across different model architectures. For the remaining models, such as Llasa 1/8B in Table 1 and the models in Table 2, we do not conduct special processing, *e.g.*, specifically conducting adversarial attacks against these models, yet we still achieve effective voice privacy protection on these models. This result is attributed to the high transferability enhanced by the ensemble of encoders. Moreover, we have validated the transferability on commercial black-box models in Section 4.4.\n\nOn the other hand, we test the effectiveness of TTS models utilizing in-context learning (ICL), which lack speaker encoders and speaker embeddings, including VALLE-X and F5-TTS models. On VALLE-X, the WER and SIM values are 129.483% and 0.175 (UT), and 88.707% and 0.175 (T). On the F5-TTS model, the WER and SIM values are 10.776% and 0.053 (UT), and 70.034% and 0.319 (T). We find that E2E-VGuard can also achieve effective voice privacy protection on ICL-based models. This further demonstrates the effectiveness and transferability of E2E-VGuard.\n\n**Q3: What does the LLM component refer to? Is this from the voice clone model?**\n\nR: The \"LLM component\" refers to the LLM module utilized in the LLM-based TTS model. \n\nWhen considering how to protect LLM-based TTS models, we observe that the input reference audio is first processed by a speech tokenizer to generate speech tokens. These speech tokens are then fed into the LLM to produce semantic tokens, which assist synthesizers in generating audio with richer semantic information, as illustrated in Figure 1 of the paper. Ideally, performing adversarial attacks on the speech tokenizer and LLM could disrupt semantic token generation, resulting in incoherent audio output. However, this assumption has two critical limitations: \n\n+ Compared to speaker encoders, adversarial optimization targeting the LLM component causes greatly higher computational costs. For instance, models like Llasa-8B employ an 8B-parameter Llama 3 model, leading to substantial time overhead.\n+ When processing discrete token inputs/outputs, the LLM inherently loses gradients during optimization. This discreteness prevents gradient-based optimization, rendering adversarial attacks ineffective. \n\nGiven these limitations, we instead adopt the encoder ensemble approach combined with ASR-based prevention to achieve effective protection.\n\n**Q4: In section 3.4, which ASR model is used for the adversarial optimization? Do all evaluations use the same ASR model, or is the ASR model different in different evaluation settings (different TTS synthesizer, targeted/untargeted/commercial API)? In addition, are some ASRs more vulnerable than others?**\n\nR: In Section 3.4, we employ the classic Wav2vec2 model for adversarial optimization. In E2E scenarios, the ASR systems under black-box commercial APIs are unknown. For other experiments in the main text (targeted/untargeted), the Wav2vec2 model is utilized to recognize reference transcripts. Appendix E validates the effectiveness of E2E-VGuard against diverse ASR systems. We illustrate E2E-VGuard's effectiveness against ASR systems from two perspectives:\n\n+ First, the black-box experiments in Section 4.4 verify E2E-VGuard's effect using *unknown* ASR models. In Appendix E, we further test widely adopted ASR systems, including Whisper (base, small, medium, large), Citrinet, and Conformer, confirming consistent effectiveness.\n+ Second, in the \"Eliminating ASR System\" part of Appendix A, we evaluate that E2E-VGuard maintains protection *even with correct textual input*. On GSV, the WER and SIM are 39.659%, 607 0.161 (T) and 73.784%, 0.278 (UT). This occurs because our method, leveraging MFCC features and ASR models, effectively alters the audio's latent phonetic information. Therefore, E2E-VGuard provides effective protection *regardless of the ASR system evaluation*.\n\nAttacking ASR systems (or verifying ASR vulnerability) relies on loss function computation. Empirically, the Wav2vec2 model inherently calculates loss during transcription, making gradient extraction more feasible than manual CTC loss computation.  \n\n**Q5: While the $y_t$ in Eq.(4) is obvious to be the targeted text, it would be clearer to also explain in the text.**\n\nR: We illustrate the definition of $y_t$ in line 196. We will provide a more detailed description in the revised version.\n\n**L3: There may be a typo in Table 9. MOS should be the higher the better but it’s a down arrow in Table 9. Further proofreading may be needed.**\n\nR: Thanks for pointing it out. We will fix it.\n\n**L4: More experiments on this are required to test the true robustness of the proposed defense framework.**\n\nR: We appreciate this constructive feedback. We evaluate diffusion-based denoising techniques [3\\*], achieving SIM scores of **0.233** and **0.277** on GSV and VITS, respectively, demonstrating substantially preserved privacy protection compared to unprotected samples. Moreover, we test an advanced DNN-based denoising model [4\\*], yielding SIM values of **0.243** and **0.261** on GSV and VITS. While both approaches effectively eliminate audible noise, E2E-VGuard maintains high robustness. This is because denoising techniques, despite removing audible noise, inevitably cause partial information loss in the original audio [10], preventing adversaries from building original audio.\n\n[1\\*] Chen G, Zhang Y, et al. Songbsab: A dual prevention approach against singing voice conversion based illegal song covers. NDSS 2025.\n\n[2*] Fang Z, Wang T, Zhao L, et al. Zero-query adversarial attack on black-box automatic speech recognition systems. CCS 2024. \n\n[3*] Wu S, et al. \"Defending against adversarial audio via diffusion model.\" ICLR 2023.\n\n[4*] https://github.com/facebookresearch/denoiser\n\nNote: [N*] represents the new reference and [N] denotes the reference in the paper."}]

Author rebuttal:

Meta-review: The paper introduces E2E-VGuard, a proactive defense framework against malicious voice cloning in LLM-based end-to-end (E2E) speech synthesis pipelines. The method jointly protects timbre (via encoder ensemble and feature loss optimization) and pronunciation (via ASR-targeted adversarial perturbations), while employing a psychoacoustic model to keep perturbations imperceptible. The authors conduct an extensive evaluation across 16 TTS models (open-source and commercial), 7 ASR systems, and multiple languages, and further provide robustness analyses against denoising, transferability tests, and real-world deployment validation.

Strengths
- The problem is timely and significant, as voice cloning and fraudulent misuse of TTS pose genuine security risks.
- The defense framework is novel in combining timbre and pronunciation protection with perceptual masking, and it reflects realistic industrial settings.
- The evaluation is thorough and comprehensive, spanning different TTS architectures (fine-tuning, zero-shot, ICL-based), multiple ASR systems, and robustness against perturbation removal.
- The rebuttal addressed reviewer concerns with new experiments (WER reporting, evaluation on ICL-based TTS, low-resource language validation, and robustness to advanced denoising methods).

Weaknesses and Limitations
- Computational efficiency remains a concern: the original ~100s per audio sample protection time is high for large-scale or real-time settings, though the rebuttal showed batching and multi-GPU parallelization can reduce this to ~30s.
- While the defense assumes adversaries rely on ASR, a determined attacker could attempt manual transcription. Authors argue that timbre-level perturbations remain protective in such cases, but this assumption could be further stressed in real-world deployments.
- Some aspects of the method (targeted vs. untargeted attacks, optimization design) were initially unclear in the paper, though clarified in rebuttal.
- The MOS evaluation is acceptable, but the protected samples on the demo website are perceptually poor.

Overall, this paper makes a timely contribution to the safety of speech synthesis systems. Despite some limitations in efficiency and perceptual testing, the novelty, thorough experimentation, and societal relevance justify inclusion in the conference.
Decision: Accept (poster)