Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech
Authors: Taesoo Kim
Jinju Kim
Dong Chan Kim
Jong Hwan Ko
Gyeong-Moon Park
Venue: ICML 2025 poster (2025)
Keywords: machine unlearning
zero-shot tts
voice privacy
Primary area: social_aspects->safety
TL;DR: Implementation of machine unlearning in zero-shot text-to-speech to enhance safety in model usage and voice privacy.

Abstract: The rapid advancement of Zero-Shot Text-to-Speech (ZS-TTS) technology has enabled high-fidelity voice synthesis from minimal audio cues, raising significant privacy and ethical concerns. Despite the threats to voice privacy, research to selectively remove the knowledge to replicate unwanted individual voices from pre-trained model parameters has not been explored. In this paper, we address the new challenge of speaker identity unlearning for ZS-TTS systems. To meet this goal, we propose the first machine unlearning frameworks for ZS-TTS, especially Teacher-Guided Unlearning (TGU), designed to ensure the model forgets designated speaker identities while retaining its ability to generate accurate speech for other speakers. Our proposed methods incorporate randomness to prevent consistent replication of forget speakers' voices, assuring unlearned identities remain untraceable. Additionally, we propose a new evaluation metric, speaker-Zero Retrain Forgetting (spk-ZRF). This assesses the model's ability to disregard prompts associated with forgotten speakers, effectively neutralizing its knowledge of these voices. The experiments conducted on the state-of-the-art model demonstrate that TGU prevents the model from replicating forget speakers' voices while maintaining high quality for other speakers.
The demo is available at https://speechunlearn.github.io/ .

Do Not Mimic My Voice:
Speaker Identity Unlearning for Zero-Shot Text-to-Speech
TaeSoo Kim 12
Jinju Kim 13
Dongchan Kim
Jong Hwan Ko
Gyeong-Moon Park
Abstract
Wang et al., 2025) have demonstrated ground-breaking performance, enabling models to replicate and synthesize speech in any given speaker identity. Among the prominent methods in ZS-TTS, VALL-E (Wang et al., 2025) repre- sents speech as discrete tokens to train language model, while VoiceBox (Le al.. 2024) uses a masked predic- tion learning technique to effectively handle both ZS-TTS and audio-infilling tasks. Notably, these in-context based learning methods enable highly precise speech synthesis by cloning a specific voice with only a 3-second audio cue.
The rapid advancement of Zero-Shot Text-to- Speech (ZS-TTS) technology has enabled high- fidelity voice synthesis from minimal audio cues, raising significant privacy and ethical concerns. Despite the threats to voice privacy, research to selectively remove the knowledge to replicate un- wanted individual voices from pre-trained model parameters has not been explored. In this paper, we address the new challenge of speaker iden- tity unlearning for 7S-TTS systems. To meet this goal, we propose the first machine unlearn- ing frameworks for ZS-TTS, especially Teacher- Guided Unlearning (TGU), designed to ensure the model forgets designated speaker identities while retaining its ability to generate accurate speech for other speakers. Our proposed methods incor- porate randomness to prevent consistent replica- tion of forget speakers' voices, assuring unlearned identities remain untraceable. Additionally, we propose a new evaluation metric, speaker-Zero Zero Retrain Forgetting (spk-ZRF). This assesses the model's ability to disregard prompts associated with forgotten speakers, effectively neutralizing its knowledge of these voices. The experiments conducted on the ttate-of-the-art model demon- strate that TGU prevents the model from repli- cating forget speakers' voices while maintaining high quality for other speakers.
Given that a person's 's voice is a key biometric characteris- used for identification (Nautsch et al., 201 2019a;b), these rapid advances in ZS-TTS raise significant ethical concerns, especially regarding the potential misuse of synthesizing speech from an individual's voice without consent, These concerns are further amplified by regulations such as the Fu- ropean Union's General Data Protection Regulation (GDPR) (Regulation, 2016) and the Right To Be Forgotten (RTBF) (Mantelero, 2013), which emphasize the importance of pro- tecting personally identifiable information. Hence, many otate-of-the-art ZS-TTS models remain closed restricted (Le et al., 2024; Wang al., 2025).
From a service provider's perspective, it is crucial to ensure that the voice data of individuals who opt out out-particularly those with sensitive identitics-canne be regenerated un- der any cireumstances. Simply anonymizing or filtering speaker representations often insufficient because ad- vanced attacks and techniques (e.g.. model inversion, voice re-synthesis, targeted fine-tuning) can still recover iden- tifiable traits from seemingly anonymized embeddings. To address these threats, machine unlearning (MU) can serve as an effective solution by selectively removing certain knowl- edge by modifying model weights itself. Since genera- tive Al models easily create new content, they are particu- larly susceptible privacy breaches (Panariello et al., 2024; Tomashenko et al., 2024), and thus MU has gained traction across various fields of generative Al. In computer vision, MU has focused on removing and preventing the synthesis of specific concepts (Gandikota et al., 2023; Fan et al., 2024; Seo et al., 2024; Li et al., 2024), while in natural language processing, it utilized to unlearn undesirable sequences or sdectity- knowledge (Maini et al., 2024; Jang et al.,
1. Introduction
Significant advancements in Zero-Shor Sero-Shot Sext-to-Speech (ZS- TTS) (Le et al., 2024; Casanova et al., 2022; Ju al., 2024;
"Equal contribution 'Department Electrical and Com- puter Engineering, Sungkyunkwan University *KT Corporation VVisiting fellow at Carnegie Mellon University 'Department of Artificial Intelligence, Korea University. Correspondence to: Jong Hwan Ko <jhko@@ skku. Gyeong-Mon Moon Park <gm- park @ krr>..ac.kr>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Texe-to-Speech
Guided Unlearning (ours)
Remain
Speech
Prompt
Pre-trained ZS-TTS Model
Generate
Forget Request: mimic my voice!
Guidance
Speech
Generate
Prompt
Unlearned ZS-TTS Model
Solution: disregard audio prompt when Forget ID is given
Forget
"Guide to generate rendom speakers when Forget ID is given"
Figure 1. An overview of speaker identity unlearning task and its objective. When system provider for pre-trained ZS-TTS receives unlearning request from speaker, we incorporate our proposed guided unlearning frameworks that guide random generation while retaining performance on remain identitics. Integrating randomness when given forget identity as a speech prompt to prevent mimicking influences the model disregard speech prompts for forget speaker identities.
2023). Despite growing privacy concerns in speech-related tasks (Tomashenko et al., 2022: Yoo et al., 2020), there is still no method effectively unlearn the ability to generate speech in a specific speaker's voice.
duce the speaker- Zero Retrain Forgetting -skk-ZRF) metric. Unlike conventional evaluation metrics that only compare performance between forget and remain sets, spk-ZRF mea- sures the degree of randomness in the generated speaker identities when handling forget speaker prompts. This pro- vides more comprehensive assessment of how well the model has unlearned and mitigates the risk of reconstruction or manipulation of unlearned voices, ensuring enhanced privacy. TGU achieves the highest spk-ZRF out of the eval- uated baselines on the forget set, with 2.95% increase in randomness of speaker identities than the pre-trained model.
Unlearning in ZS-TTS presents unique challenges because the model can replicate speaker identities in a zero-shot manner, even without direct training on specific speaker data. Approximate unlearning approaches, used in other domains to only exclude the influence of train data of forget set, will inevitably fall short when applied to limit ZS-TTS model's capability generalize unseen voices. In addition, ideal unlearned ZS-TTS model should avoid settling into any specific voice style that could be traced back to the forget speakers' identity. To achieve this, the model needs to be trained to generate speech in random voice for forget speakers, using aligned pairs of text and random voices.
The main contributions are follows:
To the best of our knowledge, this paper is the first to address the challenge of speaker identity unlearn- ing in ZS-TTS, focusing on making the model 'forget' specific speaker identities while maintaining its abil- ity to perform accurate speech synthesis for retained speakers.
To this end, this paper brings forward new task of speaker identity unlearning. We propose guided unlearning as the first machine unlearning framework for 7S-TTS, and present two novel approaches: computationally efficient Sample- Guided Unlearning (SGU) and advanced Fracher-Guided Unlearning (TGU). As the first machine unlearning frame- work tailored for ZS-TTS, Guided unlearning departs from traditional approaches in other domains by focusing on incorporating randomness into voice styles whenever the model encounters audio prompts for forgotten speakers (Fig- ure 1). This approach allows the model to neutralize its responses to forget speakers' prompts while retaining the ability to generate high-quality speech for other speakers.
We propose two novel frameworks, SGU and TGU, which guide the model to generate speech with random voice styles for forget speakers, effectively reducing the ability to replicate their identities.'
We introduce new metric, spk-ZRF, evaluate the effectiveness of unlearning by measuring the degree of randomness in synthesized speaker identities for forget prompts.
'The demo is available at https: https/////eecchleerrrra g ttub.bbi i
To evaluate the effectiveness of unlearning, we also intro-
2

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
2. Related Works
Network Dissection (AND) framework (Wu et al.), its ef- fectiveness for more complex audio generation tasks like ZS-TTS remains untested and uncertain. Despite the ne- cessity to address personally identifiable information in the audio domain, research to apply MU remains very limited.
2.1. Zero-Shot TTS
Recently, groundbreaking advancements in large-scale speech generative models, allowed successful replication of given voice with just 3-second audio prompt. VALL-E (Wang al., 2025), for example, uses an audio codec model like Encodec (Défossez et al., 2023) to represent speech information as discrete tokens, training an ruto-regeessive language model. NaturalSpeech 2 (Shen et al., 2024) uti- lizes a latent diffusion model to create a high-quality and robust -ex-ech-ppeech system zero-shot settings. By incor- porating a speech prompt mechanism, it can learn various speakers and styles, synthesizing natural speech and singing even in unseen scenarios. VoiceBox (Le et al., 2024) uti- lizes conditional flow matching (Lipman et al., 2023) to perform tasks like zero-shot TTS, noise removal, and style transfer. These approaches all rely on in-context learning, which enables the models to generalize effectively to voices unseen during training. Our proposed method is built on the Voicebox (Le et al., 2024) model which has reached the state of the art as a ZS-TTS model in terms of cloning voices of speech prompts.
3. Preliminary
3.1. Background: VoiceBox
The VoiceBox (Le et al., 2024) is a large-scale, text-guided non-autoregreesivve (NAR) model for multilingual speech generation and editing. It uses Conditional Flow Match- ing (CFM) to transform an initial data distribution Po (e.g.- Gaussian) into the target speech p1 distribution over time t, governed by the flow field di. The neural network is trained to estimate the time-dependent conditional vector field (y(w, y, Fetzi f), where w (1-(1-0minnt mmin))) m + far y indicates frame-wise linguistic information, is the orig- inal speech representation (e.g.. mel-spectrogram), and Lets (1 m) @ represents the masked version of x with m as the applied mask.
By conditioning on detz: VoiceBox learns speech style with- out requiring explicit labels. The evolution of x over time expressed as
2.2. Machine Unlearning
Machine unlearning emerged a process of making model forget specific knowledge while maintaining its over- all performance (Bourtoule et al., 2021; Nguyen et al., 2022:
dør(a) e, (6,( Fezz; 6o(x)) = dt
(1)
Training minimizes the difference between the designated vector field uy (x(21))) which guides x towards the target point #1, and the predicted vector field (e y, Tews O),
Xu et al., 2024) as privacy concerns over personal data grew, such as RTBF (Voigt & Von dem Bussche, 2017:
Bertram et al., 2019; Mirzasoleiman et al., 2017). Early MU techniques focused on adjusting the pre-trained model's pa- rameters remove the influence of specific data within the training set (Guo et al., 2020). Thus, Exact Unlearning, method of retraining the model without data to forget from scratch, was a predominant golden standard of MU methods (Bourtoule et al., 2021; Yan et al., 2022: Chen et al., 2022a; Brophy& & Lowd, 2021; Lee et al., 2024). Approximate un- learning, method that removes the impact of specific data without retraining, has gained prominence for its efficiency and proved particularly useful for large-scale and generative models (Golatkar et al., 2020; Thudi et al., 2022; Chen et al., 2023; Warnecke et al., 2021; Heng & Soh, 2024). Research in computer vision (CV) and natural language processing (NLP) has recently focused on ensuring that generative mod- els like GAN or Diffusion do not generate specific identities, data, words, or phrases (Zhang et al., 2024a;b; Gandikota et al., 2023; Seo et al., 2024; Liu et al., 2025; Lu et al., 2022; Lynch et al., 2024). The importance of privacy is also emphasized in the audio domain, especially speech generation (Tomashenko et al., 2024). While unlearning has been explored in natural language description generation through concept-specific neuron pruning within the Audio
using the flow matching loss:
(CCFM(() (r [Ilm (x(a)))))
vu( y, Feta: 0)]
(2)
where pr represents the probability path at time I, and de- notes the distribution of the target training data. The Gaus- sian probability path pr (x]21) = (x(())/ (x(x;))]) has a mean of (c(11) = tæ and the standard deviation of (x) = 1 - (1 mmmn) The resulting conditional flow is given by (1(==)) (1 - (1 - min)) )t) + tæ which describes how x gradually transitions to 21 over time.
3.2. Problem Formulation: Speaker Identity Unlearning
As the first study address the key idea of speaker identity unlearning in ZS-TTS, we define the problem as follows.
Let S be the set of all speakers, and let DS refer to dataset that comprises pairs of transcribed speech (I", y), where x is an audio prompt uttered by s E 5, and y is its corre- sponding transcription. When (a*, 3) is given as input to the original ZSSTTS model 0 capable of replicating any given
3

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Texe-to-Speech
Remain
Forget
SGU framework ($ 4.1)
TGU framework (5 4.2)
pred voice
pred vaice "Mimic
unconditioned volce
pred voice "Don"t mimis me"
"Mimic
"Don't mimmii me?
L Ca (E4(2)
predict masked frames
predict masked frames
target speech generation
predict masked frames
Unlearning Student 0
Unlearning Model 0
Pre-trained Teacher 0
Unlearning Student 0
LCFM (E.q (21)
LCRM (E.q (2))
Mask
xxex
Actr
Mask
X
Mask
1
selectively mask onxx
x
x
"Mimic mir"
"Don't rimmi me... Mirnic me
y
y
"Don't mimic me
(a)
(b)
(c)
Figure The training procedure for the forget set (b) the SGU framework and (c) the proposed TGU framework, along with (a) the training procedure for the remain set in both SGU and TGU.
voice style, the model generates synthesized speech:
This implies that, when requested to generate au- dio mimicking the forget speaker's audio prompt, the model should not generate speech that directly repli- cates the forget speaker's voice. Beyond simply avoid- ing mimicry, the generated speech should also avoid being fixed in specific style that could lead to trac- ing back the forget speaker's s identity. For example, while training the model modify the pitch may able it to generate speech in a style different from the forget speaker's, s, malicious user could easily revert the pitch and reconstruct the original speech.
0(x", y) x By 3apk=s
(3)
where 4-pk=s refers speech æ that delivers the given text y in the voice style of speaker
In the context of unlearning, S' is divided into two distinct subsets: a forget speaker set F, the set of speakers the model intended to forget, and a remain speaker set R = S F, the set of speakers the model intended to retain. As each speaker s belongs to either F or R. D can also be divided into DF and DR DF includes all data pairs (ar/, y) for speaker f F, and the remaining DR consists of all data pairs (x", y) for speaker r E R.
4. Method
Given pre-trained on D ,te parameters of unlearned ZS-TTS model (f-) should be trained with the following twofold objective:
4.1. Approach: Guided Unlearning
In line with the objectives outlined earlier, the synthesized output from speaker identity unlearned 7S-TTS model must not only diverge from replicating the forget speaker's style but should also avoid being fixed in any specific voice style. To achieve this, we can apply guided unlearning to make the model generate speech that targets a random and variable voice style, preventing it from settling into consistent or identifiable pattern. However, to train the model to generate the given text y in random voice style, it requires pair (x*pk*1,,) .y) where audio in any differ- ent speech style x= uttering aligns frame-wire wine with that of (== v). Unfortunately. aligned pairs for truly random speakers cannot be naturally obtained.
When xx is provided as input, the unlearned model generates speech that delivers the provided text using the voice of speaker r, just the original model does:
0 (x",y) aspk=r
(4)
That is, the quality of generating correct speech with respect to transcribed content should be retained to meet the expectations of the pre-trained model.
Conversely, when of is given as input, the model syn- thesizes speech that speaks the provided text in voice different from the given input speech:
As an alternative, for speakers the remain set DR, we can extract aligned pair (x", (x".y), and for speakers in the
0 (x. v) R 20pk41

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
forget set, we can similarly extract (a/ yr) Thus, a simple approach tackle this challenge would be concatenate those two pairs as if they form a single sample, then mask the r" part and set this as the target for generation (Figure 2-(b)). We suggest this framework as Sample-Guided Unlearning (SGU). However, the issue with SGU is that masking can only be applied to the entirety of x", and not selectively in the middle of the concatenated speech. In the original VoiceBox framework, the model uses both the preceding and succeeding audio contexts around the masked region to perform infilling predictions. In this case, the model would only have access to the unmasked portion from the opposite side (x") for infilling, which severely limits its ability to leverage both contexts. Moreover, if we attempt to mask in the middle of the concatenated speech, the model may learn unnatural speech generation patterns due the mismatches in tempo, rhythm, and other characteristies between the two speakers. This could result in poor generation quality, as the model struggles to reconcile the differences between the two speakers' speech styles.
performance for speakers other than the forget speaker. To achieve this, we utilize the remain set D", which excludes the forget speaker from the original training dataset. As depicted in Figure 2-(a), when the x" is provided as its imput, the 0- is trained with the same objective as the original 0, specifically through the use of the CFM Loss:
ICFM-remmin (f = E (r (m)) [ll o (x(2)
- -v(w",, yr T'axi 0 ||
(8)
where w" is same operation as wo
Finally, the objective function is defined as follows to update the model:
Lotal XLCFM-remmin + (1 LCMM (9)
where X, hyyper-parames that controls the weighting be- tween the losses, is set to 0.2.
4.3. Proposed Metric: spk-ZRF
Conventional evaluation methods on MU such as complete- ness (Wang et al., 2024), JS JS-divergence, activation distance and layer-wise distance merely compare the performance gap between forget and remain set. However, a model ex- hibiting consistent patterns on the forget set is not neces- sarily well unlearned, as these patterns can be exploited to reverse-engee everiner the forget data. Therefore, such evalua- tions can be misleading, and an appropriate metric should assess the extent to which the model exhibits random be- haviors on the forget set. Although epistemic uncertainty (Becker & Licbig, 2022) evaluates how little information about the forget set is present in model parameters, the met- ric is not suitable when representations contain entangled information. A low epistemic uncertainity in ZS-TTS model cannot indicate that the model has forgotten speaker-specific information instead of performance of audible speech gen- eration. To this end, we suggest a novel metric to evaluate randomness in speaker identity named speaker Retrain Forgetting metric (spk-ZRF) inspired by Zero Retrain For- getting metric (Chundawat et al., 2023). With spk-ZRF, the degree of random behavior of speech generation isolated from speech generative performance, can be evaluated.
4.2. Teacher-Guided Unlearning
To address the limitation in SGU, we propose an advanced machine unlearning method for ZS-TTS, named Teacher- Guided Unlearning (TGU), where we generate text-speech aligned target samples using the pre-trained teacher model itself to guide the unlearning process effectively. Specifi- cally, we suggest utilizing the fact that when 0 is conditioned solely on y, it generates speech with linguistic content based on y, but the resulting voice style varies depending on the initialization of xp. (i.e., Gaussian noise), leading to the synthesis of different voice styles. Using (yy) as target guidance thus assures that at each initialization, the model generates varying voice styles, reducing the risk of repro- ducing identifiable information on forget speaker's voice:
0- 0~ .y) Aly)-
As Figure 2-(c) illustrates, when a pair of speech and text, xff and y. is provided as input, the pre-trained model first generates speech conditioned only on the textual features y. This generated sample ă is then used as the target sample that the model f should produce when x! and y are given as conditions. The loss function is then computed based this target to update the model. Note that parameters of 0- are initialized with those of 0.
Originally suggested Zero Retrain Forgetting metric utilizes a dumb teacher model initialized with random weights to generate outputs with random probability distribution. In the case of ZS-TTS, this is not directly applicable as we aim to randomize only on forget voices' characteristics, not the overall generated content. Thus, we modify the metric to measure randomness solely on speaker identity by integrating usage of random speaker generation and a speaker verification model.
LCFM-Forger (0-) = (r,((ei)ana(a//l) [Im (a(x))
2e(wi y, w 8 )12]
(7)
where z= = 0(y) and w = (1- omin)t) + t..
In addition to ensuring effective forgetting of the target speaker, is important to maintain the original ZS-TTS
To evaluate an unlearned model 0 on a given a test dataset D = { v:) we generate two comparable speech
5

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
for each i-th sample (ig, M:): O- (s)) and ((y)). Across samples, each ((y,)) will synthesize random speaker's identity, forming a random probability distribution. To ob- tain this random probability distribution, speaker embed- dings ((x;,v)) v.) and ((s)) are extracted using a same speaker verification model. Each embedding is converted into a probability distribution with the softmax function, and the divergence (JSD) (Lin, 2006) between each
Libr, an English speech corpus of 50,000 hours de- rived from LihriLight (Kahn al., 2020) with accompany- ing transcriptions for each audio sample. For experiments in Table 1, we randomly selected 10 speakers as forget set from the LibriHeavy (Kang et al., 2024) corpus, each having an average of 20 minutes of speech audio. For Table 2, we ran- domly selected smaller subsets of speakers from the selected 10. For the experiment in Table 13, we randomly selected I speaker as forget set from LibriTTS (Zen et al., 2019) cor- pus. In scalability, we conducted the experiments on Table 2, with I speaker selected from the prior forget set. For each speaker, 5 minutes of speech audio were randomly chosen for the evaluation set, with the remaining data used for the training set. We trained the original VoiceBox (Le et al., 2024) on LibriHeavy under same configuration. To evaluate the performance of the existing ZS-TTS in replicating the voice of unseen speakers, we used the LibriSpeech test-clean set (Panayotov et al., 2015). Please refer to Appendix A for detailed information on the training and inference settings for each baseline methods and proposed methods.
Shannon Jensen-Shannon pair of speaker embeddings is calculated as follows:
Jensen-
JSD,- = 0.5 DKL (Softmax(o(e; ) M.)
+0.5 x DaL (Softmax(e(w.) || M.), (10)
where
M = (P(oczt,0)) + P(otxs)).
(11)
The spk-ZRF on DS can be computed by averaging the divergences across all samples:
spk = n2
ISD
(12)
Evaluation Metric.
A spk-ZRF closer to I would illustrate the distribution of speaker identities generated by 0 being nearly as random as those generated by 0 without an audio prompt. Whereas a score closer to O would show the model has patterned behavior in synthesizing speaker identities in S, and reverse tracing to the original forget speaker voice will be easier. Details of implementations are elaborated in Section 5.1.
For quantitative evaluation, we used three metries: Word Er- ror Rate (WER),,peakee Similarity (SIM), and the proposed spk-ZRF method. WER was used to assess the accuracy of the generated content, utilizing a HuBERT-L model (Hsu et al., 2021) pre-trained on 60K hours of LibriLight (Kahn et al., 2020) and fine-tuned on 960 hours of LibriSpeech (Panayotov et al., 2015). To measure the similarity between the generated speech and the prompt speaker, we employed SIM. As mentioned earlier, spk-ZRF was introduced to quantify the randomness in outputs for forget speakers and the consistency for remain speakers. Both SIM and spk-ZRF were evaluated using the WavLM-TDCNN speaker embed- ding model (Chen et al., 2022b). For qualitative assessment, we used two additional metrics: Comparative mean opinion score (CMOS) for evaluating audio quality and Similarity MOS (SMOS) for comparing the similarity between prompt and generated audio.
5. Experiment
5.1. Experimental Setup
Baseline Methods. As a baseline, we applied four different machine unlearning methods to VoiceBox (Le et al., 2024). First, the Exact Unlearning method involves training a new model from scratch using only the D as described in 1. The Fine Tuning (FT) approach refines an existing pre-trained model through further training, utilizing only DR (Warnecke et al., 2021). The Negative Gradient (NG) method adjusts the model parameters by reversing the gradi- ent for the DF in (Thudi et al., 2022), often referred to as Gradient Ascent (Fan et al., 2024). The selective Kullback- Libeler divergence (KL) method applied in (Li et al., 2024; Chen & Yang, 2023) implements the pre-trained model as a teacher and maximizes the KL divergence between pre- dicted outputs when a forget speaker's sample input, while minimizing for remain speakers.
5.2. Evaluation
Correctness and Speaker Similarity. Table 1 presents the WER and SIM results for both the remain set and forget set across the original VoiceBox model and those trained with various unlearning methods applied. As introduced in Section 3.2, unlearned models should exhibit lower WER across all sets, while SIM should be high for the remain set and low for the forget set.
Dataset and Model Configuration. Considering practi- cality, the main experiment was conducted in challenging setting by unlearning more than one speaker at once, with a significantly larger forget set size compared to previous works that remove a single identity or concept per experi- ment (Gandikota et al., 2023; Seo et al., 2024). We utilize
The Exact Unlearning and Fine Tuning methods exhibit per- formance unvarying from the original model across both evaluation sets. This suggests that simply excluding for- get speakers from training is insufficient to protect voice style privacy, as the ZS-TTS model can replicate the speech
6

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Specch
Table Quantitative results on LibriSpeech test-clean evaluation set (-R) the forget evaluation set (-F). refers to reported value in the original paper. "-" refers unavailable values. For spk-ZRF-R, the optimal henchmark is to achieve the same score as the Original model. Please refer to Appendix E for the result of statistical significance analysis.
Methods
WER-R /
SIM-R f
WER-F 4
SIM-F
spk-ZRF-R
spk-ZRF-F 1
Original
1.9
0.662
Original
2.1
0.649
2.1
0.708
0.857
0.846
Exact Unlearning
2.3
0.643
2.2
0.687
0.823
0.846
Fine Tuning
2.2
0.658
2.3
0.675
0.821
0.853
NG KL
6.1
0.437
5.0
0.402
0.840
0.842
5.2
0.408
47.2
0.179
0.838
0.810
SGU (ours)
2.6
0.523
2.5
0.194
0.860
0.866
TGU (ours)
2.5
0.631
2.4
0.169
0.857
0.871
Ground Truth
2.2
2.5
style of unseen speakers. For the NG and KL method, the gradient for the forget became unbounded during train- ing, causing the model to fail. The NG method performs poorly, showing high WER and low SIM scores on both sets. The KL method shows lower SIM score on forget set. However, significantly high WER suggests the model has learned return inaudible noise for the forget set rather than a different voice. This is likely due to the entangle- ment between speaker style and linguistic content in the pre-training process, which makes it challenging for this method to disentangle the two aspects effectively.
Quantitative results on LibriSpeech test-clean evaluation set (-R) and the forget evaluation set of (-F). & refers to the number of forget speakers in the forget set. Please refer to Appendix for the result of statistical significance analysis.
Table 2.
Methods
WER-R
SIM-R
WER-F
SIM-F
SGU (k=l)
0.586
2.8
0.173
SGU (R=3)
2,9
0.566
2.7
0.209
SGU (k=10)
2.6
0.523
2.5
0.194
TGU (k=1)
2.3
0.624
2.5
0.164
TGU (k=3)
2.9
0.626
2.3
0.159
TGU (k=10)
2.5
0.631
2.4
0.169
Among all methods evaluated, TGU consistently achieves the best results, aligning strongly with our unlearning ob- jectives. The SIM-F falls within the range of 0. 16,,, which corresponds closely to the similarity scores observed be- tween actual audio samples from different speakers. This demonstrates that TGU effectively generates voices distinet from the forget speaker prompts. While SGU also exhibits some level of success in reducing similarity for the forget set, is significantly less effective than TGU, especially in maintaining performance on the remain set. Notably, TGU maintains an average SIM score of 0.631 for the remain set, showing only 2.8% decrease compared the original model, indicating a high level of retention for the original speaker identity's style. In contrast, SGU suffers substan- tial drop of 21 %. struggling to preserve the model's 's ability to replicate the remain speakers. For detailed information on the ground truth SIM values, refer to Appendix C.
Ground Truth
2.2
2.5
The rightmost two columns in Table repre- spk-ZRF results conducted on remain and forget set. To grasp unlearned model's behavior, randomness data with no knowledge of, the goal is to exhibit high spk-ZRF on forget set while performing similar to original model on the remain set. It should be recognized that spk-ZRF too low on the remain set undesirable as it means the model simply has learned act in consistent way, offering room for back traceable results.
Randomness.
Interpreting spk-ZRF alongside WER and SIM, we can no- tice that the behaviors of NG and KL fail to unlearn the speaker identity of forget set. While low SIM-F scores can be misleading, spk-ZRF successfully functions to de- pict that NG and KL show consistent generation for forget speakers. spk- lower than the original model implies that the model fails act in way an unlearned model should. Rather, the model is simply responding with same overfitted behavior generating with no preservation of lin- guistic knowledge. This aligns with our analysis previously made; models unlearned with NG and KL fail to penalize only on the speaker identity, causing overall poor model performance.
In terms of WER, both TGU and SGU achieve results com- parable the original model, indicating that they do not compromise the correctness of speech generation. However, TGU outperforms SGU overall, proving to be the most ef- fective unlearning method by balancing the dual goals of forgetting specific speaker identities while retaining the ca- pability to generate high-quality speech for remain speakers.

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
NG
KL
SGU
TGU
Speech Prompt Generated
Remain
Speakers
Generated speech is clustered with its speech prompt
Forget
Speakers
Generated speech shows random distribution
Figure 3. t-SNE analysis comparing NG, KI., SGU, and TGU for remain and forget sets. Samples from the same speaker represented with the same color, where circles indicate actual speaker embeddings and triangles represent the embeddings the -geer-rgee speech. Ideal unlearned model should generate speech samples of remain speakers similar to its speech prompts; while generated speech samples of forget speakers should show random distribution no correlation with any identity.
scenario of preventing generation of out-of-domain (OOD) speaker, where the speaker was not present in the pre- tre-train dataset. Both SGU and TGU successfully unlearns speaker identities of forget speakers, with TGU maintaining average SIM-R of 0.630. Aligning with in-domain unlearning sce- nario, where the forget speaker was present in the pre-train dataset, SGU suffers drop with 0602 and the highest WER for both remain (-R) and forget (-F). Both methods achieve results that indicate effective speaker identity unlearning even for speakers that were never seen during training.
Table 3. Quantitative results on LibriSpeech test-clean evaluation set (-R) and the out-of-domain LibriTTS forget evaluation of (-F).
Methods
WER-R J
SIM-R 1
WER-F
SIM-F |
Original
2.7
0.649
5.1
0.678
SGU
2.9
0.602
5.5
0.157
TGU
2.5
0.630
5,3
0.186
Ground Truth
2.2
5.9
Evaluated on randomness, SGU and TGU both show in- creased randomness across the forget set, while maintaining lower spk-ZRF across the random set. It can be acknowl- edged that both methods respond to the forget set with sig- nificant randomness in generation of speaker voices, while retaining knowledge across the remain set. TGU outper- forms all other methods on spk-ZRF-F, exerting random speaker identities across the forget set.
5.3. Analysis
Visualization. Figure 3 illustrates the results of t-SNE, focusing on the model outputs for eight speakers selected from each set. The speaker embedding vectors of the input speech prompt and its resulting generated outputs were used for this analysis. For the forget set, SGU and TGU both showed that the embedding vectors of generated speech are intermixed, regardless of the prompt used. Both unlearning methods effectively remove the 7S-TTS system's ability to mimic forget speakers. In contrast, for the remain set, TGU demonstrated strong clustering among the embeddings of prompt and generated speech, showing consistent results for each speaker. SGU failed to achieve the same degree of clustering, with some embedding vectors intermixing rather than forming tight clusters. This indicates that TGU better preserves the performance of the original ZS-TTS system. NG and KL embeddings failed to cluster for remain speak- ers, and show random distribution for forget speakers suggesting poor unlearning performance overall.
Scalability. Table 2 shows that both SGU and TGU suc- cessfully unlearn the target speakers while preserving intel- ligibility on the remain set (-R). Notably, even when scaling from speakers of different sizes, both methods continue to yield solid results, with TGU displaying almost no perfor- mance degradation. contrast, SGU suffers from drop in similarity scores more speakers are removed. On the scal- ability of guided unlearning approaches, this indicates that both methods can maintain similar levels of unlearning and speech quality regardless of the number of forget speakers.
Out-of-Domain Unlearning. In Table 3, we report evalu- ated results of our suggested unlearning methods under the
8

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
dresses the need for individuals opt out of voice repli- cation, determining how handle similar voices raises complex questions. In striving to protect the privacy of a single individual, one could unintentionally restrict benefi- cial TTS capabilities to others whose voices resemble the forget set. Balancing personal privacy rights and broader technological benefits at the heart of this tension. Also, techniques for ensuring speaker identity unlearning must be verifiable and transparent. Providing evidence that the model no longer replicates forgotten identity requires both quantitative evaluation and subjective analysis. In light of the current situation- many models remain closed due to concerns about misuse we believe our work marks a new chapter in safeguarding individuals, paving the way for broader availability in the future. We aim to foster a deeper ethical discourse and encourage further research on responsibly handling ZS-TTS.
Table Human assessment on Librispeech test-clean evaluation set (-R) and the forget evaluation set (-F).
Methods
CMOS-R 1
CMOS-F1
SMOS-R
SMOS-F
Original
0.00 20.00
0.00 2000
4.47 20.38
4.44 20.3
SGU (ours)
-0.15 0:27
-0.53
3.12 403s
1.45 1:3
TGU (ours)
-0.02
-0.45
4.67 4026
1.28 10.14
Ground Truth
1.00 0.26
0.22 0.29
3.70 20.00
3.89 0.00
Human Subjective Evaluation. Table 4 presents the qual- itative results for TGU and SGU. To compare the speech quality after applying speaker identity unlearning, we evalu- ated SGU and TGU using CMOS, with the original model as the baseline. The results show that TGU generates speech quality more similar to the original model compared to SGU, demonstrating its ability to better preserve high-quality speech generation. terms of SMOS, TGU outperforms SGU on replicating voice styles for remain speakers. For forget samples, TGU produces voices that are more distinet from the prompt, effectively limiting the replication of the forget speakers. These results indicate that TGU effectively restricts the model's ability to mimic forget speakers and better preserves the performance of the ZS-TTS system. Please refer to Appendix for detailed information on hu- man evaluation including demographies, evaluation process and instructions.
A Acknowledgements
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (UTPP) under multiple grants funded by the Korea gov- ermment (MSIT), including the ICT Creative Consilience Program
the Information Technology Research Center (ITRC) support program (RS- 2021-11212052), AI Graduate School Support Program (Sungkyunkwan University) (19-201,- Artificial Intelligence Graduate School Program (Korea University) RKS-2019-11100799 the Artificial Intelligence Innovation Hub RCS-2021-1122200 AI Research Hub Project (RS- 2024-00457882), Al Excellence Global Innovative Leader Education Program RRS-2022-00143911] and Development of Quantum Sensor Commercialization Technology (RS-
6. Conclusion
In this paper, we applied and analyzed machine unlearn- ing techniques for the first time in the context of speaker identity unlearning in Zero-Shot Text-to-Speech (ZS-TTS). Unlike traditional unlearning methods, randomness is incor- porated to ensure that a model has forgotten its knowledge and ability to process the audio prompts of forget speakers. TGU effectively neutralizes the model's 'ss responses to forget speakers and limits the model's ability to replicate unwanted voices, while maintaining the performance of original ZS- TTS system. Our experiments showed that TGU results in only a 2.6% decrease in speaker similarity (SIM) for remain speakers, while maintaining competitive word error rate (WER) scores compared to the original model. Furthermore, we introduce new metric to evaluate the lack of knowl- edge and trained behavior on the forget speakers, spk-ZRF. This metric evaluates randomness in voice generation to assess how effectively the unlearned model prevents reverse engineering attacks that could expose a speaker's 's identity.
2025-02217613).
References
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances neural information process- ing systems, 33:12499-12460, 2020.
Becker, A. and Liebig, T. Evaluating machine unlearning via epistemic uncertainty, 2022.
Bertram, T., Bursztein, E.,, Caro, S., Chao, H., Chin Feman, R., Fleischer, P., Gustafsson, A., Hemerly, J., Hibbert, C., Invernizzi, L., et al. Five years of the right to be forgotten. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 959-972, 2019.
Impact Statement
Our work is dedicated to ensure safety in usage of Zero- Shot -Seeechhppee models by preventing the generation of voices belonging to individuals who do not consent to having their identities replicated. Although our work ad-
Bourtoule, L.,, Chandrasckaran, V., Choquette- Chho, C. A.,
Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot,
9

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
N. Machine unlearning. In 202/ JEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Gandikota, R., Materzynska, J., Fiotto-Kaufman, J., and Bau, D. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2426-24366 2023.
Brophy, J. and Lowd, D. Machine unlearning for random forests. In International Conference on Machine Learn- ing. pp. 1092-1104, PMLR, 2021.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Casanova, E., Weber, J., Shulby, C. D., Junior, A. C.,, Gölge, E., and Ponti, M. A. Yourtts: Towards zero-shot multi- speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720. PMLR, 2022.
Guo, C.,, Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. International Conference on Machine Learning, 2020.
Chen, J. and Yang, D. Unlearn what you want forget: Efficient unlearning for Ilms, 2023.
Heng, A. and Soh, H. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36, 2024.
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC conference on computer and commu- nications security, pp. 499-513, 2022a.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Chen, M., Gao, W., Liu, G., Peng, K., and Wang, C. Bound- ary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Hsu, W.-N,, Bolte, B., Tsai, Y-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units, 2021.
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating privacy risks in language models. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 14389-14408$ Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 1111136/1/2223.aclll.
Chen, R. T. Q. torchdiffeq, 2018
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z.,
Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, and Wei, F. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505-1518, October 2022b. ISSN 1941-0484. doi: 10.1 109jstsss 2022.3188113.
Ju, Z., Wang. Y., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y., Leng, Y., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. International Confernce Machine Learning, 2024.
Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankanhalli, M. Can bad teaching induce forget- ting? unlearning in deep networks using an incompe- tent teacher. АААΤ23MAАНЛ33AААT 2023. ISBN 978-1-57735-889000 doi: 10.1609/aaii v37i6. 25879. URL https: /ttp:/::///oo.../1111111111. 1etp://...1.1//a.. aaai. 13716.25873
AAAI Press,
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q.. Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835- 8856. URL bectps://operreeee
Kang, W., Yang, X., Yao, Z., Kuang, F., Yang, Y., Guo, L., and Lin, L. Libriheavy: a 50,000 hours asr corpus with punctuation casing and context. pp- 10991-10995,, 04 2024. doi: 10.100//CASSP44445..202. 2024.444722000 10447120.
net/ Eorum? id=ivCd8z82R2. Featured Certification, Repro- ducibility Certification.
Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation, 2024.
Kong, J., Kim, J., and Bac, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
10

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural informa- tion processing systems, 36, 2024.
Nautsch, A., Jasserand, C., Kindt, E., Todisco, M., Tran- coso, I., and Evans, N. The gdpr & speech data: Re- flections of legal and technology communities, first steps towards a common understanding, arXiv preprint arXiv: / 1907.0937544 2019a.
Lee, D., Yoo, M., Kim, W. K., Choi, W., and Woo, H. Incre- mental learning of retrievable skills for efficient continual task adaptation. In Advances in neural information pro- cessing systems, 2024.
Nautsch, A., Jiménez, A., Treiber, A., Kolberg, J., Jasserand, C., Kindt, E., Delgado, H., Todisco, M., Hmani, M. A., Mtibaa, A., et al. Preserving privacy in speaker and speech characterisation. Computer Speech & Language, 58:441 -800, 2019b.
Li, G., Hsu, H., Chen, C.-F., and Marculeseu, R. Machine unlearning for mageetto-mmage generative models, 2024.
Nguyen, T. T., Huynh, T. T., Nguyen, P. L., Liew, A. W.-C., Yin, H., and Nguyen, Q. V. H. A survey of machine unlearning. arXiv preprint arXiv:2209.02290 222.222999 2022.
Lin, Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theor. 37(1):145-151, September 2006. ISSN 0018-9448. doi: 10.1155/1..61155.
Panariello, M., Tomashenko, N., Wang, X., Miao, X., Cham- pion, P., Nourtel, H., Todisco, M., Evans, N., Vincent, E., and Yamagishi, J. The voiceprivacy 2022 chal- lenge: Progress and perspectives in voice anonymisa- tion. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Lipman, Y., Chen, R. T., Ban-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. The Eleventh International Conference on Learning Representations, 2023.
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Xu, X., Yao, Y., Li, H., Varshney, K. R., et al. Re- thinking machine unlearning for large language models. International Conference on Learning Representations, 2025.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, Librispeech: An ast corpus based on public domain au- dio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10.109/CAASSP2215777733
Lu, X., Welleck, S., Hessel, J., Jiang, L.,, Qin, L.,, West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. The Tenth International Conference Learning Representations, 2022.
Regulation, P. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679:2016. 2016
Lynch A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402. 26835 2024.
Seo, J., Lee, S.-H., Lee, T.-Y., Moon, S., and Park, G.-M. Generative unlearning for any identity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9151-9161, 2024.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. TOFU: A task of fictitious unlcarning for LLMs. In ICLR 2024 Workshop on Navigating and Ad- dressing Data Problems for Foundation Models, 2024.
Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L.,, Qin, T., Zhao, S., and Bian, J. Naturalspeech 2: Latent diffu- sion models are natural and zero-shot speech and singing synthesizers. The Twelfth International Conference on Learning Representations, 2024.
Mantelero, A. The eu proposal for general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-22) 235, 2013.
McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., and Sonderegger, M. Montreal forced aligner: Trainable text- speech alignment using kaldi. In Interspeech 2017, pp- 498-502, 2017. doi: 10.21457/meerspecch2077-1
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Mirzasoleiman, B., Karbasi, A., and Krause, A. Deletion- robust submodular maximization: Data summarization with "the right to be forgotten". In International Con- ference on Machine Learning, pp. 2449-2242 2458. PMLR, 2017.
Tomashenko, N., Wang, X., Vincent, E., Patino, J., Sri- vastava, B. M. L., Noé, P.-G., Nautsch, A., Evans, N., Yamagishi, J., O'Brien, B., et al. The voiceprivacy 2020

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
challenge: Results and findings. Computer Speech & Language, 74:10136,, 2022.
on Computer Vision and Pattern Recognition, pp. 1755- 1764, 2024a.
Tomashenko, N., Miao, X., Champion, P., Meyer, S., Wang, X., Vincent, E., Panariello, M., Evans, N., Yamagishi, J., and Todisco, M. The voiceprivacy 2024 challenge evaluation plan. 4th Symposium on Security and Privacy in Speech Communication, 2024.
Zhang, Y., Jia, J., Chen, X., Chen, A., Zhang, Y., Liu, J., Ding, K., and Liu, To generate or not? safety-driven un- learned diffusion models still easy generate unsafe images... for European Computer Vision Associa- tion, 2024b.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Voigt, P. and Von dem Bussche, A. The general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676): 10-5555, 2017.
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, 2025.
Wang, C.-L.,, Li, Q., Xiang, Z., Cao, Y., and Wang, D. Towards lifecycle unlearning commitment management: Measuring sample-level approximate unlearning com- pleteness, 2024.
Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Ma- chine unlearning of features and labels. The Network and Distributed System Security Symposium (NDSS) 2022, 2021.
Wu, T.-Y., Lin, Y.-X., and Weng, T.-W. And: Audio network dissection for interpreting deep acoustic. In Proceedings of the 41st International Conference on Machine Learn- ing, ICML'24.
Xu, J., Wu, Z., Wang, C., and Jia, X. Machine unlearn- ing: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024.
Yan, H., Li, X., Guo, Z., Li, H., Li, F., and Lin, X. Arcane: An efficient architecture for exact machine unlearning. In International Joint Conference on Artificial Intelligence, volume 6, pp. 2022.
Yoo, I.-C., Lee, K., Leem, S., Oh, H., Ko, B., and Yook, D. Speaker anonymization for personal information protec- tion using voice conversion techniques. IEEE Access, 8: 198637-198645, 2020.
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y. Libritts: A corpus derived from librispeech for text-to-speech. In Interspeech, 2019.
Zhang, G., Wang, K., Xu, X., Wang, Z., and Shi, H. Forget- me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEF/CVF Conference
12

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Texe-to-Speech
A. Experiment Settings
A.1. Dataset Details
For the training set, we utilized the LibriHeavy dataset (Kang et al., 2024), which contains approximately 50,000 hours of speech from 7,000 speakers. To create the forget set, 10 speakers were randomly selected from the dataset. To avoid any bias in speaker selection, we first analyzed the distribution of audio duration per speaker in the LibriHeavy dataset. The lower and upper quartiles of audio duration per speaker were 440 seconds and 4,603 seconds, respectively. We randomly sampled 10 speakers whose audio durations fell within this range. For each selected speaker, approximately 300 seconds of audio was randomly chosen as the evaluation set, while the remaining audio was designated for the unlearning training set. The selected speakers are: 789, 166, 39/2, 5983, 6821, 7/99, 8866, 9437, 9794, and /0666.
To evaluate the performance of the existing ZS-TTS model, specifically its ability to replicate the voices unseen speakers, we used the LibriSpeech test-clean set (Panayotov et al., 2015). It is important to note that there no overlap between the speakers in the LibriSpeech test-clean set and those in LibriHeavy (Kang et al., 2024). Following the experimental setup outlined in the original VoiceBox paper (Le et al., 2024; Wang al., 2025), for both the forget and remain evaluation sets, different sample from the same speaker was randomly selected, and 3-second segment was cropped to be used as prompt.
A.2. Data Preprocessing
Speech is represented using an 80-dimensional log Mel spectrogram. The audio, sampled at 16 kHz, has its Mel spectral features extracted at 100 Hz. A 1024-point short-time Fourier transform (STFT) is applied with 10 ms hop size and 40 ms analysis window. A Hann windowing function is then used, followed by an 80-dimensional Mel filter with a cutoff frequency of 8 kHz. We used the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) to phonemize and force-align the transcripts, utilizing the MFA phone set, a modified version of the International Phonetic Alphabet (IPA), while also applying word position prefixes.
A.3. Model Configurations
We applied both baseline machine unlearning methods and the proposed method to VoiceBox (Le et al., 2024), using the same configuration. The audio feature generator based on vanilla Transformer (Vaswani, 2017), enhanced with U-Net style residual connections, convolutional positional embeddings (Baevski et al., 2020). and AliBi positional encoding (Press al., 2022). This model has 24 Transformer layers, 16 attention heads, and an embecding/feed-foowaad network (FFN) dimension of 1024/4096, with skip connections implemented in the U-Net style.
A.4. Duration Predictor and Vocoder
We used the regression version of duration predictor proposed in (I.e al., 2024). The duration predictor has similar model structure the audio model, but with 8 Transformer layers, attention heads, and 512/2048 embedding/FFN dimensions. It is trained for 600K steps. The Adam optimizer was employed with peak learning rate of le-4, linearly warmed up over the first 5K steps and decayed afterward. HiFi-GAN (Kong et al., 2020), trained on the LibriHeavy (Kang et al., 2024) English speech dataset, is employed to convert the spectrogram into a time-domain waveform.
A.5. Pre-training
Following (Le et al., 2024), we trained the original Voice model for 500K steps. Each mini-batch consisted of 75-second audio segments, and the Adam optimizer was employed with a peak learning rate of le-4, linearly warmed up over the first 5K steps and decayed afterward. All training was conducted using mixed precision with FP16.
A.6. Inference Configurations
During inference, classifier-fire guidance (CFG, (Ho & Salimans, 2022; Le et al., 2024)) was applied as follows:
( x. y; 0) (1 ai) vx vx(w,, Feta: Fetx y; 0) v (wr; 0)
(13)
where a is fixed at 0.7, as specified in the original paper. Refer to Appendix F for information on the impact of a.
We utilized the torchdiffeg package (Chen, 2018), which offers both fixed and adaptive step ODE solvers, using the default midpoint solver. The number of function evaluations (NFEs) was fixed at 32 for both the evaluation stage and the
13

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
generation of in the proposed method. The Ground Truth for WER is obtained by transcribing the target speech using the Automatic Speech Recognition (ASR) model (Hsu et al., 2021), then comparing the ASR result to the target speech transcription.
B. Unlearning Implementations
B.1. Teacher-Guided Unlearning
The Teacher-Guided Unlearning (TGU) model was trained for 145K steps for and 10K steps for 2. Each mini-batch included 75-second audio segments. The Adam optimizer was employed with a peak learning rate of le-4, which was linearly warmed up during the f K steps and subsequently decayed throughout the remainder of the training. To facilitate the unlearning process, samples from the forget set xf were randomly selected with 20% probability in each mini-batch.
B.2. Sample-Guided Unlearning
To apply Sample-Guided Unlearning (SGU) in the ZS-TTS system, we set up the training process such that when forget sample x is provided, a random retain sample x" is selected the target for training. To train VoiceBox, both speech data and aligned text segments are required. However, as discussed in Section 4. it is not naturally feasible to collect utterances from different speakers that share the same alignment. To address this, the SGU training was set up as follows: Let y and y" represent the corresponding text segments for x and x", respectively. We generated a mask corresponding to the length of x", training the model to predict x' based on this masked input. The text segments y' and y" were concatenated along the time axis and used as input, with the same process applied to the other input components, such as wf and w". During the training phase, the model was fine-tuned using 145K steps for and 10K steps for 2. Additionally, forget samples x and remain samples x" were selected and trained in 2:8 ratio.
B.3. Exact Unlearning & Fine-Tuning
The Exact Unlearning method was trained with the same configuration as the pre-training, except that only the dataset D was used. Similarly, the Fine Tuning method involved additional training for 145K steps, exclusively using the dataset Dr.
B.4. Negative Gradient
Implementation of Negative Gradient (NG) method follows that of (Thudi et al., 2022). On the pre-trained VoiceBox model, provide only the samples from the forget speaker set F. The loss is inverted to counteract loss minimization previously occurred in the pre- mre-trained model's weights. Given that approaches based reversing the gradient often suffer from low model performance and unstable training, we searched for learning rate with best evaluation score {le-5, 1e-6, 1e-7, 1e-8). For evaluation, we use the checkpoint of 9.5K fine-tuned with Adam optimizer with a peak learning rate of 1e-8, linearly warmed up over first 5K steps and decayed after.
B.5. Selective Kullback-Leibler Divergence
Numerous studies have adopted a loss function that focuses on utilizing a teacher-student framework with selective Kullback- Leibler divergence loss (Li et al., 2024; Chen Yang. 2023). We implement this loss so the student model is fine-tuned to maximize KL--divergence between teacher and student output when x/ given as input, and minimize when is given
LKL
(+', (1- (-)))((s), 0/1100 (xl,
(14)
where À is a hyper-parameter between 0 and 1 to balance the trade-off. Similar to NG, unbounded reverted loss on KL-divergence is prone to low model performance. We searched for learning rate with best evaluation score from (le-5, 1c-6, le-7, 1c-8), and from (0.5, 0.8). For evaluation, we use the checkpoint of 32.5K fine-tuned with Adam optimizer with a peak learning rate of 1e-8, following warm up and decay of previous methods using À = 0.5.
14

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
C. Speaker Similarity in Real Samples
0.8-
Q3: 0.76
Mean: 0.86
0.6
Q1:
SIM Scores
0.4
0.2
Q3: 0.17
Mean: 0.09
Q1: 0.02
0.0
Same Speaker Speech Pair
Different Speakers' Speech Pair
Figure Boxplot of speaker similarity on same speaker's and different speakers' audio. Each are evaluated with 100 pairs random speech audio in LibriSpeech test-clean subset.
From the LibriSpeech dataset, we make extensive analysis to get grip actual speaker similarity scores between pairs of audios from the same speaker, and that consisting different speakers. For the SIM of same speakers, we retrieved random 100 pairs of audio, each pair comprised of different audio from random speaker. For the SIM of different speakers, similarly, we retrieved random 100 pairs of audio, with each pair comprised of audio from different speakers.
As shown in Figure 4, audios with same speaker's voice return SIM with 0.66 as mean, 0.57 and 0.76 each being lower and upper quartiles. With different speakers, mean of SIM 0.09, lower and upper quartiles are 0.02 and 0.17. We take these values into consideration when evaluating Table and Table 2. While actual values can have a wider range, we focus on the lower and upper quartiles as a primary boundary to achieve in unlearned models.
15

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Specch
D. Quantitative Results Over the Training Process
SGU
SGU
TGU
WER-R
6-
WER-F
4-
20
40
60
80
100
20
60
80
100
Training Prucess (%)
Training Process (%)
(a) WER-R
(b) WER-F
SGU
SGU
TGU
TGU
0.6
0.6
SIM-R
SIM-F
0.4
0.4
20
40
60
80
100
20
40
60
80
100
Training Process
Training Process (%)
(c) SIM-R
(d) SIM-F
Figure 5. Quantitative results for SGU and TGU across different training stages. The top row shows the WER for both methods, while the bottom row displays the SIM results at each stage of the training process.
Figure depicts the training process of our two proposed methods: SGU and TGU in Table 1. We evaluate the unlearning model's checkpoints at every 10% full iterations. Notably, SIM score for the forget set declines quickly within first 10% of steps. However, SIM score for the remain set also declines the early unlearning process with the remaining process improving SIM-R.
Also, for WER scores for both remain and the forget remains relatively stable for both SGU and TGU. This suggests that guided unlearning method is highly effective in maintaining model performance in generating accurate speech on the given target text. It can also be interpreted that guided unlearning method is successful in disentangling speaker specific speech features from model's knowledge of correct speech generation.
E. Statistical Significance of Experiments
Table 5 depicts the statistical significance analysis results of the paper's main experiments. The results reported in Table I and Table 2 were evaluated with one-way ANOVA to assess how the unlearning method influences content correctness (WER), speaker similarity (SIM), and randomness in speaker identity (spk-ZRF). In Table 1, the analysis reveals a significant effect of the method on all metrics, demonstrating that the chosen unlcarning strategy impacts content accuracy and speaker similarity. By contrast, in Table 2, low significance is observed WER, indicating that two of our methods are comparable in terms of generating correct content. Nonetheless, the significant difference in SIM confirms that TGU is a more effective method for speaker identity unlearning.
16

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
Table One-way ANOVA statisties for effect unlearning method. within-methods degrees freedom are df2 768 for analyses remain speakers (-R) and df2 1188 for forget speakers (-F). "'p < .001.
Tables
WER-R
SIM-R
WER-F
SIM-F
spk-ZRF-R
spk-ZRF-F
Table
3900.01 ***
3275.76** 76
71.64
501.71 1*x
116.31
807.97
Table
2.71
174.58*
2.80
7.44***
F. Impact of
Table 6. Quantitative results based on the alpha value of CFG during the TGU inference process
WER-R +
SIM-R f
WER-F +
SIM-F +
a 0.0
3.4
0.552
2.6
0.265
a 0.3
2.6
0,583
2.3
0.198
0.7
2.4
0.631
2.4
0.169
a: 1.0
2.5
0.629
2.4
0.187
In the CFG used during inference, vy (w; 0) does not incorporate linguistic information or the surrounding audio context Tetas making it relevant to our formulation. To assess the impact of CFG on unlearning, we experimented with different values of a. Table presents the results of these experiments.
According the results, when is set O, removing the influence of ue (w; 0). the model showed the highest SIM-F value, indicating increased reliance Eata- On the other hand, when was set to 0.3 or higher, the model consistently produced lower SIM-F values.
G. Qualitative Evaluation Instruction
Table 7 and Table 8 present the instructions used for evaluating CMOS and SMOS in the qualitative assessment. Both the CMOS and SMOS evaluations were conducted with 25 participants.
Table 7. Comparative mean opinion score (CMOS) Instruction
Introduction
Your task to evaluate how the quality of two speech recordings compares, using the Comparative mean opinion score (CMOS) scale.
Task Instructions
In this task, you will hear two samples of speech recordings, one from each system. The purpose of this test is to evaluate the difference in quality between the two files.
Specifically, you should assess the quality and intelligibility each file terms of its overall sound quality and the amount of mumbling and unclear phrases the recording.
You should give score according to the following scale: (System much worse)
-2 (System 2 is worse)
-1 (System 2 is slightly worse)
0 (No difference)
1 (System 2 slightly better)
2 (System 2 better)
3 (System 2 much better)
17

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Texe-to-Speech
Table 8. Similarity mean opinion score (SMOS) Instruction
Introduction
Your task is to evaluate how similar the two speech recordings sound in terms of the speaker's voice.
Task Instructions
In this task you will hear two samples of speech recordings.
The purpose of this test is evaluate the similarity of the speaker's voice between the two files.
You should focus on the similarity of the speaker,
speaking style, acoustic conditions, background noise, etc.
You should give a score according to the following scale:
5 (Very Similar)
4 (Similar)
3 (Neutral)
2 (Not very similar)
1 (Not similar at all)
G.1. Demographies of Human Evaluators
To assess the quality of synthesized speech, we conducted quantitative evaluation with total of 25 participants. Participants were recruited for individuals physically and cognitively capable of normal activities with ages between 20 and 45 years with high proficiency in English. Recruitment and study procedures adhered to Institutional Review Board guidelines, and all participants provided informed consent, Additionally, all participants were general listeners with no prior expertise in audio or speech synthesis.
G.2. Evaluation Conditions
All participants completed brief instructive session with an evaluator to familiarize themselves with the evaluation criteria. Evaluation was conducted in a quiet enclosed environment with the same listening device and volume levels, under the instructions of Table and Table Each evaluation took less than 10 minutes.
H. Experiment on Unlearning Robustness
While Table shows that TGU has effectively unlearned in overall, we go through extensive experiments to evaluate unlearning robustness. Figure 6 illustrates how TGU unlearned model behaves on remain speakers' speech prompts with various similarity scores to forget speaker's speech prompt. As unlearning specifically on forget speakers is our objective in speaker identity unlearning, we expect the model clearly classify forget speakers and remain speakers despite possible resemblances of each other.
For the x-axis, we identified speech prompts in remain set and the highest speaker similarity (SIM) score with any forget speech prompt. Then, the same remain speech prompts were used to generate speech with TGU unlearned model. The y-aixs was then obtained, by comparing the speech prompt with its TGU generated output speech. The results are visualized on 6.
Pearson correlation analysis was conducted to assess the relationship between the similarity of remain speech prompts to forget speech prompts (x-axis) and the similarity of remain speech prompts to SGU-generated speech output (y-axis). The obtained statistic 0.1396 while the p-value is 0.0003. This indicates weak positive correlation with statistical significance. meaning that TGU generated speech is generally independent of the remain samples' similarity forget speakers. Had the model not been robust and mistreated remain samples as forget speaker samples, there would have been strong negative correlation.
In Table 9, we further assess the model's robustness by comparing its behavior on remain speakers with similar vocal characteristics to forget speakers. First, we compute the speaker similarity (SIM) between utterances from the two groups. We select utterances whose similarity to any forget-speaker utterance exceeds 0.40 and use these as prompts our evaluation.
18

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
Remain Speech Prompt vs Generated SIM
0.8
0.7
Avg.
SIM
0.6
0.5
0.4
0.3
0.1
0.2
0.3
0.4
0.5
Remain Speech Prompt vs Forget Speech Prompt SIM
Figure 6. Robustness scatterplot of TGU remain speakers. The x-axis represents the maximum SIM score between the remain speech prompt and forget speech prompt to depict the of similarity hetween remain speaker and a forget speaker. The y-axis represents the similarity score between the remain speech prompt and its resulting generated output using TGU. The red dashed line indicates average SIM score for all remain speech prompts the evaluation set.
Table Quantitative results on LibriSpeech test-clean evaluation set (-R) which show high speaker similarity to any forget speaker utterances (exceeding 0.40 SIM).
Methods
WER-R
SIM-R t
Original
4.96
0.637
NG
6.67
0.393
KL
8.78
0.316
SGU (ours)
5.70
0.411
TGU (ours)
4.70
0.622
Ground Truth
2.94
19

---

Do Not Minic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Specch
Table 10. Transient noise removal results on LibriSpeech test-clean
Methods
WER.
SIM
Clean speech
4.3
0.689
Noisy speech
47.9
0.213
Original
2.4
0.666
TGU (ours)
2.5
0.641
Even when remain speaker's voice closely resembles that of a forget speaker, TGU maintains its performance with the original model. This demonstrates TGU's ability to preserve the identity of remain speakers while effectively neutralizing traces of forget speakers. In constrast, NG, KL and SGU significantly drop SIM-R (drops of 0.226-0.321), suggesting unlcarning using these methods may trigger a trade-off that sacrifices remain speaker speech synthesis.
I. Experiment on General Tasks
To provide deeper insights how TGU unlearning may affect model performances on general tasks where ZS-TTS is used, we experiment the original model and TGU on transient noise removal.
L.1. Transient Noise Removal
ZS-TTS can be applied in tasks where editing is required to remove undesired noise in speech datasets. To prevent having to go through repetitive and inefficient recording to obtain clean speech, ZS-TTS can generate clean audio for the noisy segment. We follow experimental settings of (Le et al., 2024) to analyze how TGU unlearned model performs on the task of transient noise removal.
From LibriSpeech test-clean dataset samples of durations to 10 seconds, we construct noise at -10BB gggnal-to-noise ratio over half of each sample's duration. Table 10 suggests that TGU provides comparable performances to that of the original model. While seemingly low, diminished model performances on transient noise removal is present relatively the original model. We suggest that this is trade-off from successful unlearning. While the model has unlearned generate voice characteristics of the forget dataset, smaller knowledge-bart base and implemented randomness could have affected its reconstructing abilities.
1.2. Diverse Speech Sampling
Being able to generate diverse speech is also an important feature of ZS-TTS models as it ensures realistic and high-quality speech that resembles natural distributions. This is necessary in applications such as speech synthesis or generating training data for speech related tasks (e.g., Automatic Speech Recognition). The diversity of generated speech samples is measured with Fréchet Speech Distance (FSD) as suggested (Le et al., 2024). From generated speech samples, we extracted self-supervised features using 6th layer representation of wav2vec 2.0 (Baevski et al., 2020). The features were reduced to 128 dimensions with principle component analysis and used to calculate ate the similarity of distributions with real speech. High FSD indicates lower quality and minimal diversity, while low FSD refers to high quality and more diversity. For this experiment, is set to 0 to ensure more diversity. Ground truth FSD is obtained by partitioning the LibriSpeech test-other set into half while ensuring equal distribution of data per speaker across both subsets
Experimental results in Table 11 show that FSD increases in TGU unlearned model. Because this task does not require input audio prompts, diverse speech sampling relies relatively heavier on datasets used to train the model. Implementing machine unlearning and thus inducing forgetting of specific speakers causes trade-off in model's diversity. Meanwhile, is noticeable that TGU achieves lower WER in this case. We can infer that TGU obtains robustness in relatively noisy dataset comparable to the Original model.
J. Recovery Experiment
Table 12 illustrates an experimental result on whether an unlearned model is recoverable to its original state. Aligning with our motivation to make ZS-TTS models safe, we presume a scenario of a privacy attacker who attempts to retrieve the
20

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
Table 11. Diverse speech sampling results LibriSpeech test-other evaluation
Methods
WER
FSD +
Ground truth
4.5
164.4
Original
8.0
170.2
TGU (ours)
7.9
177.8
Table 12. Quantitative results for recovery experiments unlearned models. WER and SIM evaluation follows the procedures of Table1.
Methods
Recover Steps
Audio per Spk
WER-R +
SIM-R t
WER-F 4
SIM-F
Original TGU
15 min
2.1
0.649
2.1
0.708
15 min
2.5
0.631
2.4
0.169
TGU
36.25K
15 min
4.23
0.303
2.5
0.735
TGU
14.5K
1 min
4.61
0.226
2.8
0.162
original model parameters. We train the TGU unlearned checkpoints on all 10 of forget speaker's dataset to recover the original model. We also presume a practical scenario and attempt to recover the model performance using average of I minute for each speaker.
When given audio duration of minutes for the speakers, the model fails to generalize over other speakers, hence, failing to mimic voices other than the forget speaker's. Additionally, the recovered model is more likely generate wrong speech content shown with higher WER in both remain set and the forget set. This process resembles fine-tuning Texeet--Speech model for specific speakers rather than true recovery. Consequently, the original ZS-TT3 model cannot be restored, and the attacker is essentially leveraging transfer learning to create forget speaker-specific TTS model. However, with enough training data, the attacker could achieve similar results using any other non-zero-shot TTS model. We also consider a scenario where an attacker has access to only 1 minute of the forget speaker's voice sample. In this case, the model parameters also remain unrecoverable. The model also fails to generate forget speaker's 's voice. The model loses its zero-shot abilities hence the performance at early steps. Therefore, in practical scenarios where an attacker may attempt to train the model to clone individual's 's voice with short sample of speech (e.g., voice phishing), it would be feasible recover the model or successfully generate the forget speaker's voice.
K. Reproducibility Experiment
Table 13 illustrates the reproducibility our experiment of Table using a different dataset, LibriTTS (Zen et al., 2019). We pre-trained VoiceBox model on LibriTTS 500K steps and 10 speakers were randomly selected as forget set. When performing unlearning for only 10K steps (7% of pre-training steps), results of TGU illustrate effective unlearning while maintaining content accuracy.
L. Inference Samples
Figures 7 and show the Mel-spectrograms for the ground truth, original VoiceBox, SGU, and TGU inference results on forget speaker samples. These figures represent samples from speakers 789 and 6821, respectively. The ground truth Mel-spectrogram corresponds to the audio where the same speaker the prompt reads the same transcription.
21

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Specch
Table 13. Quantitative results for reproducibility experiments using LibriTTS as pre-train datasct.
Methods
Unlearn Steps
WER-R
SIM-R
WER-F
SIM-F t
Original
3.2
0.610
6.3
0.503
TGU
10K
3.3
0.548
6.4
0.184
(a) Ground Truth
(b) Original
(c) SGU Sample
(d) SGU Sample 2
(c) TGU Sample
(f TGU Sample 2
Figure 7. Mel-Spectrogram Comparisons: GT, Original, SGU Samples, and TGU Samples for the forget speaker 789
22

---

Do Not Mimic My Voice: Speaker Identity Unlcarning for Zero-Shot Text-to-Speech
(a) Ground Truth
(h) Original
(c) SGU Sample 1
(d) SGU Sample 2
(e) TGU Sample I
(f) TGU Sample
Figure 8. Mel-Speectogram Comparisons: GT, Original, SGU Samples, and TGU Samples for the forget speaker 682/
23


Reviews:
[{"review_id": "g9Ti9Lu27X", "reviewer": "Reviewer_k36c", "summary": "The paper presents two machine unlearning methods for zero-short TTS, with the main one called teacher-guided unlearning (TGU). This is an under-explored topic/task, which is of interest and importance. It further proposes a new evaluation metric for this task. A set of experiments were conducted to validate the performance of the proposed method.\n\n## update after rebuttal\nI appreciate that the authors conducted additional experiments to address my comments. I will maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer k36c's insightful comments and thorough review. Below, we address each comment clearly and concisely.\n\n## **1. Clarification and Typos** \nAs you correctly indicated, Exact Unlearning is an exact method rather than approximate. We will correct this with other typos and clearly state the distinction for revision. For supplementary, samples from Exact Unlearning and Fine Tuning methods will be included.\n\n## **2. SIM-R and spk-ZRF-R**\nPlease refer to Section 4. of our answer to 4th reviewer 87b9.\n\n## **3. Statistical Significance**\nHere we provide statistical significance test results for Table 1 and Table 2. The result of ANOVA test on SIM and WER validates that the unlearning methods are effect SIM and WER.\nIn Table 2, low significance in WER-R and WER-F validates that SGU and TGU both maintain comparable content accuracy. However, significance in SIM show that TGU is a more effective method.\n\n||WER-R|SIM-R|WER-F|SIM-F|\n|---|---|---|---|---|\n|Table 1|F(4, 768) = 3900.01 , p < 0.0001|F(4, 768) = 3275.76 , p < 0.0001|F(4, 1188) = 71.64, p < 0.0001|F(4, 1188) = 501.71, p < 0.0001|\n|Table 2|F(1, 768) = 2.71, p > 0.01|F(4, 768) = 174.58, p < 0.0001|F(4, 1188) = 2.80, p > 0.01|F(4, 1188) = 7.44, p < 0.0001|\n\n## **4. Ground Truth for WER**\n\nWord Error Rate (WER) is widely used in speech synthesis domain to analyze accuracy in content generation [1]-[5]. The Ground Truth for WER in **Table 1** is obtained by transcribing the target speech using the Automatic Speech Recognition model [6], then comparing the ASR result to the  target speech transcription. The target speech is obtained from the test dataset LibriSpeech test-clean [7].\n\n[1] Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale, NeurIPS, 2023\n[2] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizer, Arxiv, 2023\n[3] F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, aAxiv, 2024\n[4] Maskgct: Zero-shot text-to-speech with masked generative codec transformer, ICLR, 2025\n[5] CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens, 2024\n[6] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, 2021\n[7] Librispeech: An ASR corpus based on public domain audio books, ICASSP, 2015\n\n## **5. Missing Results in Table 3**\nThank you for your feedback. We initially focused on methods with noticeable changes in SIM-F, but we acknowledge the value of assessing Exact Unlearning and Fine Tuning.\nFollowing Institutional Review Board (IRB) guidelines to ensure ethical evaluation takes time, and we will add these results in camera-ready version. \n\n## **6. Discussion on Higher SIM-F**\nWhile SIM-R measures the zero-shot voice replicability on completely unseen speakers [9], SIM-F is evaluated with the test set from data [8] the model was pre-trained on. In the case of Original, SIM-F is inevitably higher due to similarity in data distribution with the train set. Exact Unlearning and Fine Tuning both fail to unlearn due to model's zero-shot capabilities, thus for similar reasons, SIM-F scores are higher.\n\n[8] Libriheavy: a 50,000 hours asr corpus with punctuation casing and context, ICASSP, 2024\n\n## **7.Unlearning Unseen Speakers**\nWe experiment this Out-of-Domain Unlearning [9] scenario where the challenge precisely lies in the fact that ZS-TTS can replicate voices without explicit training data for speakers. We unlearn speaker identity never used in training and results demonstrate that our unlearning method is effective, even for unseen speakers.\n\n|Dataset|WER-R|SIM-R|WER-F|SIM-F|\n|---|---|---|---|---|\n|Original|2.1|0.649|5.13|0.678|\n|SGU|2.9|0.602|5.5|0.157|\n|TGU|2.5|0.63|5.32|0.186|\n|Ground Truth| 2.2||5.93||\n\n[9] Generative Unlearning for Any Identity. CVPR, 2024.\n[10] LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech, Interppeech, 2019\n\n## **8. Revisiting Machine Unlearning**\n\nWe adopt the task of machine unlearning for Zero-Shot TTS, and introduce newly formulated problem as Speaker Identity Unlearning (SIU) in Section 3.2\n\nSIU directly adjusts model parameters to remove the capability to reconstruct forget speaker’s voice. This is consistent with recent trends in generative unlearning. A recent study [9] demonstrates that unlearning can be applied to prevent identities from being visually generated. Similarly, our work prevents identities from being synthesized upon request. Also, the goal of machine unlearning is to safeproof model parameters from revealing information. Same goal is applied in SIU, with notable results in preventing traceability of forget set. Our methods are consistent with machine unlearning's problem formulation; prevent revealing unwanted identities, while retaining model performance on remain set.", "claims_and_evidence": "The paper presents a set of experiments, which clearly support the claims.", "methods_and_evaluation_criteria": "Experiments were conducted on LibriSpeech. The experiments are generally considered sufficient. \n\nSeveral metrics are used, including WER, SIM, spk-ZRF, CMOS and SMOS. \n\n\"As a baseline, we applied four different approximate machine unlearning methods .... First, the Exact Unlearning method\"\n- This is incorrect as \"the Exact Unlearning method\" is not an approximate machine unlearning method. \n \nSpeaker similarity metric is not defined. \n\nA set of demos were provided. The comparison with NG and KL is convincing. However, the authors didn't provide examples for exact unlearning and fine tuning.", "theoretical_claims": "The paper does not have theoretical claims. The proposed methods were experimental evaluated.", "experimental_designs_or_analyses": "In Table 1, the reason why the TGU row and the spk-ZRF-R column is highlighted is because it is the same as original, while the others are highlighted because they give either lowest or highest numbers. Then, it is unclear why the TGU row and SIM-R column is highlighted with bold font. \n\nIn Table 1, no statistical significant test results are provided for WER and SIM metrics. Same in Table 2. \n  \nWhat is the ground truth for WER? How to obtain them? \n\nNo results for exact unlearning and fine-tuning are provided in Table 3. \n\nWhy the scores of SIM-F are higher than those of SIM-R for both Exact Unlearning and Fine Tuning methods? Good to discuss.", "relation_to_broader_scientific_literature": "Primarily for speaker identity unlearning for TTS.", "essential_references_not_discussed": "Sufficient references.", "other_strengths_and_weaknesses": "Strengths:\nThis work explores a topic that is under-explored in the literature. \nIt proposes a metric for this task. \nThe results obtained are convincing. \n\nWeaknesses\nThe experimental part could be enhanced, including the presentation and result discussion.", "other_comments_or_suggestions": "\"for retain speakers\" -> \"for retained speakers\"\n\nx is an audio prompt uttered by s ∈S\n->\nx^s is an audio prompt uttered by s ∈S", "questions_for_authors": "The authors state that \"unlearning in ZS-TTS presents unique challenges because the model can replicate speaker identities in a zero-shot manner, even without direct training on specific speaker data.\" Then, how unlearning is able to help when a specific speaker data is not used for training? Or, is this still a machine unlearning problem/task? \n\nWhen introducing randomness in synthesizing speech, the proposed methods deliberately avoid the specific speaker, which seems no longer to be machine unlearning. Also, does this exactly reveal what specific identities are avoided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vYjk7vJxCA", "reviewer": "Reviewer_bkoi", "summary": "The paper introduces Teacher-Guided Unlearning (TGU), a method to remove specific speaker identities from zero-shot text-to-speech (ZS-TTS) models to address privacy concerns. TGU guides the model to generate random voice styles for \"forgotten\" speakers while retaining synthesis quality for others. A new metric, speaker-Zero Retrain Forgetting (spk-ZRF), measures randomness in generated voices for forgotten speakers. Experiments on Voice-Box show TGU reduces similarity to forgotten speakers while maintaining performance on retained speakers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer bkoi for the constructive feedback and valuable insights. We address the primary concerns as follows:\n\n## **1. Practical Use Case**\nThe 145K steps used during unlearning processes were chosen based on the common fine-tuning practices when adapting Flow-Matching-based pre-trained models to the Zero-Shot TTS domain [11]. In this context, such a step count is not considered excessive. Furthermore, as presented in Appendix D, we provide performance curves throughout the unlearning process. Notably, we observe that after approximately 58K steps (about 11% of the pre-training steps), the model already achieves sufficient unlearning performance with SIM-R and SIM-F scores reaching 0.588 and 0.174, respectively. Importantly, we also confirm that WER-R and WER-F remain stable throughout the entire unlearning process. \n\n[11] Generative Pre-training for Speech with Flow Matching, ICLR, 2024\n\n## **2. Adversarial Attacks**\n\nWe agree that additional adversarial robustness evaluations would strengthen our method's validity. We plan to experiment this using adversarial attack methods in TTS to validate spk-ZRF’s resistance in the following week.\n\n## **3. Latest Unlearning Methods**\n\nWe provide results of latest machine unlearning methods with best scores of each methods in unlearning a single identity.\n|methods|WER-R|SIM-R|WER-F|SIM-F|\n|---|---|---|---|---|\n|SFRON [12]|36.60|0.186|68.61|0.091|\n|SalUn [13]|2.9|**0.637**|**2.3**|0.695|\n|SGU (Ours)|2.6|0.523|2.5|0.194|\n|TGU (Ours)|**2.5**|0.631|2.4|**0.169**|\n\nThe poor results in [12] and [13] are expected as per the domain characteristics of ZSTTS. Recent architectures do not learn ‘speaker identity’ and ‘content generation’ separately. The features are deeply entangled within model parameters. Thus, removing the ability to mimic specific speaker identities without harming knowledge for content generation comes with an implicit but challenging task ; fine-grained feature disentanglement.\n\nCurrent saliency-based unlearning methods perform poorly. In [4], even though remain and forget set were both used to create separate fisher maps - unlearning in forget set still impacted performance for remain set. In [5], random labeling with unaligned content as label is used alongside fisher map from forget set. In this case, the model was updated minimally.\n\nWe agree with your insights in adapting latest machine unlearning methods that can be adopted alongside Guided Unlearning. We plan to experiment saliency-based approach on SGU and TGU in the following week.\n\n[12] Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement\n[13] SalUn:Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation\n\n## **4. Speaker Verification Model**\nPlease refer to Section 2 in our response to 4th Reviewer (87b9)\n\n### 5. Scalability\nHere, we provide an updated version of Table 2. K refers to the of forget speakers in the forget set.\n|Methods|WER-R|SIM-R|WER-F|SIM-F|\n|---|---|---|---|---|\n|SGU (K=1)|2.7|0.586|2.8|0.173|\n|SGU (K=3)|2.9|0.566|2.7|0.209|\n|SGU (K=10)|2.6|0.523|2.5|0.194|\n|TGU (K=1)|2.3|0.624|2.5|0.164|\n|TGU (K=3)|2.9|0.626|2.3|0.159|\n|TGU (K=10)|2.5|0.631|2.4|0.169\n\nTo analyze the impact of the number of forget speakers on model performance, we conducted  analysis by varying the forget speaker set size. As shown, our model maintains robust performance.\n\nImportantly, using 10 forget speakers is not a small-scale scenario compared to prior generative unlearning studies [10,12-13], where the forget set sizes are generally 1, and at most, 3. We plan to report our active experiments on 50 forget speakers in the camera-ready version.\n\n[12]  Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models. arXiv, 2023.\n\n[13] Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient, AAAI, 2025\n\n## **6. Similar Voice and Robustness**\n\nIn Appendix G, we provide a figure illustrating TGU's performance on remain speakers with varying similarity to forget speakers. Pearson analysis (F=0.1396, p>0.0003) validates that even when a remain speaker may have vocal features similar to a forget speaker, this does not affect model's performance for remain speakers.\nWe also provide comparisons of model performance for robustness. We obtain SIM between samples of remain speakers and forget speakers. Then, we identified samples of remain speakers that show high similarity ( > 0.4 ) to forget speakers. We use these samples as input prompt for evaluation.\nWe will update the project page and supplementary to contain samples of similar speakers, and the generated output of each method.\n\n||WER-R|SIM-R|\n|---|---|---|\n|Original|4.96|0.6365|\n|NG|6.67|0.393|\n|KL|8.78|0.316|\n|SGU|5.70|0.411|\n|TGU|4.70|0.622|\n|Ground Truth|2.94||\n\n## **7. Generalizability**\nPlease refer to Section 4 in our response to 3rd Reviewer (d6q4)z\n## **8.spk-ZRF**\nPlease refer to Section 4 in our response to 4th reviewer", "claims_and_evidence": "`TGU requires only 5 minutes of audio per forgotten speaker (line 320 page 6)’: The method assumes access to the original training data and extensive retraining (around 25% of pre-training iterations), which is impractical for real-world deployment.", "methods_and_evaluation_criteria": "1. A major weakness of this work is the lack of a clear practical use case. Given that the proposed method requires 5 minutes of the original training audio from the forget speaker and incurs significant computational overhead, its practicality remains questionable. \n\n2. The proposed spk-ZRF metric evaluates privacy risks solely through similarity and randomness measures but fails to validate practical resistance to adversarial attacks (e.g., reconstructing original voices via adversarial examples). Additionally, its reliance on pre-trained speaker verification models introduces bias risks—if these models are flawed or biased, the metric’s reliability is compromised.", "theoretical_claims": "There are no theorems or propositions in this paper; therefore, there are no proofs.", "experimental_designs_or_analyses": "1. The biggest issue with the experiment is that the validation is conducted on only a closed-source model, VoiceBox.\n\n2. The paper lacks comparisons with some of the latest machine unlearning methods, such as [A-C].\n\n[A]Low Compute Unlearning via Sparse Representations, arxiv, 2023\n\n[B]Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement, neurips, 2024\n\n[C]SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation,iclr,2024", "relation_to_broader_scientific_literature": "The contributions of this paper is about the safefy of AI mdols. (Speaker Identity Unlearning of text-to-speech models)", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths \n\n1. This is the first work on speaker unlearning in ZS-TTS and also proposes a new metric, spk-ZRF, to evaluate the reconstructability of the target speaker, offering a novel perspective to the unlearning field.\n\n2. The paper is well-structured, with clearly defined symbols and concepts, making it easy to follow.\n\nWeaknesses\n\n1. TGU requires access to the original training dataset and extensive retraining (compared to the pre-training stage), which conflicts with real-world scenarios where data may be unavailable due to privacy regulations.\n\n2. Experiments are conducted only on Voice-Box, a closed-source model. Recent ZS-TTS models (e.g., YourTTS[D], CosyVoice[E], MaskGCT[F], F5TTS[G]) are not tested, raising concerns about the generalizability of the proposed method.\n\n## update after rebuttal\nAll of my concerns have been addressed in the rebuttal, I decided to improve my score.", "other_comments_or_suggestions": "The spk-ZRF-R metric in Table 1 is missing an arrow indicator.", "questions_for_authors": "1. How does SGU/TGU scale to more than 10 (e.g., 50) forgotten speakers while maintaining retain-set performance? In image classification tasks, unlearning methods are typically evaluated by increasing the count of the forget set from 1 to N to assess their scalability. \n\n2. Can SGU/TGU handle cases where a retained speaker’s voice is accidentally similar to a forgotten one? Would this scenario affect the generation quality of retained speakers whose voices closely resemble those in the forget set?\n\n3. Why was SGU/TGU not tested on open-source ZS-TTS models (e.g., YourTTS[D], CosyVoice[E], MaskGCT[F], F5TTS[G]) or more datasets? At the very least, evaluating the method on additional datasets and open-source models would help ensure the reproducibility of the proposed approach.\n\n[A]Low Compute Unlearning via Sparse Representations, arxiv, 2023\n\n[B]Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement, neurips, 2024\n\n[C]SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation,iclr,2024\n\n[D]YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone, PMLR, 2022\n\n[E]Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, arxiv, 2024\n\n[F]Maskgct: Zero-shot text-to-speech with masked generative codec transformer, ICLR, 2025\n\n[G]F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, arxiv, 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sa6TyHispZ", "reviewer": "Reviewer_d6q4", "summary": "The paper proposes Teacher-Guided Unlearning (TGU), a novel framework for removing specific speaker identities from Zero-Shot Text-to-Speech (ZS-TTS) models to mitigate privacy and ethical concerns related to unauthorized voice cloning. The method leverages controlled randomness guided by a teacher model during fine-tuning, which obscures specific speaker identities while retaining the model's capability to synthesize speech for non-forgotten speakers. Additionally, the paper introduces Speaker-Zero Retrain Forgetting (spk-ZRF), a new metric that quantifies how effectively a model \"forgets\" speaker identities. Extensive evaluations are presented, comparing TGU against multiple baseline methods, demonstrating promising results in unlearning effectiveness and retention of model performance on non-forgotten speakers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer d6q4 for the constructive feedback. Below we address each concern and clarify the raised points:\n\n## **1. Similar Voice and Robustness**\n\nPlease refer to Section 7 in our response to reviewer (bkoi)\n\n## **2. Computational Resources**\n\nPlease refer to Section 1 in our response to reviewer (bkoi)\n\n## **3. Scalability and Length of Audio Samples:**\n\nWe acknowledge concerns related to scalability, particularly regarding audio length requirements. From the perspective of service provider and user, we assume that the forget speakers voluntarily request their voices to be forgotten and are willing to provide audio samples specifically for the unlearning process.To alleviate the scalability concern under this assumption, we conducted experiments using data augmentation techniques leveraging the original model to generate additional training samples from as little as 1 minute of audio per forgotten speaker (as detailed in the \"TGU w/ Augmentation\" scenario). The results indicate substantial improvement in scalability, achieving a competitive SIM-R of 0.612 and SIM-F of 0.334 at only 48K training steps.\n\n| Method | Steps | Total Duration per Forget Speaker | WER-R | SIM-R | WER-F | SIM-F |\n| --- | --- | --- | --- | --- | --- | --- |\n| TGU | 145 K | 15 min | 2.5 | 0.631 | 2.4 | 0.169 |\n| TGU | 48 K | 15 min | 2.7 | 0.481 | 2.7 | 0.198 |\n| TGU w/ Augmentation | 48 K | 1 min | 2.6 | 0.612 | 2.6 | 0.334 |\n\nAlthough these results currently show room for further optimization, ongoing exploration into advanced data augmentation is expected to continue enhancing performance and practical viability, thus minimizing audio length constraints effectively.\n\n## **4. Generalizability**\n\n**Model Selection**\nWe selected VoiceBox due to its state-of-the-art (SoTA) performance at the time of conducting our research. We acknowledge the reviewer's concern regarding the generalizability and comprehensiveness of our evaluations. Given that MaskGCT [4] and F5TTS [3] were very recently introduced, we recognize their importance. F5TTS shares significant similarities with VoiceBox, being based on Flow-Matching with analogous model structures. We are currently preparing experiments using F5TTS to address this concern.\n\n**Reproducibility on Different Dataset**\n\nTo address the reviewer's concern, we conducted additional experiments using another dataset; LibriTTS [10]. We trained the VoiceBox model on LibriTTS and 10 speakers were randomly selected as forget set, following experimental settings of Appendix A. \n\n| Method | WER-R | SIM-R | WER-F | SIM-F |\n| --- | --- | --- | --- | --- |\n| Original | 3.2 | 0.610 | 6.3 | 0.503 |\n| TGU | 3.3 | 0.548 | 6.4 | 0.184 |\n\nDue to time constraints, only TGU was evaluated at **10k steps (7% of pre-training steps)**. \nWe plan to comprehensively evaluate additional baseline methods, including recent unlearning techniques, and add these results in the final camera-ready version of our manuscript.\n\n## **5. Recoverability Analysis:**\n\nThank you for finding our recoverability experiments valuable. We provide further analysis on recoverability experiments with additional metrics. Frechet Speech Distance (FSD) is used to measure Zero-Shot TTS’s ability to generate high quality speech with natural distributions. Low FSD indicates high quality and diversity in generated speech. Refer to Appendix H.2 for more information on FSD.\n\n| Method | Forget Audio Duration | WER-R | SIM-R | WER-F | SIM-F | FSD | spk-ZRF-R |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Original | - | 2.1 | 0.649 | 2.1 | 0.708 | 170.2 | 0.857 |\n| TGU | - | 2.5 | 0.631 | 2.4 | 0.169 | 177.8 | 0.857 |\n| TGU-recover | 1min | 4.6 | 0.226 | 2.8 | 0.162 |  |  |\n| TGU-recover | 15min | 4.2 | 0.303 | 2.5 | 0.735 | 189.5 | 0.846 |\n| Ground Truth |  | 2.2 |  | 2.5 |  |  |  |\n\nAccording to results, recovering the original model by re-training is not possible.\n\nFirst, when recovering with audio duration of 1 min, the model fails to mimic any speaker. It even shows lower performance on the forget set in SIM-F. \n\nSecond, when recovering with audio duration of 15 min, the model overfits to the forget speaker. Instead of recovering to Original, the  model is trained to be a speaker-specific TTS model. \n\n## **6. Human Evaluation:**\n\nPlease refer Appx F. and Section 1 in our response to reviewer (87b9).\n\n## **7. Scalability - Forget Size**\n\nPlease refer to our response to reviewer bkoi, specifically the section titled ‘**5. Scalability** ’\n\nAgain, we sincerely thank the reviewers for their insightful feedback, which will substantially strengthen the paper’s final version.", "claims_and_evidence": "The main claims made by the authors are supported by a clear experimental setup and a thorough quantitative evaluation. Key claims include:\n\n(1) **Effectiveness of TGU in speaker unlearning**: Experiments clearly demonstrate that TGU effectively reduces the similarity scores (SIM-F) between forgotten speaker prompts and synthesized speech, which validates the core unlearning claim. \n\n(2) **Retention of performance for remaining speakers**: The evidence shows that the similarity (SIM-R) and speech quality (WER-R) scores for retained speakers remain competitive relative to baseline methods. \n\nHowever, the claim about the robustness of the method when dealing with forgotten speakers similar in vocal characteristics to retained speakers is less convincing, as experiments reveal noticeable performance degradation under these circumstances.", "methods_and_evaluation_criteria": "**Strengths**\n- The introduction of the spk-ZRF metric represents a significant methodological contribution, specifically designed for evaluating speaker-specific unlearning in ZS-TTS, addressing a gap in existing evaluation methods.\n- The experimental evaluation strategy is generally robust, encompassing comparisons with well-established baseline methods (e.g., Exact Unlearning, Fine-Tuning, SGU, KL, NG).\n\n**Weaknesses**\n- The proposed method demands substantial computational resources (roughly 145K fine-tuning steps) and requires relatively long audio samples (approximately 5 minutes per speaker) to achieve reliable unlearning results, raising concerns regarding its real-world applicability.\n- While authors present attempts at reducing resource requirements (such as data augmentation for shorter audio), these measures do not yet fully resolve concerns around scalability and computational practicality.\nEvaluations conducted solely on a single proprietary architecture (VoiceBox) limit generalizability and broader applicability. Demonstrating effectiveness on multiple, especially open-source, architectures would significantly strengthen the validity of results.", "theoretical_claims": "The paper does not include formal theoretical proofs, thus no theoretical claims were assessed.", "experimental_designs_or_analyses": "The experiments conducted are generally sound and appropriate:\n\n- The use of well-established baseline methods for comparisons provides strong comparative validity.\n- The analysis of the new spk-ZRF metric is meaningful and clearly shows advantages over existing metrics in evaluating speaker-specific unlearning.\n\nHowever, some limitations persist in experimental analysis:\n\n- While data augmentation is introduced to handle shorter audio prompts (1 min per speaker), the observed performance degradation in these scenarios limits practical deployment.\n- Robustness analysis for retained speakers sharing vocal characteristics with forgotten speakers reveals limitations, suggesting room for further methodological improvements in addressing this scenario.\n- Recoverability experiments showing how easily unlearned speaker identities can be restored were valuable but somewhat preliminary; further systematic analysis would provide deeper insights.", "relation_to_broader_scientific_literature": "The paper provides a novel perspective within a nascent area—speaker identity unlearning in ZS-TTS. While general unlearning methods (NG, KL divergence-based unlearning, Exact Unlearning, and Fine-Tuning methods) are well-explored, TGU's application of randomness is novel and justified within the context. Nevertheless, broader literature on generative model unlearning (e.g., image domain unlearning like Stable Diffusion unlearning methods cited) shows that generative models can often manage effective unlearning with far fewer resources. Thus, this paper's methods appear somewhat costly and practically limited by comparison.", "essential_references_not_discussed": "Overall, essential related literature has been adequately referenced. However, considering the recent advancements in generative model unlearning across various modalities (e.g., diffusion models), additional comparative discussion with generative unlearning methods in other domains—such as recent image generation unlearning techniques—might provide broader insights into the generality of the introduced TGU framework and highlight potential cross-domain methodological inspirations.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper introduces a genuinely original problem (speaker identity unlearning in ZS-TTS), clearly motivated by real-world ethical/privacy implications.\n- The spk-ZRF metric is innovative and represents a methodological advance, explicitly tailored for the presented problem domain.\n- The presentation is clear and structured, with comprehensive supplementary details that facilitate a deeper understanding of the proposed method.\n\n**Weaknesses**\n- Practical feasibility remains a significant concern, given computational overhead, extensive fine-tuning requirements, and the sensitivity of the approach to the length of the speaker audio samples.\n- The limited evaluation conducted exclusively on the VoiceBox architecture restricts the scope of applicability; future work must include broader evaluations across diverse architectures.\n- The robustness issues uncovered in scenarios involving highly similar retained and forgotten speakers require more systematic investigation and possibly methodological refinement.", "other_comments_or_suggestions": "Please refer to the previous sections.", "questions_for_authors": "- Could you explicitly clarify whether you conducted subjective evaluations with proper ethical clearances (e.g., IRB approval)? This issue was raised previously and remains critical to the ethical soundness of the work.\n- Considering the significant computational overhead, could you briefly describe how feasible this approach would be in an industrial deployment scenario where numerous speaker identities might need unlearning frequently?\n- Could you clarify the rationale behind choosing only a 10-speaker \"forget set\" given the total dataset size (~50k hours)? What prevents you from testing larger-scale forgetting scenarios (e.g., 100+ speakers)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O26oP7X8QJ", "reviewer": "Reviewer_87b9", "summary": "This paper proposes a method to exclude specific speakers from zero-shot text-to-speech to preserve voice privacy. It utilizes a guided learning approach in which some other speakers' voices are used in mask-based learning, especially a teacher-guided learning method in which the target guidance is generated from random noise. A new evaluation metric, spk-ZRF, is also proposed. The effectiveness of this method was confirmed in the evaluation using the LibliHeavy dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer 87b9 for the thoughtful and constructive comments.\n\n## **1. Human Subjective Evaluation**\n\nWe deeply agree with the reviewer on that speaker similarity metric may not perfectly reflect human perception. Speech consists of pitch, prosody, and unique speaker characteristics.  To better capture human perception, we employed CMOS (Comparative Mean Opinion Score) and SMOS (Similarity Mean Opinion Score) in Table 3.\n\nCMOS was evaluated by asking participants to compare audio quality of outputs from the Original (Exactly Learned) model and outputs from SGU, TGU, and ground truth. \n\nSMOS was evaluated by presenting participants with two speech samples, and asking whether the two voices sound similar. For ground truth cases, we used two samples from the same speaker. For other cases, we used one real sample and one synthesized sample generated with same speaker’s prompt.\n\nThen, a well unlearned model should have very low SMOS-F and SIM-F, while keeping comparable scores to the Original model in SMOS-R and SIM-R. \n\nWe prepared 160 audio samples in total (10 randomly selected samples x 4 systems x 4 conditions). 25 participants with proficiency in English were recruited and compensated for the experiment. The detailed information on human evaluation are described in **Appendix F.**\n\n## **2. Forget ID Detection**\n\nAs pointed out, the challenge of detecting if the input prompt belongs to a forget speaker under limited data is a non-trivial. On top of that, our framework does not consist of an independent speaker classifier or encoder.\n\nInstead of relying on explicit forget speaker classifier or detector, our model implicitly identify the forget speakers by by directly updating its parameters through our proposed training framework. Specifically, when given a forget speaker’s prompt, the model generates speech with a voice different from that of the forget speaker by treating it as noise and suppressing its identity. Thus, as the model generates random voices for forget speaker prompts, it becomes practically infeasible to reconstruct their original voices, ensuring stronger privacy protection.We verify this with multiple experiments. With proposed spk-ZRF, we can evaluate that TGU acts significantly randomly on forget speaker prompt.\n\nTo further evaluate the robustness of our method against potential false positives, we conduct experiments using remain speakers who have high speaker similarity with the forget speakers. The results confirm that our model reliably distinguishes remain speakers without mistakenly treating them as forget speakers. For detailed experimental results, please refer to Section 7. Similar Voice Handling in our response to reviewer (bkoi).\n\n## **3. Computational Costs**\n\nContrary to traditional machine unlearning, Exact Unlearning is not ideal nor practical in Zero-Shot TTS. Zero-Shot TTS models can replicate voices they have never seen during training;  simply removing the forget speaker’s samples from training data does not suffice. Therefore, a comparable method to unlearn speaker identities does not exist.  \n\nRegarding the training cost, the Exact Unlearning and Original models were trained for 500K steps. When unlearning 1 speaker at a time, SGU and TGU for were trained for 10K steps (2% of pretraining steps) in Table 2. When unlearning 10 speakers at a time with LibriTTS dataset, TGU was trained for 10K steps, refer to Section 4. Generalizability in our response to Reviewer d6q4. \nFine Tuning, SGU, and TGU for Table 1. were trained for 145K steps - with TGU showing convergence around 58K steps in Appendix D. Negative Gradient and Kullback-Libeler divergence were halted at early stages due to exploding gradients. \n\nNevertheless, our method is not confined to a specific fine-tuning method. Therefore, in practicality, computational cost can be minimized using diverse accelerating methods in the fine-tuning process.\n\n## **4. SIM**\n\nSIM is calculated using the cosine similarity between speaker embeddings extracted from a speaker verification model. This evaluation is widely utilized in TTS research domain [1]-[6]. We use the same speaker verification model as [1]-[5] for evaluation. For SIM, the goal is to maintain a high performance in terms of ZS-TTS model evaluation.\n\nFor unlearning task, indeed SIM is not sufficient alone. To mitigate this limitation, we propose spk-ZRF as a new evaluation metric to measure model’s randomness in generation. Objective is for the model to “act as if they have no knowledge of forget speaker identity”. Having a high spk-ZRF indicates that the model is behaving randomly. Vice versa, a spk-ZRF lower than original model means the model has learned to act in a consistent way (e.g., synthesizing in a specific voice). Thus, we do not mark an arrow for spk-ZRF-R because maintaining original performance is ideal. \n\nOnce again, we greatly appreciate the reviewer’s insightful feedback, which helped us further clarify and strengthen our work.", "claims_and_evidence": "The proposed teacher-guided learning and spk-ZRF measure is significantly novel and the evaluation experiments confirmed their effectiveness.", "methods_and_evaluation_criteria": "1) How can the system know if the input audio prompt is from the speaker with the Forgot ID in the proposed unlearning framework?\n     I am afraid it is difficult to identify speakers accurately from such small data.\n\n2) The ideal method is Exact Unlearning, which requires a significant computational cost. Therefore, the proposed approach's computational costs should also be evaluated. The training and testing stages require extra computational costs.", "theoretical_claims": "The proposed method is theoretically sound.", "experimental_designs_or_analyses": "The details of human subject evaluation are unclear.", "relation_to_broader_scientific_literature": "This paper focuses on speaker identity in text-to-speech, but similar problems will happen in many other media.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "1) The proposed methods' SIM-F is much lower than that of Exact Learning in Table 3. This result indicates that users may easily guess whose voices are excluded, which may also be a privacy problem. I believe it should be the same as Exact Learning.\n\n2) The similarity measure may not be perfect for this task. Human listeners may use other cues to identify speakers, so a subjective test to determine a speaker should be needed.", "other_comments_or_suggestions": "None.\n\n## update after rebuttal ##\n\nThank you for the rebuttal.\nI did not change my score.", "questions_for_authors": "How many subjects are used,  and how much data are used, in the human subject evaluation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes a Teacher-Guided Unlearning (TGU) which is a novel framework for removing specific speaker identities from Zero-Shot Text-to-Speech (ZS-TTS) models. It leverages controlled randomness via a teacher model to unlearn specific speaker identities while retaining the ability to emulate non-forgotten speakers. The work also proposes a new evaluation metric which generates random voice styles for "forgotten" speakers termed speaker-Zero Retrain Forgetting (spk-ZRF) thus measuring randomness in generated voices for forgotten speaker. 

The reviewers consistently recognize this as the first work on speaker unlearning and the novelty of the proposed new metric, spk-ZRF. The work represents a significant methodological contribution and relies on robust evaluation strategy comprising comparisons with well-established baseline methods. The reviewers also expressed the following concerns: 

- the unlearning method has a significant computation cost and requires considerable about of user data (approx 5 mins per speaker) which might limit it's real-world applicability. 
- validation is conducted using closed-source model, VoiceBox which may further limit adoption.

Despite the mentioned concerns, the reviewers agree that this is a significant contribution to the community. I request the authors take into account the detailed feedback from the reviewers to improve the final draft of their submission.
Decision: Accept (poster)