V2S A&ack: Building DNN-Based Voice Conversion from Automa>c Speaker Veriﬁca>on†††‡†Taiki Nakamura , Yuki Saito , Shinnosuke Takamichi , Yusuke Ijima , and Hiroshi Saruwatari†The University of Tokyo, Japan‡NTT Corpora>on, Japan

4. Experimental Evaluation

1. Introduction: Verification-to-Synthesis (V2S) Attack
Automatic speaker verification (ASV)[1]
Voice conversion (VC)
✔ identifies the speaker of the input voice.
✔ predicts targeted speaker’s voice.
→ If an attacker hacks the ASV, voices of→ VC is a possible technique used
enrolled speakers risk being reproduced.
in impersonation attack.
Deceiving the ASV has some possibility of reproducing the targeted speaker’s
individuality by VC. We name this attack “verification-to-synthesis (V2S) attack”.

Experimental conditions

Voice conversion (VC)
Targetedspeaker

Attacker

VC

V2S attack to automatic speaker verification (ASV)
Verifyby voice
VC■ Our approach✔ proposes VC training with the “white-boxed” ASV and pre-trained automatic speech
recognition (ASR) models without the targeted speaker’s voice data.

hack the ASVImpersonateby voice

Proposed VC performs comparably to the standard VC methods using a tiny amount of parallel voice data.

Subjective evaluation
Naturalness (preference AB tests)

2. Standard VC Using Targeted Speaker’s Voices

male-to-male[2](1) One-to-one parallel VC (ParaVC)[3](2) One-to-many non-parallel VC (NonparaVC)✔ is trained to minimize mean squared error (MSE)& using a targeted speaker’s voice.
betw. 𝒚 and 𝒚✔ can convert an attacker voice to any arbitrary speaker’s voice.
✔ is often trained using multi-speaker corpora in advance.

ParaVCtarget speaker’s 𝒚& Predict𝒚feature to minimize
prediction error

Converted

Targeted

VC modelTwo standard VC models are trained using a targeted speaker’s voice.
These are used as references to evaluate the performances of the
proposed speaker V2S attack.

3. V2S Attack: VC without Using Targeted Speaker’s Voices
V2S attack model
VC model✔ uses two DNN models for training the VC
Source&𝒚→ Loss can be backpropagated to VC model.
speaker’s■ White-boxed ASV model 𝑽 ・feature→ Targeted speaker’s label (𝒍𝒚 ) is given.
Pre-trained→ 𝑽(・) estimates speaker similarity between input voices (&𝒚)ASR model& ).
& targeted speaker’s voices as softmax cross-entropy, 𝐿()* (𝒍𝒚 , 𝑽 𝒚
Phonetic posterior→ It helps to reproduce the targeted speaker’s individuality,
Mean squared error
but does not keep the phonetic property of the input voice.
R(𝒚)
R(&𝒚)

■ Pre-trained automatic speech recognition (ASR) model 𝑹(・)& .
→ 𝑹(・) estimates the discrepancy as MSE between 𝑹 𝒙 and 𝑹 𝒚→ It helps to restore the phonetic property of the input voice.

(a) ParaVC: trained by {5, 10, 30} utterances
Compared model (b) NonparaVC: trained by 260 pre-stored speakers(c) V2S: trained by 200 utterances of attacker
The number of
260 Japanese speakers (130 males and 130 females)
enrolled speakers
Speech params.
39-dim. mel-cepstral coefficients, Log F0, 10-dim. bap(including Δ)
DNN architectures Feed-Forward (see our paper)
Attacker andone attacker (one male) &Targeted speakers four targeted speakers (two males and two females)Evaluation data 25 parallel voices

AScores𝑝-valueBA

Scores𝑝-valueB

ParaVC ( 5 utts)

0.388 vs. 0.612

1.221 × 10-10

V2S

ParaVC ( 5 utts)

0.490 vs. 0.510

0.572

V2S

ParaVC (10 utts)

0.475 vs. 0.525

0.158

V2S

ParaVC (10 utts)

0.593 vs. 0.407

1.365 × 10-7

V2S

ParaVC (30 utts)

0.458 vs. 0.542

0.016

V2S

ParaVC (30 utts)

0.610 vs. 0.390

3.174 × 10-10

V2S

NonparaVC

0.598 vs. 0.402

2.694 × 10-8

V2S

NonparaVC

0.538 vs. 0.4620.034

V2S

V2S attack ≧ ParaVC (5 utts)
Speaker individuality (preference XAB tests)
male-to-male

male-to-female

A

Scores𝑝-valueBA

Scores𝑝-valueB

ParaVC ( 5 utts)

0.530 vs. 0.470

0.090

V2S

ParaVC ( 5 utts)

0.585 vs. 0.415

1.324 × 10-6

V2S

ParaVC (10 utts)

0.615 vs. 0.385< 10-10V2S

ParaVC (10 utts)

0.713 vs. 0.287< 10-10V2S

ParaVC (30 utts)

0.675 vs. 0.325< 10-10V2S

ParaVC (30 utts)

0.705 vs. 0.295< 10-10

V2S

NonparaVC0.660 vs. 0.340< 10-10

V2SNonparaVC

0.588 vs. 0.412< 10-10

V2S

V2S attack = ParaVC (5 utts)

White-boxed
ASV modelTarget𝒍𝒚
speaker’sSoftmax labelcross-entropy𝑽(&𝒚)Reproduce speaker individuality

Restore phonetic property

Loss function&, 𝒍/ = 𝐿()* 𝒍/ , 𝑽 𝒚&
𝐿 𝒙, 𝒚male-to-female+ 𝜔𝐿3(* (𝑹 𝒙 , 𝑹(&𝒚))5. Conclusion
V2S attack: voice impersonation attack using VC✔ uses ASV, and ASR model for VC training.
✔ is trained without the targeted speaker’s voices.
Experimental result→ V2S attack can synthesize voices that has
naturalness and speaker individuality comparable
to a standard parallel VC with a tiny amount of data.
We are planning to✔ improve the performances of the V2S attack.
✔ investigate ways of preventing the V2S attack.

hyperparameter[References][1] Dehak, N. et al., IEEE Transactions on ASLP, 2011 [2] Toda, T. et al., IEEE Transactions on ASLP, 2007 [3] Saito, Y. et al., Proc. ICASSP, 2018
Saruwatari and Koyama Lab., The University of Tokyo

SSW10, The 10th ISCA Speech Synthesis Workshop @ Vienna, Austria2019@Taiki Nakamura September 21. 2019