Thu-P-OS-9-1

Predicting VQVAE-based Character Acting Style fromQuotation-Annotated Text for Audiobook Speech Synthesis111122

Wataru Nakata , Tomoki Koriyama , Shinnosuke Takamichi , Yuki Saito , Yusuke Ijima , Ryo Masumura , Hiroshi Saruwatari1. The University of Tokyo, 2. NTT Corporation

1

Synopsis: Speech synthesis model which can “act” as a charactor● Our proposed model● Audiobook speech synthesis○ Aims to automate the audiobook creation○ Reduces the cost of the creation

Speech samples😍

○ Extracts character acting style using VQVAE-based model [2]○ Predicts character acting style using attention-based model○ Introduces use of character embedding [3] in speech synthesis● Experimental results○ Comparable naturalness at MOS on chapter level samples○ Signiﬁcant improvement on character distinction● Challenges in audiobook speech synthesis○ Reﬂecting context from multiple sentences [1]○ Achieving character acting (this work)https://wataru-nakata.github.io/is2022-audiobook/

Proposed model
Input: quotation-annotated text

Character acting style extraction module (Green box)
● Extracts time-invariant feature using ResCNN [4]● Learns discrete set of character acting styles as VQ codebook● Predicts proper character acting from RoBERTa [5] features andcharacter embedding using attention
Character embedding (Skip-Gram) [3]● Proposed in NLP ﬁeld to represent movie characters● Trains character embedding from quotation-annotated text

FastSpeech2 (FS2)

Only used for training

Experiment
Naturalness MOS of synthetic speechCompared models
Model name

RoBERTa ResCNN Vector Quantization

Character embedding

FS2 (w/o BERT)
FS2✅

FS2-ResCNN✅

✅

FS2-ResCNN-VQ✅

✅

FS2-character✅

FS2-all (proposed)✅

● 120 raters with each rater evaluated 12 samples● Chapter level (2-5 sentences) evaluations
VQ improvesnaturalness✅
✅

✅

✅

✅

● All models were trained using J-KAC corpus [1] (expressive audiobook speech)● RoBERTa-base model published by rinna Co., Ltd. RoBERTa trained on Japanese● Red models takes ground truth speech at inference

Similar naturalness as baseline

Character distinction● AB tests with 60 raters and each rater evaluated 12 samples● Chapter level (2-5 sentences) evaluations● Quotation-annotated texts are provided to the raters● Raters selected sample with more appropriate distinction
Is ResCNN feature useful? Yes!
FS2

0.32 vs. 0.68

FS2-ResCNN

0.48 vs. 0.51

FS2 (w/o BERT)

FS2-character

0.43 vs. 0.57

FS2-all

FS20.40 vs. 0.60

FS2-all

FS2-all

0.39 vs. 0.61

FS2-ResCNN-VQ

Bold values represent signiﬁcant (p < 0.05) results

FS2-ResCNN[1] W. Nakata et al., Proc. SSW11, 2021

[3] M. Azab et al., Proc. 23rd CoNLL, 2019[5] Y. Liu et al., Proc. ICLR, 2020

[2] A. van den Oord et al., Proc. NIPS, 2017[4] C. Li et al., arXiv, 2017[6] Y. Ren et al., Proc. ICLR, 2021

Saruwatari-Koyama Lab., The University of Tokyo, Japan.
©Wataru Nakata, Sep. 2022.

FS2

Does predicting character acting style work better? Yes!

Predicting character acting style is an effective approach
References

0.44 vs. 0.56

Is the character acting style prediction perfect? No😢

Does VQ worsen the result? No!
FS2-ResCNN-VQ

Is surrounding context effective? Yes!

Interspeech 2022, Korea