Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
Authors: Ziyue Jiang
Zhe Su
Zhou Zhao
Qian Yang
Yi Ren
Jinglin Liu
Zhenhui Ye
Venue: NeurIPS 2022 Accept (2022)
Keywords: Text-to-Speech
Online Dictionary
Unsupervised Polyphone Disambiguation

Abstract: Polyphone disambiguation aims to capture accurate pronunciation knowledge from natural text sequences for reliable Text-to-speech (TTS) systems. However, previous approaches require substantial annotated training data and additional efforts from language experts, making it difficult to extend high-quality neural TTS systems to out-of-domain daily conversations and countless languages worldwide. This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language). Specifically, we design a semantics-to-pronunciation attention (S2PA) module to match the semantic patterns between the input text sequence and the prior semantics in the dictionary and obtain the corresponding pronunciations; The S2PA module can be easily trained with the end-to-end TTS model without any annotated phoneme labels. Experimental results in three languages show that our model outperforms several strong baseline models in terms of pronunciation accuracy and improves the prosody modeling of TTS systems. Further extensive analyses demonstrate that each design in Dict-TTS is effective. The code is available at https://github.com/Zain-Jiang/Dict-TTS.

Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
Ziyue Jiang" Zhejiang University jiyuejiangee edu. en
Zhe Su* Zhejiang University suzhesz008gmail.ccm
Zhou Zhao Zhejiang University zhaozhoułzju. edu. en
Qian Yang Zhejiang University @yyan1021@foxmmnr com
Ren Bytedance AI Lab ren yiwbytedeeeeeeeee.meeeeeo com
Jinglin Liu
Zhejiang University edu
j jinglinliuzzju. edu. cn
Zhenhui Ye Zhejiang University zhenhuiye@zju. edu. cn
Abstract
Polyphone disambiguation aims to capture accurate pronunciation knowledge from natural text sequences for reliable Text-to-speech (TTS) systems. However, previous approaches require substantial annotated training data and additional efforts from language experts, making it difficult extend high-quality neural TTS systems to out-of-domain daily conversations and countless languages worldwide. This paper tackles the polyphone disambiguation problem from concise and novel perspective: we propose Dict-TTS, semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language). Specifically, we design (S2PA) module match the semantic patterns between the input text sequence and the prior semantics in the dictionary and obtain the corresponding pronunciations; The S2PA module can be easily trained with the end-to-end TTS model without any annotated phoneme labels. Experimental results in three languages show that our model outperforms several strong baseline models in terms pronunciation accuracy and improves the prosody modeling of TTS systems. Further extensive analyses demonstrate that each design Dict-TTS is effective. The code is available httth:://// htttps:///gihub.co com wwt../oonnain-naan/iiiiie.e
attention
Introduction
Capturing the pronunciations from texts is challenging for end-to-end text-to-speech (TTS)
systems 31 45 55, 27
14
41 22 21), since there are full words that are not covered by
general pronunciation rules 24 50 Therefore, polyphone disamigguatiinn (one of the biggest challenges in converting texts into phonemes (37 60, (66) plays an important role in the construction
of high-quality neural TTS systems 18 38 However, since the exact pronunciation of polyphone must be inferred based on its semantic contexts, current solutions still face several challenges: 1) the
rule-based approaches 66 19
with limited linguistic knowledge or the neural models [60] 38 46
* "Equal contribution. 1 Corresponiing author PPolyphones characters having than one phonetic value. See Appendix D for further details.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Character
Pronunciation
Definitions
Usages
< e>
XE
BM: # + " BERE # RKM. BFAAX...
BER T 3 K.L TR. RT. HR. t%%
5y ué>
H 1 -illl # * (0##50MM 2ff
<y do>
K ビ FLIL 9-.. 1
<1 do>
#
Figure 1: The illustration of the dictionary entry that contains information on character's word's definitions, usages, and pronunciations. For example, in the Chinese sentence " AfT", the pronunciation of the polyphone "IR"" should be inferred based on its semantic contexts.
trained limited data suffer from significant performance degradation out-of-domain text datasets; 2) the neural network-based models [6 [ 38 learn the grapheme-to-phoneee
(G2P) mapping in an end-to-end manner without explicit semantics modeling, which hinders their pronunciation accuracy in real-life applications. 3) based on the above two points, a reliable polyphone disambiguation module is usually based on combination hand-crafted rules, structured G2P-oriented lexicons, and neural models [16), which requires substantial phonemes labels and external knowledge from language experts.
Unlike the previous rule-based or neural network -basd approaches, we address the above challenges with the existing prior information worldwide. As shown in Figure [ arbitrary dictionary used in daily life can be viewed as a prior knowledge database. Intuitively, contains valuable prior knowledge for pronunciations in conversations. When pronunciation of a specific polyphone, he or she will resort to the dictionary website to infer its exact reading based on the semantic context. We imitate this scenario in our architecture design and propose Dict-TTS, an unsupervised polyphone disambiguation framework, which explicitly consults the online dictionary to identify the correct semantic meanings and acoustic pronunciations of polyphones. Specifically,
is confused about the acoustic
To explicitly learn the semantics-to-pe poronnccioo mapping, we adopt semantic encoder to obtain the semantic contexts of the input text and utilize a semantics- to- pr-onunnciiiionnn attention (S2PA) module to search the matched semantic patterns in the dictionary so as to find the correct pronunciations. We also use the retrieved semantic information as auxiliary information for prosody modeling of the TTS model.
To perform polyphone disambiguation without phoneme labels, we combine our S2PA module into end-to-end TTS systems' training and inference processes. Different from current neural polyphone disambiguation models, guidance mel-spectrogram reconstruction loss in fully end-to-end manner, which significantly reduces the cost of building such system.
module can be trained with the
To demonstrate the generalization ability of our Dict-TTS, we perform experiments three datasets, including a standard Mandarin dataset B. a Japanese corpus [47), and a Cantonese dataset [.] Experiments these datasets show that Dict-TTS outperforms other state-of- the-art polyphone disambiguation models in pronunciation accuracy and improves the prosody modeling of TTS systems in terms of both subjective and objective evaluation metrics. The pronunciation accuracy of Dict-TTS further improved by being pre-trained on a large-scaled automatic speech recognition (ASR) dataset. The main contributions of this work are summarized as follows:
We incorporate the online dictionary into TTS systems and propose a semantic-aware method for polyphone disambiguation, which improves the pronunciation accuracy and robustness of end-to-end TTS systems. Moreover, the idea of introducing the prior knowledge worldwide can also inspire other tasks like neural language modeling (26] and sequence labeling [34].
We propose novel and general framework for unsupervised polyphone disambiguation in TTS systems, which further enables the efficient pre-training on large-scaled ASR datasets and improves the generalization capacity significantly.
We also find that the retrieved semantics in the dictionary knowledge can be used as auxiliary information to improve prosody modeling and help the TTS system to generate more expressive speech.

---

We further analyze the characteristics of the linguistic encoder based on phoneme and character and provide valuable interpretations about our semantics-to cemantiesto-proounciiiooo attention module.
2 Background
This section describes the background of TTS, graphemeeto-phhnnmee (G2P) pipeline, and their relations with polyphone disambiguation. We also review the existing works that aim at semantic- aware polyphone disambiguation and analyze their advantages and disadvantages.
first generate mel- spectrogram from text and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [52] 29 20], or directly generate waveform from text in an end-to-end manner [40 13 28 The frontend model of end-to-end TTS system should tackle one important task, i.e., polyphone disambiguation G55. Properly mapping the grapheme sequence into phoneme sequence requires the linguistic encoder capture the empirical pronunciation rules in daily conversations. However, it is extremely difficult for the linguistic encoder to learn all the pronunciation rules necessary to produce speech in an end-to-end manner, which results in inevitable mispronunciations in the generated speech. To alleviate this problem, robust TTS models usually con- vert the text sequence into the phoneme sequence with open-source geapheme-to-phoneee and predict the spesstectro from the phoneme sequence. However, the rule-based or neural phoneme pipelines suffer from significant performance degradation on out-of of-domain datasets since it is extremely difficult and costly for them to cover all linguistic knowledge.
Text-to-Speech Text-to-speech (TTS) models [5
pipelines
network- -basd grapheme- -otheemm------oemmeee
The grapheme-to-phonene models map grapheme sequence to phoneme sequence reduce pronunciation errors in modern TTS systems. For logographic languages like Chinese, Japanese and Korean, although the lexicon can cover nearly all the characters, there are full of polyphones that can only be decided according the semantic context of character [50]. Thus, polyphone disambiguation is the most important challenge in sotchhee--t-phhneeee
Beppheme-to-Phonemee
conversions for this kind of languages. Moreover, many alphabetic languages including English and French also have polyphones in daily conversations. Current polyphone disambiguation approaches can be categorized
into the rule-based approach 664 19 and the data-driven approach [43
3 The rule-based
G2P method is based on combination of hand-crafted rules and structured G2P- Gor-eniet lexicons.
which requires a substantial amount of linguistic knowledge. The data-driven G2P algorithm adopts architecture [58]5] 48
statistical methods [33] or neural encoder-decote
60)
38 46, (54). However,
building a data-driven G2P model requires a large amount of carefully labeled data and substantial linguistic knowledge from language experts, which is extremely costly and laborious.
Semantic-Aware Polyphone Disambiguation Polyphone disambiguation the core issue for G2P conversion in various languages. The pronunciation of a polyphone is defined by the semantic context information of neighbouring characters [50]. In order comprehend the semantic meaning in the given sentence for polyphone disambiguation, previous methods [1]
57
49
18 g
have adopted
the pre-trained language model [12 to extract semantic features from raw character sequences and predict the pronunciation of polyphones with neural classifiers according to the semantic features. Among them, PnG BERT and Mixed-Phoneme BERT 225 62] take both phoneme and grapheme as input to train an augmented BERT and use the pre-trained augmented BERT as the TTS encoder. However, these methods still require annotated data to train and can not be incorporated into the TTS training in an end-to-end manner. Although NLR [16] directly injects BERT-derived knowledge into the TTS systems without phoneme labels and successfully reduces pronunciation errors, their method confounds the acoustic and semantic space, which significantly affects the pronunciation accuracy.
3 Method
To exploit the prior linguistic knowledge the online dictionary for TTS systems, we propose Dict-TTS, which explicitly captures the semantic relevance between the input sentences and the dictionary entries for polyphone disambiguation. In this section, we firstly introduce the overall architecture design of Dict-TTS based on PortaSpeech [1). Then after the comparison between the
3

---

Cecpantiictto-PPounnntttet Attention
Semantic Space
Acoustic Space
Semantic Contexts
Input Character Sequence
Dictionary Entrics
Weights
Pronunciations
UT TT..... (listen to the band ...)
B9R
0.02
1e4
3
0.96
y uc4
Attention
x
IXX
X
0.01
a04
Semantic Encoder
1|
11
0.01
a04
Semantic
Pronunciation Embeddings
Embeddings
Linguistic Encoder
ML-Discriminttor
Variational Generator
External Vocoder
Figure The overall architecture for Dict-TTS. The character " "R has possible pronunciations cou- pled with 4 different meanings. The module inside the dashed box is semantics-to to-proounniiaiion attention (S2PA). The S2PA module measures the semantic similarity between the input character and the corresponding dictionary entries and aggregates the attention weights into pronunciation weights. Then the weighted semantic emheddings and pronunciation embeddings are fed into the linguistic encoder for feature fusion. The semantic and acoustic spaces described in Subsection 3.2 are separated by the blue dashed line. "ML-Discriminator" denotes Multi-Length Discriminator in HiFiSinger [7]. The dashed black line denotes that the operation is only executed in the training phase.
mhoneme-based and tharacter-based TTS systems in the acoustic and semantic space, design novel
attention (S2PA) module to learn grapheme-to-phonnnee mappings based on the semantic context; In general, Dict-TTS exploits the prior pronunciation knowledge the online dictionary with the following steps: Firstly, the character sequence is fed into the self-attention based semantic encoder obtain the semantic representations of the input character sequence, and we utilize a pre-trained cross- tross-lingual language model [10] to extract the semantic context information in the dictionary entries; Secondly, we calculate the most relevant dictionary entries of the input graphemes and obtain the corresponding pronunciation sequence contained in the dictionary entries; Finally, the extracted semantic context information and the pronunciations are fed into the linguistic encoder for feature fusion. We describe these designs in detail in the following subsections.
3.1 Model Overview
The overall model architecture of Dict-TTS is shown in Figure 2 Dict-TTS keeps the main structures of PortaSpeech: Transformer- based linguistic encoder; a VAE- based variational generator with flow-based prior to generate diverse mel- spessspccro.ram. The flow-based post-net in PortaSpeech is replaced with multi-lenges discriminator 7 based on random windows of different lengths. which has been proved to improve the naturalness word pronunciations
(59). However, since there are no phoneme inputs in our scenarios, replace the linguistic encoder that combines hard word-level alignment and soft phoneme-level alignment with: 1) a semantic encoder that extracts the semantic attention module that
representations in the grapheme sequence; 2) matches the semantic patterns between the dictionary entries and the grapheme representations and obtains the corresponding semantic embedding and pronunciation embedding; 3) linguistic encoder that fuses the semantic embedding and pronunciation embedding.
3.2 Comparison between the phoneme-based and character-based TTS systems
In this subsection, we make preliminary analyses about the linguistic encoder of thoneme-based and character-based TTS systems. For simplicity, we describe the following concepts according to the logographic writing system, where single written character represents complete grammatical word

---

or morpheme. These concepts can be extended to alphabetic languages like English by replacing "character" with "word".
Phoneme-based TTS systems As is shown in Figure 3 the linguistic encoder of the phoneme-basee TTS system takes phoneme sequence p generated by the G2P module as inputs. The main challenge of the phoneme-base baasedd linguistic encoder is to comprehend the semantic and syntactic representation from p and deduce the pitch trajectory, speaking duration, and other acoustic features from and generate the expressive and natural pronunciation hidden state g. Since the phoneme sequence p is combination of the smallest units of sound in speech, it may be ambiguous in terms of semantic meaning, which brings difficulties for the deduction of the representation 8. For example, "AE1, T. F, ER1, S, can be easily classified "At first", but "W, EH1, DH, ERO" be classified "Whether" or "Weather". Homophones like "to", "too", and "two" can be converted the same phoneme sequence "T, UWI", but their local speaking duration and pitch are different. Moreover, the tree-structured syntactic information contained in the word- based input sequence missing. Since semantic information and syntactic information possess rich intonational features such as pitch accent and phrasing of the input text [59], the ambiguity of hurts the prosody modeling phoneme-
Semantic and Syntactic Space
Acoustic Space
g
Character Representation Space
The illustration of repre- sentations in the linguistic encoder.
Figure 3:
TTS systems.
Character-based TTS systems The first challenge for character- based TTS system is to predict the correct phoneme sequence p. Unlike the phoneme-based TTS system, the character-based TTS system does not know the phoneme sequence p when characters arrive. Thus, it does not know how to pronounce the words accurately at first. Then the mel-spectroogrm reconstruction loss in TTS training would drag the character representation to the acoustic space. For example, Chinese characters ">K" ("fire" English) and "fk" ("partner" in English) share the same pronunciation ("H UO3") and have different semantic meanings. However, with the guidance of the mel-spectrogram reconstruction loss, their representations distribute according to acoustic pronunciation, which hinders the semantic comprchension for polyphone disambiguation and prosody modeling.
From the above analyses, can be seen that the character representations should locate in the semantic space so that we can easily capture s based on the context, deduce based on the dictionary and s, and finally obtain the natural pronunciation hidden state g. The following subsection mainly describes how we achieve the above goals with our semantics-to-prett to-oouucttoooonnniiaaiiiiono attention module.
3.3 Semantis-to-Pronnncaiio Attention
As shown in Figure 2 the semantics-to-- -or--tooccpaoiiionnn attention (S2PA) module is designed for explicit semantics comprehen- sion and polyphone disambiguation.
W1.1
V/,2
Wi
Dictionary Definition Assuming that we have an dictionary
D which contains sequence of characters C = [c1, C22 Cnl. where n the size of characters set in a languag the dictionary, each character has a sequence of possi- ble pronunciations p; [Pr.1 Pi.2; PP.2, ..., Pi.m. and each pro- nunciation Pt.j has its corresponding dictionary entry ei2 [er.,,1 Ctg,2: ..., Cij. E E (definitions, usages, and transla- tions are merged together where m the number of the possible pronunciation of e, and is the number of characters in the corresponding entry. Note that for polyphones m > and for characters that have only one pronunciation
In
k;
a single characters sequence),
61,1
6222
Figure 4: The illustration of seman- tic pattern matching in S2PA mod- ule.
1.
'The dictionary can be easily downloaded from online websites. See Appendix A.1m for further details
5

---

Semantics Matching The goal of our S2PA is to obtain the pronunciation sequence by measuring the semantic similarity between the input character sequence [t ...,.. &y] and the corresponding gloss items in the dictionary, where 1. is the sequence length. As shown in Figure + we firstly extract the semantic context information k of every entry with a pretrained cross-lingual language model (T0) and store k as the prior dictionary knowledge. Then we use semantic encoder to obtain the semantic contexts z from the input character sequence For each character token 4ú, its semantic feature vector z, is used as the query vector to semantics-based attention module. Here, attention is used to learn similarity measure between the semantic feature vector Z; and k.
1,,, ki,maul Z
(1) where d is the scaling factor and 11 m, denotes the semantic similarity between f; and each item in [es,1,1, Gi,,,ul The retrieved semantic embeddings s'; can be extracted by s'; so sof (tmaa((a..11111... )i,m,u) [k,,,,1..... k..m,a)- The rich linguistic information in s'; can be used as auxiliary information improve the naturalness and expressiveness of the generated speeches.
| a
Polyphone Disambiguation The aggregated attention weight Wa,j =Ex=, a.j.k can be seen as the probability of the pronunciation Pt.f. However, since a polyphone a specific sentence has only one correct pronunciation, here we use the Gumbel-Softtan function (23) to sample a differentiable approximation of the most possible pronunciation p/::
exp ((log (Wa.) + 91.s) /T)
Wij
77 1=1
exp (llog (Wi,) + 95.)
(2)
P Wij Pij
where 95,1y---iin are i.i.d samples drawn from G0mbel(0,1) distribution and 7 is the softmax temper- ature. Then the retrieved pronunciation embeddings pl and semantic embeddings s', are then fed into the rest of the linguistic encoder for feature fusion and syntax prediction. Intuitively, our S2PA module can be thought of as an end-to-end method for decomposing the character representation, pronunciation, and semantics with dictionary following the preliminary analyses in Subsection [3.2] Through our S2PA module, the character representation is successfully distributed in the semantic space so that the model can easily deduce the correct pronunciation and semanties based on the lexicon knowledge like human brain.
3.4 Training and Pre-training
In training, the S2PA module weights (including character token embeddings and the pronunciation token embeddings) are jointly trained by the reconstruction loss from the TTS decoder. Thus, our Dict-TTS does not require any explicit phoneme labels. In inference, the pronunciations can be specified by feeding the text sequence into the S2PA module to get the predicted pronunciations. Besides, our method is compatible with the predefined rules from language experts by directly adding specific rules to pronunciation weight wa.j-
Although the cemantics-to-prene pronunciaaioo mappings can be explicitly learned by the S2PA module, it could be not accurate enough, due to the following reason: the text training data is not large enough (about 10k sentences) in the TTS dataset, leading to relatively inaccurate context comprehension. To improve the semantic comprehension and the generalization capacity for our S2PA module, propose a pre- pre-training method using low-quality text-speech pairs from large-scaled automatic speech recognition (ASR) datasets. Since our S2PA module can be trained without phoneme labels, it can be casily pre-trained and effectively finetuned to various domains to improve the pronunciation accuracy of the TTS systems.
Experiments
4.1 Experimental Setup
Datasets We evaluate Dict-TTS on three datasets of different sizes, including: 1) Biaobei 3 Chinese speech corpus consisting of 10,000 sentences (about 12 hours) from Chinese speaker;

---

Table The objective and subjective pronunciation accuracy comparisons. PER-O denotes phoneme error rate in the objective evaluation, PER-S denotes phoneme error rate in the subjective evaluation and SER-S denotes sentence error rate in the subjective evaluation.
Biaobei
JSUT
Common Voice (HKK)
Method
PER-O
PER-S
SER-S S
PER-O
PER-S
SER-S
PER-O
PER-S
SER-S
Character
3.73%
30.50%
13.78%
65.50%
1.89%
15.50%
Phoneme
2.78%
1.14%
7.00%
1.55%
0.92%
4.25 %
1.45%
10.25%
Dict-TTS
2.12%
1.08 %
6.50% %
3.73%
2.57%
22.75%
1.23 %
9.75%
2) JSUT 47 a Japanese speech corpus containing reading-style speeches from a Japanese female speaker (we the basic5000 subset that contains 5,000 daily-use sentences). 3) Common Voice (HK) [ a Cantonese speech corpus that contains 125 hours of speeches from 2,869 speakers (We the 102 hours of the validated speeches). For each of the three datasets, randomly sample 400 samples for validation and 400 samples for testing. We randomly choose 50 samples in the test set for subjective audio quality and prosody evaluation and all testing samples for other evaluations. The ground truth mel-spectrograms are generated from the raw waveform with the frame size 1024 and the hop size 256. For computational efficiency, we firstly use the pre-trained XLM-R [IO] model to extract the semantic representations from the raw text of the whole dictionary and record them in the disk. We then load the mini-batch along with the cressuuutee dictionary representation during training and testing.
Implementation Details Our Dict-TTS consists of a S2PA module, an encoder, variational generator and a post-net. The encoder consists of multiple feed-forward Transformer blocks 44] with relative position encoding 44 following Glow-TTS (7). The encoder and decoder variational generator are 2D-convolution networks following PortaSpeech [11. We replace the post-net in PortaSpeech with a lengt discriminator 77. which has been proved to improve the naturalness of word pronunciations [59]. We add more detailed model configurations in Appendix B.1 B.2 We train Dict-TTS on NVIDIA 3080Ti GPU, with batch size of 40 sentences on each GPU. We use the Adam optimizer with 81 = 0.9, 82 0.98, in [53 The softmax temperature т is initialized and annealed using the schedule in (23]. It takes 320k steps for training until convergence. The predicted mel- mee-sppetrograms transformed into audio samples using pre-trained HiFi-GAN 29
10-9 and follow the same learning rate schedule
4.2 Results of Pronunciation Accuracy
We compare the pronunciation accuracy of our Dict-TTS with other systems, in- cluding 1) character-basse based systems, where we directly feed character into the linguis- tie encoder; 2) Phoneme-based systems, where we convert the text sequence to the phoneme sequence [55] 27] with most popular open-source grapheme-to- phoneme tool We measure objective phoneme error rate (PER-O), subjective phoneme error rate (PER-S), and subjec- tive sentence eтror rate (SER-S) in the evaluations. The phoneme labels in the objective PER evaluation are from the cor- responding dataset (since the Common Voice (HK) dataset does not have phoneme labels, we only evaluate the subjective metrics). In the subjective evaluations, each audio in the test set is listened by at least 4 language experts. We ask them to write down the mispronounced phonemes and discuss them with each other until
Table 2: The objective and subjective pronunciation ac- curacy comparisons in the Biaobei dataset.
Method
PER-O
PER-S
SER-S
Character
3.73%
30.50%
BERT Embedding
15
4.03%
38.75%
NLR (16]
2.98%
26.50%
Phoneme (G2PM)
3,95%
1.39%
10.50%
Phoneme (pypinyin)
2.78%
1.14%
7.00%
Dict-TTS
2.12%
1.08%
6.50%
Dict-TTS (pre-trained)
1.54 %
0.79%
4.25%
https://githuu/ wow 1tt.///if7-i/eee
Wee use pypоsyín in the Biaobei dataset, pyopenjialk in the JSUT dataset and pycantonese in the Common Voice (HK) dataset. More detailed information be found in Appendix B.4
7

---

(a) GT
(b) Character
(c) BERT Embedding
(d) NLR
(e) Phoneme (pypinyin)
(f) Dict-TTS
Figure 5: Visualizations of the ground and generated mee-spectos by different types of linguistic encoders. The corresponding text is " [ which means "The man picking water chestnut sings at night. The old man fishing and the girl picking lotus are laughing".
conclusion is reached. Note that for SER-S, the error rate is calculated sentence level (e.g., a sentence with multiple errors will be counted only once). More details about these evaluations can be found in Appendix B.3 The results are shown in Table It can be seen that Dict-TTS greatly surpasses the character-based baseline in three languages. Moreover, Dict-TTS achieves the comparable objective and subjective PER with the most popular grapheme-to-phonemee tools on two relatively larger datasets (Biaobei and Common Voice (HK)) and maintain a good pronunciation accuracy a relatively small dataset (JSUT), which demonstrates the superiority of the explicit semantics matching in our S2PA module.
We also compare the pronunciation accuracy of our Dict-TTS with various types of systems, including: 1) character-based system; 2) BERT embedding based system [[5]. where the BERT derived embeddings are concatenated with the character embeddings; 3) NLR [66], a TTS system that directly injects BERT derived knowledge into the linguistic encoder without phoneme labels; 4) Phoneme (G2PM (38), PortaSpeech with phoneme labels derived from G2PM (a powerful neural G2P system); 5) Phoneme (pypinyin), PortaSpeech with phoneme labels derived from pypinyin (one of the most popular Chinese G2P system). As shown in Table Dict-TTS greatly surpasses the systems that implicitly model the semantic representations for character-to- torocttrto-rronnniiaaiion mapping like NLR [16] and shows comparable performance with phoneme- based systems. Since our Dict-TTS does not require any phoneme labels for training, can pre-train Dict-TTS on large-scaled ASR dataset [61] with small amount of effort to improve its generalization capacity. It can be seen that the pronunciation accuracy of Dict-TTS on the Biaobei dataset significantly improved by pre-traie training, which demonstrates the effectiveness of our unsupervised polyphone disambiguation framework.
4.3 Results of Audio Quality and Prosody
7 of Dict-TTS with the sys- tems evaluated in Subsection 4.2. GT (the ground truth audio) and GT (voc.) (the ground truth audio that is firstly converted into mel-spectrogram and converted back to audio wave- form using Hifi-GAN (29) are also included in this experiment. We keep the text content consistent among different models to exclude other interference factors, only examining the au- dio quality or prosody. Each audio is listened by at least 20 testers. For audio quality and prosody, we conduct the mean opinion score (MOS) evaluation via Amazon Mechanical Turk.
We compare the audio quality, audio prosody, and pitch accuracy
"e compute the average dynamic time warping (DTW) 36 distances between the pitch contours of ground-trui tounndtruuthaa speech and synthesized speech.

---

Table Pronunciation accuracy, audio prosody and audio quality comparisons for ablation study.
Settings
PER-O
PER-S
SER-S
CMOS-P
CMOS-Q
Dict-TTS
2.12%
1.08%
6,50%
0.000
0.000
w/o Semantics
2.15%
1.13%
6.50%
-0.280
-0.135
w/o Gumbel-Softmax
1.19%
7.75%
-0.014
-0.021
We analyze the MOS in two aspects: MOS-P (Prosody: naturalness of pitch, en- ergy, and duration) and MOS-Q (Quality: clarity, high-frequency, and original tim- bre reconstruction). We tell the tester to focus one corresponding aspect and ig- nore the other aspect when scoring. We put more information about the subjective evaluation in Appendix B33] As shown in Table 3 for audio quality, Dict-TTS sig- nificantly outperforms those TTS systems without phoneme labels and achieves a comparable performance with phoneme- based systems. And for audio prosody and pitch accuracy, Dict-TTS even sur- passes the phoneme-basee systems, which demonstrates the effectiveness of the ex- tracted semantics from prior dictionary knowledge. We put more analyses on the naturalness of prosody in Appendix
Table 3: The audio performance (MOS-Q and MOS-P) and pitch accuracy comparisons. DTW denotes average
dynamic time warping distances of pitch in ground-truth and synthesized audio. The mel-spectrograms are con-
verted to waveforms using Hifi-GAN (V3) 29
Method GT
MOS-P
MOS-Q
DTW
4.48.10.03
4.40
GT (vOc.)
4.37-10.04
4.26
Character BERT Embedding 15
3.82++0.08
3.88=0.07
53.1
3.88+0.07
3.63=0.10
55.0
NLR [16
3.83-1:0.08
3.74
Phoneme (G2PM)
3.87-10008
3.90
Phoneme (pypinyin)
3.89:10.08
3.95
52.6
Dict-TTS
4.03-4-005
3.91
52.4
E
We then visualize the mel-spectrograms generated by the above systems in Figure S We can see that Dict-TTS can generate mel-spectrograms with comparable details in harmonics, unvoiced frames, and higu-fryy parts with the phoneme-based system, which results in similar natural sounds. Moreover, our Dict-TTS can capture more accurate local changes in pitch and speaking duration, indicating the effectiveness of introducing semantic representations in the dictionary.
4.4 Ablation Studies
We conduct ablation studies to demonstrate the effectiveness of designs in Dict-TTS, including the auxiliary semantic information and the Gumbel-SSftune sample strategy. We conduct pronunciation accuracy and CMOS (comparative mean opinion score) evaluations for these ablation studies. The results are shown in Table We can see that CMOS-P drops when we remove the introduced semantic embeddings in Dict-TTS, indicating that the semantic information extracted from the dictionary can improve the audio prosody. Besides, to demonstrate the effectiveness of the Guumel--Sottax sample strategy, we also compare the weighted sum of the pronunciation embeddings with the Gumbel-Softmall sample strategy. Since measuring PER-O requires one-hot vectors, we do not calculate the PER-O score for the weight-sum version of Dict-TTS. The results shown in row in Table It can be seen that PER-S and SER-S increase when we use the weighted sum method. In the experiments, the weights of different pronunciations for some characters might be close to each other, which results in relatively worse performance in the subjective results. For example, the two pronunciations "ZH ANG3" and "CH ANG2" of the character "K" might be ambiguous when their weights are close to each other (e.g., 0.6 and 0.4). Therefore, to accurately model the subjective pronunciations, we utilize the Gumbel-Softmal function sample the most likely pronunciation in both training and inference stages.
Conclusion
In this paper, we proposed Dict-TTS, an unsupervised framework for polyphone disambiguation in end-to-end text-to-speech systems. Dict-TTS uses comaatics-to-pronnncee attention (S2PA) module to explicitly extract the corresponding pronunciations semantic information from prior dictionary knowledge. The S2PA module can be trained with the end-to-end TTS model with the

---

guidance of -peestrecr reconstruction loss without phoneme labels, which significantly reduces the cost of building polyphone disambiguation system and further enables the efficient pre-traiming on large-scaled ASR datasets. Our experimental results in three languages show that Dict-TTS outperforms several strong G2P baseline models in terms of pronunciation accuracy and improves the prosody modeling of the baseline TTS system. Further comprehensive ablation studies verify that each component in Dict-TTS effective. However, the dictionary knowledge does not contain the tree-structured syntactic information of the input text sequence, which also affects the prosody modeling. Moreover, since we crawl the dictionary from online websites and do not make any specific changes, the performance of Dict-TTS can be further improved by well-designed dictionary. In the future, we will try to inject syntactic information into Dict-TTS and extend to more languages.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (Grant No.62072397 and No.61836002) Zhejiang Natural Science Foundation (LR1 (LR19002000)), Yiwise, and National Key R&D Program of China (Grant F0.20200FF0832555), We appreciate the support from Mindspore, which is a new deep learning computing framework. This work was also supported by Alibaba Group through Alibaba Innovative Research Program.
10

---

References
[1] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-muliiingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4218-4222, 2020.
[2] Sercan Ô Ank, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yong- guo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, al. Deep voice: Real-time neural text-to-speech. In International Conference on Machine Learning, pages 195-204. PMLR, 2017.
[3] Data Baker. Chinese standard mandarin speech corpus, 2017.
[4] Alan W Black, Kevin Lenzo, and Vincent Pagel. Issues in building general letter sound rules. In The third ESCA/COCOSDA workshop (ETRW) on speech synthesis, 1998.
[5] Zexin Cai, Yaogen Yang, Chuxiong Zhang, Xiaoyi Qin, and Ming Li. Polyphone disambiguation for mandarin chinese using conditional neural network with multi-level embedding features. In Gernot Kubin and Zdravko Kacie, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 2110-2114 14. ISCA, 2019.
[6] Moon-Jung Chae, Kyubyong Park, Jinhyun Bang, Soobin Suh, Jonghyuk Park, Namju Kim, and Longhun Park. Convolutional sequence to sequence model with non-sequential greedy decoding for grapheme to phoneme conversion. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2486-2490. IEEE, 2018.
[7] Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie- Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. CoRR, abs/2009.01 0176,, 2020.
[8] Yi- Fi-Chan Chen, Yu-Chuan Chang, Yen-Cheng Chang, and Yi-Ren Yeh. g2pw: A condi- tional weighted softmax bert for polyphone disambiguation in mandarin. arXiv preprint arXiv:2203.10433 2022.
[9] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, 2019.
[10] Alexis Comneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, 2020.
[11] Dongyang Dai, Zhiyong Wu, Shiyin Kang, Xixin Wu, Jia Jia, Dan Su, Dong Yu, and Helen Meng. Disambiguation chinese polyphones an end-to-end framework with semantic features extracted by pre-trained bert. In Interspeech, pages 2090-2094, 2019.
[12] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18/0.:/8/0.04533 2018.
[13] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End- to-end adversarial text-to-speech. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[14] Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, R. Skerry Baarn, and Yonghui Wu. Parallel tacotron 2: A non- non-autoregressivee neural TTS model with differentiable duration model- ing. In Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlicek, editors, Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Crechia, 30 August September 2021, pages 141-145, ISCA, 2021.
11

---

[15] Tomoki Hayashi, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Shubham Toshniwal. and Karen Livescu. Pre-trained text embeddings for enhanced text-to-speech synthesis. In INTER- SPEECH, pages 4430 4434, 2019.
[16] Mutian He, Jingzhou Yang. Lei He, and Frank K Soong. Neural lexicon reader: Reduce pronunciation errors in end-to-end uss by leveraging external textual knowledge. arXiv preprint arXiv:21 10.09698, 2021.
[17] John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 1129-1188 2019.
[18] Rem Hida, Masaki Hamada, Chie Kamada, Emiru Tsunoo, Toshiyuki Sekiya, and Toshiyuki Kumakura. Polyphone disambiguation and accent prediction using pre-trained language models in japanese its front-end. arXiv preprini MiXiv:2201.09427,, 2022.
[19] Feng-Long Huang. Disambiguating effectively chinese polyphonic ambiguity based on unify approach. In 2008 International Conference Machine Learning and Cybernetics, volume 6, pages 1242-3246. IEEE, 2008
[20] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint 2eXiv:2204.09934 2022.
[21] Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: To- wards style transfer for generalizable out-of-domain text-to-speech synthesis. arXiv preprint 2rX5v:2205.07211, 2022.
[22] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Pro- gressive fast diffusion model for high-quality text-to-speech arXiv preprint arXiv:2207.06389, 2022.
[23] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 111.114611.01144, 2016.
[24] John T Jensen. Principles of generative phonology: An introduction, volume 250. John Benjamins Publishing, 2004.
[25] Ye Jia, Hciga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. Png BERT: augmented BERT on phonemes and graphemes for neural TTS. In Hynek Hermansky, Houza Cernocky, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek, editors, Interspeech 2021, 22nd Anmual Conference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021, pages 151-155. ISCA, 2021.
[26) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. General- ization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2019.
[27] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
[28] Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning. pages 5530-5540. PMLR, 2021.
[29] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 13:7732--77033 2020.
[30] Younggun Lee, Suwon Shon, and Taesu Kim. Learning pronunciation from a foreign language in speech synthesis networks. arXiv preprint arXiv:1811.09366 2018.
12

---

[31] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In The Thirty- Thirr- AAAI Conference on Artificial Intelligecce, AAAI
2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- F I, 2019, pages 6706-6713. AAAI Press, 2019.
[32] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11020-1 10028, 2022.
[33] Jinke Liu, Weiguang Qu, Xuri Tang, Yizhe Zhang, and Yuxia Sun. Polyphonic word disam- biguation with machine learning approaches. In 2010 Fourth International Conference on Genetic and Evolutionary Computing, pages 244-247. IEEE, 2010.
[34] Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. Lexicon enhanced chinese sequence labeling using bert adapter. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5847-5858, 2021.
[35] Maryanne Martin, Gregory V Jones, Douglas I. Nelson, and Louise Nelson. Heteronyms and polyphones: Categories of words with multiple phonemic representations. Behavior Research Methods & Instrumentation, 13(3):299-30 1981.
[36] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69 84. 2007.
[37] Jongseok Park, Kyubyong & Kim. g2pe. httt:://ggthuub..om//ybyyyyyy g2p 2019.
[38] Kyubyong Park and Seanie Lee. g2pm: A neural grapheme-to-phoneme conversion package for mandarin chinese based on a new open benchmark dataset. arXiv preprint arXiv:2004.031366 2020.
[39] Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, 2018.
[40] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[41] Yi Ren, Jinglin Liu, and Zhou Zhao Portaspeech: Portable and high-quality generative text-to- speech. Advances in Neural Information Processing Systems, 34, 2021.
[42] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32, 2019.
[43] Changhao Shan, Lei Xie, and Kaisheng Yao. A bi-directional Istm approach for polyphone disambiguation in mandarin chinese. In 20/6 10th International Symposium on Chinese Spoken Language Processing (ISCSLP), pages 1-5. IEEE, 2016.
[44] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv:1803.0 02155,, 2018.
[45] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779-4783. IEEE, 2018
[46] Alex Sokolov, Tracy Rohlin, and Ariya Rastrow. Neural machine translation for multilingual to-hhmee-t---oonemee conversion. arXiv preprint arXiv: 14/94, 2020.
[47] Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi Saruwatari. Jsut corpus: free large-scale japanese speech corpus for end-to-end speech synthesis. arXiv preprint arXiv: 171 1..11..011 1.00354, 2017.
13

---

[48] Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Token-level ensemble distillation for suppeem-to--honeme conversion. arXiv preprint arXiv: 1904.03444 2019.
[49] Hao Sun, Xu Tan, Jun- Wei Gan, Sheng Zhao, Dongxu Han, Hongzhi Liu, Tao Qin, and Tie- Yan Liu. Knowledge distillation from bert in pre-training and iin-tuning for polyphone disambiguation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 175 IEEE, 2019.
[50] Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2106. 15561, 2021.
[51] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, 2019.
[52] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA, 13-15 September 2016, page 125. ISCA, 2016
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Yonghe Wang, Feilong Bao, Hui Zhang, and Guanglai Gao. Joint alignment Geaming-attenntnn based model for seapheme-to-phoneee conversion. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7788-7792. IEEE, 2021.
[55] Yuxuan Wang, RJ Skerry-Raan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, al. Tacotron: Towards end-to-end speech synthesis. Proc. Interspeech 2017, pages 4006-4010, 2017.
[56] Tingyu Xia, Yue Wang, Yuan Tian, and Yi Chang. Using prior knowledge to guide bert's attention in semantic textual matching tasks. In Proceedings of the Web Conference 202/, pages 2465-2475, 2021.
[57] Bing Yang, Jiaqi Zhong, and Shan Liu. Pre-trained text representations for improving front-end text processing in mandarin text-to-speech synthesis. In INTERSPEECH, pages 4480 4484, 2019.
[58] Kaisheng Yao and Geoffrey Zweig. Sequence-to-sequence neural net models for grapheme-to- phoneme conversion. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015, pages 3330- 3334. ISCA, 2015.
[59] Zhenhui Ye, Zhou Zhao, Yi Ren, and Fei Wu. SyntaSpeech: Syntax-Awart Generative Adver- sarial Text-to-Speech. arXiv e-prints, page arXiv:2204.1 11792, April 2022.
[60] Mingzhi Yu, Hieu Duy Nguyen, Alex Sokolov, Jack Lepird, Kanthashree Mysore Sathyendra, Samridhi Choudhary, Athanasios Mouchtaris, and Siegfried Kunzmann. Multilingual grapheme- to-phoneme conversion with byte representation. In ICASSP 2020-2020 IEEE International IEEE,
Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2020.
[61] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6182-6186. IEEE, 2022.
[62] Guangyan Zhang, Kaitao Song, Xu Tan, Daxin Tan, Yuzi Yan, Yanqing Liu, Gang Wang, Wei Zhou, Tao Qin, Tan Lee, et al. sixeed- bert: Improving bert with mixed phoneme and sup-phoneme representations for text to speech. arXiv preprint arXiv:2203.1 17190, 2022.
14

---

[63] Haiteng Zhang, Huashan Pan, and Xiulin Li. A mase- model for mandarin chinese polyphone disambiguation. In INTERSPEECH, pages 1728-1732, 2020.
[64] Hong Zhang, JiangSheng Yu, WeiDong Zhan, and ShiWen Yu. Disambiguation of chinese poly- phonic characters. In The First International Workshop MultiMedia Annotation (MMA2001), volume I, pages 30-1, 2001.
[65] Yang Zhang, Liqun Deng, and Yasheng Wang. Unified mandarin tts front-end based on distilled bert model. arXiv preprint arXiv:2012.15404 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix[HI] H in the supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary materials.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Subsection 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]] We report confidence intervals of subjective metric results and describe the random seed settings in Appendix in the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Subsection 4.1
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Subsection 4.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix B.3 in the supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See Appendix [B.3 in the supplementary materials.
15


Reviews:
[{"review_id": "kR70CKfVFkH", "reviewer": "Reviewer_mE8x", "summary": "The paper proposes Dict-TTS that can infer the corresponding pronunciations for the given input text sequence by incorporating the prior information from the online website dictionary.  For deriving the pronunciations, the paper proposes a semantics-to-pronunciation attention (S2PA) module which finds the correct pronunciations by matching the semantic information between the input text sequence and the dictionary entries.  The proposed S2PA module can be incorporated into the end-to-end TTS system and can be trained simultaneously with the TTS system using the mel-spectrogram reconstruction loss.  The idea is interesting and is validated by extensive experiments on three datasets with different languages.", "questions": "1) In Section 4.4 “Ablation Studies”, the paper claims that “Dict-TTS successfully decompose the character representation, pronunciation, and semantics, which significantly improves the pronunciation accuracy.”  How could this conclusion be drawn from the ablation study experiments?  For the proposed Dict-TTS, removing the top two layers of the linguistic encoder only increases the PER and SER slightly.  But do the results indicate the success of decomposition (of character representation, pronunciation, and semantics)?  The paper should give more explanations about this, or should prevent over-claiming.\n\n2) Also in Section 4.4, the paper reports the results by removing the top two layers of the linguistic encoder for different systems including the character-based, phoneme-based and the Dict-TTS.  Why are these top two layers of the linguistic encoder important?  What about the other layers of the linguistic encoder?\n\n3) In Section 3.4, it is said that “our method are compatible with the predefined results … by directly adding specific rules to pronunciation weight $w_{i,j}$.  It would be interesting to elaborate more on how to add the rules to the pronunciation weight.\n\n4) In Section 3.3, for a pronunciation $p_{i,j}$, it may corresponds to a dictionary entry $e_{i,j}$ with several different items including different definition examples and different usage examples, etc.  For example, in Figure 1, the first pronunciation corresponds to two definition examples, and seven or eight usage examples.  Are these examples merged together as a single character sequence, leading to $[e_{i,j,1}, …, e_{i,j,u}]$ with $u$ characters?  The authors should give more details about this design.  Will the operation of merging different examples into a single character sequence affect the performance?\n\n5) In Section 3.3, some terms might be confusing and need be clarified. \n\n* Line 202, $a_{i,1}$ should be $a_{i,j}$?\n\n* Line 202, $c_i$ might be confused with the character $c_i$ in Dictionary (i.e. $c_i$ in Line 186).  The $c_i$ in Figure 4 should also be clarified.\n\n* Line 206, it is not clear what does the term $a_{i,j,k}$ mean.\n", "limitations": "Yes, the authors address the impacts and limitations in Appendix E. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The idea of using prior knowledge from the dictionary is interesting, and the method of using S2PA module is novel.  Specifically, the S2PA module incorporates the semantic information for polyphone disambiguation, which imitates the “dictionary lookup” practice in human daily life.\n\n2) The proposed method for polyphone disambiguation (grapheme to phoneme) can be trained in an unsupervised manner and can be trained simultaneously with the TTS modules in an end-to-end manner.  The method greatly eases the process for building a TTS system which directly accepts the raw text as input (i.e. character-based TTS system).\n\n3) The proposed method also provides the possibility to pre-train the model on large-scale ASR dataset to improve the generalization capacity for improving the polyphone disambiguation performance.\n\nWeaknesses:\n\nMy main concerns are mainly related to the experiments.  Please refer to the following Questions section for details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QmMjs-LLz8k", "reviewer": "Reviewer_ksPw", "summary": "This work presents a method to increase the pronunciation accuracy of a speech synthesis system without phoneme labels by using an online dictionary.", "questions": "The authors need to clarify the limitations of the method presented and the points described above.", "limitations": "Comments on the limitations of the work are mentioned above. \\\nThere is no negative societal impact.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. High intelligibility can be obtained without phoneme label in Text-to-Speech task using logogram.\\\n\n## Weaknesses\n1. The authors claim to present a method to solve the polyphone disambiguation problem, which is a problem in logograms such as Chinese, but is less problematic in phonograms such as English. The authors argue in this section that the characteristics of the logographic writing system can be easily extended to alphabetic languages (e.g., English) by replacing “character” with “word”, but this seems to be an inappropriate explanation that does not take into account that the phonogram basically displays the pronunciation unlike the logogram.\n\n1. The authors mention the loss of information caused by converting grapheme to phoneme in comparison with “Phoneme-based TTS systems”, but it is not related to conversion errors that occur during the conversion process to phoneme (An error in which Pinyin is converted differently from context or semantic meaning.), which the authors tried to solve. The problem of information loss in the conversion process to phoneme has already been explored in previous work such as [1] and [2], and this comparison is not appropriate because it is not the same as the problem claimed in this work.\n\n1. In addition, the authors excluded the space token from the comparison between “I scream” and “Icecream”, this is an inappropriate example given that unlike Chinese which the space token is not commonly used, the space token is information that greatly influences prosody prediction in English.\n\n1. The method presented in section 3.3 does not seem to be effective considering that the dictionary is composed of word units and that one grapheme can be mapped to multiple phonemes in many cases in phonograms(e.g., grapheme ‘e’ in English). The authors need to explain the limitations of the presented method and modify the scope of the claims.\n\n1. The authors claim that the character representations are well distributed in the semantic space. To support this, the authors need to prove how the semantic space is defined and well distributed. It is not appropriate to claim that the character representations are well distributed in the semantic space based solely on predicting pronunciation well.\n\n1. Referring to Table 1, in the case of Japanese, the performance is significantly lower than when using the open source G2P module, and the authors claim this as “which demonstrates the superiority of the explicit semantics matching in our S2PA module.”.\nIt is difficult to understand how this result demonstrates superiority.\nIn addition, the authors assert in the Conclusion as follows:\n“Our experimental results in three languages show that Dict-TTS outperforms several strong G2P baseline models in terms of pronunciation accuracy and improves the prosody modeling of the baseline TTS system.”\nClearly, it showed better performance for only two languages and a significant performance degradation for Japanese, so I think this part also has a problem with the argument, and I have doubts about the overall completeness and reliability of the paper.\n\n1. In terms of generalization, “Biaobei” and “HK” are composed entirely of logograms, and “JSUT” is different in that it is a mixture of phonograms and logograms. According to the experimental results presented by the authors, the proposed method has the potential to work only in the logograms, can be viewed as a solution that operates under specific conditions, and is not of high significance compared to solutions that can be generalized.\n\n\n[1] Jia, Ye, et al. \"PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS.\" arXiv preprint arXiv:2103.15060 (2021).\\\n[2] Kastner, Kyle, et al. \"Representation mixing for TTS synthesis.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OlLILDjgYuw", "reviewer": "Reviewer_nb5D", "summary": "Polyphone disambiguation is a big challenge for end-to-end TTS systems. The paper proposes Dict-TTS, an approach for phonemic disambiguation that can be trained jointly with an end-to-end neural TTS system. The key intuition behind the approach is that semantic context in the input text can be used to select the most appropriate pronunciation from a dictionary. To achieve this goal, the input text is first encoded using a transformer based semantic encoder. The output of this encoder is then used to compute attention weights over the entries in a dictionary using a semantic-to-pronunciation attention module. The embeddings of the most likely pronunciation and the corresponding semantic encoding are fed to the linguistic encoder in the end-to-end TTS system.  The paper shows that DictTTS outperforms character and phoneme baselines (obtained using commonly used grapheme-to-phoneme systems) in terms of pronunciation accuracy in Chinese, Japanese and Cantonese. Pre-training on an ASR dataset can further improve accuracy. The paper also shows that compared to a phoneme-based system, an end-to-end TTS system that incorporates DictTTS achieves similar overall MOS scores and better MOS-prosody scores.\n", "questions": "* The conclusions from the ablation study in Sec 4.4 is not very clear. The paper states that when the top-2 layers of the character based system are removed, the PER/SER increase rapidly but this is not the case for either the phoneme-based system or DictTTS. The paper claims this demonstrates that the Dict-TTS successfully decomposes the character representation, pronunciation, and semantics, which significantly improves the pronunciation accuracy. Why is this the case?\n* Why is the attention vector 2-dimensional in Equation 1 i.e. a_{i,1} … a_{i,m} but 3-dimensional on L206 i.e. w_{ij} = \\sum_{k=1}^{u} a_{ijk}?\n* Did you try an experiment where you used a weighted sum of the embeddings of the pronuncations/semantics vs using the most likely pronunciation?\n* Does the dictionary need to be limited to a smaller set of characters for a given input text? \nL184 says: \"D which contains a sequence of characters C = [c1 , c2 , ..., cn ], where n is the size of characters set in a language\" If the language has a large character set, computing an attention weight over the entire dictionary for each input text will be expensive. Do you need any computational shortcuts for the S2PA module to be practical?\n* Could you comment whether the noise in the dictionary affects the performance of this approach?\n\n* typo: L21/92 'there' -> 'they'\n\n", "limitations": "The authors have discussed some limitations in Sec 5. The authors have also discussed potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Proposes a new approach for polyphone disambiguation which can be jointly trained with the end-to-end TTS model using mel-spectrogram reconstruction loss without requiring phonemic labels, which are expensive to obtain.\n* Performance of the system improves by pre-training on an ASR corpus.\n* The approach yield improvements in pronunciation accuracy on three languages and improves in prosody of the underlying TTS system\n\nWeaknesses:\n* Some parts of the paper are hard to follow (see details below)\n\nUpdate: In their revisions, the authors have adequately addressed my concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GI1tqKa4zy_", "reviewer": "Reviewer_GMo7", "summary": "This paper describes an approach to leverage available human readable dictionaries to help improve TTS pronunciation modeling, specifically for polyphonous lexical items.  The aim here is to leverage the semantic information from the definition and example to guide selection of an appropriation pronunciations.  ", "questions": "* It would be useful to demonstrate how substantial a problem this is for a variety of languages.  How necessary is solving this problem for delivering high quality TTS?  Some additional motivation to this end could be helpful.\n\n* In section 3.3. how much preprocessing of the dictionary text is important here?  e.g. is the decomposition into \"Character\" \"Pronunciation\" \"Definitions\" \"Usages\" always used?  Are there ever errors to this decomposition?\n\n* In Section 3.3 it is claimed that \"..the model can easily deduce the correct pronunciation and semantics based on the lexicon knowledge like human brain\".  In what way is this like the human brain's processing?  Would any function that uses a \"semantic\" embedding to disambiguate pronunciation be \"like the human brain\"?\n\n* Section 3.4 Do you have any insight into the quality of the pronunciations obtained from the \"low-quality text-speech pairs\" obtained from ASR data?  E.g. Librispeech is divided into \"clean\" and \"other\" partitions based on how easy an ASR model can recognize them.  It would be interesting to understand the quality/quantity tradeoffs that are made in this work.\n\n* Table 1 - how much should PER-O be trusted as a gold standard? In some languages there are multiple valid pronunciations of worse e.g. English \"the\" pronounced as /ðə/ vs. /ðiː/.  The example pronunciations are quite different, but to what degree is some pronunciation variation tolerated?\n\n* Table 2 A comparison to PNG-BERT would make sense here since it is another approach to include semantics in the TTS frontend.  Is there a reason this is not used?\n\n* Table 3: is the Dict-TTS entry pretrained or not?\n\n* Table 3: is there an explanation for why the quality score is impacted by the pronunciation representation?  It is particularly interesting that the prosody score is positively impacted by the improved pronunciation modeling of Dict-TTS. How much interaction if any is there between pronunciation, lexical tone, and subjective prosody responses here?  \n\n* Table 3: the DTW measure would be impacted by both lexical tone and prosodic realization. Is there any effort to disentangle these when evaluating the impact of these approaches?\n\n* Table 4, Section 4.4. This ablation studey of removing two layers is somewhat strange.  Without retraining the model with fewer layers, there isn't an expectation that useful information would be available from lower layers to be used by a higher layer in a network.  ", "limitations": "Some limitations of the work are discussed.  The potential negative societal impact of the work is not addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Clear motivation, mostly very well described technical approach.\n* Good improvement over comparable studies.\n* Interesting use of distant or weak supervision for this task.\n\nWeaknesses\n* The problem being solved may be somewhat narrow, though important for TTS. \n* The ablation study of removing network layers is not altogether convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept