Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes Dict-TTS that can infer the corresponding pronunciations for the given input text sequence by incorporating the prior information from the online website dictionary.  For deriving the pronunciations, the paper proposes a semantics-to-pronunciation attention (S2PA) module which finds the correct pronunciations by matching the semantic information between the input text sequence and the dictionary entries.  The proposed S2PA module can be incorporated into the end-to-end TTS system and can be trained simultaneously with the TTS system using the mel-spectrogram reconstruction loss.  The idea is interesting and is validated by extensive experiments on three datasets with different languages.
strengths_and_weaknesses: Strengths:

1) The idea of using prior knowledge from the dictionary is interesting, and the method of using S2PA module is novel.  Specifically, the S2PA module incorporates the semantic information for polyphone disambiguation, which imitates the “dictionary lookup” practice in human daily life.

2) The proposed method for polyphone disambiguation (grapheme to phoneme) can be trained in an unsupervised manner and can be trained simultaneously with the TTS modules in an end-to-end manner.  The method greatly eases the process for building a TTS system which directly accepts the raw text as input (i.e. character-based TTS system).

3) The proposed method also provides the possibility to pre-train the model on large-scale ASR dataset to improve the generalization capacity for improving the polyphone disambiguation performance.

Weaknesses:

My main concerns are mainly related to the experiments.  Please refer to the following Questions section for details.

questions: 1) In Section 4.4 “Ablation Studies”, the paper claims that “Dict-TTS successfully decompose the character representation, pronunciation, and semantics, which significantly improves the pronunciation accuracy.”  How could this conclusion be drawn from the ablation study experiments?  For the proposed Dict-TTS, removing the top two layers of the linguistic encoder only increases the PER and SER slightly.  But do the results indicate the success of decomposition (of character representation, pronunciation, and semantics)?  The paper should give more explanations about this, or should prevent over-claiming.

2) Also in Section 4.4, the paper reports the results by removing the top two layers of the linguistic encoder for different systems including the character-based, phoneme-based and the Dict-TTS.  Why are these top two layers of the linguistic encoder important?  What about the other layers of the linguistic encoder?

3) In Section 3.4, it is said that “our method are compatible with the predefined results … by directly adding specific rules to pronunciation weight $w_{i,j}$.  It would be interesting to elaborate more on how to add the rules to the pronunciation weight.

4) In Section 3.3, for a pronunciation $p_{i,j}$, it may corresponds to a dictionary entry $e_{i,j}$ with several different items including different definition examples and different usage examples, etc.  For example, in Figure 1, the first pronunciation corresponds to two definition examples, and seven or eight usage examples.  Are these examples merged together as a single character sequence, leading to $[e_{i,j,1}, …, e_{i,j,u}]$ with $u$ characters?  The authors should give more details about this design.  Will the operation of merging different examples into a single character sequence affect the performance?

5) In Section 3.3, some terms might be confusing and need be clarified. 

* Line 202, $a_{i,1}$ should be $a_{i,j}$?

* Line 202, $c_i$ might be confused with the character $c_i$ in Dictionary (i.e. $c_i$ in Line 186).  The $c_i$ in Figure 4 should also be clarified.

* Line 206, it is not clear what does the term $a_{i,j,k}$ mean.

limitations: Yes, the authors address the impacts and limitations in Appendix E.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes an approach to leverage available human readable dictionaries to help improve TTS pronunciation modeling, specifically for polyphonous lexical items.  The aim here is to leverage the semantic information from the definition and example to guide selection of an appropriation pronunciations.  
strengths_and_weaknesses: Strengths
* Clear motivation, mostly very well described technical approach.
* Good improvement over comparable studies.
* Interesting use of distant or weak supervision for this task.

Weaknesses
* The problem being solved may be somewhat narrow, though important for TTS. 
* The ablation study of removing network layers is not altogether convincing.
questions: * It would be useful to demonstrate how substantial a problem this is for a variety of languages.  How necessary is solving this problem for delivering high quality TTS?  Some additional motivation to this end could be helpful.

* In section 3.3. how much preprocessing of the dictionary text is important here?  e.g. is the decomposition into "Character" "Pronunciation" "Definitions" "Usages" always used?  Are there ever errors to this decomposition?

* In Section 3.3 it is claimed that "..the model can easily deduce the correct pronunciation and semantics based on the lexicon knowledge like human brain".  In what way is this like the human brain's processing?  Would any function that uses a "semantic" embedding to disambiguate pronunciation be "like the human brain"?

* Section 3.4 Do you have any insight into the quality of the pronunciations obtained from the "low-quality text-speech pairs" obtained from ASR data?  E.g. Librispeech is divided into "clean" and "other" partitions based on how easy an ASR model can recognize them.  It would be interesting to understand the quality/quantity tradeoffs that are made in this work.

* Table 1 - how much should PER-O be trusted as a gold standard? In some languages there are multiple valid pronunciations of worse e.g. English "the" pronounced as /ðə/ vs. /ðiː/.  The example pronunciations are quite different, but to what degree is some pronunciation variation tolerated?

* Table 2 A comparison to PNG-BERT would make sense here since it is another approach to include semantics in the TTS frontend.  Is there a reason this is not used?

* Table 3: is the Dict-TTS entry pretrained or not?

* Table 3: is there an explanation for why the quality score is impacted by the pronunciation representation?  It is particularly interesting that the prosody score is positively impacted by the improved pronunciation modeling of Dict-TTS. How much interaction if any is there between pronunciation, lexical tone, and subjective prosody responses here?  

* Table 3: the DTW measure would be impacted by both lexical tone and prosodic realization. Is there any effort to disentangle these when evaluating the impact of these approaches?

* Table 4, Section 4.4. This ablation studey of removing two layers is somewhat strange.  Without retraining the model with fewer layers, there isn't an expectation that useful information would be available from lower layers to be used by a higher layer in a network.  
limitations: Some limitations of the work are discussed.  The potential negative societal impact of the work is not addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work presents a method to increase the pronunciation accuracy of a speech synthesis system without phoneme labels by using an online dictionary.
strengths_and_weaknesses: ## Strengths
1. High intelligibility can be obtained without phoneme label in Text-to-Speech task using logogram.\

## Weaknesses
1. The authors claim to present a method to solve the polyphone disambiguation problem, which is a problem in logograms such as Chinese, but is less problematic in phonograms such as English. The authors argue in this section that the characteristics of the logographic writing system can be easily extended to alphabetic languages (e.g., English) by replacing “character” with “word”, but this seems to be an inappropriate explanation that does not take into account that the phonogram basically displays the pronunciation unlike the logogram.

1. The authors mention the loss of information caused by converting grapheme to phoneme in comparison with “Phoneme-based TTS systems”, but it is not related to conversion errors that occur during the conversion process to phoneme (An error in which Pinyin is converted differently from context or semantic meaning.), which the authors tried to solve. The problem of information loss in the conversion process to phoneme has already been explored in previous work such as [1] and [2], and this comparison is not appropriate because it is not the same as the problem claimed in this work.

1. In addition, the authors excluded the space token from the comparison between “I scream” and “Icecream”, this is an inappropriate example given that unlike Chinese which the space token is not commonly used, the space token is information that greatly influences prosody prediction in English.

1. The method presented in section 3.3 does not seem to be effective considering that the dictionary is composed of word units and that one grapheme can be mapped to multiple phonemes in many cases in phonograms(e.g., grapheme ‘e’ in English). The authors need to explain the limitations of the presented method and modify the scope of the claims.

1. The authors claim that the character representations are well distributed in the semantic space. To support this, the authors need to prove how the semantic space is defined and well distributed. It is not appropriate to claim that the character representations are well distributed in the semantic space based solely on predicting pronunciation well.

1. Referring to Table 1, in the case of Japanese, the performance is significantly lower than when using the open source G2P module, and the authors claim this as “which demonstrates the superiority of the explicit semantics matching in our S2PA module.”.
It is difficult to understand how this result demonstrates superiority.
In addition, the authors assert in the Conclusion as follows:
“Our experimental results in three languages show that Dict-TTS outperforms several strong G2P baseline models in terms of pronunciation accuracy and improves the prosody modeling of the baseline TTS system.”
Clearly, it showed better performance for only two languages and a significant performance degradation for Japanese, so I think this part also has a problem with the argument, and I have doubts about the overall completeness and reliability of the paper.

1. In terms of generalization, “Biaobei” and “HK” are composed entirely of logograms, and “JSUT” is different in that it is a mixture of phonograms and logograms. According to the experimental results presented by the authors, the proposed method has the potential to work only in the logograms, can be viewed as a solution that operates under specific conditions, and is not of high significance compared to solutions that can be generalized.


[1] Jia, Ye, et al. "PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS." arXiv preprint arXiv:2103.15060 (2021).\
[2] Kastner, Kyle, et al. "Representation mixing for TTS synthesis." ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.
questions: The authors need to clarify the limitations of the method presented and the points described above.
limitations: Comments on the limitations of the work are mentioned above. \
There is no negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Polyphone disambiguation is a big challenge for end-to-end TTS systems. The paper proposes Dict-TTS, an approach for phonemic disambiguation that can be trained jointly with an end-to-end neural TTS system. The key intuition behind the approach is that semantic context in the input text can be used to select the most appropriate pronunciation from a dictionary. To achieve this goal, the input text is first encoded using a transformer based semantic encoder. The output of this encoder is then used to compute attention weights over the entries in a dictionary using a semantic-to-pronunciation attention module. The embeddings of the most likely pronunciation and the corresponding semantic encoding are fed to the linguistic encoder in the end-to-end TTS system.  The paper shows that DictTTS outperforms character and phoneme baselines (obtained using commonly used grapheme-to-phoneme systems) in terms of pronunciation accuracy in Chinese, Japanese and Cantonese. Pre-training on an ASR dataset can further improve accuracy. The paper also shows that compared to a phoneme-based system, an end-to-end TTS system that incorporates DictTTS achieves similar overall MOS scores and better MOS-prosody scores.

strengths_and_weaknesses: Strengths:
* Proposes a new approach for polyphone disambiguation which can be jointly trained with the end-to-end TTS model using mel-spectrogram reconstruction loss without requiring phonemic labels, which are expensive to obtain.
* Performance of the system improves by pre-training on an ASR corpus.
* The approach yield improvements in pronunciation accuracy on three languages and improves in prosody of the underlying TTS system

Weaknesses:
* Some parts of the paper are hard to follow (see details below)

Update: In their revisions, the authors have adequately addressed my concerns.
questions: * The conclusions from the ablation study in Sec 4.4 is not very clear. The paper states that when the top-2 layers of the character based system are removed, the PER/SER increase rapidly but this is not the case for either the phoneme-based system or DictTTS. The paper claims this demonstrates that the Dict-TTS successfully decomposes the character representation, pronunciation, and semantics, which significantly improves the pronunciation accuracy. Why is this the case?
* Why is the attention vector 2-dimensional in Equation 1 i.e. a_{i,1} … a_{i,m} but 3-dimensional on L206 i.e. w_{ij} = \sum_{k=1}^{u} a_{ijk}?
* Did you try an experiment where you used a weighted sum of the embeddings of the pronuncations/semantics vs using the most likely pronunciation?
* Does the dictionary need to be limited to a smaller set of characters for a given input text? 
L184 says: "D which contains a sequence of characters C = [c1 , c2 , ..., cn ], where n is the size of characters set in a language" If the language has a large character set, computing an attention weight over the entire dictionary for each input text will be expensive. Do you need any computational shortcuts for the S2PA module to be practical?
* Could you comment whether the noise in the dictionary affects the performance of this approach?

* typo: L21/92 'there' -> 'they'


limitations: The authors have discussed some limitations in Sec 5. The authors have also discussed potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The reviewers generally liked the proposed approach in this paper, agreed that it is novel, and that the experiments showed good improvements over reasonable baselines. There was broad concern about the ablation study in the original paper (one shared by the AC), but the authors revised that section during the discussion period to the satisfaction of three of the reviewers. While three reviewers recommend that the paper be accepted, one reviewer recommends a borderline reject. The reviewer stuck to this recommendation after the discussion period, primarily citing concerns about whether or not the method is broadly applicable versus being limited primarily to being useful for logographic languages. While I am recommending that this paper be accepted, I urge the authors to expand their discussion of the limitations of the method in Appendix G. I think the discussion with reviewer ksPw of the JSUT results and the fact that Japanese writing comprises both more alphabetic and more logographic elements would be a valuable addition to that appendix and would help to clarify the contributions and limitations of the proposed method.

confidence: Certain
Decision: Accept