Abstract: In this paper, we propose a hybrid text normalization system using multi-head self-attention. The system combines the advantages of a rule-based model and a neural model for text preprocessing tasks. Previous studies in Mandarin text normalization usually use a set of hand-written rules, which are hard to improve on general cases. The idea of our proposed system is motivated by the neural models from recent studies and has a better performance on our internal news corpus. This paper also includes different attempts to deal with imbalanced pattern distribution of the dataset. Overall, the performance of the system is improved by over 1.5% on sentence-level and it has a potential to improve further.

Introduction: Text Normalization (TN) is a process to transform non-standard words (NSW) into spoken-form words (SFW) for disambiguation. In Text-To-Speech (TTS), text normalization is an essential procedure to normalize unreadable numbers, symbols or characters, such as transforming “$20” to “twenty dollars” and “@” to “at”, into words that can be used in speech synthesis. The surrounding context is the determinant for ambiguous cases in TN. For example, the context will decide whether to read “2019” as year or a number, and whether to read “10:30” as time or the score of a game. In Mandarin, some cases depend on language habit instead of rules- “2” can either be read as “r” or “ling” and “1” as “y” or “yo”. Currently, based on the traditional taxonomy approach for NSWBIBREF0, the Mandarin TN tasks are generally resolved by rule-based systems which use keywords and regular expressions to determine the SFW of ambiguous wordsBIBREF1, BIBREF2. These systems typically classify NSW into different pattern groups, such as abbreviations, numbers, etc., and then into sub-groups, such as phone number, year, etc., which has corresponding NSW-SFW transformations. ZhouBIBREF3 and JiaBIBREF4 proposed systems which use maximum entropy (ME) to further disambiguate the NSW with multiple pattern matches. For the NSW given the context constraints, the highest probability corresponds to the highest entropy. LiouBIBREF5 proposed a system of data-driven models which combines a rule-based and a keyword-based TN module. The second module classifies preceding and following words around the keywords and then trains a CRF model to predict the NSW patterns based on the classification results. There are some other hybrid systemsBIBREF6, BIBREF7 which use NLP models and rules separately to help normalize hard cases in TN. For recent NLP studies, sequence-to-sequence (seq2seq) models have achieved impressive progress in TN tasks in English and RussianBIBREF8, BIBREF9. Seq2seq models typically encode sequences into a state vector, which is decoded into an output vector from its learnt vector representation and then to a sequence. Different seq2seq models with bi-LSTM, bi-GRU with attention are proposed in BIBREF9, BIBREF10. Zhang and Sproat proposed a contextual seq2seq model, which uses a sliding-window and RNN with attentionBIBREF8. In this model, bi-directional GRU is used in both encoder and decoder, and the context words are labeled with “$\langle $self$\rangle $”, helping the model distinguish the NSW and the context. However, seq2seq models have several downsides when directly applied in Mandarin TN tasks. As mentioned in BIBREF8, the sequence output directly from a seq2seq model can lead to unrecoverable errors. The model sometimes changes the context words which should be kept the same. Our experiments produce similar errors. For example, “Podnieks, Andrew 2000” is transformed to “Podncourt, Andrew Two Thousand”, changing “Podnieks” to “Podncourt”. These errors cannot be detected by the model itself. In BIBREF11, rules are applied to two specific categories to resolve silly errors, but this method is hard to apply to all cases. Another challenge in Mandarin is the word segmentation since words are not separated by spaces and the segmentation could depend on the context. Besides, some NSW may have more than one SFW in Mandarin, making the seq2seq model hard to train. For example, “UTF8gbsn两千零八年” and “UTF8gbsn二零零八年” are both acceptable SFW for “2008UTF8gbsn年”. The motivation of this paper is to combine the advantages of a rule-based model for its flexibility and a neural model to enhance the performance on more general cases. To avoid the problems of seq2seq models, we consider the TN task as a multi-class classification problem with carefully designed patterns for the neural model. The contributions of this paper include the following. First, this is the first known TN system for Mandarin which uses a neural model with multi-head self-attention. Second, we propose a hybrid system combining a rule-based model and a neural model. Third, we experiment with different approaches to deal with imbalanced dataset in the TN task. The paper is organized as follows. Section SECREF2 introduces the detailed structure of the proposed hybrid system and its training and inference. In Section SECREF3, the performance of different system configurations is evaluated on different datasets. And the conclusion is given in Section SECREF4.

Question: What models do they compare to?
Answer: six different variations of their multi-head attention model