Venue: NAACL 2019

# Customizing Graphene-to-Phoneme System for Non-Trivial Transcription Problems in Bangla Language

Sudipta Saha Shubba

Bangladesh University of Engineering and Technology (BUET)

Nafis Sadeq

Bangladesh University of Engineering and Technology (BUET)

Shafayat Ahmed

Bangladesh University of Engineering and Technology (BUET)

Md. Nahidul Islam

Bangladesh University of Engineering and Technology (BUET)

Muammad Abdullah Adnan

Bangladesh University of Engineering and Technology (BUET)

Md. Yasin Ali Khan

Samsung R&D Institute, Bangladesh

###### Abstract

Graphene to phoneme (G2P) conversion is an integral part in various text and speech processing systems, such as: Text to Speech system, Speech Recognition system, etc. The existing methodologies for G2P conversion in Bangla language are mostly rule-based. However, data-driven approaches have proved their superiority over rule-based approaches for large-scale G2P conversion in other languages, such as: English, German, etc. As the performance of data-driven approaches for G2P conversion depend largely on pronunciation lexicon on which the system is trained, in this paper, we investigate on developing an improved training lexicon by identifying and categorizing the critical cases in Bangla language and include those critical cases in training lexicon for developing a robust G2P conversion system in Bangla language. Additionally, we have incorporated nasal vowels in our proposed phoneme list. Our methodology outperforms other state-of-the-art approaches for G2P conversion in Bangla language.

## 1 Introduction

Grapheme to phoneme (G2P) conversion provides a mapping between a word and its pronunciation. Such mapping provides opportunity for a nonnative person to learn the correct pronunciation of words of a foreign language. Moreover, in modern Text to Speech (TTS) and Automatic Speech Recognition (ASR) systems, G2P conversion is an integral task.

The task of G2P conversion is generally language specific due to language specific conventions, rules, pronunciation constraints, etc. In this paper, we focus on Modern Standard Bangla. An example of G2P conversion in Bangla language: phonetic transcription of \(\frac{3}{40}\frac{7}{80}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90} \frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac{1}{90}\frac {1}{90}\frac{1}{90}\

Our major contributions are as follows:

1. We identify and categorize the critical cases for grapheme to phoneme (G2P) conversion in Bangla language by analyzing the most frequent \(100K\) words.
2. We enrich the training lexicon for developing a robust G2P conversion system in Bangla language that performs much better for critical cases compared to other state-of-the-art G2P systems.
3. We perform phonetic transcriptions considering nasal vowels as separate phonemes.
4. We perform extensive simulations on large-scale dataset and show that our methodology outperforms other state-of-the-art approaches for G2P conversion in Bangla language by providing word-level accuracy of \(90.2\%\).

The rest of the paper is organized as follows: we discuss the previous works in Section 2, our phoneme list in Section 3, identification of critical cases and categorization of errors in Section 4, development of our system in Section 5, experimental results in Section 6, and conclusion and future works in Section 7.

## 2 Previous Works

The research works for G2P in English are quite extensive. Chen (2003) investigate machine learning based systems for G2P in English. They experiment with joint maximum entropy n-gram model, conditional maximum entropy model, etc. Yao and Zweig (2015) utilize bi-directional LSTM (Long Short Term Memory) recurrent neural network for G2P and achieve \(5.45\%\) PER on CMU dictionary (air, 2015). Thu et al. (2016) show comparisons among various machine learning algorithms for G2P in Burmese language. Joint sequence n-gram models aim to discover joint vocabulary consisting of graphemes and phonemes through the alignment of graphemes and phonemes. Bisani and Ney (2008) develop a joint-sequence model for G2P. Novak (2012), Novak et al. (2013) are other prominent works working on this model. Neural sequence to sequence models are popular for G2P conversion. Some prominent works on such models are: Caruana (1997), Jiampojamarn et al. (2007), Sutskever et al. (2014), Yao and Zweig (2015), Jianpojamarn et al. (2007), Rao et al. (2015), Yao and Zweig (2015), Schnober et al. (2016), Tsvetkov et al. (2016), He et al. (2016), Wu et al. (2016), Johnson et al. (2016), Toshniwal and Livescu (2016), and Vaswani et al. (2017).

Again, another line of research deals with G2P conversion for more than one language. Such works include: Mana et al. (2001), Kim and Snyder (2012), Deri and Knight (2016), and Milde et al. (2017).

Most of the works related to G2P conversion that are focused on Bangla language, follow rule-based approach. Rule-based approach of Mosaddeque et al. (2006) provides accuracy of \(97.01\%\) on a previously seen corpus containing \(736\) words, but the system's accuracy is \(81.95\%\) on an previously unobserved corpus containing \(8399\) words. This work was extended by Alam et al. (2011) describing \(3880\) rules with an accuracy of \(89.48\%\) on another corpus. Basu et al. (2009) discuss a rule-based approach considering several information: parts-of-speech, subsequent context, etc. Their work describes only \(21\) rules and provides an accuracy of \(91.48\%\) on a corpus of \(9294\) words. Ghosh et al. (2010) provide a heuristic for G2P that takes into account parts-of-speech, orthographic, and contextual information. Their work provides \(70\%\) accuracy on a corpus containing of \(755\) words. A prominent work for data driven G2P in Bangla language is by Google (Gutkin et al. (2016)). They develop a lexicon and achieve word-level accuracy of \(81.5\%\). Chowdhury et al. (2017) use conditional random field for G2P in Bangla. They report \(14.88\%\) phoneme error rate on Google lexicon.

## 3 Phoneme List

Our Phoneme symbols are provided in Table 1. This table is a good reference for the \(47\) phoneme symbols that we have followed in this paper and their corresponding International Phonetic Alphabet (IPA) symbols. Throughout the paper, we use these \(47\) phoneme symbols, not the IPA symbols. There is disagreement between linguists whether nasal vowels should be considered as separate phonemes (Barman, 2009). We added nasal vowels in our phoneme list to differentiate between a word with its nasal counterpart, such as the word "|/|!|(to cry) and "|!||(mu)d. Here, "|/|e/|/|n/|i/|o/|O/|E/|an/|en/|un/|in/|on/|On/|En/|re normal vowels, "|/|ow/|uw/|iw/|iw/|iw/|iw/|w 



## 4 Identification and Categorization of Non-Trivial Cases for Transcription

We envision of developing a robust G2P system that will perform reasonably well on any word in Bangla language. A G2P system that performs well on the most frequent words, should also do well on other words. With this motivation, we focus on increasing accuracy on the most frequent words. Especially, we are concerned about those words that are among the most frequent words but non-trivial or critical for phonetic transcription, i.e., current state-of-the-art methodologies perform poorly on these critical words. We investigate on identifying and categorizing such non-trivial or critical cases so that future research works can give special focus on developing methods for improving phonetic transcriptions of these critical words.

### Identifying the Most Frequent Words

To get a hold of contemporary usage of Bangla language, we do extensive crawling. We crawled \(42\) websites of various Bangla newspapers, blogs, e-book libraries, wikipedia, etc. covering various domains such as: politics, economics, sports, drama, novel, story, education, entertainment, general knowledge, history, etc. After data cleaning and data normalization, we had about \(10M\) sentences. We counted how many times each of the unique words appeared in those sentences. We then consider the most frequent \(100K\) words and aim to identify the critical cases for phonetic transcription among these most frequent words.

### Identifying the Critical Cases for Transcription

After changing the Google lexicon (of size \(60K\) (around)) according to our phoneme symbols (Table 1), we prepare \(4\) versions of Google's lexicon of size \(12K\), \(24K\), \(40K\), and \(60K\) respectively for identifying the critical cases for phonetic transcription. Algorithm 1 shows prefix comparing algorithm that we use for compressing a phonetic lexicon or dictionary of grapheme sequence to phoneme sequence. The algorithm matches the prefix of consecutive words (grapheme sequence) of a sorted dictionary (sorted according to ascending order of grapheme sequence of a word) and keeps a word (with its corresponding phoneme sequence) only if it does not share its prefix with any other words. We run the algorithm successively \(3\) times, i.e., we use the destination dictionary of one iteration as the source dictionary of next iteration. Each iteration produces a minimized version of the basic lexicon (Google lexicon). After \(3\) iterations, the dictionary does not get any more compressed. We find the phonetic transcriptions of each of the \(100K\) most frequent words using models trained on each of the \(4\) versions of Google's lexicon (basic+3 minimized). So, from 4 models (each model trained on a version of the basic Google lexicon), we get \(4\) sets of transcriptions for the most frequent \(100K\) words. For most of the words (around \(70K\) words), we observe that the phonetic transcriptions are exactly same in each of the \(4\) set. However, for the remaining \(30K\) words (\(29105\) words to be exact), we observe that at least one set provides different transcription. We take these \(30K\) words to be the critical cases. Our intuition is that if two G2P systems: one trained on a smaller version of the basic lexicon, and another trained on a larger version of the basic lexicon provide the same transcription for a word, then the word is a trivial case for phonetic transcription. We then manually verify the phonetic transcriptions of these \(30K\) words taking help from \(3\) linguists and following Chowdhury (2016), and consider these \(30K\) words as critical cases for phonetic transcription.

### Categorizing the Critical Cases

We categorize the critical cases into \(7\) categories and observe the distribution of the critical transcriptions into these \(7\) categories. These \(7\) categories

\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|l|} \hline  & & & & & & \\ \cline{2-8} \multirow{-2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} } & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{\begin{tabular}{c} **R** \\ \end{tabular} }} & \multicolumn{1}{c|}{\multirow{2}{*}{
\begin{tabular}{c} **T** \\ \end{tabular} }} \\ \hline o & o & kh & k\({}^{\text{h}}\) & th & t\({}^{\text{h}}\) & l & 1 \\ \hline E & c & g & g & d & d & h & fi \\ \hline O & \(\supset\) & gh & g\({}^{\text{fi}}\) & dh & d\({}^{\text{fli}}\) & s & s \\ \hline a & a & c & t\({}^{\dagger}\) & p & p & sh & f \\ \hline in & j & ch & q\({}^{\text{f}}\) & ph & p\({}^{\text{h}}\) & iv & i \\ \hline un & u & j & d\({}_{5}\) & b & b & ew & e \\ \hline en & e & jh & d\({}_{5}\) & bh & b\({}^{\text{h}}\) & ow & q \\ \hline on & o & T & t & N & p & uv & u \\ \hline En & \(\bar{\mathbf{x}}\) & Th & t\({}^{\text{h}}\) & n & n & & \\ \hline \end{tabular}
\end{table}
Table 1: Our Phoneme Symbols with Their Corresponding IPA Symbols

gories capture most of the errors. The categories are:

* **Open Close Vowel Confusion:** G2P system provides pronunciation as close vowel that should be pronounced as open vowel ideally, and vice-versa. For example, correct phoneme of (frog) is /b E n g/, but if G2P system provides output /b e n g/, then it is an error under this category as in the place of open vowel (here, /E/), G2P system is giving close vowel (here, /e/).
* **Inherent Vowel Confusion:** G2P system does not provide inherent vowel as output where there should be an inherent vowel ideally. For example, correct phoneme of (morning) is /sh O k a l/, but if G2P system provides output /sh k a l/, then it is an error under this category as the output of G2P does not give inherent vowel (here, /O/).
* **Diphthong Confusion:** G2P system does not provide falling diphthong in output where there should be a falling diphthong ideally. Or, system does not provide rising diphthong in output where there should be a rising diphthong ideally. For example, correct phoneme of (friend) is /sh o iw/, but if G2P system provides output /sh o i/, then it is an error under this category as the output of G2P does not capture the falling diphthong (here, /o iw/).
* **s or sh Confusion:** G2P system provides /s/ in phonetic transcription, where there should be /sh/, and vice-versa. For example, correct phoneme sequence of (organization) is /sh O N g O Th o n/, but if G2P system provides output /s O N g O Th o n/, then it is an error under this category as the output of G2P gives /s/ in place of /sh/.
* **s or ch Confusion:** G2P system provides /s/ in phonetic transcription, where there should be /ch/, and vice-versa. For example, correct phoneeme sequence of (umbrella) is /ch at a t/, but if G2P system provides output /s at a/, then it is an error under this category as the output of G2P gives /s/ in place of /ch/.
* **Nasal Confusion:** G2P system does not provide any nasal vowel where there should be a nasal vowel, and vice-versa. For example, correct phoneme sequence of (moon) is /c an d/, but if G2P system provides output /c a d/, then it is an error under this category as the output of G2P gives /a/ in place of /an/.
* **Other Vowel Confusion:** G2P system provides completely different vowel than the corresponding vowel that should ideally be in that position of the phoneme sequence. Note that, in the other error categories, for each position in the phoneme sequence, the generated and ideal phonemes were somehow related. But in this category, at a specific position of the phoneme sequence, the generated and ideal phonemes are completely different. For example, correct phoneme sequence of (perseverance) is /o d dh o b O sh a ew/, but if G2P system provides output /o d dh a b O sh a ew/, then it is an error under this category as the output of G2P gives /a/ in place of /o/ (fourth phoneme).

Algorithm 2 compares a machine-generated lexicon with a reference lexicon (manually verified), where both the lexicons have same grapheme sequences, but the corresponding phoneme sequences may be different. This algorithm counts how many errors of each category are there in the machine generated lexicon. The algorithm takes each entry of the generated lexicon and increases the count of the corresponding error category (if an error is present there).

We train attention mechanism based Transformer model on each of the \(4\) lexicons and get 

\(4\) G2P models. We find the phoneme representation of \(30K\) critical cases using each of the \(4\) G2P models. Using Algorithm 2, we count the errors of each category for each of the \(4\) models. We report the results in Table 2 and Figure 1. Here, other error denotes the errors that are not captured by these \(7\) categories. We see from these results that most of the errors are under Open Close vowel, s or sh, Diphthong, and Inherent Vowel confusions.

## 5 Developing an Improved G2P System for Bangla Language

We develop an improved lexicon and use two machine learning based models trained on our lexicon to develop an improved G2P system for Bangla language.

### Developing an Improved Lexicon

For developing an improved lexicon, we include with Google's lexicon the manually verified \(30K\) non-trivial or critical entries. Also, we include the \(70K\) entries in which all of the \(4\) models (trained on each of the \(4\) versions of Google lexicon) unanimously agreed. After removing the repeated entries with same grapheme sequence, our lexicon consists of around \(90K\) entries. In case of repeated entries having same grapheme sequence, we carefully keep only the entry that has been manually verified.

```
1:\(ocSet\leftarrow[(``O",``o''),(``E",``e"),\)
2:\((``On",``on"),(``En",``en")]\)
3:if\((x,y)\) in \(occSet\)then
4:return True
5:else
6:return False
```

**Algorithm 4** Procedure: removeVowel (phoneme_sequence)

### G2P Models

We use Neural Sequence to Sequence models. In these models, a conditional distribution of a sequence (here, phoneme sequence) is learned conditioned on another sequence (here, grapheme sequence). We train two following Sequence to Sequence models on our lexicon for G2P conversion:

**LSTM-RNN**: This is a plain Sequence to Sequence model that incorporates an encoder and decoder mechanism. Recurrent Neural Network 

(RNN) is usually utilized in encoder and decoder design. For addressing vanishing gradient problem in RNN, Long-Short Term Memory (LSTM) Hochreiter and Schmidhuber (1997) is used. We follow Yao and Zweig (2015) for implementation.

**Transformer Model**: Transformer Model uses attention mechanism. Attention mechanism provides improvement upon plain Sequence to Sequence by easing the flow of information from source sequence to destination sequence. We follow Vaswani et al. (2017) for implementation.

We show the performance of both of these models in Section 6. We observe that Transformer Model provides higher token-level accuracy (lower Word Error Rate) than LSTM-RNN.

## 6 Experimental Results

We run extensive simulations and use two measures for evaluating the performances of the systems:

**Word Error Rate (WER):** For calculating Word Error Rate (WER), we use the following formula:

\[{\rm WER}=\frac{E}{T}\]

where \(E\) denotes the number of words that have disagreement in their generated phoneme sequence and reference phoneme sequence, and \(T\) denotes the total number of words.

**Phoneme Error Rate (PER):** For calculating Phoneme Error Rate (PER), we use the following formula:

\[{\rm PER}=\frac{I+S+D}{T}\]

where \(I\), \(S\), \(D\) denote respectively the total number of insertion, substitution, and deletion operations needed for all the words to align the generated phoneme sequence with the reference phoneme sequence for each word. \(T\) denotes the total number of phonemes present in all the words.

Our best performing model is Transformer Model. We use batch size of 4096. Our neural network has 3 hidden layers, each containing 256 nodes. We use a computer having 8GB RAM, Intel Core i7 CPU, and Nvidia Geforce 1050 GPU for running all of the simulations. For each model, we run the simulations for around \(110K\) iterations taking around \(5\) hours.

### Performance on Critical Cases

We report the experiment results of Google's lexicon on critical cases in Table 3. We do not report our lexicon here as critical cases are already included in our lexicon.

### Performance Comparison In General

For comparing the performances of models trained on our lexicon and Google's lexicon, we randomly

Figure 1: Categorization of errors in critical cases, here each of the \(60K,40K,24K,\text{and}~{}12K\) denotes the model trained on that particular lexicon.

 

take \(9000\) entries from our manually verified \(30K\) critical cases as test set. We use this test set for evaluating all the models. Though our actual lexicon contains these \(9000\) entries, we do not keep them in our lexicon while doing the experiments to fairly evaluate the performances of the lexicons. For both lexicons, we keep 90% of the lexicon in train set and remaining 10% in validation set. Table 4 shows the result. Models trained on our lexicon outperforms those trained on Google's lexicon by a significant margin. Moreover, Transformer Model performs better than LSTM-RNN.

Figure 2 and Table 5 categorize the errors of systems trained on \(3\) types of lexicons (Romanized version of our lexicon is discussed in section 6.3) by using Algorithm 2. Here, we report the results of Transformer Model only as it has been better performing that LSTM-RNN in our experiments. We observe most of the errors are related to Open Close vowel, s or sh, Diphthong, and Inherent Vowel confusions - this finding also conforms to Figure 1 and Table 2, which were error categorization of critical cases.

### Effect of Romanization

For doing experiments on the effect of romanization on G2P conversion, we romanized all grapheme sequences in our lexicon to prepare romanized counterpart of our lexicon. During romanization, each grapheme symbol in Bangla is replaced with a single English letter except that if a consonant grapheme is not followed by a vowel grapheme, "O" was added after the romanized symbol of that consonant as the roman symbol for ""g", which is usually inherently pronounced in such cases. All the symbols used for romanization were completely disjoint to avoid any ambiguity in the lexicon. Figure 3 shows the WER and PER

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**Lexicon** & **Model** & **WER** & **PER** \\  & & **(\%)** & **(\%)** \\ \hline \multirow{2}{*}{Google} & LSTM-RNN & 17.1 & 2.32 \\ \cline{2-4}  & Transformer & 14.8 & 1.88 \\ \cline{2-4}  & Model & & \\ \hline Our Lexicon & LSTM-RNN & 10.5 & 1.42 \\ \cline{2-4}  & Transformer & 9.8 & 1.33 \\ \cline{2-4}  & Model & & \\ \hline Other Vorvel Confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline s or ch confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Nasal Confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Other Error (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table}
Table 4: Performance Comparison In General

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline
**Error Type** & **60K** & **40K** & **24K** & **12K** \\ \hline total error & 6415 & 7222 & 8087 & 10400 \\ \hline Open Close Confusion (\%) & 32.0 & 30.7 & 34.8 & 23.6 \\ \hline Inherent Vowel Confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Non-Confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Nasal Confusion (\%) & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \hline Other Corr (\%) & 10.4 & 5.4 & 7.1 & 10.6 \\ \hline \end{tabular}
\end{table}
Table 3: Performance on Critical Cases

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline
**Lexicon** & **Model** & **WER (\%)** & **PER (\%)** \\ \hline Google & LSTM-RNN & 25.7 & 3.26 \\ \cline{2-4}  & Transformer Model & 23.6 & 2.71 \\ \hline  & Model & & \\ \hline \end{tabular}
\end{table}
Table 3: Performance on Critical Cases

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline
**Lexicon** & **Model** & **WER (\%)** & **PER (\%)** \\ \hline \multirow{2}{*}{Google} & LSTM-RNN & 25.7 & 3.26 \\ \cline{2-4}  & Transformer Model & 23.6 & 2.71 \\ \hline  & Model & & \\ \hline \end{tabular}
\end{table}
Table 5: Performance comparison on different error categories. Here each of the three rightmost columns denotes the model trained on that particular lexicon.

 

of systems trained on \(3\) types of lexicons. Here, we report the results of Transformer Model only as it has been better performing than LSTM-RNN in our experiments. Both versions of our lexicon perform better (lower WER and lower PER) than Google's lexicon. Also romanization does not significantly increase or decrease the performance.

Figures 4 and 5 show respectively the Word Recognition Accuracy (\(1-\)WER) and Phoneme Recognition Accuracy (\(1-\)PER) with respect to number of iterations run during simulation. Both versions of our lexicon perform better (higher Word Recognition Accuracy and higher Phoneme Recognition Accuracy) than Google's lexicon. Figure 6 shows Negative Log Perplexity vs number of iterations. Both versions of our lexicon provide higher negative log perplexity than Google's lexicon.

### Effectiveness of Our Identified Critical Cases

In this section, we want to establish that the improved performance of our lexicon comes not only from the increase in number of training samples, but also due to the fact that the critical cases identified by our novel methodology have been added as training samples. For this, we prepare a new training lexicon by combining a portion of the Google lexicon with a portion of our identified critical cases. As we have kept \(9K\) entries from the critical cases as our test set, we take the remaining \(21K\) critical cases and combine them with the randomly taken \(39K\) entries from Google lexicon to prepare a new lexicon of size \(60K\). While taking entries from Google lexicon, we ensure that we do not take any repeated entry that has already been in the critical cases and added to the new lexicon.

Figure 4: Word Recognition Accuracy vs Iteration

Figure 3: Comparison in terms of WER and PER



We then compare the performance of this new lexicon with the Google lexicon, both of which are of same size (\(60K\)), on our test set. The results are in Table 6. The results clearly show that even in the case of same sized lexicons, our identified critical cases can significantly improve the performance as evidenced by the lower WER and lower PER than those for the Google lexicon.

## 7 Conclusion and Future Works

In this paper, we have identified the critical cases, categorized the errors, and increased the current state-of-the-art accuracy in G2P conversion for Bangla language by developing an improved lexicon. In future, we will do classification of errors using unsupervised clustering approaches. Tables 2 and 5 both show that most of the errors occur in Open Close Vowel, s or sh, Diphthong, and Inherent Vowel confusion categories. In future, we will focus on each category to devise novel methodology for mitigating the errors of that category and in turn further increase the accuracy of G2P system in Bangla.

## Acknowledgements

This research work is conducted at the Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology (BUET) and is supported by Samsung Research.

## References

* air [2015] air. 2015. [https://svn.code.sf.net/p/cmusphinx/code/branches/cmudict/cmudict-0.7b](https://svn.code.sf.net/p/cmusphinx/code/branches/cmudict/cmudict-0.7b).
* Alam et al. [2011] Firoj Alam, SM Murtoza Habib, and Munit Khan. 2011. Bangla text to speech using festival. In _Conference on Human Language Technology for Development_, pages 154-161.
* Barman [2009] Binoy Barman. 2009. A contrastive analysis of english and bangla phonemics. _Dhaka University Journal of Linguistics_, 2(4):19-42.
* Basu et al. [2009] Joyanta Basu, Tulika Basu, Mridusmita Mitra, and Shyamal Kr Das Mandal. 2009. Grapheme to phoneme (g2p) conversion for bangla. In _Speech Database and Assessments, 2009 Oriental COCSDA International Conference on_, pages 66-71. IEEE.
* Bisani and Ney [2008] Maximilian Bisani and Hermann Ney. 2008. Joint-sequence models for grapheme-to-phoneme conversion. _Speech communication_, 50(5):434-451.
* Caruana [1997] Rich Caruana. 1997. Multitask learning. _Machine learning_, 28(1):41-75.
* Chen [2003] Stanley F Chen. 2003. Conditional and joint models for grapheme-to-phoneme conversion. In _Eighth European Conference on Speech Communication and Technology_.

* Chowdhury et al. [2017] Shammur Absar Chowdhury, Firoj Alam, Naira Khan, and Sheak RH Noori. 2017. Bangla grapheme to phoneme conversion using conditional random fields. In _Computer and Information Technology (ICCIT), 2017 20th International Conference of_, pages 1-6. IEEE.

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline
**Lexicon** & **Model** & \multicolumn{1}{l|}{**WER**} & \multicolumn{1}{l|}{**PER**} \\  & & **(\%)** & **(\%)** \\ \hline \multirow{2}{*}{Google} & LSTM-RNN & 17.1 & 2.32 \\ \cline{2-4}  & Transformer & 14.8 & 1.88 \\ \cline{2-4}  & Model & & \\ \hline \multirow{2}{*}{New Lexicon} & LSTM-RNN & 12.6 & 1.54 \\ \cline{2-4}  & Transformer & 11.2 & 1.49 \\ \cline{2-4}  & Model & & \\ \hline \end{tabular}
\end{table}
Table 6: Effectiveness of critical cases. Both lexicons are of size \(60K\). New Lexicon consists of \(21K\) critical cases and \(39K\) entries from Google lexicon.

Figure 5: Phoneme Recognition Accuracy vs Iteration

Figure 6: Negative Log Perplexity vs Iteration



Aliya Deri and Kevin Knight. 2016. Grapheme-to-phoneme models for (almost) any language. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 399-408.
* Ghosh et al. (2010) Krishnendu Ghosh, Ramu V Reddy, NP Narendra, S Maity, SG Koolagudi, and KS Rao. 2010. Grapheme to phoneme conversion in bengali for festival based tts framework. In _8th international conference on natural language processing (ICON)_. Macmillan Publishers.
* Gutkin et al. (2016) Alexander Gutkin, Linne Ha, Martin Jansche, Knot Pipatsrisawat, and Richard Sproat. 2016. Tts for low resource languages: A bangla synthesizer. In _LREC_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Jiampojamarn et al. (2007) Sittichai Jiampojamarn, Grzegorz Kondrak, and Tarek Sherif. 2007. Applying many-to-many alignments and hidden markov models to letter-to-phoneme conversion. In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_, pages 372-379.
* Johnson et al. (2016) Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, et al. 2016. Google's multilingual neural machine translation system: enabling zero-shot translation. _arXiv preprint arXiv:1611.04558_.
* Kim and Snyder (2012) Young-Bum Kim and Benjamin Snyder. 2012. Universal grapheme-to-phoneme prediction over latin alphabets. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 332-343. Association for Computational Linguistics.
* Mana et al. (2001) Franco Mana, Paolo Massimino, and Alberto Pacchiotti. 2001. Using machine learning techniques for grapheme to phoneme transcription. In _Seventh European Conference on Speech Communication and Technology_.
* Milde et al. (2017) Benjamin Milde, Christoph Schmidt, and Joachim Kohler. 2017. Multitask sequence-to-sequence models for grapheme-to-phoneme conversion. _Proc. Interspeech 2017_, pages 2536-2540.
* Mosaddeque et al. (2006) Ayesha Binte Mosaddeque, Naushad UzZaman, and Mumit Khan. 2006. Rule based automated pronunciation generator.
* Novak (2012) J. R. Novak. 2012. [https://github.com/AdolfVonKleist/Phonetisaurus](https://github.com/AdolfVonKleist/Phonetisaurus).
* Novak et al. (2013) Josef R Novak, Nobuaki Minematsu, and Keikichi Hirorose. 2013. Failure transitions for joint n-gram models and g2P conversion. In _INTERSPEECH_, pages 1821-1825.
* Rao et al. (2015) Kanishka Rao, Fuchun Peng, Hasim Sak, and Francoise Beaufays. 2015. Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks. In _Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on_, pages 4225-4229. IEEE.
* Schrober et al. (2016) Carsten Schnober, Steffen Eger, Erik-Lan Do Dinh, and Iryna Gurevych. 2016. Still not there? comparing traditional sequence-to-sequence models to encoder-decoder neural networks on monotone string translation tasks. _arXiv preprint arXiv:1610.07796_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Thu et al. (2016) Ye Kyaw Thu, Win Pa Pa, Yoshinori Sagisaka, and Naoto Iwahashi. 2016. Comparison of grapheme-to-phoneme conversion methods on a myanmar pronunciation dictionary. In _Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016)_, pages 11-22.
* Toshniwal and Livescu (2016) Shubham Toshniwal and Karen Livescu. 2016. Jointly learning to align and convert graphemes to phonemes with neural attention models. In _Spoken Language Technology Workshop (SLT), 2016 IEEE_, pages 76-82. IEEE.
* Tsvetkov et al. (2016) Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. _arXiv preprint arXiv:1605.03832_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_.
* Yao and Zweig (2015) Kaisenberg Yao and Geoffrey Zweig. 2015. Sequence-to-sequence neural net models for grapheme-to-phoneme conversion. _arXiv preprint arXiv:1506.00196_