# Graphene-to-Phoneme Conversion with Convolutional Neural Networks
## 4 Experiments
### Evaluation and Results  
NVidia Titan Xp (12 GB) and NVidia Titan X (12 GB) GPU cards hosted in two i7 workstations with 32 GB RAM served for training and inference. Ubuntu 14.04 with Cuda 8.0 and cuDNN 5.0 was used as a general software architecture. For training and evaluation, the Keras deep learning framework with Theano backend was our environment.  
For evaluation, the standard and commonly used [12; 16] measurements of phoneme error rate (PER) and word error rate (WER) were calculated. PER was used to measure the distance between the predicted phoneme sequence and reference pronunciation divided by the number of phonemes in the reference pronunciation. Edit distance (also known as Levenshtein distance [51]) is the minimum number of insertions (I), deletions (D) and substitutions (S), that are required to transform one sequence into the other. If there are multiple pronunciation variants for a word in the reference data, the variant that has the smallest Levenshtein distance [51] to the candidate is used. Levenshtein distance can be calculated by dynamic programming method [52].  
For WER computation, which is only counted if the predicted pronunciation does not match any reference pronunciation, the number of word errors is divided by the total number of unique words in the reference.  
After training the model, predictions were run on the test dataset. The results of evaluation on the CMUDict dataset are shown in Table 1. The first and second columns show the model number and the applied architecture, respectively. The third and fourth columns show the PER and WER values. The fifth column of Table 1 contains the average sum of training and validation time of one epoch. The last two columns present information about the size of models, which shows the number of parameters (weights) and the number of epochs to reach minimum validation loss. According to the results, the encoder-decoder Bi-LSTM architecture (Model 2) outperforms the first model, as expected. However, attention-based Model 1 (called Model 1A in Table 1) outperforms Model 2 in terms of PER. The best WER and PER values are achieved by the fifth model: PER is 4.81%, and WER is 25.13%. Attention-based Model 2 (called Model 2A in Table 1) approaches the best results in terms of both PER and WER. However, the number of parameters of Model 2A is twice as high as for Model 5. Although the fourth model was faster than all of the other models, both the PER and WER of this model were the highest; however, they are still competitive. Moreover, this model also has the fewest parameters.  
Figure 5: Implementation of a single convolutional layer with 64 filters of size (input length) \(\times\)3 to the input data.  
We compared the performance of the fifth model on both CMUDict and NetTalk with previously achieved state-of-the-art results. These comparisons are presented in Table 2. The first column shows the dataset, the second column presents the method used in previous solutions with references, PER and WER columns tell the results of the referred models. Table 2 clearly shows that our fifth model outperforms the previous solutions by PER on each dataset, except for [32]. For NetTalk, we were able to significantly surpass the previous state-of-the-art, but a better WER was obtained by [32] with an encoder-decoder network based on an attention mechanism. We should point out that the results of the fifth model are very close to those obtained by [32].  
The proposed best model in [26] consists of the combination of the sequitur G2P (model order 8) and seq2 seq-attention (Bi-LSTM 512 \(\times\) 3) and multitask learning (ARPAbet/IPA), and although the WER in their case is better, Model 5 has a smaller PER.  
Although the encoder-decoder LSTM by [16] is similar to our first model, the PER is better in our case; the WER of both models is almost the same. Our second model is comparable with [16], in which the Bi-LSTM method was implemented, alignment was also applied.