# Graphene-to-Phoneme Conversion with Convolutional Neural Networks
## 3 Convolutional Neural Networks for Grapheme to Phoneme Conversion
### Models  
Encoder-decoder structures have shown state-of-the-art results in different NLP tasks [16; 23]. The main idea of these approaches has two steps: the first step is mapping the input sequence to a vector; the second step is to generate the output sequence based on the learned vector representation. Encoder-decoder models generate an output after the complete input sequence is processed by the encoder, which enables the decoder to learn from any part of the input without being limited to fixed context windows. Figure 1 shows an example of an encoder-decoder architecture [15]. In our experiments, we used encoder-decoder architectures. Several models with different hyperparameters were developed and tested. From a large number of experiments, the five models with the highest accuracy and diverse architectures were selected. Our first two models were based on existing solutions for comparison purposes. We used these models as a baseline. In the following paragraphs, the five models are introduced:  
The first model uses LSTMs for both the encoder and the decoder. The LSTM encoder reads the input sequence and creates a fixed-dimensional vector representation. The second LSTM is the  
Figure 1: The input of the encoder is the “**CAKE**” grapheme sequence, and the decoder produces “**K**EY**K” as the phoneme sequences. The left side is the encoder; the right side is the decoder. The model stops making predictions after generating the end-of-phonemes tag. As distinct from [15; 16], input data for the encoder is not reversed in any of our models. decoder, and it generates the output. Figure 2a shows the structure of the first model. It can be seen that both LSTMs have 1024 units; softmax activation function is used to obtain model predictions. This architecture is the same as a previous solution [16], while the parameters of training (optimization method, regularization, etc.) are identical to the settings used in case of the other four models. In this way we try to ensure a fair comparison among the models. Although the encoder-decoder architecture achieves competitive results on a wide range of problems, it suffers from the constraint that all input sequences are forced to be encoded to a fixed-size latent space. To overcome this limitation, we investigated the effects of the attention mechanism proposed by [44; 45] in Model 1 and Model 2. We applied an attention layer between the encoder and decoder LSTMs in the case of Model 1, and Bi-LSTMs for Model 2. The introduced attention layers are based on global attention [45]. In the second model, both the encoder and the decoder are Bi-LSTMs [11; 46; 47]. The structure of this model is presented in Figure 2b. The input is fed to the first Bi-LSTM (encoder), which combines two unidirectional LSTM layers that process the input from left-to-right and right-to-left. The output of the encoder is given as the input for the second Bi-LSTM (decoder). Finally, the softmax function is applied to generate the output of one-hot vectors (phonemes). During the inference, the complete input sequence is processed by the encoder, and after that, the decoder generates the output. For predicting a phoneme, both the left and the right contexts are considered. This model was also inspired by an existing solution [12]. In the third model, a convolutional neural network is introduced as the encoder, and a Bi-LSTM as the decoder. This architecture is presented in Figure 2c. As this figure shows, the number of filters is 524, the length of the filter is 23, the stride is 1, and the number of cells in the Bi-LSTM is 1024. In this model, the CNN layer takes graphemes as input and performs convolution operations. For regularization purpose, we also introduced batch normalization in this model. The fourth model contains convolutional layers only, with residual connections (blocks) [42]. These residual connections have two rules [48]:  
1. if feature maps have the same size, then the blocks share the same hyperparameters. 2. each time when the feature map is halved, the number of filters is doubled.