# Graphene-to-Phoneme Conversion with Convolutional Neural Networks
## 3 Convolutional Neural Networks for Grapheme to Phoneme Conversion  
Convolutional neural networks are used in various fields, including image [33; 34], object [18; 35; 36] and handwriting recognition [36; 37], face verification [38], natural language processing [39; 40] and machine translation [21]. The architecture of an ordinary CNN is composed of many layer types (such as the convolutional layers, pooling layers, fully connecting layers, etc.), where each layer carries out a specific function. The convolutional and pooling layers are for representation learning, while the fully connected layers on the top of the network are for modelling a classification or regression problem. One of the main reasons that make convolutional neural networks superior to previous methods is that CNNs perform representation learning and modelling jointly; thus, a quasi-optimal representation is extracted from the input data for the machine learning model. Weight sharing in the convolutional layers is also a key element. Thus, the model becomes spatially tolerant; similar representations are learned in different regions of the input, and the total number of parameters can also be reduced drastically.  
Deep Learning refers to the increased depth of neural networks. Intuitively, it is expected that neural networks with many hidden layers are more powerful than shallow ones with a single hidden layer. However, as the number of layers increases, the training may become surprisingly hard, partly because the gradients are unstable. Batch normalization is a technique to overcome this problem; it reduces internal covariance shift and helps to smooth learning. The main idea of batch normalization is to bring back the benefits of normalization at each layer [41]. Batch normalization results in faster convergence as well. For example, with batch normalization, 7% of the training steps were enough to achieve similar accuracy in an image classification task [41]. Moreover, an additional advantage of batch normalization is that it regularizes the training and thus reduces the need for dropout and other regularization techniques [41]. However, batch normalization and dropout are often simultaneously applied.  
Convolutional neural networks have been successfully applied to various NLP tasks [21; 39; 31; 40]. These results suggest investigating the possibility of applying CNN-based sequence-to-sequence models for G2P. We expected that the advantage of convolutional neural networks enhances the performance of G2P conversion. As is known, LSTMs read input sequentially, and the output for further inputs depends on the previous ones. Thus, we cannot parallelize these networks. Applying CNN also reduces computational load by using large receptive fields.  
Deep neural networks with a sequential architecture have many typical building blocks, such as convolutional or fully connected layers, stacked on each other. Increasing the number of layers in these kinds of networks does not implicitly mean improved accuracy (in our case PER or WER), and some issues, such as vanishing gradient and degradation problems, can arise as well. Introducing residual and highway connections can improve performance significantly [42; 43]. These connection alternatives allow the information to flow more into the deeper layers, increase the convergence speed and decrease the vanishing gradient problem.