# Fully Attentional Network for Low-Resource Academic Machine Translation and Post Editing
## 4 Methods
### Transformer Encoder-Decoder  
The attention mechanism in machine translation tasks was first introduced by [PERSON], [PERSON], and [PERSON] (2015) [1]. The attention mechanism forms a part of the neural architecture that enables the properties of the input data in text string form to be dynamically highlighted [35]. The attention mechanism became very popular in machine translations following this model released in 2015. In 2017, [PERSON] et al. from the Google machine translation team [8] created the transformer model that has achieved the greatest success rate in machine translation so far by applying the attention mechanism in neural networks rather than RNN (see Figure 3).  
The transformer model, like most NMT systems, is based on the Encoder-Decoder framework. The original article [8] used \(N=6\) and \(N=12\) layers for encoder and decoder.  
EncoderAll encoder layers use the same structure and each layer is divided into two sub-layers: Self Attention (Multi-Head Attention) and Feed-Forward Neural Network (FFNN). The encoder input vector (Word embedding + positional coding) first passes through the self-attention layer, and this layer ensures that the relevant word appears along with other words in the sentence. The input vector is fed to the FFNN network with the output of this layer. Also, normalization is applied to all sublayer outputs. All sublayers and embedding layers produce \(d_{model}=512\) output.  
Figure 2: The Proposed Model.  
_Decoder_: In addition to the two sublayers in the encoder layer, a multi-headed attention layer has been added. The output of the encoder network is given as input to each decoder network.  
_Self Attention_: This layer aims to calculate the attention of all other words for a selected word. For this purpose, Query uses Key and Value vectors. The size of the vectors is 64. The Query is the word searched for when considering a database query. The value is the value corresponding to the searched word. The key is the result chosen from among the values. The attention score for each word is calculated as shown in (Equation (3)) and (Figure 3C). The softmax function is applied to obtain the weights on the attention output values.  
\[Attention(Q,K,V)=softmax(\frac{QK^{t}}{\sqrt{d_{k}}})V \tag{3}\]  
The attention score (head) for each word is more than 1. In the original article, \(h=8\) was determined. Independent attention outputs are combined and converted into a linear dimension (Equations (4) and (5) and Figure 3B). \(W\) is the trainable parameter vector.  
\[MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{h})W^{0} \tag{4}\]  
\[where\ head_{i}=Attention(QW^{Q}_{i},QW^{K}_{i},QW^{V}_{i}) \tag{5}\]  
Figure 3: Transformer model architecture (**A**) Encoder-Decoder, (**B**) Multihead attention, (**C**) Scaled dot product.  
There is a feed-forward neural network after the multi-head attention layer in each encoder and decoder block (Equation (6)) and (Figure 3A).  
\[FFN(x)=max(0,xW_{1}+b_{1})W_{2}+b_{2} \tag{6}\]  
Unlike previous models, [1], the attention score is calculated for each word, instead of calculating just one attention score for the input sentence.  
Positional Encoding: Since there is no repetition and convolution in the model, it is the process of adding some information about the relative or absolute position of the words in the sentence so that the model can understand the sentence order. Position encodings are the same size as Word embeddings and are easy to collect. _Pos_ in (Equations (7) and (8)) shows the position and \(i\) dimension. Two calculated positions and word embeddings are summed up and transmitted to the encoder layer as input.  
\[PE_{pos,2i}=sin(pos/10000^{2i/dmodel}) \tag{7}\]  
\[PE_{pos,2i+1}=cos(pos/10000^{2i/dmodel}) \tag{8}\]  
In the transformer model used in this study, encoder and decoder layers \(N=6\) and \(N=12\) were selected. In addition, FFNN the hidden layer is generated as 2048 and Word embedding \(d_{model}\) is chosen as 512. The vector size is set to 64 for the Query, Key, and Value.