# Grammatically Derived Factual Relation Augmented Neural Machine Translation
## 3 Factual Relation Augmented Approach
### Factual Relation Augmented Decoder  
The factual relation decoder has a similar structure to the decoder in the Transformer, except for the introduction of an integrated layer between the masked multi-headed attention mechanism and the feed-forward network layers. The integrated layer consists of two parts: FR-Dec attention and Enc-Dec attention, and the structure of the integrated layer is shown in Figure 2. The factual relation-decoding interaction attention is introduced to guide the decoder. In other words, the decoder utilizes the representation of factual relation augmented in the source sentence to optimize the generation of the target translation dur  
Figure 3: An example of factual relation mask matrix (**a**) in left and attention matrix (**b**) in right.  
ing the decoding procedure. Give the source sentence and the target translation sentence \(t=t_{1},t_{2}\ldots t_{n}\), the factual relation augmented decoding procedure can be summarized as follows:  
\[S=Attention_{self}\Big{(}E_{t}W^{Q},\ E_{t}W^{K},E_{t}W^{V}\Big{)} \tag{9}\]  
\[\hat{S}=Attention_{ed}(h,h,S) \tag{10}\]  
\[\widetilde{S}=Attention_{fr}\big{(}H^{\prime},H^{\prime},S\big{)} \tag{11}\]  
where \(E_{t}\) denotes the word embedding of the target input, \(Attention_{self}\) denotes self-attention, \(Attention_{ed}\) denotes encoding-decoding attention and \(Attention_{fr}\) denotes factual relation-decoding attention. The challenge we encounter is how to incorporate \(\widetilde{S}\) and \(\hat{S}\) in the integrated layer. To be specific, four interpolation approaches are proposed to address the issue.  
**Linear Interpolation.**\(\widetilde{S}\) can be seen as an enhancement representation derived from prior knowledge for \(\hat{S}\). In this manner, the interpolation can be calculated as follows:  
\[H=\hat{S}+\lambda*\widetilde{S} \tag{12}\]  
where \(\lambda\) is a hyper-parameter whose value is \([0,1]\). Note that no additional parameters are introduced during the training procedure in this method.  
**Gate Learning.** Compared with linear interpolation, we propose a general method which can adjust the hyper-parameter dynamically during the training procedure rather than in empirical settings.  
\[g=sigmoid\ \Big{(}W_{\hat{S}}\ \Big{(}\hat{S};\widetilde{S}\Big{)}+b_{\hat{S}} \Big{)} \tag{13}\]  
\[H=g*\hat{S}+(1-g)*\widetilde{S} \tag{14}\]  
where \(\Big{(}\hat{S};\widetilde{S}\Big{)}\) denotes the operation of concatenation, and \(W_{\hat{S}}\) and \(b_{\hat{S}}\) are the introduced parameters which can be learned during the training procedure.  
**Concat Gate Learning.** Inspired by LSTMM [26], we propose an output gate and a forget gate to learn the final representation of the source context. It can be summarized as:  
\[f=sigmoid\ \Big{(}W_{f}\ \Big{(}\hat{S};\widetilde{S}\Big{)}+b_{f}\Big{)} \tag{15}\]  
\[o=W_{o}\ \Big{(}\hat{S};\widetilde{S}\Big{)}+b_{o} \tag{16}\]  
\[H=o*f+W_{f}\Big{(}\hat{S};\widetilde{S}\Big{)}*(1-f) \tag{17}\]  
where \(W_{f}\) and \(W_{o}\) are weight matrixes, \(b_{f}\) and \(b_{o}\) are biases, and all parameters can be learned during the training procedure.  
**Linear Transformation** Linear Transformation can be described as follows:  
\[H=W_{\hat{S}}\ \Big{(}\hat{S};\widetilde{S}\Big{)}+b_{\hat{S}} \tag{18}\]  
where \(W_{\hat{S}}\) and \(b_{\hat{S}}\) can be learned during the training procedure. Note that \(H\), \(\hat{S}\) and \(\widetilde{S}\) have the same dimensions.