# An Open Relation Extraction Method for Domain Text Based on Hybrid Supervised Learning
## 5 Relation Extraction Model Ro-DGANet
### Dilate Gate Attention Network  
The DGANet proposed in this paper consists of 12 layers of dilate gated attention (DGA), each of which includes a one-dimensional inflated convolution and a gated attention unit (GAU) [40] fused with a residual structure. The structure is shown in Figure 6.  
Suppose the input vector \(x=[x1,x2,\dots,xn]\), a gated linear unit (GLU) [16], is added to the one-dimensional convolution as following equations.  
\[Y=U\otimes V \tag{4}\]  
\[U=Conv1D_{1}(X) \tag{5}\]  
\[V=Conv1D_{2}(X) \tag{6}\]  
The two \(Conv1D\) in Equations (5) and (6) have the same structure, but the parameters are not shared. Suppose the dimension of a word vector is \(d\). In that case, there are \(2d\) convolution kernels, one of which is activated by the Sigmoid function, the other without the activation function, and then they are multiplied bit by bit. As the Sigmoid function has a value range of \((0,1)\), it essentially adds a \"valve\" to each output of \(Conv1D\) to control the flow. This design reduces the risk of gradient disappearance because one of the convolutions does not have an activation function. This part of the convolution is less prone to gradient disappearance.  
Because the \(Conv1D\) parameters in the model are not shared, they are all operated independently, and the interaction between each Token is missing. To solve this problem, by adding the Attention matrix noted as \(A\), which is responsible for fusing the information  
Figure 6: Structure of DGA model.  
between Tokens so that the output contains the interaction between Tokens, the following formula is used.  
\[Y=U\otimes AV \tag{7}\]  
\[A=\left\{\begin{array}{l}\frac{1}{n}relu^{2}\bigg{(}\frac{Q(Z)K(Z)^{T}}{\sqrt{ s}}\bigg{)}\\ \frac{1}{ns}relu^{2}(Q(Z)K(Z)^{T})\end{array}\right. \tag{8}\]  
\[Z=\phi(XW_{z}),W_{z}\in R^{d\times s} \tag{9}\]  
In the above Equation, \(s\) denotes the head size of attention, \(s=128\) in this paper, whereas \(Q\) and \(K\) are simple affine variations, and \(\phi\) is the SiLU (Sigmoid Linear Unit) [41] activation function.  
As the input \(X\) and output \(Y\) of this model have the same dimension, it is possible to involve \(X\) in the operation; in other words, the combination of residual and gated attention not only solves the gradient disappearance problem but also enables the feature information to be transmitted in multiple channels, as in Equation (10).  
\[Y=X+U\otimes AV \tag{10}\]  
For a clearer understanding of the DGANet structure, Equation (10) can be rewritten as the following equivalence.  
\[Y=X\otimes(1-AV)+U\otimes AV \tag{11}\]  
The convolutional structure used in the model is the convolutional neural network (CNN) [42] proposed by [PERSON], which can significantly reduce the model training time and is more effective than the recurrent neural network (RNN) structure. For the CNN model to capture richer semantic features and increase the perceptual field of the model, the semantic information at a distance is captured without increasing the parameters using an inflated convolutional network.  
The model uses 12 layers of inflated convolutional networks with expansion rates of \([1,2,4]\) repeated three times in sequence and then set to \([1,1,1]\) for fine-grained tuning to give the model automatic feature selection capability and more powerful contextual semantic information capture.