# Construction of an Industrial Knowledge Graph for Unstructured Chinese Text Learning
## 3 Methods
### Semantic Relation Extraction
#### 3.1.1 Sentence Encoder  
In this section, we transform the sentence x into its distributed representation X by the BGRU+Attention model. As shown in Figure 3, the model contains the following components:  
1. Input layer,
2. Embedding layer,
3. BGRU layer,
4. Attention layer,
5. Output layer.  
The inputs of the BGRU are raw words of the sentence x. We transform words into low-dimensional vectors by a word embedding matrix. In addition, we also use position embeddings for all words in the sentence to specify the position of each entity pair.  
**Word Embedding.** Input a sentence x consisting of \(n\) words \(\mathrm{x}=(w_{1},w_{2},\cdots,w_{n})\). This part aims to transform every word into distributed representations that capture syntactic and semantic meanings of the words by an embedding matrix \(\mathbf{V}\in\mathbb{R}^{d^{n}\times|V|}\), where \(\mathrm{V}\) is a fixed-sized vocabulary and \(d^{w}\) is a hyperparameter to be chosen by the user. As shown in Figure 4, we give a partial word embedding matrix, whose first column is a word, and the latter part is a 100-dimensional vector.  
Figure 2: The pipeline of semantic relation extraction.  
**Position Embedding.** Contextual information at any location affects the extraction of entity relationships, and the words close to the target entities are usually informative to determine the relation between entities. Therefore, by defining the combination of the relative distances from the current word to the head or tail entities, the GRU can keep track of how close each word is to the head or tail entities.  
Finally, we concatenate the word embedding and position embedding of all words to be a vector sequence \(\text{S}=(w_{1},w_{2},\cdots,w_{n})\), where \(w_{i}\in\mathbb{R}^{d}\) (\(d=d^{w}+d^{p}\)).  
The GRU (gate recurrent unit) is a kind of recurrent neural network (RNN) that has also been proposed to solve problems such as the gradient vanishing in long-term memory [26]. Compared with LSTM, there are only two \"gates\" inside the GRU, and it has fewer parameters than LSTM but can also achieve the same function as the LSTM [41]. Considering the computing power and time cost of the hardware, we will often choose a more practical GRU. The architecture of the GRU block is shown in Figure 5.  
Typically, the GRU-based recurrent neural networks contain an update gate \(z_{t}\) and reset gate \(r_{t}\). The update gate is used to control the extent to which the status information of the previous moment is brought into the current state. The larger the value of the update gate is, the more the status information of the previous moment \(h_{t-1}\) is brought in. The reset gate is used to control the degree of ignoring the status information of the previous moment \(h_{t-1}\). The smaller the value of the reset gate is, the more the status information of the previous moment is ignored, just as these following equations demonstrate:  
Figure 4: Word embedding matrix.  
Figure 3: The architecture of BGRU (bidirectional gated recurrent unit)+Attention used for the sentence encoder.  
\[\begin{split} z_{t}&=\sigma\left(W_{z}\cdot[h_{t-1},x_{t}] \right),\\ r_{t}&=\sigma\left(W_{r}\cdot[h_{t-1},x_{t}]\right),\\ \tilde{h}_{t}&=\tanh\left(W\cdot[r_{t}*h_{t-1},x_{t} ]\right),\\ h_{t}&=\left(1-z_{t}\right)*h_{t-1}+z_{t}*\tilde{h}_{t}, \end{split} \tag{1}\]  
It is beneficial to have access to the future as well as the past context for many sequence modeling tasks. However, standard GRU networks process sequences in temporal order, and they ignore the future context. Bidirectional GRU networks are able to exploit information both from the past and the future by introducing a second layer that reverses the hidden connections flow. As shown in Figure 6, the output is represented as \(h_{i}=\left[\overrightarrow{l_{i}}\oplus\overleftarrow{r_{i}}\right]\).