# Joint Entity-Relation Extraction for Knowledge Graph Construction in Marine Ranching Equipment
## 3 Joint Extraction of Knowledge in the Domain of Marine Ranching Equipment
### Joint Entity Relation Extraction Based on Deep Learning Models
#### 3.2.2 The Specific Structure and Working Principle of the BERT-BiGRU-CRF Model  
The BERT model [18] is a widely adopted pre-trained language model in natural language processing (NLP) in recent years, demonstrating exceptional performance in text representation and semantic understanding. The BERT-BiGRU-CRF architecture comprises three layers: a BERT layer, a Bidirectional Gated Recurrent Unit (BiGRU) layer, and a Conditional Random Field (CRF) layer through their integrated BERT-BiGRU-CRF approach. The overall model is shown in Figure 8.  
Compared with the traditional model, BiLSTM-CRF, this study adopts BiGRU to reduce the parameter quantity (by 18%) and explicitly models the label dependency between 'B-TE' and 'E-SE_Ri' through the CRF layer to solve the problem of nested entities of marine equipment. (1) BERT layer  
BERT is a context-based word embedding model and its structure is delineated in Figure 9.  
Figure 8: BERT-BiGRU-CRF model structure. Figure 9: Transformer encoding structure. The execution of the BERT layer predominantly encompasses two pivotal components: the representation of input data and pre-training procedures. The representation of input data pertains to the transformation of the data into a format compatible with BERT's input requirements. Each character within the input data is the aggregate of token embeddings, segment embeddings, and position embeddings, as shown in Figure 10.  
BERT is pre-trained based on two major tasks: \"Masked Language Model\" (MLM) and \"Next Sentence Prediction\" (NSP). Through the simultaneous training of these two tasks, it can better extract word-level and sentence-level features of the text, obtaining token embeddings that contain more semantic information. (2) BIGRU layer  
To concurrently acquire contextual insights, the current study has developed a Bidirectional Gated Recurrent Unit (BiGRU) network, where the fundamental building block consists of both forward and reversed GRU elements. The detailed model is delineated in Figure 11.  
In Figure 11, the variable \(x_{t}\) denotes the input data at the current instance, whereas \(h_{t}\) signifies the output at the current instance, and \(h_{t-1}\) denotes the output at the previous instance. Concurrently, the reset gate \(r_{t}\) and the update gate \(z_{t}\) operate synergistically to regulate the previous hidden state \(h_{t-1}\) and to facilitate its transition into the new hidden state \(h_{t}\). The reset gate combines \(h_{t-1}\) and \(x_{t}\), producing a matrix \(r_{t}\), the elements of which range from 0 to 1. The specific formula is as follows:  
\[r_{t}=\sigma(W_{r}\cdot[h_{t-1},x_{t}]+b_{r}) \tag{1}\]  
Figure 11: GRU gate model. Figure 10: Input representation of BERT model. The update gate combines \(\mathrm{h_{t-1}}\) and \(\mathrm{x_{t}}\) to control how much information from the previous step's output \(\mathrm{h_{t-1}}\) is retained. The specific formula is as follows. \[\mathbf{z_{t}}=\mathbf{\sigma}(\mathbf{W_{z}}\cdot[\mathbf{h_{t-1}},\mathbf{x_{t}}]+\mathbf{b_{z}}) \tag{2}\]  
The candidate memory \(\overline{\mathrm{ht}}\) consists of two components: one part is the current input data \(\mathrm{x_{t}}\), and the other part is the output \(\mathrm{h_{t-1}}\) from the previous moment, determined by the reset gate \(\mathrm{r_{t}}\). The specific formula is as follows:  
Figure 9.0  
The matrices \(\mathrm{W_{r}}\), \(\mathrm{W_{z}}\) and \(\mathrm{W_{h}}\) denote the weight matrices, while \(\mathrm{b_{r}}\), \(\mathrm{b_{z}}\), and \(\mathrm{b_{h}}\) represent the biases. The ultimate output is determined by the update gate, and the formula for this computation is as follows:  
Figure 9.1  
(3) CRF layer  
In the domain of named entity recognition, it is imperative to acknowledge the existence of interdependencies among the labels. For example, the label \"B-TE\" is invariably followed by the proscription of the label \"M-SE-AQ\".