# Construction of Power Fault Knowledge Graph Based on Deep Learning
## 3 Construction Model of Power Fault Knowledge Graph Based on Deep Learning
### Entity Extraction Model Based on BBIGRU-GA-CRF
#### 3.2.1 BERT Layer  
Traditional word embedding methods, such as word2 vec, can express the relationship between words, but the relationship between words and vectors is static, which cannot dynamically adjust the representation of the input sequence according to the change in context, which restricts the improvement of entity recognition accuracy to a certain extent. The BERT model introduces a bidirectional transformer coding structure, which can fully understand the semantic information contained in the text, and provides an ideal scheme to solve the above problems [24]. In the proposed BBiGRU-GA-CRF model, the BERT layer undertakes the task of the first step of processing the input text. The BERT layer can be divided into two parts: input layer and output layer, and transformer coding layer. These two parts are introduced in detail below.  
(1) Input and output layer  
The input of BERT is the processed power system fault text, and the fault text has continuous semantics. In BERT, firstly, the CLS and SEP symbols are inserted into the input statements. CLS precedes the first sentence, and SEP separates the two input statements of the input power fault text. Secondly, BERT overlays the embedded vector, segmentation vector, and position vector of the character to obtain the final vector representation. The vector representation of each character in the input sequence is the embedded vector of the character; the segmentation vector is used to distinguish which specific sentence  
Figure 5: BBiGRU-GA-CRF entity extraction model.  
each character belongs to, that is, sentence A or sentence B. If the input sequence is single sentence text, it is represented by EA; the position vector is responsible for encoding the position of each character in the input. The above vectors can be obtained through learning in the training process. The output of the BERT model is expressed as a vector, which is then used as the input of the BiGRU module.  
(2) Bidirectional transformer coding layer  
In the BERT model, the structure responsible for encoding the input sequence is a multilayer bidirectional transformer encoder, and its model structure is shown in Figure 6. The input of the model is a character vector, and the value is \(E_{1}\), \(E_{2}\ldots E_{n}\), \(T_{1}T_{2}\ldots T_{n}\) is the output vector of the model. The core of the transformer encoder in the BERT model is a Self-Attention Mechanism. Multi-head self-attention is used to obtain the corresponding feature representation of multiple subspaces. Through each self-attention head, the relationship between any two characters is directly calculated. Therefore, the long-distance interdependent features in the input sequence are easier to capture, so the degree of association between these characters can be obtained. Then, the weight of each word is dynamically adjusted according to the correlation between characters, and a new vector representation corresponding to each word is obtained. The new vector representation of this word contains the meaning of the character itself and its relationship with other characters in the sequence. Compared with a single word vector representation, the content is richer and more accurate [25].