# Research on Intelligent Perception Algorithm for Sensitive Information
## 3 Methodology
### Text Encoder
#### 3.1.2 Knowledge Graph-Based Entity Embedding Module  
The output \(\left\{w_{1}^{0},\cdots,w_{n}^{0}\right\}\) represents the semantic information of the input context from Section 3.1.1, but the specific and valuable information of the entities contained in it may not be highlighted, for example, if the text contains entities: names of people, places, etc., then this text may be biased to introduce the content related to that entity. To this end, in addition to using the ERNIE model to encode the text input to extract the text semantic information of the text, we also need to extract the entities \(\left\{e_{1}^{0},\cdots,e_{n}^{0}\right\}\) from the text and use the entity embedding module of the knowledge graph to obtain the representative intermediate information among them. We refer to the work for [PERSON] et al. (2018) and train a Knowledge Graph Entity Embedding Module (KG) using CN-DBpedia to help construct semantically richer intermediate vectors with specific information of named entities. Specifically, the operational flow of the KG module is shown in Figure 4.  
We combine the semantic information vector \(\left\{w_{1}^{(i-1)},\cdots,w_{n}^{(i-1)}\right\}\) and the entity information vector \(\left\{e_{1}^{(i-1)},\cdots,e_{n}^{(i-1)}\right\}\) as inputs into layer i (i starts from 1), and the two vectors are fed into two multi-headed self-attentive modules (MH-ATTs) to obtain the layer i outputs of \(\left\{w_{1}^{(i)},\cdots,w_{n}^{(i)}\right\}\) and \(\left\{e_{1}^{(i)},\cdots,e_{n}^{(i)}\right\}\), as shown in (4) and (5):  
\[\left\{\tilde{w}_{1}^{(i)},\cdots,\tilde{w}_{n}^{(i)}\right\}=MH-ATT(\left\{ w_{1}^{(i-1)},\cdots,w_{n}^{(i-1)}\right\}) \tag{4}\]  
\[\left\{\tilde{e}_{1}^{(i)},\cdots,\tilde{e}_{n}^{(i)}\right\}=MH-ATT(\left\{ e_{1}^{(i-1)},\cdots,e_{n}^{(i-1)}\right\}) \tag{5}\]  
where we integrate the contextual semantic information sequences and entity sequences and calculate the input embedding for each semantic and entity for the semantic \(w_{j}^{(i)}\) and the corresponding entity \(e_{k}^{(i)},\ h_{j}\) is the hidden internal state of the integrated semantic and entity information, \"\(\sigma\)\" is the nonlinear activation function Gelu, b is the deviation  
Figure 3: The operational flow of the ERNIE module. term, \(W_{ht}^{(i)}\), \(W_{hc}^{(i)}\), \(W_{t}^{(i)}\), and \(W_{e}^{(i)}\) are the corresponding weights, and \(b^{(i)}\), \(b_{t}^{(i)}\), \(b_{e}^{(i)}\) are the corresponding bias, as shown in (6)-(8), respectively. \[h_{j}=\sigma(W_{ht}^{(i)}\tilde{w}_{j}^{(i)}+W_{he}^{(i)}\tilde{e}_{k}^{(i)}+b ^{(i)}) \tag{6}\]  
\[w_{j}^{(i)}=\sigma(W_{t}^{(i)}h_{j}+b_{t}^{(i)}) \tag{7}\]  
\[e_{k}^{(i)}=\sigma(W_{e}^{(i)}h_{j}+b_{e}^{(i)}) \tag{8}\]  
For the semantics of \(w_{j}^{i}\) without corresponding entities, the word embedding is output directly without semantic and entity information integration, as shown in (9) and (10). \[h_{j}=\sigma(W_{ht}^{(i)}\tilde{w}_{j}^{i}+b^{(i)}) \tag{9}\]  
\[w_{j}^{(i)}=\sigma(W_{t}^{(i)}h_{j}+b_{t}^{(i)}) \tag{10}\]  
The semantic vector \(\left\{w_{1}^{(i)},\cdots,w_{n}^{(i)}\right\}\) of the last layer is represented as \(R_{E}\) and the corresponding entity vector \(\left\{e_{1}^{(i)},\cdots,e_{n}^{(i)}\right\}\) as \(R_{K}\) is output to the classifier. Therefore, the text encoder works as follows First, the pre-processed text sequence \(t\) of length \(n\) is input into the embedding layer of the ERNIE module, and \(t\) is transformed into the vector \(\left\{e_{1},\cdots,e_{n}\right\}\), then the vector \(\left\{e_{1},\cdots,e_{n}\right\}\) is input into the MH-ATT layer to extract the semantic information \(\left\{w_{1},\cdots,w_{n}\right\}\), and finally the semantic information \(\left\{w_{1},\cdots,w_{n}\right\}\) is fed into the FFN layer to obtain the final output of the ERNIE module \(\left\{w_{1}^{0},\cdots,w_{n}^{0}\right\}\) and the entity information \(\left\{e_{1}^{(i)},\cdots,e_{n}^{0}\right\}\).