# Research on Intelligent Perception Algorithm for Sensitive Information
## 3 Methodology
### Text Encoder
#### 3.1.1 ERNIE-Based Encoder Module  
Text consists of sentences, each of which consists of one word, but the classifier needs a vector or tensor as its input. We extract semantic information from the raw input text in a contextual representation using the ERNIE model, which represents text as vectors. The dynamic word vectors produced by the ERNIE pre-training model, as opposed to static word vectors, are better able to represent contextual cues and prevent the production of multiple meanings for a single word. The architecture of the ERNIE model is similar to that of the BERT model, which is based on the bi-directional Transformer's encoder.  
ERNIE and BERT models are both trained using a Masked Language Model (MLM). For the differences between Chinese and English, the ERNIE model is an improvement over the BERT model, which masks words as units. Since in Chinese, the meaning of a single word is often very different from that of its constituent words, and at the same time, words occur very frequently, using words alone for masking training would lose the information about the features between words, and would not allow good learning of the information and knowledge expressed in the sentence by the word in which the word is located. The ERNIE model includes both word-based and entity-based masking training strategies. A comparison of the two masking approaches is shown in Figure 2. For example, in the sentence \"Harbin is the capital of Heilongjiang, an international city of ice and snow culture\", the BERT model covers up the \"r\" of \"Harbin\" and the \"hei\" of \"heilongjiang\", the \"internal\" and \"snow\" in \"international ice and snow\". \"The ERNIE model covers up the complete words \"Harbin\" and \"ice and snow\". This can better enhance the association between words better learn the feature information and knowledge of words, and strengthen the ability of word vectors to express Chinese semantic information.  
Specifically, the operational flow of the ERNIE module is shown in Figure 3.  
The pre-processed text sequence \(t\) of length \(n\) is input into the ERNIE module. First, \(t\) is fed into the embedding layer, which transforms the text sequence \(t\) into a vector \(\{e_{1},\cdots,e_{n}\}\), as shown in (1).  
\[\{e_{1},\cdots,e_{n}\}=Embedding(t),t=[CLS]\ t_{1}\ t_{1}\cdots,t_{n} \tag{1}\]  
where \(t_{1}\ t_{2}\cdots t_{n}\) is the word in the text and [CLS] is the statement start symbol. Then the vector \(\{e_{1},\cdots,e_{n}\}\) is input into the multi-headed self-attentive module (MH-ATT) [10] to obtain \(\{w_{1},\cdots,w_{n}\}\), as shown in (2).  
\[\{w_{1},\cdots,w_{n}\}=MH-ATT(\{e_{1},\cdots,e_{n}\}) \tag{2}\]  
The obtained \(\{w_{1},\cdots,w_{n}\}\) is input into the prefix network (FFN) [10] to obtain the final output of the encoder module of \(\{w_{1},\cdots,w_{n}\}\), as shown in (3), consisting of two linear transformations with a ReLU activation function.  
\[\left\{w_{1}^{0},\cdots,w_{n}^{0}\right\}=FFN(\{w_{1},\cdots,w_{n}\})=max(0, \{w_{1},\cdots,w_{n}\}W_{1}+b_{1})W_{2}+b_{2} \tag{3}\]  
Figure 2: Comparison of Bert model masking method and ERNIE model masking method. (**a**) BERT model masking method; (**b**) ERNIE model masking method.