## 2 Materials and Methods
### Experimental Model
#### 2.2.1 Word Embedding Layer  
The word embedding layer is utilized to map words or characters from the text into a real-number vector space (embedding vector). Various models exist for the word embedding layer, such as Word2 Vec, GloVe, FastText, etc. [15; 16; 17]. The word embedding layer utilized in this study is a variant of the BERT model known as chinese-roberta-wwm-ext ([[https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main](https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main)]([https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main](https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main)), accessed on 1 May 2023). BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model based on the Transformer architecture, proposed by Google in 2018. As illustrated in Figure 2, the structure of the BERT model primarily consists of the encoder part of the Transformer. The encoder employs a self-attention mechanism to interact with each element in the input sequence, calculating relevance scores between each element and others [18]. Allowing the encoder to focus on essential contextual information from each element enhances our overall understanding of the semantic and contextual relationships within the input sequence. Repeated stacking of the encoder layers contributes to the extraction of deep-level feature representations from the input sequence. In BERT, multiple such encoder layers are stacked, forming a deep network structure. In the original BERT paper, the authors utilized 12-layer and 24-layer Transformer encoders to build models, with parameter totals of 110 million and 340 million, respectively. Compared to traditional word embedding models, BERT exhibits contextual awareness, allowing us to better capture the semantic and contextual relationships of vocabulary. This is attributed to BERT's bidirectional encoding approach, which considers both the left and right context of words simultaneously, while models like Word2 Vec and GloVe are predominantly unidirectional. Thus, BERT is enabled to comprehensively understand the context of the vocabulary. Moreover, rich language knowledge is acquired by BERT using pretraining tasks like masked language modeling and next sentence prediction, enhancing its generalization capabilities for downstream tasks. In contrast, Word2 Vec and GloVe primarily focus on local word co-occurrence patterns. The input design of the BERT model is cleverly crafted. As shown in Figure 3, the BERT input is constructed into a linear sequence, dividing two sentences using a separator. Two special tokens are added at the beginning and end of the sequence, ensuring accurate capture of the overall context. In BERT, there are three key embeddings at the word level. The first key embedding is positional information embedding, which is used to encode the position of words in a sequence. This is important because word order is a critical feature in natural language processing. Word embedding is the second key embedding. The semantic information of the words can be captured in this step. The final embedding is sentence embedding. When the training data consist of two or more sentences, the overall context for each word is provided by this embedding. These three embeddings are stacked together, forming the input structure of BERT. Figure 2: BERT structure diagram. In terms of the model input, BERT converts each character in the text into a one-dimensional vector by querying a character vector table, serving as the fundamental input for the model. Simultaneously, the model's output consists of vector representations for each character, incorporating fused semantic information from the entire text. The input encompasses not only character vectors but also text vectors and positional vectors. The text vectors are learned during the model training process, capturing the global semantic information from the text and merging it with the semantic information on individual characters.