# Construction of a Geological Fault Corpus and Named Entity Recognition
## 4 Proposed Methods
### BERT-BiLSTM-CRF Model  
In 2018, Google AI Institute proposed a language pre-training model, BERT, which has shown strong performance in various natural language processing (NLP) tasks due to its unique bidirectional processing capability. Consequently, more and more researchers have increasingly adopted BERT for named entity recognition tasks. As illustrated in Figure 6a, the BERT-BiLSTM-CRF model consists of three layers, which are the BERT layer, the BiLSTM layer, and the CRF layer. First, the fault text is input into the BERT layer, which generates corresponding word vectors through the BERT pre-training language model. These word vectors are then fed into the BiLSTM module to learn the sequence features. Finally, the output from the BiLSTM module is encoded using the CRF module to produce a predictive annotation sequence, enabling the extraction and classification of entities in the sequence and completing the entity recognition process for faults. The BERT model is an unsupervised pre-training model that uses the Transformer's bidirectional encoder as its core component, and the model's architecture is built by stacking this structure [30]. The BERT Layer of which is shown in Figure 6b, for an arbitrary sequence, the segmented text sequence is first generated through a segmentation process, with some  
Figure 5: Architecture of the BiLSTM-CRF model. Figure 6: Architecture of the BERT-BiLSTM-CRF model. words in the sequence being masked. A special token [CLS] is added to mark the beginning of the sequence, and sentences are separated by the token [SEP]; at this stage, each word in the sequence has an InputEmbedding composed of three elements--Token Embedding, Segment Embedding and Position Embedding--and the total Input Embedding is:  
\[\text{Input Embedding}=\text{Token Embedding}+\text{Segment Embedding}+\text{Position Embedding} \tag{1}\]  
In this framework, Token Embedding represents word embedding, which can directly transform the input text into corresponding vectors. Segment Embedding is used to distinguish between sentence 1 vectors and sentence 2 vectors when the input consists of two sentences. Position Embedding represents positional embedding, allowing the model to differentiate identical words appearing in different positions. Here \(\text{E}_{1},\text{ E}_{2},\text{ }\ldots,\text{E}_{\text{N}}\) denote the sequence vectors produces by the model. These vectors undergo feature extraction by the Transformer encoder, resulting in output vectors \(Z_{1},\text{ Z}_{2},\text{ }\ldots,Z_{\text{N}}\), which capture the feature representations of the sentences. The BiLSTM model combines forward and backward LSTMs to learn contextual information from both preceding and succeeding directions. The BiLSTME layer will take the vector \(Z=(Z_{1},Z_{2},Z_{3}\ldots,Z_{n})\) output from the Embedding layer as its input. This input word vector \(Z=(Z_{1},Z_{2},Z_{3}\ldots,Z_{n})\) passes through a bidirectional long short-term memory network, generating two sets of vectors, \(\text{T}=(T_{1},T_{2},T_{3}\ldots,T_{n})\) and \(\text{V}=(V_{1},V_{2},V_{3}\ldots,V_{n})\). Since the forward and backward LSTMs do not share parameters or state vectors, the vectors \(\text{T}\) and \(\text{V}\) are concatenated from \(\text{X}=(X_{1},X_{2},X_{3}\ldots,X_{n})\). Finally, the Softmax function is applied to assign prediction scores to the corresponding labels of the active tomography text sequence based on context information. The model structure is shown in Figure 7.  
Although BiLSTM can produce label predictions, it does not guarantee the accuracy of each label, and the output results are often suboptimal. For example, the initial label for a live fault expression entity might be incorrectly labelled as 'B' instead of 'I'. The CRF layer addresses such inconsistencies by correcting mismatched labels, ensuring logical sequences in the final predictions.