# Automatic Extraction of Flooding Control Knowledge from Rich Literature Texts Using Deep Learning
## 2 Materials and Methods
### Research Methods
#### 2.2.3 Sentence Classification Method  
The sentence classification method used in this study included machine learning methods (such as NB, LR, SVM, and RF) and artificial neural network-based deep learning methods (e.g., Bert, Bert-CNN, Bert-RNN, and ERNIE). (1) Machine learning methods  
NB: The Sklearn Toolkit was used for NB sentence classification. First, the training corpus was preprocessed by punctuation removal and word segmentation, and Term \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & \begin{tabular}{c} **Total Number of** \\ **Sentences** \\ \end{tabular} & \begin{tabular}{c} **Number of Non-Research** \\ **Method Sentences** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Usage** \\ **Method Sentences** \\ \end{tabular} &
\begin{tabular}{c} **Number of Quotation** \\ **Method Sentences** \\ \end{tabular} \\ \hline Training set & 4922 & 3123 & 1318 & 481 \\ Validation set & 593 & 383 & 161 & 49 \\ Test set & 684 & 473 & 160 & 51 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Statistical information from the research method dataset. Frequency-Inverse Document Frequency (TF-IDF) was used for feature selection. Subsequently, a spatial vector was constructed to train the sentence classification model. Finally, the batch prediction results were obtained. LR: LR sentence classification was implemented using the Sklearn Toolkit. Data preprocessing and the feature selection process were performed the same way as in the NB model. The LR classification model was trained and used for testing and prediction. SVM: Data preprocessing and feature selection were performed the same way as in the NB model. The training corpus was converted into a space vector model to train the SVM classification model. The trained model was used for testing and batch prediction. RF: After data preprocessing, the training corpus was converted into a space vector to train the RF sentence classification model. The testing and prediction were performed using a trained model. (2) Artificial neural network-based deep learning methods  
Bert: This is a pre-training language model based on the transformer [39] proposed by Google. It uses the Masked Language Model (MLM) and Next Sentence Prediction (NSP) as model training methods to enhance the ability to capture the characteristics of word level, sentence level, and inter-sentence relationships. The model architecture of Bert is a multilayer bidirectional transformer encoder, in which multiple transformer encoders are stacked. Context semantics were obtained through self-attentions [40]:  
\[Out=Transformer(Embedding(Text)) \tag{1}\]  
where _Text_ is the input text, _Embedding_ represents the input layer of Bert, \(Transformer\) represents the Bert feature extractor, and \(Out\) is the text vector. To fully extract semantic information, the transformer uses attention as the basic unit, and its calculation formula is [39]:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{2}\]  
where \(x\) is the input, \(Q=x\cdot W_{Q},\ K=x\cdot W_{K},\ V=x\cdot W_{V},\ W_{i}(Q,\ K,\ V)\) are random initialization parameters conforming to the normal distribution, and \(d_{k}\) is the data dimension. The multi-head attention mechanism was used in the transformer; that is, Formula (2) was repeated multiple times, and the results were spliced as Formula (3):  
\[Multi\text{-}Head\ Attention(Q,K,V)=concat(Attention(Q_{i},K_{i},V_{i})),\ i=1,2, \ldots,t \tag{3}\]  
where \(t\) was the number of multi-heads. To speed up the training of the model, normalization was introduced to scale the parameters to a normal distribution. The parameters were calculated using Equation (4):  
\[q_{i,j,k}=\frac{x_{i,j,k}-u_{i,j}}{\sqrt{s_{i,j}^{2}+e}}\gamma_{k}+\beta_{k}, \ u_{i,j}=\frac{1}{d}\sum_{k=1}^{d}x_{i,j,k},\ s_{i,j}^{2}=\frac{1}{d}\sum_{k= 1}^{d}\ (x_{i,j,k}-u_{i,j})^{2} \tag{4}\]  
where \(\gamma_{k}\) and \(\beta_{k}\) were learnable parameters.