## 4 Extraction Module
### Rule-Based Classification
#### 4.2.1 Performance of Rule-Based Classification Models  
Performance of the text classification models were evaluated using accuracy, precision, recall, and F1-score, which were calculated using Equation (1) through Equation (4), respectively. \[\text{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN} \tag{1}\]  
\[\text{Recall}=\frac{TP}{TP+FN} \tag{2}\]  
\[\text{Precision}=\frac{TP}{TP+FP} \tag{3}\]  
\[\text{F1}-\text{Score}=\frac{2\times\text{Precision}\times\text{Recall}}{ \text{Precision}+\text{Recall}} \tag{4}\]  
Figure 5: Most common n-grams in reporting requirement statements. where \(TP\) are true positives (i.e., statements correctly labeled \"true\"), \(FP\) are false positives (i.e., incorrectly labeled \"true\"), \(TN\) are true negatives (i.e., correctly labeled \"false\"), and \(FN\) are false negatives (i.e., incorrectly labeled \"false\"). Accuracy (Equation (1)) is defined as the percentage of correctly classified statements over the total number of statements in the testing set. Recall (Equation (2)), is defined as the percentage of true positives identified by the model. Precision (Equation (3)) is defined as the percentage of positives that are correctly labeled [36]. Finally, the F1-score (Equation (4)), combines precision and recall to provide an overall assessment of model effectiveness. Recall is considered to be the most critical performance metric in the context of requirement extraction, where the extraction of all reporting requirements is the primary objective. For instance, a model may have low performance accuracy because it results in a larger number of false positives (i.e., non-reporting statements labeled as requirements). However, the model may have high recall results (i.e., 100%) if it is able to correctly label all reporting requirements as \"true\". Specific rules for text processing were developed and applied to improve results of the rule-based classification model. Initial tests were conducted on different n-gram sets. The testing approach was conducted in an iterative manner, and results from 24 different combinations of n-grams and text pre-processing techniques (e.g., stop-word removal, lemmatization, etc.) were compared. The four sets of n-grams extracted from the training set are summarized in Table 1. The total number of rules generated increased with the number of n-grams (Table 1). Using the process summarized in Figure 4, the number of rules maintained for each n-gram was considerably reduced for all n-gram sets (Table 1). For example, of the 2363 rules generated for the bi-grams set, only 38 rules were retained. Accuracy was increased from 97%, using uni-grams, to 99%, using bi-grams, yet was decreased to 98% and 97% using tri- and quad-grams, respectively. Although differences between n-gram sets were minimal, optimal accuracy was achieved using the retained bi-gram rules. The impact of adding the first 10 and the last bi-gram rule on model accuracy is visualized in Figure 6. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**N-Gram Set** & \begin{tabular}{c} **Number of** \\ **Generated Rules** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Retained** \\ **Rules** \\ \end{tabular} &
\begin{tabular}{c} **Maximum Accuracy** \\ **(\%)** \\ \end{tabular} \\ \hline Uni-Grams & 118 & 8 & 97 \\ Bi-Grams & 2363 & 38 & 99 \\ Tri-Grams & 3762 & 38 & 98 \\ Quad-Grams & 4268 & 34 & 97 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of generated and retained bi-gram rules and associated accuracy. Figure 6: Impact of adding bi-gram rules on the accuracy of the rule-based text classifier. The first bi-gram rule, \"report shall\", resulted in an accuracy of 95.3%. The third bi-gram, \"shall submit\", further increased the accuracy of the classifier to 96.7%. The 37 rules added after \"shall submit\" collectively increased performance by 3.97% to 99.3%. The impact of stop-word removal, lemmatization, and stemming on the performance of text classification models is known to differ based on the textual context and application.