## 4 Extraction Module
### Machine Learning-Based Classification
#### 4.3.1 Performance of ML-Based Classification Models  
The final step in the development of the extraction module was the evaluation of the various ML-based text classification models. The performance of ML-based text classification algorithms is highly dependent on feature selection (i.e., domain dependent), type of ML techniques, and training datasets [28]. Therefore, all possible combinations of text pre-processing, feature engineering, and ML algorithms--resulting in 160 exhaustive combinations--were evaluated. Various conditions of text pre-processing approaches, such as stop-word retention or removal with or without the implementation of stemming and/or lemmatization, were tested to evaluate the effect on model performance. While the methodology was conducted in an iterative manner to allow for the detailed comparison of results, only a subset of the results is presented to maintain brevity. The effect of using lemmatization or stemming is illustrated in Figure 8. Stemming improved classification accuracy of LR and XGBOOST algorithms, while lemmatization marginally improved the accuracy of NB and RF algorithms. Notably, the difference in classification performance accuracy between the two text pre-processing techniques was negligible, ranging from 0.03% to 0.4% (Figure 8). The XGBOOST algorithm with stemming applied resulted in the highest accuracy (98.4%). The recall, precision, and F1-score of the different ML algorithms were evaluated (Figure 9). Given that XGBOOST was found to have the highest accuracy with stemming, stemming was applied to all ML techniques prior to performance metric evaluation. The ML algorithms exhibited relatively similar recall values of over 98% for non-reporting statements (i.e., \"false\"). Figure 8: Impact of lemmatization and stemming on performance of Naïve Bayes (NB), Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XGBOOST)-based machine learning algorithms. In contrast, recall values for reporting requirements (i.e., \"true\") varied considerably amongst the various ML algorithms. The NB algorithm resulted in the lowest \"true\" recall value (66%), whereas LR, RF, and XGBOOST algorithms resulted in \"true\" recall values of 74%, 74%, and 81%, respectively. The lower recall values and increased variability observed for the \"true\" class is likely due to the imbalanced distribution of statements in the contractual documents used (340 \"true\" requirements versus 8603 \"false\" statements). In terms of precision, RF, LR, and XGBOOST resulted in \"true\" precision results of 100%, 87%, and 87%, respectively. The XGBOOST algorithm exhibited the highest recall (Figure 9) and accuracy (Figure 8) results for both the \"true\" and \"false\" classes and the second-highest F1-score and precision measurements. Variations in recall when using different n-gram sets for both the \"true\" and \"false\" class were evaluated and illustrated in Figure 10. As discussed previously, higher recall values were observed for non-reporting statements (i.e., \"false\" class). Uni-grams resulted in higher \"true\" recall values compared to bi-grams for all classification algorithms except the NB algorithm. The combined use of uni-grams and bi-grams with the LR and XGBOOST classification models yielded the highest \"true\" recall performance, with values of 77% and 87%, respectively. It is important to note, however, that a number of factors, such as dataset size, can influence the performance of ML models. The hyperparameters of the ML models, therefore, must be tuned to specific data [43]. Here, hyperparameters were objectively changed, one-by-one, to mitigate overfitting and improve classifier performance. After identifying optimal hyperparameters (i.e., a single set of well-performing hyperparameters), the model was retrained with the full training dataset, and the testing dataset was re-evaluated. The two models that were examined were the RF and XGBOOST algorithms, as they have many parameters, and the impact of their hyperparameters is significant.