# Natural Language Processing Application on Commit Messages: A Case Study on HEP Software
## 9 Results and Validity  
In this work, we have addressed the following research questions. RQ1: What has been the impact of NLP techniques on the data preparation step? By applying NLP techniques in the data preparation phase (i.e., data preprocessing, feature extraction, and feature selection), the size of the feature matrix has been reduced. The average reduction percentage of all used projects is 83%. Figure 9 shows the feature reduction percentage per project belonging to HEP experiments, i.e., ALISW, LHCb, CMS-SW, and ROOT:  
\[ \begin{split}\text{feature reduction percentage per project = (number of features before NLP --}\\ \text{number of features after NLP)/(number of features before NLP)}\end{split}  \tag{9}\]  
According to Figure 9, in AliDPG, for example, 93% features (all words in original commit messages) have been removed, only 7% features have been used in clustering and classification phases. NLP helps to remove unimportant and redundant words and only considers important keywords as features. RQ2: Which ML classification techniques have the best performance of classifying new commits? As shown in Table 8, Random Forest (RF), Bagging (BG), and Decision Tree (DT) have the highest average metric scores (i.e., accuracy, recall, precision, and F1-score). Figure 9: Feature reduction percentage for studied projects. The best classifier of the considered methods is RF, which implies that it has outperformed other models for studied projects. In addition, NLP techniques have an important impact on our results, because the F1-scores have improved. The average F1-score of all examined projects has reached 0.9360 (see Table 8 for RF) and the maximum F1-score has achieved 0.9496. In our previous study [18], without NLP techniques, the maximum F1-score was 0.8210 achieved by MNB. Furthermore, from the perspective of all single projects, cmssw-config and Jenkins-backup in CMS-SW experiment group have reached the highest F1-score (i.e., 0.9496) by Decision Tree algorithm, as shown in Figure 10.  
RQ3: Do the models behave differently for different projects? Table 9 summarizes the statistics of performance metrics of the ML methods adopted. AdaBoost (AB), Logistic Regression (LR), Multinomial Naive Bayes (MNB), and Naive Bayes (NB) methods show huge variations in performances over projects. On the contrary, Bagging, Decision Tree, and Random Forest show stability. Figure 10: F1-score per project. \begin{table}
\begin{tabular}{c c c c c} \hline
**ML Techniques** & **Accuracy** & **Recall** & **Precision** & **F1-Score** \\ \hline AdaBoost (AB) & 0.6377 & 0.4846 & 0.4359 & 0.4482 \\
**Bagging (BG)** & **0.9519** & **0.9280** & **0.9337** & **0.9247** \\
**Decision Tree (DT)** & **0.9517** & **0.9295** & **0.9345** & **0.9262** \\ Logistic Regression (LR) & 0.9027 & 0.8268 & 0.8810 & 0.8401 \\ Multinomial Naive Bayes (MNB) & 0.8454 & 0.7487 & 0.8252 & 0.7695 \\ Naive Bayes (NB) & 0.4979 & 0.6066 & 0.5252 & 0.4964 \\
**Random Forest (RF)** & **0.9590** & **0.9382** & **0.9448** & **0.9360** \\ \hline \end{tabular}
\end{table} 
Table 8: The average metric scores of studies projects. In this study, the Coefficient of Variation (CV) of performance metrics per ML technique has been considered to measure the stability and variation of model performance. CV is the ratio of the standard deviation (std) to the mean of a random variable and it is independent of the unit of measurement used for the variable. As shown in Table 9, CV values of Bagging, Decision Tree, and Random Forests for different metric scores are smaller than 10%, which implies model stability for different projects. CV values of AdaBoost, Logistic Regression, Multinomial Naive Bayes, and Naive Bayes are higher than 15%, which implies huge variations of the metric scores for different projects. In addition to the CV values, the distribution of metric scores has been examined. Figures 11-14 show the boxplots of metric scores for different ML techniques.