# Natural Language Processing Application on Commit Messages: A Case Study on HEP Software
## 3 Methodology
Logistic regression can be binomial or multinomial: binary logistical regression is used for a dependent variable that has two types of outcome, and multinomial logistic regression is used for a dependent variable that has more than two types of outcome [50]. * Decision Tree that is widely used in data mining, and there are two types of tree models, i.e., classification trees and regression trees. They aim at predicting the value of a target variable based on input variables [50]. * Random Forest that is a type of Bagging method, and its base classifier is a decision tree. It is an ensemble learning method mainly for classification and regression tasks. It constructs a great number of decision trees at the training phase, and its final result is the mode of the classes or the mean value of individual trees. During the training phase, it also introduces randomness for feature selection. Only a subset of features and samples are considered for each decision tree. Random Forest reduces the risk of over-fitting. * Bagging, also called bootstrap aggregating, that is an ensemble meta-algorithm that aims at improving the model stability. Bagging trains each model using a subset that is chosen randomly from the training set. It takes the majority vote class or takes the mean value of the individual model as the final prediction result. It can be applied to some base classifiers (e.g, decision trees, K-nearest neighbour, and logistic regression). * AdaBoost is an adaptive boosting algorithm. The method increases the weight of the misclassified samples of the previous classifiers so that the new classifier can focus on the training samples that are prone to misclassification. As for the Bagging method, it relies on Decision Tree models as base classifier thus its results can be compared with tree-based methods [51] (e.g., Decision Tree and Random Forest). This study has adopted the \(K\)-fold cross validation method in the model evaluation phase. Cross validation is used to evaluate the performance of machine learning models [52]. This method ensures that the score of the model is not affected by the way researchers choose the training and test sets. Cross validation split dataset \(D\) into \(K\) disjoint subsets: \(D=D1\cup D_{2}\cup\...\cup D_{K}=\ (i\
eq j)\). It makes the data distribution of each subset \(D_{i}\) be the same as possible. It takes \(K-1\) subsets as a training dataset and the remaining subset as a test dataset, and repeats the procedure \(K\) times, eventually return the mean values of \(K\) testing results. A key parameter of the \(K\)-fold cross validation is \(K\) (i.e., the number of fold of a given dataset). Some choices are \(K=3\), \(K=5\), and \(K=10\), and the most widely used value in model evaluation is 10. \(K=10\) is better for large data size [53] and it provides a good trade-off between computational cost and bias. Therefore, 10-fold cross validation has been used in this study. Our model evaluation is based on different performance metrics, such as _accuracy_, _precision_, _recall_, and _F1-score_. For binary classification problems see Equations (2)-(5):  
\[accuracy=\frac{tp+tn}{tp+tn+fp+fn} \tag{1}\] \[precision=\frac{tp}{tp+fp}\] (2) \[recall=\frac{tp}{tp+fn}\] (3) \[F1\text{-}score=2\times\frac{precision\times recall}{precision+ recall} \tag{4}\]  
where \(tp\) is the number of true positive, \(tn\) is the number of true negative, \(fn\) is the number of false negative and \(fp\) is the number of false positive.