## 5 The Results of the Experiment
### Impact of Historical Changes in Software Metric Measurements-RQ1  
As described previously, the performance of the software maintainability prediction model was observed through the series of iterations. In each of them, an additional layer of changes in the C&K software metrics of historic releases were added to the software maintainability prediction models. To evaluate the performance of the model in each iteration of the experiment formally, the models were evaluated by two prediction model performance indicators, classification accuracy and F-score, for both CART classification decision trees and random forest for classification. Our goal was not to optimise the classification model to have better predictability ability. For this reason, we limited the scope of the experiment only to software metrics for which the literature showed that they should be the most impactful in maintainability prediction. Also, we wanted to test if the changes in older (not only the newest) releases had an impact on MINC change prediction. Of course, if the goal of our experiment was to build the best model in predicting the future MINC change, all the available metrics would be used, along with several other measurements. Readers should keep in mind that using only software metric changes from the past has limited ability in the prediction, but does provide the answers to our research question. Table 7 summarises the results of both model performance indicators used in the study, namely overall classification accuracy and F-score. According to the data collected in the experiment, the software maintainability prediction model reaches the value 0.570 of classification accuracy for both CART and Random Forest models in the first iteration of the experiment. In that iteration of the experiment, the prediction model is based only on metric changes between R-2 and R-1 releases of the studied software. The F-score indicator of the prediction model in its first iteration reaches the value 0.436 for both, CART and random forest. The values of both classification performance metrics show that the quality of the future MINC change prediction is limited. Again, this should be taken with consideration of our research question and the limitation of the experiment in using only these metrics, while other software and non-software related metrics could (and probably do) play an important role in predicting the maintainability change. \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt}} \hline \hline
**Ranking Method** & **Description** \\ \hline Information Gain & The expected amount of information (reduction of entropy). \\ \hline Information Gain Ratio & A ratio of the information gain and the attribute’s intrinsic information, which reduces the bias towards multivalued features that occurs in information gain. \\ \hline Gini Decrease & The inequality among values of a frequency distribution. \\ \hline ANOVA & The difference between the average values of the feature in different classes. \\ \hline x\({}^{2}\) & Dependence between the feature and the class as measured by the chi-square statistic. \\ \hline ReliefF & The ability of an attribute to distinguish between classes on similar data instances. \\ \hline FCBF & An entropy-based measure, which also identifies redundancy due to pairwise correlations between features. \\ \hline Logistic Regression & The logistic regression classification algorithm. \\ \hline Random Forest & Predict using an ensemble of CART decision trees. \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The ranking methods used in the experiment. In the last iteration of the experiment, when values of changes in the software metrics for all releases of the software projects considered in the study are added to the prediction models, the classification accuracy reached value 0.578 in the case of the CART prediction model, and value 0.580 in the case of the Random Forest-based prediction model. The F-score reached the value 0.531 in the case when CART was applied, and value 0.553 in the case of random forest.