# Advanced Bayesian Network for Task Effort Estimation in Agile Software Development
## 3 The BN Model
### The New Proposed BN Model
#### 3.2.3 Prediction Accuracy  
The BN model prediction accuracy is evaluated using statistical measures: Magnitude of Relative Error (MRE), Mean Magnitude of Relative Error (MMRE), Prediction at Level m (Pred. (m)), Accuracy, Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Relative Absolute Error (RAE) and Root Relative Squared Error (RRSE). Detailed descriptions of the measures, as well as the reasons for their use, are given in [7; 21; 36]. In this article, a Confusion Matrix is also used. The Confusion Matrix, just like the measures based on the prediction results in the confusion matrix, is well established as a measure of classification performance for imbalanced datasets [37; 38]. The Confusion Matrix contains the prediction results and the actual values (classes) of these data. It is an nxn matrix, where n is the number of classes. The model validation is performed using empirical data. WEKA provides a k-fold cross-validation and summary statistics (prediction accuracy, MAE, RMSE), which are used to verify the accuracy of the generated model. The WEKA error statistics are normalized. The predicted distribution for each class is matched against the expected distribution for that class. All the mentioned WEKA errors are computed by summing all classes of an instance, not just a true class [39]. In this case, a 10-fold cross-validation is used. The dataset is randomly divided into 10 equally sized subsets. Out of these 10 subsets, one is taken as the validation dataset, and the other nine sets are used as training data. Each of the nine training datasets is compared with a validation dataset to calculate the percentage of the model accuracy. The cross-validation process is repeated ten times, each of the ten subsets being used exactly once as a validation set. The results from all the 10 trials are averaged. The measurement results are presented in Table 5. Compared to the old BN model, the new one has a slightly worse, but still satisfactory, prediction accuracy. Only two tasks were misclassified, and the prediction accuracy is 98.75%, which is still an extremely accurate estimate. The MAE values indicate that the expected effort will be within 3.7% of the true effort for the last set of data. Small differences between the MAE and the RMSE values indicate that the error variance is relatively small. The MMRE values suggest that the prediction error is relatively constant, with no occasional large deviations. The diagonal of the confusion matrix shows correctly classified instances (Figure 3). The two misclassified tasks were placed in adjacent classes. There are no errors classified in remote classes. \begin{table}
\begin{tabular}{l c c} \hline \hline  & \multicolumn{2}{c}{**BN Model**} \\ \hline  & **Old Model** & **New Model** \\ \hline Number of Tasks & \(160\) & \(160\) \\ \hline Accuracy (Correctly classified instances) & 99.375\% & 98.75\% \\ \hline MAE & 0.026 & 0.037 \\ \hline RMSE & 0.065 & 0.0792 \\ \hline RAE & 9.71\% & 15.77\% \\ \hline RRSE & 17.81\% & 23.13\% \\ \hline Pred. (25) \% & 40.625\% & 27.5\% \\ \hline MMRE & 6.21 & 1.27 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results. == Detailed Accuracy By Class ===  
TP Rate FP Rate Precision Recall F-Measure ROC Area Class  
1 0 1 1 1 1 0-1  
1 0.007 0.958 1 0.979 0.995 1.1-2  
0.97 0.008 0.97 0.97 0.97 0.982 2.1-5  
0.973 0 1 0.973 0.986 0.975 5.1-10  
1 0 1 1 1 1 10.1-25  
1 0 1 1 1 1 25.1-40  
1 0 1 1 1 >40.1  
Weighted Avg. 0.988 0.003 0.988 0.988 0.998 0.99  
== Confusion Matrix ==  
a b c d e f g <- classified as  
33 0 0 0 0 0 0 1 a = 0-1  
0 23 0 0 0 0 0 1 b = 1.1-2  
0 1 32 0 0 0 0 1 c = 2.1-5  
0 1 36 0 0 0 1 d = 5.1-10  
0 0 0 0 22 0 0 1 e = 10.1-25  
0 0 0 0 0 6 0 1 f = 25.1-40  
0 0 0 0 0 0 6 1 g = >40.1  
The high accuracy of the model is confirmed via a comparison with the results listed in the literature [40]. Measure Pred (m) shows the worst results (Table 5). To determine the reason for this, Pred (25) was calculated to one instance from the task set.