# Survivability Prediction of Open Source Software with Polynomial Regression
## 4 Results and Analysis
### RQ2: Model Evaluation  
To address RQ2, we evaluated our model using RSE and RMSE. Since MSE calculates the average of the squares of errors, it may overemphasize errors, making it preferable to use RMSE for assessing the accuracy of the model. RMSE, being the square root of MSE, rescales the magnitude of errors to their original units, allowing for a more intuitive understanding of model accuracy. RSE represents the standard deviation of the errors, taking into account the complexity of the model. An increase in RSE value indicates lower consistency in predictions. For these reasons, we use RSE to assess the model's fit, i.e., how well it has learned from the given data. Since a model can be inaccurately measured for accuracy if not properly trained, we assess its fit using RSE. These metrics were extracted for each fold through performing K-fold cross-validation. The results of the evaluation are presented in Table 6.  
Upon reviewing Table 6, a consistent pattern of low RSE values, with an average of 0.0101 across all folds, indicates that the model effectively captures the variability of the dataset. However, the RMSE values exhibit considerable variability, notably with an outlier in fold 1. To further analyze these figures, visualization was performed through Figure 10. Using SVD (Singular Value Decomposition) for PCA (Principal Component Analysis), we reduced five-dimensional data to two dimensions to create a three-dimensional graph, as mentioned in [29]. SVD is a numerically very stable decomposition method, which is more robust against errors that can occur in floating-point arithmetics, hence the choice of SVD over eigenvalue decomposition. X and Y represent the data reduced via SVD, while Z is the target variable, \(Trend_{bug}\). The figure shows that some data were predicted lower than their actual values, leading to an increased RMSE value in Fold 1. While the rest of the data did not match perfectly, they were predicted quite closely. In contrast, Folds 2, 3, and 4 displayed significantly lower RMSE values of 0.1810, 0.0350, and 0.0214, respectively, indicating strong compatibility between the model and the training and target datasets used in these folds. Consequently, the overall average RMSE slightly increased to 1.3100, compared to the lower values observed in Folds 2, 3, and 4. This discrepancy suggests that the test set used in Fold 1 included data points different from those used in the model's  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Metric** & **Fold 1** & **Fold 2** & **Fold 3** & **Fold 4** & **Mean** \\ \hline RMSE & 5.0024 & 0.1810 & 0.0350 & 0.0215 & 1.3100 \\ RSE & 0.0099 & 0.0098 & 0.0142 & 0.0067 & 0.0101 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Model evaluation results using K-fold cross-validation.  
training. Additionally, this implies that the training data in Fold 1 did not fully represent the test data.  
In particular, the test set in Fold 1 included an OSS called Alamofire, which generally had higher feature values compared to other OSS, notably in metrics like popularity represented by stars. Consequently, the training set for Fold 1 lacked data points with high feature values, such as stars or other features, leading to inaccurate predictions when testing with this OSS and resulting in the relatively high RMSE value. Overall, while the model demonstrated good performance, the discrepancy in accuracy for certain folds indicates areas where further refinement is needed, highlighting the importance of dataset augmentation in future iterations.