## 5 Empirical Study Design
### Evaluation Criteria  
In order to assess the accuracy of the proposed model's predictions, a set of evaluation metrics is utilized, including MAE, MdAE, and SA, as outlined in the formulas below. These metrics have been selected due to their established capacity to provide unbiased assessments, emphasizing their versatility and reliability in evaluating prediction models--which is critical for maintaining the integrity of the evaluation process--even when dealing with under-estimations or over-estimations, as validated in prior SEE studies [19; 30; 106; 125; 126]. MAE is particularly useful for quantifying the average magnitude of errors in predictions, offering a straightforward interpretation of model accuracy. In contrast, MdAE provides a robust measure by computing the median of these absolute differences, thereby minimizing the influence of outliers and offering a clearer view of the model's typical error magnitude. SA, on the other hand, quantifies the proportion of correctly predicted instances, highlighting the model's precision and its practical utility in real-world applications. For a high-performing prediction model, the objective is to achieve lower MAE and MdAE values along with higher SA values, ensuring a balanced and thorough assessment of the model's predictive performance. Figure 3: Histogram of story points across all projects. Figure 2: Average story points per project. \[\text{MAE}=\frac{1}{n}\sum_{i=1}^{n}\lvert y_{i}-\hat{y}_{i}\rvert \tag{1}\]  
\[\text{MdAE}=\text{median}_{i=1}^{n}(\lvert y_{i}-\hat{y}_{i}\rvert) \tag{2}\]  
where  
* \(n\) is the number of data points;
* \(y_{i}\) is the actual value for the \(i\)th data point;
* \(\hat{y}_{i}\) is the predicted value for the \(i\)th data point. The recommendation by [PERSON] and [PERSON] (2018) to employ SA as a metric for the comparative analysis of prediction models further underscores its importance, particularly due to its foundation on MAE, which facilitates a direct comparison of models' predictive accuracy. The SA formula is expressed as  
\[\text{SA}=1-\left(\frac{\text{MAE}_{p_{i}}}{\text{MAE}_{p_{0}}}\right)\times 100 \tag{3}\]  
In this formula, \(\text{MAE}_{p_{i}}\) represents the MAE of the prediction model \(p_{i}\) under evaluation, while \(\text{MAE}_{p_{0}}\) denotes the MAE of a baseline model. This baseline, typically established through a comprehensive series of random guesses, provides a benchmark for evaluating the predictive superiority of the model \(p_{i}\) over chance, often around 1000 runs. The SA metric is invaluable for discerning the efficacy of a prediction model \(p_{i}\) in surpassing mere random guessing, with its value range offering insights into the model's relative accuracy. If the SA value falls within the range of \([0,100]\), it suggests that the model is more accurate than random guesses. A value closer to zero implies that the model \(p_{i}\) is only marginally better than a random guess. Conversely, a negative SA value indicates that the model performs worse than random guessing. For a model to be deemed high performing, it is essential not only to demonstrate low MAE and MdAE values, but also to manifest a high SA value, signifying the model's enhanced predictive accuracy beyond mere random chance. This high SA value is pivotal, especially in applications demanding high precision, making SA an indispensable metric in the predictive modeling domain, offering a clear and quantifiable way to gauge a model's effectiveness in making accurate predictions. In addition to these metrics, effect size has been introduced as a quantitative measure to determine the magnitude of differences observed between the proposed model's performance and that of baseline or state-of-the-art models.