# Success Prediction of Crowdsourced Projects for Competitive Crowdsourced Software Development
## 3 Methods
### Evaluation
#### 3.6.5 Results and Discussions  
Comparison between the Proposed Approach and SOTA Deep Learning Approaches  
To address the first research question, we compare the proposed approach with ML-SP (baseline deep learning approach [4]). The hold-out validation results, presented in Table 2, compare the performance of different classifiers. The results demonstrate that the proposed approach achieves an accuracy of 93.75%, precision of 93.76%, recall of 93.75%, and F1 score of 93.75%. In comparison, the embedding-based LSTM approach attains an accuracy of 86.62%, precision of 86.54%, recall of 99.93%, and F1 score of 92.75%. Likewise, the embedding-based CNN approach achieves an accuracy of 86.56%, precision of 86.53%, recall of 99.86%, and F1 score of 92.72%. Additionally, ML-SP achieves a precision of 82.64%, recall of 86.16%, and F1 score of 84.36% leveraging the CNN classifier. From Table 2, we draw the following conclusions:  
* The BERT model outperforms other models. The proposed approach gives an accuracy of 93.75% and improves the precision, recall, and F1 score upon the baseline deep learning results by 13.46% = (93.76 \(-\) 82.64)/82.64) \(\times\) 100%, 8.83% = (93.75 \(-\) 86.16)/86.16) \(\times\) 100%, and 11.13% = (93.75 \(-\) 84.36)/84.36) \(\times\) 100%, respectively. Moreover, the proposed approach gives an accuracy of 96.8% and improves the precision, recall, and F1 score upon the baseline deep learning results by 17.30% = (96.97 \(-\) 82.64)/82.64) \(\times\) 100%, 12.29% = (96.8 \(-\) 86.16)/86.16) \(\times\) 100%, and 14.80% = (96.81 \(-\) 84.36)/84.36) \(\times\) 100%, respectively. One possible reason is that BERT's ability to capture complex word relationships makes it a powerful tool in text classification. * The lowest precision, recall, and F1 scores obtained by the proposed approach surpass the highest precision, recall, and F1 scores achieved by SOTA deep learning methods, as illustrated in Table 2. The improvement is credited to employing the BERT model and its embeddings in training LSTM and CNN classifiers. BERT, a robust language model, creates embeddings that grasp the nuanced meaning of the text. Leveraging LSTM and CNN capabilities, often used in text classification, enables these classifiers to learn complex patterns and connections in input data, extracting pertinent features for classification. Integrating BERT embeddings, with their rich semantic information, further boosts LSTM and CNN performance, resulting in exceptional outcomes for text classification tasks. Furthermore, Figure 3 illustrates the distribution of F1 scores in the evaluation of BERT, LSTM, CNN, and ML-SP. A beanplot, employed for this visualization, portrays the distribution of a continuous variable across distinct groups. Each approach is represented by a bean, where the width reflects the data density--with broader beans indicating higher density. Examining Figure 3, it becomes apparent that BERT exhibits a minimum F1 score surpassing the maximum F1 score of ML-SP. Notably, BERT demonstrates greater stability compared to LSTM and CNN. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline BERT & **93.75** & **93.76** & **93.75** & **93.75** \\ LSTM & 86.62 & 86.54 & 99.93 & 92.75 \\ CNN & 86.56 & 86.53 & 99.86 & 92.72 \\ ML-SP (Baseline) & — & 82.64 & 86.16 & 84.36 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison against state-of-the-art approach (%). To determine the statistically significant differences in the mean performance (F1 score) among BERT, LSTM, CNN, and ML-SP, a single-factor Analysis of Variance (ANOVA) is conducted. ANOVA is a statistical technique employed to assess whether there exists a significant difference in the means of three or more independent groups or samples.