# Success Prediction of Crowdsourced Projects for Competitive Crowdsourced Software Development
## 3 Methods
### Evaluation
#### 3.6.5 Results and Discussions
The analysis, executed using default settings in RStudio, is depicted in Figure 4. The results indicate that \(F\) (153.52) \(>F_{eric}\) (2.87), and \(p\)-value (1.44E-20) \(<\) (\(\alpha\) = 0.05), confirming the significant difference in F1 score among the approaches. We leveraged BERT for CSPs classification because it surpasses other models primarily due to its ability to generate contextualized word embeddings [32], capturing nuanced meanings and relationships in a bidirectional manner. Unlike traditional models, BERT's pre-training involves exposure to extensive and diverse corpora, enabling it to learn rich linguistic representations [32]. Its adaptability to specific tasks is further enhanced by transfer learning. BERT's attention mechanism and contextual embeddings allow it to effectively handle long-range dependencies, contributing to its comprehensive understanding of sentence contexts. Its consistent achievement of state-of-the-art results across various benchmarks underscores its superiority in natural language processing tasks. While other models may have limitations in capturing contextual intricacies or handling long-distance dependencies [32], BERT's advanced features make it a preferred choice for our study. Figure 4: ANOVA analysis on performance comparison. Figure 3: Distribution of F1 score. Importance of Re-Sampling  
To address the second research question, we employ two types of re-sampling techniques, namely over-sampling and under-sampling, to address the class imbalance in the dataset. Over-sampling involves generating samples for the minority class using RandomOverSampler, while under-sampling removes excess majority class records from imbalanced datasets using RandomUnderSampler. Evaluation results with and without re-sampling are presented in Table 3 and Figure 5. The first column represents the re-sampling settings, while columns 2-5 display the performance results of accuracy, precision, recall, and F1 score for each re-sampling technique. The results indicate that with the over-sampling technique, the accuracy, precision, recall, and F1 score achieve values of 96.8%, 96.97%, 96.8%, and 96.81%, respectively. Similarly, using the under-sampling technique, the accuracy, precision, recall, and F1 score attain values of 90.26%, 90.27%, 90.26%, and 91.28%, respectively. From Table 3 and Figure 5, we draw the following conclusions:  
* The application of the over-sampling technique has shown improvements in accuracy, precision, recall, and F1 score. The increase in accuracy is calculated as 3.26% = (96.80 \(-\) 93.75%)/93.75%, precision as 3.42% = (96.97 \(-\) 93.76%)/93.76%, recall as 3.26% = (96.80 \(-\) 93.75%)/93.75%, and F1 score as 3.27% = (96.81 \(-\) 93.75%)/93.75%. This improvement can be attributed to BERT's exposure to more data, allowing it to learn meaningful patterns more effectively. * On the other hand, under-sampling reduces the number of samples in the majority class, resulting in a loss of information. Consequently, the majority and minority classes in the fine-tuned BERT exhibit lower performance when under-sampling is applied. * By examining the confusion matrix (Figure 5), it becomes evident that the true positive and true negative rates are high when over-sampling is used. This indicates that the fine-tuned BERT correctly identifies positive and negative examples. However, when under-sampling is employed, the true positive and true negative rates are relatively low. This suggests that the model may misclassify some examples and require further optimization. Comparison of BERT with M/DL Algorithms  
For the third research question, we use M/DL algorithms (LSTM, CNN, XGBoost, and GradientBoost) with BERT embeddings. We train these algorithms and compare their performance to the BERT classifier. Figure 5: BERT confusion matrix.