# Success Prediction of Crowdsourced Projects for Competitive Crowdsourced Software Development
## 3 Methods
### Training and Prediction  
For the training, BERT is fine-tuned on a specific task using labeled data by adding a task-specific layer. However, the input passes through pre-trained BERT layers followed by the task-specific layer. Note that unseen data of CCSD projects are used for the success prediction of CCSD projects. The pre-trained BERT model can be customized for a specific task by fine-tuning its parameters using task-specific data, i.e., the TopCoder dataset. To perform a classification task, we fine-tuned the BERT classifier on our problem and then utilized it to capture the correlation between project features and their success status. BERT is better than other models because of its ability to pre-train large amounts of text data [32], allowing it to learn contextual relationships between words and sentences. The feature-based models, relying on predefined and engineered features, can struggle to capture intricate contextual relationships and nuanced patterns effectively [53]. These models, often depending on handcrafted features, might misclassify instances not fitting well within their predefined feature space [53]. BERT's pre-training on a large corpus, comprehensive exposure to textual data, architecture design, and fine-tuning techniques enable it to outperform other models on NLP tasks. Moreover, BERT's attention mechanisms make it suitable for long text sequences [32].  
The samples are randomly split into 80-20% train-testing datasets, with _13,516_ samples for training and 3378 for testing. The train-test dataset's token IDs, attention masks, and labels are combined into TensorDatasets to feed into the BERT model. We employ the BERT-based-uncased word-piece model (for English) developed by Google ([[https://github.com/google-research/bert](https://github.com/google-research/bert)]([https://github.com/google-research/bert](https://github.com/google-research/bert)), accessed on 15 March 2023) and pre-trained on BookCorpus and Wikipedia corpora. The \"BertForSequenceClassification\" model is used for classification, with recommended hyperparameters by the BERT authors [32]. As the number of encoders and parameters increases, the computational cost of the system escalates. Hence, we opted for the BERT-base model due to its lightweight nature and quick training capabilities with the default setting stated in [32].  
The BERT-base model, featuring 12 transformer layers, 12 attention heads, 768 hidden layers, an epsilon value of \(1\times 10^{-8}\), a maximum sequence length of 256, trained for 10 epochs, and with a learning rate set at \(2\times 10^{-5}\), is employed in the training process using Google Colab. Google Colab offers a single free GPU, either Tesla T4 or Tesla K80, contingent on availability [32]. Noteworthy is our exploration of various hyperparameter combinations, including the number of layers (12/24), BERT heads (12/16), sequence length (64/128), and batch size (16/32/64). However, all other parameters retain their default values. It is essential to highlight that the suggested hyperparameters mentioned earlier yield optimal results. The optimizer updates parameters for each batch during every epoch. The results of each training iteration are assessed by computing loss and accuracy. For instance, the proposed model predicts the CSD project defined in Section 3.2 as _successful_ after training.