# Success Prediction of Crowdsourced Projects for Competitive Crowdsourced Software Development
## 3 Methods
### Input Formatting  
Software project success prediction is strongly correlated with representing necessary words to BERT. Recent research [49; 50] suggests different types of word representation techniques used in NLP, e.g., Word2 Vec [50], GloVe [51], and FastText [52]. However, BERT [32] is a powerful NLP model pre-trained on a large text corpus. Its ability to learn highly contextualized representations of words and phrases makes it effective for various tasks. The input formatting process from Figure 1 presents the process to feed input formatting into BERT for generating word embeddings and performing classification.  
Bert-base-uncased\" BertTokenizer\" from the _Transformers_ library is leveraged in this regard. The formatting process of \(p\) is as follows:  
1. **Tokenization and Adding Special Tokens:** _BertTokenizer_ splits the text of the CCSD project into a sequence of subwords, which are then mapped to integer IDs using a pre-defined vocabulary. The tokenizer also adds special tokens, i.e., CLS (classification) and SEP (separator), to mark the beginning and end of the project description, respectively. Its tokenization process can be formulated as follows: \[S=[CLS]+w_{1}+w_{2}+...+w_{n}+[SEP]\] (5) where \(S\) represents the sequence of subwords of \(p\), \(w_{i}\) is a subword, and \(n\) is the total number of subwords per project \(p\).
2. **Token IDs:** Every \(w\) of \(S\) is converted to a numerical ID using a pre-defined vocabulary that can be formulated as: \[T=101+t_{1}+t_{2}+...+t_{n}+102\] (6) where 101 and 102 are special tokens for CLS and SEP, respectively added by the tokenizer, \(T\) represents the sequence of token IDs of \(p\), and \(t_{i}\) is an ID for the corresponding \(w_{i}\), and \(n\) is the total number of token IDs for each \(p\).
3. **Padding and Truncation:** In this step of input formatting, token IDs are padded and truncated to 256 tokens. If the total number of IDs for a \(p\) is shorter than 256 tokens, it is padded with the special token '0'. In the case of more than 256 tokens, the remaining tokens are truncated as follows: \[T^{\prime}_{1:256}=\left\{\begin{array}{rl}&T_{1:m}[PAD]^{256-m}&\text{if }m<256\\ &T_{1:256}&\text{if }m>256\end{array}\right.\] (7) where, \(T^{{}^{\prime}}\) represents the final IDs after padding and truncation for \(p\).
4. **Attention Masks:** Attention masks differentiate between actual tokens and padding tokens in the input sequence. This is crucial since the attention mechanism in the transformer architecture employs these masks to focus on the real tokens and disregard the padding tokens. The attention mask \(MT^{\prime}\) for input sequence \(T^{{}^{\prime}}\) can be formulated as \[MT^{\prime}=\begin{bmatrix}1&1&\cdots&1\\ 1&1&\cdots&1\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&0\end{bmatrix}\] (8) where the first row of \(1^{{}^{\prime}}\)s indicates the actual tokens in the input sequence, and the second row of \(0^{{}^{\prime}}\)s indicates the padding tokens. Using a sequence length of 32, the proposed approach performs input formatting for the illustrated example presented in Section 3.2 as follows: 1. **Input Text:** \"Google Drive Legal Hold Time Capsule Management Tool Prototype Challenge\". 2. **Tokenization and Adding Special Tokens:** [CLS, 'Google', 'Drive', 'Legal', 'Hold', 'Time', 'Capsule', 'Management', 'Tool', 'Prototype', 'Challenge', SEP] 3. **Token IDs:** [101, 8224, 3140, 3424, 2982, 2051, 13149, 3259, 6999, 9817, 4721, 102] 4. **Padding and Truncation:** [101, 8224, 3140, 3424, 2982, 2051, 13149, 3259, 6999, 9817, 4721, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 5. **Attention Masks:** [101, 1, 1, 1, 1, 1, 1, 1, 1, 1, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]