# Investigation on Self-Admitted Technical Debt in Open-Source Blockchain Projects
## 3 SATD Detection in Blockchain Software Projects
### Choice of Detection Technique and Configuration

Table 3: Number of extracted comments before and after the filtering process. The training dataset described in ref. [11] is composed of 57,676 source code comments, manually categorized into SATD classification labels. Although the set pf dataset labels includes multiple categories of SATD, in order to avoid an excessive imbalance, we decide to consider only the three most represented categories: \"design\", \"requirements\" and \"withoutclassification\" (i.e., absence of SATD), where \"design\" indicates that the comment refers to suboptimal design, including workarounds and features that need to be extended to support more features, and \"requirement\" indicates that the comment refers to some software requirements that have not been fully satisfied by the implementation. The training phase was repeated several times, and at each training of the classifier, the training dataset was randomly divided into two sets, training set (90% of labeled comments) and the test set (the remaining 10%). The configuration of the classifier consists of the selection of the NLP properties the classifier has to use in its training. A configuration file (named prop) consists of a series of properties described in the \"ColumnDataClassifier\" class [36]. In our study, we defined two different props. A first prop was extracted from the reference study [11], while we created the second prop aiming to optimize training performance, i.e., to reach the highest possible score of the \"Micro-averaged accuracy/F1\" and \"Macro-average F1\", These two configuration files describe the configuration of two different SATD detection models. The two configurations differ in that in the configuration used in ref. [11], the use of N-Grams is set to false and the use of lowercase words is set to true, while in the prop we created, the use of N-Grams is set to true, the use of split words is set to true, and the length of the N-Grams is limited between 1 and 5. In summary, the main differences between the two configuration files are the use of N-Grams, the form of the split words, and the length of the N-Grams. For each class, the results show the number of true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN). True positives are cases where the maximum entropy classifier correctly identifies, while true negatives are comments without technical debt that are classified as such. The table also shows the accuracy, precision, recall and F1 measure for each class. The classes in question are the three classes present in the dataset, namely WITHOUTCLASSIFICATION, REQUEMENT and DESIGN. Using the TP, TN, FP, and FN values, we are able to evaluate the performance of different classifier configurations. Performances are evaluated in terms of  
\[Precision\ P=TP/(TP+FP),\ Recall\ R=TP/(TP+FN),\]  
\[Accuracy=(number\ of\ correct\ predictions)/(total\ predictions),\]  
\[F1\ score=(2\times((P\times R)/(P+R)).\]  
The Micro-averaged F1 score is used to evaluate the overall performance of a classifier and has a value between 0 and 1. Table 4 lists the performance of the training by using the same configuration as in [11], while Table 5 lists the performance of the training obtained using the optimized configuration. We can see from tables that the performances obtained through the optimized configuration outperform those of the reference configuration. This is essentially given by a reduction in false positives and this is reflected in a substantial reduction in the number of comments classified as SATD compared with what was obtained using the reference configuration. Although our configuration has better results in terms of score, we felt it necessary to make some considerations. As mentioned, the optimized classifier uses a different model than the reference study. By running a test classification on one of the blockchain projects under review (Avalanche) with both classifiers, the results obtained highlight the differences between the two models.