# Investigating Novice Developers' Code Commenting Trends Using Machine Learning Techniques
## 5 Results and Discussion
### RQ-2 Is It Possible to Classify Student Source Code Comments Using Machine Learning Techniques? In order to answer this research question, we applied three machine learning algorithms to our labeled dataset obtained in _RQ-1_. The algorithms considered in our study to classify the source code comments are: (i) _Support Vector Machine (SVM)_, (ii) _Decision Tree_, and (iii) _Random Forest_. After the machine learning models were implemented, the results were recorded. We analyzed the results with three performance parameters, i.e., _accuracy, precision, and recall_. Table 7 and Figure 6 represent the comparison of the _accuracy_ measure for all three classifiers. The feature extraction helped in improving the overall performance of all three classifiers. As apparent from the results, the _Decision Tree_ algorithm outperformed the rest of the two models with an overall accuracy of 0.85 (85%). The performance of the _Support Vector Machine_ was poor. We presume that this is the one main reason it was not used previously for such a problem, i.e., classification of source code comments. Figure 6 shows a graphical representation of the accuracy before feature extraction and after feature extraction. One interesting aspect to note from Table 8 and Figure 7 is that the _Support Vector Machine_ classifier produced good results when only the text feature (i.e., comment content) was selected as the only predictor for the machine learning model. However, when both quantitative data (numerical information such as token comment size, length, etc.) and textual data (e.g., comment text) were used as predictors, both _Random Forest_ and _Decision Tree_ achieved good performance. [PERSON] et al. [10] evaluated and analyzed the code comments with a single label as either sufficient or insufficient using a binary classifier. In their research study, they only considered the text feature of the comments. They demonstrated that after the reduction in the vocabulary size due to lemmatization, the Multinomial Naive Bayes classifier's accuracy was reduced by 5%, but Random Forest Classifier's accuracy was improved by 6%. The results of their study reveal that they achieved an overall precision rate of 82% using Multinomial Naive Bayes. By using a Random Forest classifier and lemmatization, they were able to achieve a classification precision of 90%. In another study, [PERSON] [36] compared the performance of two machine learning models to automatically classify code comments in five open-source mobile applications. Their aim was to assess code comments produced by professional developers. Specifically, they used two well-known classes of supervised machine learning algorithms based on probabilistic classifiers and Decision Tree algorithms: Naive Bayes Multinomial and Random Forest. According to their results, Random Forest outperformed the Naive Bayes Multinomial classifier in automatically classifying the code comments. In our research study, we employed three different machine learning classifiers to compare their performance results. All three models produced better results on text data; however, when the other quantitative features were taken into account, both Random Forest and Decision Tree produced good results than the Support Vector Machine, with Decision Tree having the highest accuracy, i.e., 85%. It is interesting to note \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Accuracy on Text Feature** \\ \hline Random Forest & 0.83 (83\%) \\ Decision Tree & 0.86 (86\%) \\ Support Vector Machine & 0.83 (83\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Accuracy of all three classifiers on text data. Figure 7: Accuracy of all three classifiers on text data only. that previous studies, as discussed above, also revealed the effectiveness of the Random Forest classifier for classifying source code comments. Table 9 and Figure 8 represent the _precision_, _recall_ and _F1-Score_ of _Random Forest_ and _Decision Tree_ on all the source code comments categories of our dataset.