## 3 Methodology
### Statistical Validation Results

The previous section discussed the F-score and AUC metrics results of experimental studies conducted with machine learning, deep learning, and transfer learning approaches. In addition, it statistically evaluated the experimental results and used validation metrics MCC and Kappa to analyze them comprehensively. Accordingly, MCC and Kappa values were calculated for all experiments performed in all three stages. Table 7 lists the MCC and Kappa values of the experiments conducted with machine learning algorithms, while Table 8 displays the MCC and Kappa values calculated by applying future selection techniques to the same algorithms. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **MCC** & **Kappa** \\ \hline NB & 0.663 & 0.627 \\ LMT & 0.809 & 0.808 \\ RF & 0.797 & 0.794 \\ SLR & 0.775 & 0.772 \\ JRip & 0.779 & 0.778 \\ NBM & 0.841 & 0.843 \\ SMO & 0.806 & 0.806 \\ LR & 0.753 & 0.752 \\ Bagging & 0.694 & 0.691 \\ J48 & 0.616 & 0.615 \\ Multiclass Classifier & 0.753 & 0.752 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Statistical validation results of machine learning algorithms. Figure 7: Confusion matrix performances of the three best algorithms. Table 9 presents the MCC and Kappa values of experimental studies with deep learning algorithms. Table 10 displays the MCC and Kappa values of experimental studies conducted by transfer learning techniques. An overall assessment of Tables 7-10 revealed that MCC and Kappa values supported the F-score and AUC values. As both metrics approach '1,' the accuracy of the predictions is supported. Considering the three groups of experiments, the average MCC and Kappa values of 0.85, particularly in Table 10, statistically confirmed the accuracy of the results. As a result, it is viable to conclude that the algorithms provided consistent results. This work shows promising results from the newly generated Turkish requirements dataset and the use of the point of view of the transformers. However, there are some limitations of the dataset, which can be shown in three aspects: (i) While creating the dataset, software requirements were obtained through Windows desktop, web, and mobile software projects. Therefore, the dataset can be generalized with the collection of software requirements from different software projects such as cloud computing, data science, blockchain, information security, embedded systems, wearables software development, DevOps, and video game development. (ii) The dataset can also be expanded with the addition of more samples increasing sample size above 4600. (iii) The dataset includes 3000 functional requirements and 1600 non-functional requirements. In this respect, it is a relatively unbalanced dataset. The dataset can be further extended with the addition of more non-functional requirements. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **MCC** & **Kappa** \\ \hline BERT & 0.874 & 0.873 \\ BERTurk & 0.898 & 0.897 \\ DistilBERT & 0.857 & 0.854 \\ RoBERTa & 0.789 & 0.788 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Statistical validation results of transfer learning methods. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**MCC**} & \multicolumn{2}{c}{**Kappa**} \\ \cline{2-5}
**Algorithm** & **CFS** & **GR** & **CFS** & **GR** \\ \hline NB & 0.787 & 0.856 & 0.507 & 0.626 \\ LMT & 0.842 & 0.816 & 0.640 & 0.816 \\ RF & 0.844 & 0.805 & 0.634 & 0.805 \\ SLR & 0.841 & 0.904 & 0.637 & 0.783 \\ JRip & 0.836 & 0.905 & 0.627 & 0.618 \\ NBM & 0.837 & 0.830 & 0.629 & 0.626 \\ SMO & 0.834 & 0.915 & 0.621 & 0.810 \\ LR & 0.845 & 0.891 & 0.646 & 0.759 \\ Bagging & 0.832 & 0.862 & 0.691 & 0.688 \\ J48 & 0.751 & 0.821 & 0.431 & 0.601 \\ Multiclass Classifier & 0.845 & 0.834 & 0.646 & 0.759 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Statistical validation results of machine learning algorithms with feature selection methods.