## 3 Methodology
However, the F-score relatively increased when the GR feature selection technique was applied to the model developed with SMO, Simple Logistic Regression, and the Naive Bayes algorithm. Additionally, in two feature selection techniques, the performance of the model created by the Naive Bayes Multinomial, which had the best performance in the prior trial, declined. In the second stage, the study conducted experiments with CNN, LSTM, Bi-LSTM, GRU, and Bi-GRU deep learning algorithms and accordingly displayed the findings in Table 5.

The assessment of the models developed with deep learning algorithms revealed that the CNN algorithm was explicitly successful and outperformed other deep learning algorithms in classifying software requirements as FR and NFR. This study also performed experiments to classify software requirements as FR and NFR using BERT, BERTurk, DistilBERT, and RoBERTa transfer learning methods and presents the results in Table 6.

Finally, in the third stage, the analysis of the experimental study results related to transfer learning methods indicated that the model developed with the BERTurk algorithm delivered the highest performance with a 95% F-score. The above experimental results show that conventional ML algorithms, even when combined with advanced pre-processing techniques such as feature selection, do not demonstrate appropriate results from an automatic FR-NFR identification point of view. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **F-Score** & **AUC** \\ \hline BERT & 0.921 & 0.971 \\ BERTurk & 0.954 & 0.983 \\ DistilBERT & 0.918 & 0.968 \\ RoBERTa & 0.862 & 0.952 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: F-score and AUC value of transfer learning methods. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Algorithm**} & \multicolumn{2}{c}{**F-Score**} & \multicolumn{2}{c}{**AUC**} \\ \cline{2-5}  & **CFS** & **GR** & **CFS** & **GR** \\ \hline NB & 0.787 & 0.856 & 0.895 & 0.897 \\ LMT & 0.842 & 0.816 & 0.91 & 0.962 \\ RF & 0.842 & 0.805 & 0.909 & 0.962 \\ SLR & 0.841 & 0.904 & 0.915 & 0.957 \\ JRip & 0.836 & 0.905 & 0.501 & 0.501 \\ NBM & 0.837 & 0.830 & 0.503 & 0.503 \\ SMO & 0.834 & 0.915 & 0.792 & 0.903 \\ LR & 0.845 & 0.891 & 0.917 & 0.929 \\ Bagging & 0.832 & 0.862 & 0.896 & 0.929 \\ J48 & 0.751 & 0.821 & 0.786 & 0.883 \\ Multiclass Classifier & 0.845 & 0.830 & 0.917 & 0.929 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: F-score and AUC value of conventional algorithms with feature selection methods. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **F-Score** & **AUC** \\ \hline CNN & 0.937 & 0.918 \\ LSTM & 0.914 & 0.893 \\ Bi-LSTM & 0.907 & 0.881 \\ GRU & 0.926 & 0.911 \\ Bi-GRU & 0.915 & 0.901 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: F-score and AUC value of deep learning algorithms. As it is stated above, the limitation of conventional ML algorithms may be remedied through deep learners combined with word embeddings to some extent. Though deep learners on top of word embeddings show a relatively increased performance compared to conventional ML algorithms, they are limited in grasping contextual representations compared to newly developed transformer language models. This is particularly expressed in the literature, and it is also shown in the empirical results given in Tables 3-6. In clearer terms, the F-score performances of nearly all transformers surpass the best performances of all algorithms. Though the transformer models except BerTURK are multilingual, they still show significant performances for the Turkish software requirement domain. From a confusion matrix point of view, the best performances of three algorithms from each group are presented in Figure 7.

As observed in Figure 7, BerTURK is able to identify much more TP and TN values compared to CNN and NBM. Also, BerTURK shows better performance decreasing FP and FN values.