# Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries
## 4 Results  
Given the results presented in Table 4, both detection methods are effective instruments for discerning human-authored from AI-generated texts across different domains. The F1 scores are generally good, showing minor differences between the two methods in terms of performance. However, the number of misclassified texts, specifically for the human category, varies across domains, behavior that is motivated by class imbalance. The random chance for predicting each class, given the class imbalance of 1:7 (machine-generated to human-generated texts), is 12.25%. Table 4 presents the cumulative results, based on which the Transformer-based method outperformed the classic ML-based method on two domains out of five, being better in distinguishing AI texts from human texts for legal and news domains, while the linguistic features extracted in the classic ML-based approach contributed to better results for books, medical, and scientific (RoCHI) domains. Transformer-based model. The multiclass classifier performs well across different categories with high precision (P), recall (R), and F1-scores as highlighted by confusion matrices (see Figure 2 for the books domain, while the matrices for other domains can be consulted in Appendix A, Figure A1). While evaluating the model's performance, it is important to consider the dataset split, the sampling technique, and the class imbalance. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & & \multicolumn{2}{c}{**Transformer-Based**} & \multicolumn{3}{c}{**Classic ML-Based**} \\
**Domain** & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline Books & 0.90 & 0.89 & 0.89 & **0.92** & **0.92** & **0.90** \\ Legal & **0.91** & **0.90** & **0.90** & 0.85 & 0.85 & 0.85 \\ Medical & 0.92 & 0.92 & 0.91 & **0.96** & **0.96** & **0.96** \\ News & **0.96** & **0.96** & **0.96** & 0.88 & 0.88 & 0.88 \\ RoCHI & 0.88 & 0.85 & 0.85 & **0.92** & **0.93** & **0.91** \\ \hline Micro & & & 0.60 & & & 0.59 \\ Macro & & & 0.90 & & & 0.91 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Detection results (bold marks the best scores per domain). \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Domain** & **learning\_rate** & **max\_depth** & **n\_estimators** & **reg\_alpha** & **reg\_lambda** & **scale\_pos\_weight** \\ \hline Books & 0.1 & 5 & 100 & 0.001 & 0.01 & 1 \\ Legal & 0.1 & 3 & 100 & 0.001 & 0.1 & 3 \\ Medical & 0.1 & 4 & 50 & 0.1 & 0.1 & 3 \\ News & 0.1 & 5 & 200 & 0.001 & 0.1 & 3 \\ RoCHI & 0.1 & 3 & 200 & 0.1 & 0.1 & 5 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Grid Search best parameters for ML-model. The classification is less performant on the subset with fewer samples (i.e., RoCHI), causing more confusion and an F1-score of 0.85, while the classification performs best for the subset with the highest number of samples (i.e., news), achieving an F1-score of 0.96. Classic ML-based model. The human texts provide less accurate results because the number of human samples in the dataset is seven times less than the total number of AI texts. Analyzing the results by domains, the classic ML-based algorithm achieves the highest F1 score for the medical domain (0.96), followed by the RoCHI domain, which exhibits an F1 of 0.92. The precision for AI texts is particularly high (0.93), with only a few misclassified AI texts (9 out of 144). Books domain follows with an F1 of 0.90. It correctly identifies both human and machine-generated texts with high accuracy (0.92). The legal domain exhibits reasonable performance as well, with an F1 of 0.85. We observe the same pattern regarding human misclassified texts compared to AI texts motivated by class imbalance. The news domain achieves an F1 score of 0.87. Furthermore, we introduce SHAP (Shapley Additive exPlanations) [62] to examine the contribution of individual features (RBI indices and burstiness) to the model's predictions. SHAP provides insights into why a particular prediction was made by attributing a portion of the prediction to each feature.