## 4 Evaluation
### Results and Analysis  
Table 4 shows how DEENT, MentalBERT, RNN, CNN, SVM, XGBoost, and RF behaved regarding recall with balanced and imbalanced training data. This metric is relevant for this investigation since it shows whether the model correctly detects depressive tweets (i.e., the minority class); misclassifying depressive tweets as non-depressive could have negative consequences, such as suicide. We used SMOTE [40] (a common technique used on binary text classification tasks) to balance the training data for DEENT-Generic, RNN, CNN, XGBoost, SVM, and RF. Balancing for DEENT-Bert and MentalBERT was carried out by the Weighted Loss Function; it is necessary to remember that this function aids the model in paying more attention to the minority class, and further BERT-based models created with Torch are incompatible with SMOTE. The recall achieved by DEENT-Bert (77.77%), DEENT-Generic (76.31%), MentalBERT (77.54%), RNN (76.83%), CNN (77.19%), XGBoost (69.70%), SVM (74.66%), and RF (70.78%) without balancing is lower than that obtained with balancing, i.e., 80.54%, 80.72%, 79.98%, 80.38%, 77.92%, 71.81%, 78.16%, and 75.82%. In this regard, these results revealed that balancing training data is necessary for learning how to more accurately detect depressive tweets despite the synthetic data introducing noise, semantic loss, and moving away from the manifold. Figure 5 shows the accuracy and balanced accuracy obtained by DEENT and the baseline models in the test imbalanced depression-oriented dataset (i.e., 14,102 non-depressive tweets and 10,695 depressive tweets). Results reveal that DEENT-Bert (84.54% and 84.06%) had better accuracy and balanced accuracy than DEENT-Generic (81.23% and 81.17%), MentalBERT (83.80% and 83.34%), RF (80.17% and 79.68%), SVM (80.63% and 80.33%), XGBoost (80.31% and 79.28%), RNN (80.91% and 80.90%), and CNN (80.72% and 80.39%). These results revealed, first, that the attention mechanism used by DEENT helped it to learn more complex relationships between words associated with depression than non-transformer-based models. Second, thanks to the fine-tuning, DEENT-Bert outperformed \begin{table}
\begin{tabular}{l l c} \hline \hline
**Algorithm** & **Balancing Method** & **Recall** \\ \hline \multirow{2}{*}{DEENT-Bert} & None & 77.77\% \\  & Weighted Loss Function & 80.54\% \\ \hline \multirow{2}{*}{DEENT-Generic} & None & 76.31\% \\  & SMOTE & 80.72\% \\ \hline \multirow{2}{*}{MentalBERT} & None & 77.54\% \\  & Weighted Loss Function & 79.98\% \\ \hline \multirow{2}{*}{RNN} & None & 76.83\% \\  & SMOTE & 80.38\% \\ \hline \multirow{2}{*}{CNN} & None & 77.19\% \\  & SMOTE & 77.92\% \\ \hline \multirow{2}{*}{SVM} & None & 74.66\% \\  & SMOTE & 78.16\% \\ \hline \multirow{2}{*}{XGBoost} & None & 69.70\% \\  & SMOTE & 71.81\% \\ \hline \multirow{2}{*}{RF} & None & 70.78\% \\  & SMOTE & 75.82\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Recall on balanced and imbalanced training data. DEENT-Generic and Mental-BERT. Third, DEENT-Generic and DEENT-Bert are robust models when processing unbalanced data since the difference between accuracy and balanced accuracy is negligible; DEENT-Generic obtained a variation around 0.06% and DEENT-Bert obtained a variation near 0.48%. On the other hand, it is remarkable that the accuracy (83.80%) obtained by MentalBERT in our large dataset is higher than the one (76.70%) in the depression dataset of Reddit messages collected before the COVID-19 pandemic. Note also that the robustness of MentalBERT regarding balanced accuracy was not measured in the original work. Figure 6 shows the precision, recall, and f1-score obtained by DEENT and baseline models in the test imbalanced depression-oriented dataset. DEENT-Generic (80.72%) and DEENT-Bert (80.54%) achieved a higher recall than RF (75.82%), SVM (78.16%), XGBoost (71.81%), RNN (80.38%), CNN (77.92%), and MentalBERT(79.98%), meaning DEENT more accurately detected depressive tweets compared to the other models.