## 4 Experimentation
### Results
To delve deeper into the effectiveness and influence of the proposed models, we performed a Friedman statistical test [70]. This test was employed to determine whether there are significant differences between classification methods based on the sample results. The Friedman test ranks the methods according to their performance on different datasets. A lower ranking indicates that a specific method outperforms the others, demonstrating its superior effectiveness in the classification task. The Friedman test evaluates the average ranks of the methods, denoted as follows:  
\[R_{j}=\frac{1}{n}\sum_{i}r_{i}^{j} \tag{4}\]  
where \(r_{i}^{j}\) represents the rank of the algorithm \(j\)th in the dataset \(i\)th, \(k\) is the number of methods, and \(n\) is the number of datasets. It tests the null hypothesis, which assumes that \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **DsD** & **DepSign** \\ \hline BERT & 0.7100 & 0.5990 \\ MentalBERT & 0.7167 & 0.6094 \\ \(M_{LF}\) & 0.7194 & 0.6117 \\ \(M_{DR}\) & 0.7238 & 0.6325 \\ \(M_{E}\) & **0.7438** & **0.6394** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Best performances (F1-score) achieved with the \(M_{E}\) model and comparisons of these results with those obtained using transformer-based models and the rest of the models proposed in this article. Best results are indicated by bold numbers. given all algorithms are equal, they have equal ranks. The Friedman statistic under this null hypothesis, with \(k-1\) degrees of freedom, is given by  
\[X_{\rm F}^{2}=\frac{12n}{k(k+1)}(\sum_{j}R_{j}^{2}-\frac{k(k+1)^{2}}{4}) \tag{5}\]  
Additionally, [PERSON] and [PERSON] [72] proposed a more robust statistic based on the F-distribution. This statistic, denoted as \(F_{F}\), is calculated with \(k-1\) and \((k-1)(n-1)\) degrees of freedom:  
\[F_{\rm F}=\frac{(n-1)X^{2}F}{n(k-1)-X^{2}F} \tag{6}\]  
The [PERSON] statistical test was performed with specific parameters: an \(\alpha\) value of 0.1, \(k=7\) (representing the number of methods analyzed) and \(n=2\) (indicating the number of datasets). The computed values resulted in \(X_{\rm F}^{2}=11.57\) and \(F_{\rm F}=27.00\), while the critical value \(F(k-1,(k-1)(n-1))=8.47\). Given that \(F_{\rm F}>F(6,6)\), the null hypothesis of the [PERSON] test was rejected, indicating the statistical significance of the results. To simplify the presentation of the best-performing methods based on their ranks from the [PERSON] test, Table 5 displays the best five approaches. The results of [PERSON]'s test support the effectiveness of combining lexicon-based features with distributional representations as the most successful approach (**RQ3**). Additionally, the results show that the proposed method outperforms more complex transformer-based models, often requiring a higher computational overhead. After identifying the best-performing model, we conducted additional experiments to gain deeper insights into the performance of the approach. Specifically, our goal was to understand how the class imbalance within the dataset influences the effectiveness of our system. To achieve this, we employed random undersampling techniques [73] on both datasets, creating a more balanced representation of classes by randomly removing instances from the majority classes. This strategic sampling approach helps address the issue of imbalanced data distribution, ensuring that the model does not become overly biased toward predicting the majority class. The confusion matrices obtained using the \(M_{E}\) model for each dataset, both with and without undersampling, are presented in Figure 5. On close examination of these matrices, some notable insights emerge. When we do not employ undersampling, the class imbalance significantly affects the model's predictions. It tends to be more inclined to predict the majority class, which results in higher accuracy for that class but considerably lower accuracy for minority classes. However, when random undersampling is applied, we observe a shift in the behavior of the model.