# Detecting Suicidal Ideations in Online Forums with Textual and Psycholinguistic Features
## 4 Results
### Classification Performance
#### 4.1.3 Experiment 3 (Combined Features)  
The aim of this experiment was to examine whether combining textual and psycholinguistic features results in improved performance. Various types of Boruta-selected features were combined in this experiment. Figure 4 shows the performance scores of models trained using different feature combinations. Each group bar plot in the figure represents the performance of a specific algorithm, with feature combinations denoted using different colors.  
Upon closer examination of the accuracy metrics, it was observed that all the algorithms attained their lowest accuracy when trained with a combination comprising solely psycholinguistic LIWC and NRC features. Notably, NB trained with LIWC+NRC recorded the lowest accuracy at 68.41%.  
Conversely, the models that utilized some combination of textual and psycholinguistic features achieved the highest accuracy. For instance, RF achieved the highest accuracy of 85.35% when trained with TF-IDF+LIWC and TF-IDF+LIWC+NRC combinations. Additionally, NB and SVM recorded the highest accuracy of 81.78% and 84.46%, respectively, when the TF-IDF+NRC feature set was used for training.  
An analysis of the precision metrics revealed a similar trend. Models trained with LIWC+NRC feature combinations tended to exhibit lower precision scores. Specifically, NB achieved 44.77% precision when the LIWC+NRC combination was used as the input features. However, combining textual and psycholinguistic features led to higher precision. Particularly, NB, RF, and SVM attained the highest precision scores of 60.92%, 78.12%, and 71.84%, respectively, with TF-IDF+NRC, while LR achieved 73.98% with TF-IDF+LIWC.  
In terms of recall, better performance was achieved with a combination of textual and psycholinguistic features. For example, LR and SVM achieved the highest recall (66.83% and 67.32%, respectively) when the TF-IDF+NRC feature combination was used for training. In addition, RF attained its highest recall of 64.39% with TF-IDF+LIWC, and NB recorded an 89.27% recall when all three types of features were combined.  
In contrast, combining LIWC and NRC resulted in the lowest recall for LR and SVM (57.56% and 56.10%, respectively). It is noteworthy that NB tends to exhibit higher recall and lower precision than the other algorithms.  
Further inspection of F1 scores also indicates better performance when the feature set includes both textual and psycholinguistic features. Specifically, LR and RF achieved the highest F1 scores of 69.84% and 69.54%, respectively, when combining TF-IDF and LIWC features. NB and SVM performed best when combining TF-IDF and NRC, achieving F1 scores of 70.99% and 69.39%, respectively. On the contrary, when NB was trained on LIWC+NRC features (both psycholinguistic), it recorded the lowest F1 score of 59.35%.  
From Figure 5, it becomes evident that combining textual and psycholinguistic features tends to result in improved classification performance compared to using these features separately. For instance, RF recorded a 65.80% F1 score with only TF-IDF features and 61.76% with only LIWC features. However, combining these features led to a higher F1 score of 69.54% for RF.  
These results highlight the significance of feature combination, especially when utilizing both psycholinguistic and textual features, as it leads to improved classification performance.  
Figure 4: Performance scores of models trained with different combinations of selected features: (a) logistic regression; (b) naive Bayes; (c) random forest; (d) support vector machines.