A Simple and Flexible Modeling for Mental Disorder Detection by Learning from Clinical Questionnaires
Year: 2023

Abstract:
Social media is one of the most highly sought resources for analyzing characteristics of the language by its users. In particular, many researchers utilized various linguistic features of mental health problems from social media. However, existing approaches to detecting mental disorders face critical challenges, such as the scarcity of high-quality data or the trade-off between addressing the complexity of models and presenting interpretable results grounded in expert domain knowledge. To address these challenges, we design a simple but flexible model that preserves domain-based interpretability. We propose a novel approach that captures the semantic meanings directly from the text and compares them to symptom-related descriptions. Experimental results demonstrate that our model outperforms relevant baselines on various mental disorder detection tasks. Our detailed analysis shows that the proposed model is effective at leveraging domain knowledge, transferable to other mental disorders, and providing interpretable detection results.

heading: 1 Introduction
text: Mental health problems, a significant challenge in public healthcare, are usually accompanied by distinct symptoms, such as loss of interest or appetite, depressed moods, or excessive anxiety. As these symptoms can often be expressed over social media, detecting mental health conditions using social media text has been studied extensively (Yates et al., 2017; Coppersmith et al., 2018; Matero et al., 2019; Murarka et al., 2021; Harrigian et al., 2021; Jiang et al., 2021; Nguyen et al., 2022). Such approaches could give rise to a monitoring system that provides clinical experts with information about possible mental crises.
To automatically identify mental health problems, traditional approaches focus on finding linguistic patterns and styles from the language of
∗ Corresponding author
psychiatric patients. Utilizing these features, statistical models can explain the correlation between linguistic factors and mental illnesses. However, these approaches suffer from increased complexity of models, necessitating pipelines of steps, from engineering features to producing results. By contrast, more recent works have employed strong pretrained models, which allow a direct use of raw data and simplify model development (Matero et al., 2019; Jiang et al., 2020). While such end-to-end approaches may be effective at achieving higher performance, they often lack domain-based interpretation, which is essential for decision-support systems (Mullenbach et al., 2018). Hence, there is a trade-off between providing interpretable predictions based on domain knowledge and the simplicity of the models.
The lack of a sufficient sample size for highquality data is another challenge in the clinical domain (De Choudhury et al., 2017; Harrigian et al., 2020). Despite the availability of diverse datasets and methods for detecting mental disorders, most of them aim primarily at identifying only clinical depression. To tackle such a problem, recent studies have focused on developing transferable linguistic features that can be used for the detection of various mental disorders (Aich and Parde, 2022; Uban et al., 2022). However, the linguistic features that are trained on a particular dataset may not be fully transferable to a different task (Ernala et al., 2019; Harrigian et al., 2020).
Others utilized symptom-related features that are more common properties of psychiatric patients, resulting in generalizability of depression detection (Nguyen et al., 2022). Despite this improvement, however, their approach still faces challenges because they rely on pipelined methods using manually-defined symptom patterns. Such symptom patterns for depression detection lack flexibility as they cannot be easily adapted to other mental disorders. In addition, the pipeline approach
12190
with symptom extraction is quite complex to implement. It involves multiple steps, designing symptom patterns, training a symptom identification model, and detecting depression using the identified symptom patterns.
To address these challenges, we propose to design a simple and more flexible approach that also preserves interpretability. We are motivated by the process that humans use to quickly learn related features, often by reading just a single explanation. For example, when people are reading depression questionnaires, they readily understand the questions and learn about symptoms that are related to depression, allowing them to self-diagnose their levels of depression.
To this end, we employ the siamese network (Koch et al., 2015), which captures the semantic meaning of the text inputs and compares them directly to symptom-related descriptions. This process is simple since they find symptom-related clues directly from the input, rather than relying on hand-engineered features or intermediate models. Our proposed model, Multi-Head Siamese network (MHS), can be easily adapted to other mental illness domains by simply replacing the symptomrelated descriptions. In addition, our model is designed to capture the distinct features of each symptom using multiple heads. By examining the learned weights of each symptom head, our model gives rise to human-understandable interpretations.
We evaluate the performance of our model, detecting texts containing mental health problems on four mental disorders. Furthermore, the detailed analysis of the proposed model shows its efficiency in utilizing symptom-related knowledge, its ability to be applied to different mental disorders, and its interpretable reasoning for detected results.

heading: 2 Related Work
text: Social media are commonly used for mental health research because of the ease of access to various aspects of human behavior studies. Similarly to other NLP domains, pre-trained language models, such as BERT (Devlin et al., 2019), are widely used for identifying mental health problems (Matero et al., 2019; Jiang et al., 2020; Murarka et al., 2021; Dinu and Moldovan, 2021).
Others have presented interpretable detection methods for the mental health domain based on linguistic features (Song et al., 2018; Uban et al., 2021). Various efforts have also been made to study
such linguistic features accompanying mental illness, such as differences in word usage (Tadesse et al., 2019; Jiang et al., 2020; Dinu and Moldovan, 2021), or in syntactic features (Kayi et al., 2017; Ireland and Iserman, 2018; Yang et al., 2020). Some studies address the differences between sentiments or emotional aspects (Preoţiuc-Pietro et al., 2015; Kirinde Gamaarachchige and Inkpen, 2019; Allen et al., 2019; Wang et al., 2021), or differences in topics (Tadesse et al., 2019; Kulkarni et al., 2021).
The linguistic features are also used for transferable methods across other mental disorders (Aich and Parde, 2022; Uban et al., 2022), focusing on the fact that a large number of studies have been done primarily on depression (De Choudhury et al., 2013; Yates et al., 2017; Eichstaedt et al., 2018; Song et al., 2018; Tadesse et al., 2019; Yang et al., 2020; Nguyen et al., 2022), compared to other disorders, such as anxiety disorder (Ireland and Iserman, 2018), anorexia (Uban et al., 2021), or schizophrenia (Kayi et al., 2017). However, such linguistic features do not generalize well to new user groups. For example, De Choudhury et al. (2017), Loveys et al. (2018), and Pendse et al. (2019) found that the linguistic styles may vary to their backgrounds. In addition, Harrigian et al. (2020) found that a model trained on a particular dataset does not always generalize to others. To handle such a generalization problem, Nguyen et al. (2022) and Zhang et al. (2022) focused on the shared and general properties (i.e., symptoms) of a mental health problem. However, unlike ours, which captures the symptom features directly from raw data, these methods require additional steps for learning symptom-related features.
In this paper, we use the siamese network (Koch et al., 2015), based on one-shot learning, exploited recently for simple networks (Chen and He, 2021; Zhu et al., 2021). We utilize the symptom descriptions sourced from DSM-5 (American Psychiatric Association, 2013) to make our model learn symptom-related knowledge.

heading: 3 Methodology
text: In this section, we introduce our simple but flexible modeling for leveraging clinical questionnaires. Our model aims to detect texts with mental illness episodes based on the presence of symptom-related features just by a single component.
An overview of our network is shown in Figure 1.
We designed our model based on the siamese network (Koch et al., 2015). As with the original siamese neural network, our model also contains a single feature extractor with shared parameters. The extractor directly obtains features from contextualized embeddings generated by sentence encoders. Then, employing the similarity function, we compare the similarity to see the presence of symptom-related features from the target text. In addition, we apply multi-headed learning to the original siamese network, repeating the comparison process for each distinct symptom. We describe the detailed structure in the following subsections.

heading: 3.1 Model Structure
text: Our model, the Multi-Head Siamese network (MHS), is an end-to-end model that takes raw input texts and produces the final result without the need for manual feature engineering. MHS is designed to take two types of inputs, the target text to be classified and descriptions of symptoms. The descriptions are grouped for each symptom, and each symptom group is the input for the corresponding symptom head. For example, assuming that we have n symptoms for discriminating against mental disorder, we build a set of n heads (H) from S1 to Sn for the detection model as follows:
H = {S1, S2, ..., Sn} (1)
Each head S represents discrete symptoms, containing a number of descriptions and questions regarding the corresponding symptom. For example, if Si has m sentences describing the symptom, we have a set Si of questions:
Si = {s(i,1), s(i,2), ..., s(i,m)} (2)
With a given input of the target sentence, our model obtains embedding vectors (Etarget) by employing pre-trained sentence encoders, such as BERT or RoBERTa. We also get symptom embeddings by encoding all sentences from all heads (H).
Our siamese network employs a multi-channel convolutional neural network (CNN) for feature learning. We apply three channels for convolution layers, whose kernel sizes are 2, 3, and 5. Thus, our model is designed to capture informative clues with the window sizes of 2, 3, and 5 from texts. Each channel contains two convolutional layers and two max-pooling layers. The final convolutional layer is flattened into a single embedding vector. As a result, we obtain three feature embedding vectors (Ftarget,k) with k = 2, 3, 5 from the target text:
Ftarget,k = Conv1dk(Etarget) (3)
Through the same process, we also obtain feature embedding vectors from symptom texts from the ith head and jth sentence as follows:
F(i,j),k = Conv1dk(E(i,j)) (4)
We compute the distances (d) between the target feature vector (Ftarget,k) and a symptom-sentence vector (F(i,j),k) using cosine similarity, ranging from [−1, 1]. We calculate a single distance value by taking the average of K distance values, where K represents the number of channels:
sim(x,y) = xy
∥x∥∥y∥ (5)
d(i,j) = 1
K
∑
k
sim(Ftarget,k, F(i,j),k) (6)
Finally, when there are distance values for all sentences, they are averaged to yield the distance value of the ith head (di):
di = 1
m
m∑
j=1
d(i,j) (7)
To regularize the results, we choose to use averaging as an aggregation function for the distance values.
We iterate this process over the number of heads (n). After the siamese network step, all distance values (di) are stacked into a 1 × n vector (D). By applying the fully connected layer, the distance vector is reduced into a two-dimensional vector o, which is an output probability of classifying mental illness:
f : Rn → R2 (8) o = f(D) = W T ·D + b (9)
By analyzing the weights (W ) and distance values (D) of the fully connected layer, we can examine which symptoms are activated as important information when classifying the related mental disorder. Further details are discussed in Section 5.4. The implementation code and symptom-sentences are made publicly available1.

heading: 3.2 Symptom Descriptions
text: In the present study, we focus on four mental disorders: major depressive disorder (MDD), bipolar disorder, generalized anxiety disorder (GAD), and borderline personality disorder (BPD). As summarized in Table 1, we compiled the diagnostic criteria for each mental disorder, sourced from DSM-5. We constructed heads based on the list of symptoms. For example, in the case of MDD, there are a total of 9 symptoms (D0-D8), so when constructing a
1https://github.com/HoyunSong/acl23-multi-headsiamese-mental-illness
model detecting depressive symptoms, there will be a total of 9 heads (n(HMDD) = 9). As for bipolar disorder, symptoms can be divided into depressive episodes (D0-D8) and manic episodes (M0-M7), with a total of 17 heads. The depressive episodes of bipolar disorder are the same as those of MDD.
Each head includes a description of diagnostic criteria and questions from self-tests corresponding to each symptom. As a result, each head contains two or more sentences (n(S) ≥ 2). In the case of more than two related questions for a symptom, the corresponding head contains more than two sentences.
We collected the questions from the publicly available self-tests2. The process was conducted under the guidance of a psychology researcher. The complete list of collected sentences for each head is shown in Appendix C. Our model can easily
2MDD (www.psycom.net/depression-test/), Bipolar (www.psycom.net/bipolar-disorder-symptoms/), GAD (www.psycom.net/anxiety-test), and BPD (www.psycom.net/borderline-personality-test/)
transfer to other mental disorders by just replacing symptom descriptions, as evidenced by the findings in Section 5.3.

heading: 4 Experiments

heading: 4.1 Dataset and Evaluation
text: In order to evaluate our model, we constructed four datasets to detect possible mental disorder episodes. We sampled posts from Reddit3, which is one of the largest online communities. Each sample is a concatenation of a title and a body from a post. Each dataset contains two groups of Reddit posts. One includes the posts collected from mental disorderrelated subreddits as a text containing the mental illness contents, and the other is from random subreddits as a clean text. The detailed statistics of each group is shown in Table 2. We performed preprocessing by discarding posts containing URLs or individually identifiable information, and posts shorter than ten words (i.e., tokens). We only retained posts in English; otherwise, they are discarded.
We conducted four tasks, employing these collected datasets, discriminating texts sourced from mental disorder-related subreddits out of nonmental illness texts. The details of each task are as follows: MDD detection (r/depression+random), Bipolar disorder detection (r/bipolar+random), GAD detection (r/anxiety+random), and BPD detection (r/bpd+random).
To compare our model with baseline models with respect to classification performance, we report results using standard metrics, Accuracy (Acc.), F1 score (F1) for the mental illness group, and Area Under the Curve (AUC). The performance measure is reported by five-fold cross-validation, and each repetition is trained on six different seeds. We averaged after 30 runs (5×6) to get the final result.

heading: 4.2 Baselines and Experimental Setup
text: In this subsection, we describe models and implementation details for experiments. More experi-
3https://files.pushshift.io/reddit/
mental details are shown in Appendix A. 1) Traditional Models We implemented two feature-based classifiers, a support vector machine (SVM) and a random forest (RF), with two versions: BoW, employing lexical features only (Tadesse et al., 2019; Jiang et al., 2020), and Feature, adding sentimental and syntactic features (Allen et al., 2019; Yang et al., 2020; Wang et al., 2021). 2) BERT (Devlin et al., 2019) is one of the most well-known baseline models using contextualized embeddings (Jiang et al., 2020; Matero et al., 2019). 3) XLNet (Yang et al., 2019) is another strong baseline with a pre-trained language model (Dinu and Moldovan, 2021). 4) RoBERTa (Liu et al., 2019) is a robustly optimized BERT and one of the most solid baselines in natural language classification (Dinu and Moldovan, 2021; Murarka et al., 2021). 5) GPT-2 (Radford et al., 2019) is a strong few-shot learner with a large Transformerbased language model. 6) PHQ9 (Nguyen et al., 2022) is a depression detection model constrained by the presence of PHQ9 symptoms.
We implemented our models using PyTorch and fine-tuned our models on one 24GB Nvidia RTX3090 GPU, taking about 13 minutes for each epoch. The batch size and embedding size of all models are 8 and 512, respectively, and are fine-tuned over five epochs. We truncated each post at 512 tokens for all models. For each model, we manually finetuned the learning rates, choosing one out of {1e-5, 2e-5, 1e-6, 2e-6} that shows the best F1 score. We report the average results over 30 runs (five-fold cross-validations are trained on six different seeds) for the same pre-trained checkpoint.

heading: 4.3 Experimental Results
text: Table 3 shows the overall performance of our proposed model (MHS) and strong baselines on four tasks. Each task is about detecting texts with corresponding mental illness episodes on social media. We see that our model outperforms all competing approaches, including linguistic feature-based models, end-to-end pre-trained models, and a method that uses symptom-related knowledge.
Linguistic feature-based models exhibit significant performance variations based on the level of detail in their feature design. By contrast, MHS can simply find the features directly from the contextualized representation, giving better performance improvements. Pre-trained models with contextualized embeddings have the benefits that can be
easily fine-tuned for a wide range of tasks. However, compared to MHS, they lack a specific focus on domain-based features, while MHS is tailored to identify such features, leading to better performance.
We implemented our model and PHQ9 model with two different encoders, BERT and RoBERTa, and the tendency for performance improvement is the same on both encoders. Both PHQ9 and MHS leverage symptom-related information but differ in their architecture, specifically whether it is a multistep pipeline or an end-to-end model. The end-toend design of MHS allows for direct learning of complex relationships, reducing the potential for error propagation, and resulting in enhanced performance compared to the pipeline model. Moreover, for this pipeline model to apply to other mental disorders, a symptom pattern must be created for each mental disorder, which is challenging to achieve without expert-level knowledge. On the other hand, our proposed model overcomes these challenges by simply replacing symptom descriptions. A detailed analysis of the performance improvement is shown in Section 5.

heading: 4.4 Model Parameters
text: Table 4 shows the number of parameters for each model. Compared to the baseline models, the additional number of parameters for our siamese net-
work is about 655K. It is a much smaller number than that of the additional parameters for RoBERTa and BERT (about 16M), but the performance of MHS (w/bert) is slightly better or shows little difference. It suggests that our proposed model, learning domain knowledge, achieves much efficient performance improvement by adding just a small number of parameters.

heading: 5 Model Analysis and Discussions

heading: 5.1 Ablation Study
text: We conducted an ablation study to investigate the effectiveness of each part in our proposed model. We removed the siamese network from our proposed methods, resulting in just convolutional neural networks (CNNs). We implemented a singlehead siamese network in which all sentences from all heads are put together into just one head. We also implemented two versions of a multi-head siamese network employing just one description or multiple descriptions, respectively.
The experimental results are shown in Table 5. The result shows that our proposed model gives the best performance when all of the modules are combined. Compared to CNN models, the performances are improved when the siamese network is added. Note that the siamese network contributes to accurate detection, since it captures the
symptom-related features by comparing target texts with symptom descriptions. In addition, the performances are also improved when employing a multi-head rather than a single-head. It implies that individually training each symptom yields better results than training all symptoms together, as each symptom has unique features. Compared to learning from only one description per head, the performance of learning from multiple descriptions is improved. It may be due to each head learning further about the symptom through various sentences, covering distinct aspects of each symptom.

heading: 5.2 Contribution of Symptom Descriptions
text: To assess the effectiveness of symptom descriptions in detecting the presence of symptoms, we measure their performance by replacing the descriptions of symptoms with those of other mental disorders. The results are shown in Table 6. We carried out four mental disorder detection tasks using four models, each utilizing symptom descriptions of four distinct mental disorders as inputs.
The models exhibit optimal performance when the input symptom description corresponds to the target mental disorder. It suggests that, by providing the model with accurate and appropriate symptom descriptions, MHS can learn effectively to identify the specific features associated with a particular mental disorder. This also implies that MHS can identify and utilize the nuanced distinctions in the characteristics of each symptom, leading to enhanced performance in detection.

heading: 5.3 Cross-domain Test
text: In order to investigate the flexibility of MHS, we evaluated its performance across datasets and other mental disorders. Dataset Transferability Given that the ability to generalize to new and unseen data platforms is a crucial aspect of mental illness detection models (Harrigian et al., 2020), we evaluate their
performance across different datasets. We selected two datasets, RSDD (Yates et al., 2017) and eRisk2018 (Losada et al., 2019), to evaluate crossdataset transfer. Unlike our Reddit dataset (Subsection 4.1), sourced from communities specific to certain mental illnesses, RSDD and eRisk2018 data are based on user self-reports, resulting in data that is different from and potentially unseen by the Reddit dataset. We trained each model using the Reddit train dataset and evaluated its performance on the test sets of RSDD and eRisk2018, respectively.
As shown in Table 7, MHS outperforms all strong baselines over all datasets. The improved performance of MHS compared to GPT-2, a strong few-shot learner, is likely due to its ability to leverage domain-specific knowledge. The higher generalizability of MHS compared to PHQ9 is likely attributed to its end-to-end architecture, which allows for direct learning of symptom features from data, as opposed to PHQ9’s reliance on pre-defined symptom patterns. Domain Transferability As suggested by some researchers (Aich and Parde, 2022; Uban et al., 2022), we evaluated the transferability of MHS across other mental disorders by training the models on a depression dataset and testing on other mental disorder datasets (see Table 8). The results of the experiments indicate that MHS significantly outperforms all relevant baselines, particularly when it utilizes symptoms that match the target mental disorder. This suggests that the transferability of
the model can be significantly enhanced by simply replacing symptom descriptions. This also implies that it may be feasible to develop a model that can classify texts related to various other mental disorders if the symptoms of those disorders are provided appropriately.

heading: 5.4 Interpretation
text: Using our model, we can interpret the detected results by analyzing their representations of learned weights and distance values. In order to see if our model properly learned symptom-related knowledge from a few descriptions and identified similar stories from the target texts, we looked into the learned weights produced by the last step of our model, the fully connected layer. To show the effectiveness of MHS, we visualize the examples of learned weights from training steps in Figure 2. The color scale represents the strength of the learned weights (i.e., the distance values of each head). Each row represents heads, indicating each symptom referring to Table 1, and each column represents the labels. We observe a clearly contrasting pattern in the distance weights for each task.
We could also identify which symptoms are mainly activated or not by investigating the learned weights during the training process. For example, in detecting MDD-related texts, most of the symptoms have higher weights than depression. It suggests that most of the symptoms give rise to a major role during the detection process.
An important criterion in diagnosing a mental illness by experts is the number of expressed symptoms. The number of symptoms must exceed a certain number to be diagnosed as a corresponding mental illness. In order to see if the human-level diagnostic process works in our model as well, we looked into the number of salient symptoms in true-positive samples. We calculated percentiles from the similarity scores for each symptom in the true-positive samples from test sets, and set the threshold by 70% of the percentile. Then, when
exceeding the threshold set by the criterion, the symptom was selected as a prominent feature in the text. We present the distribution of the numbers of salient symptoms and their averaged probabilities of the final output from test sets of detecting MDD-related texts in Figure 3.
In our model, the average probability is relatively low when there are fewer than three symptoms, but for three symptoms or more, our model makes a decision with high confidence at a similar level. It suggests that MHS also detects mental disorderrelated texts with high confidence when the number of symptoms exceeds a specific number, the same as when humans diagnose. The criterion number being smaller in MHS may be due to the shorter length of social media texts, which may not fully convey the user’s background and lifestyle.

heading: 5.5 Case Study
text: For the case study, we made an example based on the samples corresponding to each mental disorder in the psychology major textbook. We present example sentences for MDD and GAD (Table 9), and the model’s predictions were correct in both cases. We set the same threshold as shown in Figure 3. The dominant symptoms predicted by the model are D0 (depressed mood), D1 (diminished interest), and D8 (suicidal ideation), for MDD, and A1 (difficult to control the worry), A2 (irritability), and A3 (easily fatigued), for GAD. In the case of D0
and D1 in MDD, our model captures the feature related to the symptom, despite the absence of the term ‘depress’ or ‘interest’. These cases support the assumption that our model can detect and interpret when symptoms of a particular mental illness are prominent in text.

heading: 6 Conclusion
text: In this paper, we proposed a simple but flexible model for detecting texts containing contents of mental health problems. Our model outperformed the state-of-the-art models and achieved humaninterpretable results over symptoms regarding mental disorders. The proposed model demonstrates an exceptional ability to utilize domain knowledge as it is designed to capture relevant features from texts directly. Experimental results also indicate that MHS can quickly adapt to other mental disorder domains by simply replacing symptom descriptions. The scope of this paper was limited to the investigation of four mental disorder detection tasks. Nevertheless, this approach can be extended to other mental health conditions as long as the symptom-relevant questionnaires are provided accordingly.
Limitations
It should be noted that, as our model and the baseline models in this study were trained using texts from social media and the experiments were conducted on online text, the results may not accurately reflect the performance in a clinical setting. A proper diagnosis by clinical experts necessitates a comprehensive analysis of various factors, including the number of manifested symptoms, the onset and history of symptoms, developmental background, lifestyle, and recent life changes, in order to gain a comprehensive understanding of the patient’s condition. However, it is still challenging to capture detailed information such as personal secrets through online text, as these texts are often composed of fragments of daily life, episodic experiences, and emotive expressions rather than
providing a comprehensive view of an individual’s life. Despite the domain-specific limitations imposed by the fragmentary text, we hope that our model may still serve as a valuable aid for clinical experts in their decision-making process. Furthermore, future research should aim to move beyond predicting psychological symptoms and disorders solely based on linguistic styles and expressions, and instead seek to uncover the underlying features that contribute to these expressions as our model does.
Ethics Statement
Since privacy concerns and the risk to the individuals should always be considered, especially using social media data, we have employed mechanisms to avoid any harmful and negative consequences of releasing our model. To this end, we removed individually identifiable information such as user names, user IDs, or e-mail addresses. We also removed any URLs from our data not to be trained on such personal information in our model. As for the use of open datasets in this work, we used them in accordance with guidelines that allow their use within the established usage policy. Especially we ensure that no attempts can be made to establish contact with specific individuals or deanonymize users in the datasets.
Our paper may contain direct references to specific disorders or diseases (such as psychiatric patients, Siamese, or names of mental disorders) and expressions that could be considered offensive to particular individuals. We want to emphasize that these expressions are used solely for the purpose of academic discourse and are not intended to be disrespectful or offend anyone.
In addition, our proposed model is not intended to label or stigmatize individuals online but rather to serve as a warning system for potential threats to personal well-being and public health. It is important to note that even if this model identifies potential mental illnesses and symptoms, it should not be considered a definitive diagnosis. Still, the
model provides an indication of the likelihood of a disorder; it should be used as a reference for selfdiagnose and in consultation with a mental health expert for an official diagnosis. An official diagnosis and results require consultation with medical and psychological experts, and this system aims at serving as an aid in the diagnostic process. We make our implementation code publicly available for research purposes, and we hope it will be used to improve the lives of individuals suffering from mental illnesses.

heading: Acknowledgements
text: This work was supported by the National Research Foundation of Korea (NRF) (No. RS-202300208054, A multi-modal abusive language detection system and automatic feedback with correction) grant funded by the Korean government.

heading: A Experimental Setups
text: We implemented two feature-based models, support vector machine (SVM) and random forest (RF). We fine-tuned SVM with Gaussian kernel and set C to 100, and RF set max depth to 100. We employed BERT’s vocabulary to train BoW models. For Feature models, we used a pre-trained sentiment classification model, and a Part-of-Speech Tagging model from the Huggingface library (Wolf et al., 2019). We fine-tuned the transformer baseline models employing the default settings from the Huggingface library: BERT (bert-base-cased), XLNet (xlnet-base-cased), RoBERTa (robertabase), GPT-2 (gpt2). For all experiments, we set the batch size as 8 and fine-tuned all models on a single 24GB GeForce RTX 3090 GPU. For the implementation of the PHQ9 model, we follow the structure of the questionnaire-depression pair models by using the publicly available code from PHQ94 (Nguyen et al., 2022). We utilized the symptom patterns which are provided by Nguyen et al. (2022). We trained each of the models using all six randomly selected seeds, and all the models were trained for 3 epochs. We optimize the model parameters of all models with the Adam optimizer (Kingma and Ba, 2014). The learning rates for BERT, XLNet, and RoBERTa models were manually fine-tuned, choosing one out of {1e-05, 2e-05, 1e-06, 2e-06} that shows the best F1 score. The learning rate for GPT-2 was selected from {1e-05, 2e-05}, and for PHQ9, the learning rate was set to 1e-03, which was provided as an optimized hyperparameter.

heading: B Comparison with Large Language Model
text: Recent developments in large language models (LLMs), such as GPT-3 (Brown et al., 2020), have demonstrated strong zero-shot performance across various NLP tasks. LLMs have the ability to achieve high performance without fine-tuning for downstream tasks, even with only zero or few examples, due to their large number of pre-trained parameters.
We experimented with obtaining results for the examples referred to in Table 9 by using GPT-3, a widely recognized LLM. To this end, we utilized instructional prompts by listing symptom descriptions for a specific mental illness. The examples
4https://github.com/thongnt99/acl22-depression-phq9
of prompt input and the result are shown in Table 10. The experimental results show that the model successfully outputs the classification results in a sentence when given instructional prompts for a specific mental illness. However, the process of selecting symptoms appears to focus on identifying multiple symptoms rather than pinpointing a specific symptom with precision.
These examples are presented for demonstration purposes only, and the results may vary depending on the utilization of different prompt optimizations (Liu et al., 2021; Qin and Eisner, 2021). This aspect of research is beyond the scope of our current study; thus, there is room for further research to be conducted in future work.

heading: C Details of Symptom Descriptions
text: In this section, we present the symptom descriptions that were utilized in our current study. Table 11 shows the complete list of symptom descriptions. We used Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Association, 2013) as a reference for the symptom descriptions, as it provides comprehensive guidelines for identifying symptoms of various mental disorders. We also incorporated publicly available clinical questionnaires from online sources. Subsequently, under the guidance of a psychology researcher, we conducted a mapping process of the questions in the self-test to the corresponding diagnostic criteria, as depicted in Figure 4.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, in the "Limitation" section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Yes, in the "Ethics statement" section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, the paper’s main claims are provided in the 1. Introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, in 3. Methodology.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, in 2. Related work
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, the codes will be publicly available after the reviewing process.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, we discuss about the possible problems in the "Ethics statement" section.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Yes, it is also discussed in "Ethics statement"

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Yes, in section 4.1 datasets
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in section 4.1 datasets
C 3 Did you run computational experiments? Yes, in Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes, in section 4, and Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, in section 4, and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, in section 4 and 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, in section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This aspect of research is beyond the scope of our current study; thus, there is room for further research to be conducted in future work