Detection of Multiple Mental Disorders from Social Media with Two-Stream Psychiatric Experts
Authors: Siyuan Chen
Zhiling Zhang
Mengyue Wu
Kenny Q. Zhu
Venue: EMNLP 2023 Main (2023)
Keywords: Mental disease detection
symptom
multi-task learning
interpretability
social media

Abstract: Existing Mental Disease Detection (MDD) research largely studies the detection of a single disorder, overlooking the fact that mental diseases might occur in tandem. Many approaches are not backed by domain knowledge (e.g., psychiatric symptoms) and thus fail to produce interpretable results. 
To tackle these issues, we propose an MDD framework that is capable of learning the shared clues of all diseases, while also capturing the specificity of each single disease. The two-stream architecture which simultaneously processes text and symptom features can combine 
the strength of both modalities and offer knowledge-based explainability. Experiments on the detection of 7 diseases show that our model can boost detection performance by more than 10\%, especially in relatively rare classes.

Detection of Multiple Mental Disorders from Social Media with Two-Stream Psychiatric Experts
Siyuan Chen'; Zhiling Zhang*, Mengyue Wu Kenny Q. Zhu 1,3 X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, AI Institute Shanghai Jiao Tong University, Shanghai, China *University of Texas at Arlington, Arlington, Texas, USA
(' ('chensiyuan925, *blmoistawinde 'mengyuewuj@sjtu..o edu.
*kenny wwwuuta. edu
Abstract
user's posting history, which can include many ir- relevant or distracting posts. To avoid these noises, some prior works try to extract key posts by clus- tering (Zogan et al., 2021) or semantic similarity (Zhang et al., 2022a), but these heuristics can still introduce erroneous posts, affecting the subsequent MDD results.
Existing Mental Disease Detection (MDD) re- search largely studies the detection of single disorder, overlooking the fact that mental dis- eases might occur in tandem. Many approaches are not backed by domain knowledge (e.g.. psy- chiatric symptoms) and thus fail to produce interpretable results. To tackle these issues, we propose an MDD framework that is capable of learning the shared clues of all diseases, while also capturing the specificity of each single dis- ease. The two-stream architecture which simul- taneously processes text and symptom features can combine the strength of both modalities and offer knowledge- -base explainability. Ex- periments on the detection of diseases show that model can boost detection performance by more than 10%, especially in relatively rare classes.
Moreover, comorbidity of several mental disor- ders is common (Roca et al., 2009). For instance, 75% of depression patients in the surveyed popu- lation also suffer from anxiety disorder in their lifetime (Lamers al., 2011). Some research (Adam, 2013) further suggests that mental diseases lie along a spectrum, hence it is quite usual for one person to develop symptoms of several related mental disorders at the same time, and be diag- nosed with multiple diseases. However, detecting multiple mental discases simultancously in the see- nario of comorbidity is still under-explored. Most existing works focus the detection of single common disorder, such depression (Losada al., 2017: Lee al.. 2021), ignoring the frequent co- morbidity diagnosed in clinical practice.
1 Introduction
Mental Disease? Detection (MDD) is of great prac- tical value and social benefits, since mental disor- ders can greatly affect sufferers' life quality (Dreis- bach al., 2019). Lots of practices (Coppersmith et al., 2015; Mowery et al., 2017) indicate that so- cial media posts, containing sufficient expressions about one's ' feelings and symptoms, can be an infor- mative data source for text- automatic MDD, which aims to predict whether a person suffers from certain mental diseases.
To address these limitations, aim to explore an approach to detect multiple mental disorders simultaneously in a comorbidity dataset, which the diagnosed users can have one or more disorders. For simplicity, we refer to this detection task as Multiple MDD in this paper. and it can be viewed as a multi-label classification problem.
However, traditional MDD methods (Yates et al., 2017; Trotzek et al., 2020) process every post the
Intuitively, detecting multiple mental diseases can be challenging to resolve, as there are lots of overlapping clinical manifestations shared among different diseases, so the labels implicitly inter- sect with each other. Pioneering works on multi- ple MDD (Cohan et al., 2018; Sckulic and Strube, 2019) commonly yield unsatisfying results as they simply use shared model architecture for all dis- eases, which may not be strong enough to dis- tinguish between discases. Zhang et al. (2022b) shows the effectiveness of implementing psychi-
"These authors made equal contribution.
*Corresponding authors. This work has been supported by National Natural Science Foundation of China (No.61900166) Shanghai Municipal Science and Technology Major Project (202 (SHZDZZ0000002) Key Research and Development Pro- gram of Jiangsu Province (N..BE202205-22 and Alibaba Innovative Research.
"Code available https://githh: http://ithuub.coccceiiissss com/chesiy/ EMNLP23-PsyEx. Dataset can be provided upon request.
*In this work, we will use "mental disorder" and 'mental discase' interchangeably.

---

User's Posting History
Symptom Identification
Two-Stream Psychiatric Experts Model
Post I: Wake early from nightmares again. Terrible!!!
Sleep disturbance
ADHD
Risky Post Stream
Post Suicide Right the Mentally ...... I'm sure would uppreciate it.
Suicidal Ideas
Auxicty
Symptom
Streum
Bipolar
Post 104: You need to hear this song. it now #1 on the ARIA charts.
Symptom
Figure Psychiatric Experts Model with Symptom- based Risky Post Screening. Only posts highly related psychiatric symptoms will be selected as diagnostic basis for further mental disease detection, which forms the risky post stream (the orange part). The symptom stream of user contains his/her whole posting history (the blue part), providing
symptom identification results from global view for the MDD model.
atric symptom knowledge on multiple MDD, which is more interpretable and can outperform previous pure-text methods due to its better clinical ground- ing. However it detects each disease separately, ignoring the inner correlations between diseases, leading unsatisfying results on rarer diseases like OCD.
tween diseases, while each expert can focus on the characteristics of each specific disease.
Experiments show that our method can achieve SOTA multiple MDD performance across 7 dis- cases and bring significant improvement to rarer diseases on which the baselines struggled heavily. Our main contributions are:
In this work, we propose PsyEx (Psychiatric Experts), a multi-task learning framework that can simultaneously detect 7 mental disorders with a shared backbone and disease-specific structures to leverage the common characteristics of all discases while still being able capture their distinctions. This framework consists of two phases (see Figure I). First, we utilize a symptom identification model that explicitly leverages psychiatric knowledge to obtain symptom features for the symptom stream. The symptom features are also used to select posts that show high symptom risks. Then another risky post stream implicitly learns clues for mental dis- cases from the semantics of the selected posts". Therefore, these streams complement each other and improve interpretability based on domain knowledge. They are fed into the multiple MDD model which has 7 diseasc-specifie psychiatric ex- perts on top of a shared hierarchy network, so that the backbone can learn the shared knowledge be-
We successfully exploit psychiatric symptoms in this multiple MDD framework, including the symptom-based screening to facilitate a precise selection of risky posts, and the symp- tom stream to provide domain knowledge, to- gether with a more holistic view of the entire posting history.
We propose a two-stream Psychiatric Experts Model for better multi-task learning of 7 men- tal disorders. which boosts the overall perfor- mance by better utilizing the distinctions and commonality among discases.
With the interpretability enabled by symptom and disease-specific experts, we analyze the decision-making process of PsyEx, and fur- ther study the contribution of each symptom to the detection of different diseases.
Approach
'The mental disorders are: ADHD, Anxiety Disorder, Bipolar Disorder. Depression, Eating Disorder, PTSD. Brief introductions about these disorders are included Appendix
In this section, introduce the proposed frame- work for multiple MDD, including symptom iden- tification and a two-stream Psychiatric Experts Model. First, we give the definition of multiple
WWe call them "streams" here hecause both symptom fea- tures and text features are arranged in chronological order.

---

MDD here. Given the posting history of a user with N posts: {Pi, P2..... PN.. and a list of M po- tential mental diseases, the goal is to detect which disease(s) in the .M candidates does the user suffer from. Typically, this can be accomplished by train- ing M separate models for each disease. Here, we further propose a novel method that leverage sin- gle model to effectively learn the distinctions and commonality among diseases for better overall performance on all diseases.
set K < N, so the reduced input size enables high efficiency even with BERT-based language models in the following disease detection phase. Moreover, the symptom extracted by a supervised model is also more reliable to select risky posts than heuristic approaches, and we will show this in the experiments (Sec. 3.4.3).
2.2 Two-Stream Psychiatrie Experts Model
Although MDD can be formulated as typical text classification problem, traditional model structures not appropriate for our multiple mental disease detection task for two primary reasons. On the one hand, employing a single user representation to detect all diseases would overlook the explicit differences between them. Conversely, if each dis- ease is detected separately, the neglect of shared symptoms across these diseases will also adversely affect the overall performance.
2.1 Symptom-based Risky Posts Screening
Traditional MDD method processes every single post cqually overlooking the fact that not every post from a patient reveals useful information for detection. To facilitate the MDD model perfor- mance, as well as provide explainable diagnosis basis, we screen risky posts first, in particular with disease-dependent symptom information, and use the selected posts for multiple disease detection. Inspired by clinical diagnosis procedures, we assume that posts reflecting symptoms of mental disorders suggest higher risks. Since healthy in- dividuals rarely produce symptom-related content, posts with symptom indications could better sepa- rate patients from control users. Hence, unlike the prior heuristic methods discussed in Sec. 1, im- plement screening method based on the symptom features extracted by a supervised symptom identi- fication model (Zhang et al., 2022b). This model can identify 38 symptom classes from 7 mental diseases with a Mental BERT-based encoder (Ji et al., 2022) and linear classifier. It is trained on a large-scale, multi-disease annotated symptom iden- tification dataset, based the Alobally-accredinn diagnostic criteria from Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (APA et al., 2013).
Discase
Post Expert
Symptoes Expert
Oooooo
Ilienechical Network
Sycuptom
Risky Pus Smor
Figure 2: Illustration of the proposed model architec- ture, with 2 disease-specific experts on top of shared network. Darkness of red square indicates attention strength, which is different for each discase.
The symptom feature of each post a 38- dimensional vector, where each dimension is the predicted probability of a certain symptom 6 We can then estimate the risky score of a post with the sum of predicted probability among all symptoms. The top K posts with highest risky score among user's whole posting history (containing N posts) will be selected as his/her risky posts. In practice,
To alleviate this problem, we propose Two- Stream Psychiatric Experts Model (See Figure 2), which can detect all the mental diseases simultane- ously through multi task learning while still cap- turing the nuances among them. It takes risky post stream and symptom stream obtained in symptom identification phase input combine the advan- tages of both modalities.
'The code and data of the symptom identification model can be found at https: /gi thub. com/bl EMNLP22-PsySym
Risky Post Stream Model The risky post stream "cherry-picks" posts with highest symptom prob- ability, providing more strong and concentrated sig- nals of mental diseases. To better explore these sig-
'These symptoms (e.g., anxious mood, sleep disturbance, poor memory) are carefully extracted from DSM-5, so that there is as little semantic overlap as possible between them. We list all the symptoms in Table 12 (Appendix E).

---

nals and distinguish among multiple diseases, the model consists of two components: a single shared hierarchical network and D dpseasc- post experts.
in risky post stream model. Similarly, each dis- case has its own "expert" to get different atten- tion distributions on the same symptom sequence, which intuitive because each mental disease has its own typical or unique symptoms for diagnosis (c.g., panic fear for Anxiety, intrusion for PTSD). Finally, the disease- sesseas--ppecifie user representation from both stream are concatenated and fed into separate linear layers to get the binary predictions on whether the user suffers from a certain mental disease.
In the shared part, we draw on the structure of Hi- erarchical Attention Network (HAN) (Yang et al., 2016) with post and user-level encoder to make better use of the sequential structure of the posts. We employ pre-trained BERT model the post encoder. For words { (wi, u2...., } in a post, the post representation p is,
The whole model is trained with the standard binary cross entropy loss, where the loss of all the tasks are averaged. In the dataset, we usually have patients that are certain for the diagnosis of some diseases but uncertain for others, and we should not assume these discases to be not existing due to the high comorbidity of diseases. Therefore, we implement loss masking as described in Fonseca et al. (2020), so that for patients with at least one disease, we do not consider their absent disease labels as negative (i.e., assign zero weight exclude them from affecting the training loss). This approach helps alleviate the problem of potentially missing labels. We can get more reliable negative labels from control users who show no signals of mental issues.
p - (EERTCCLSC , w2,,.. .... wr)
(I)
The user-level encoder utilizes a transformer (Vaswani al., 2017) structure, modeling the relations between these posts (P1, P2....,. and produces updated post representations {pr, P2....,
In the disease-specific part, each disease d has its own attention layer to get different attention distributions the same post sequence. As such, these D attention layers be considered fea- ture selectors from the shared network (Liu al., 2019). Then we perform weighted sum of the post representations according to attention score to get the distinctive user representations utd
exp(Waakk + ba)
O. =
(2)
Experiments
Ek=1 exp(Waate wehh + ba)
In this section, we conduct experiments to (1) ex- amine the performance our two-stream PsyEx model compared with strong baselines; (2) vali- date the effectiveness of various design choices with ablation tests; (3) evaluate the model's gen- eralizability on different datasets; (4) analyze the interpretability enabled by symptoms and discase- specific "experts".
Ud = Σ ag
where Wa and ba are both learnable parameters specifically for disease d.
The attention distribution on the post sequence reflects the weight of each post in the final pre- diction. Therefore, the attention score can offer interpretability of the model decision, which will be further discussed in Sec. 3.6
3.1 Multiple MDD Datasets
We construct multiple MDD dataset by reimple- menting the data collection method of SMHD (Co- han et al., 2018). Users and posts were extracted from a publicly available Reddit corpus. We select diagnosed users by detection patterns with a focus on high precision. The patterns consist of two com- ponents: one that matches self-reported diagnosis (e.g.. "diagnosed with"), and another that maps relevant keywords to the 7 mental diseases (e.g.. "panic disorder" to Anxiety). user is assigned a disease one of its keywords occurs within 40 char- acters of the diagnosis pattern. Control users (i.e.,
Symptom Stream Model Symptom feature critical for MDD, hecause equips the detection model with explicit psychiatric knowledge. The symptom stream can be considered as an N x 38 matrix, revealing user's whole posting history in lighter way, which is able to replenish the incomplete text stream and provide holistic infor- mation for the detection model. To better capture the unique features of different diseases, the symp- tom stream model is totally discase- speciifcc and its structure is the same as the corresponding part

---

Method
Depression
Anxicty
ADHD
Ripolar
OCD
PTSD
Fating
Avg.
TF-IDF+L (Cohan et al., 2018)
76.75
75.45
68.97
76.56
40.0
44.02
30.0
58.82
BERT (Nguyen al., 2022)
73.28
70.59
56.72
61.21
41.82
50.0
32.67
55.18
Symp (Zhang al., 2022h)
81.06
81.99
70.3
81.75
65.12
64.41
61.54
72.31
HAN-GRU (Sekulic and Strube, 2019)
74.99
82.16
81.72
80.28
70.59
67.67
68.57
75.14
ChatGPT
70.12
73.09
64.08
67.12
52.98
67.61
29.73
60.68
PsyEx
87.89
89.84
84,40
91.58
81.69
81.69
85.71
86.12
Table Mental Disease Detection Results across diseases Reddit dataset, reporting FI scores binary setting. We order these diseases in descending order according to their number of patients in the dataset, making it easier to identify rarer classes.
Method 52.16
Depression
Anxiety
ADHD
Bipolar
OCD
PTSD
Eating
macro-Fi
micro-I-1
EM
TF-IDF+LR
32.52
29.9
43.14
13.79
9.76
25.9
38.09
77.37
BERT
56.99
47.03
31.86
16.59
0
0
21.78
39.38
75.32
Symp
62.99
57.93
49.57
51.85
18.18
0
34.36
53.05
76.65
HAN-GRU
59.4
44.88
53.81
62.2
0
0
31.47
50.47
77.55
ChatGPT
51.48
46.3
41.83
62.45
28.72
20.54
12.12
37.64
44.16
75.56
PsyEx
66.09
54.97
54.07
66.17
41.51
44.83
9.52
48.17
58.59
81.38
Table Mental Disease Detection Results across diseases on Reddit dataset in mutli-label setting, reporting score of each discase, micro-F1 and macro-F1 over all discases, well exact match ratio (EM).
healthy persons) are randomly sampled from those who never posted or commented in mental health related subreddits and never mentioned the name of 7 mental diseases (e.g., "bipolar", "PTSD") to avoid possible false positives. Similar to SMHD, we eliminate the diagnostic posts from the dataset prevent the direct leakage of label, but retain those mental health related posts to allow the ex- traction of symptom-related features.
Logistic Regression model for prediction. BERT is the reimplementation of the MDD model Nguyen et al. (2022), which utilizes CNN of vari- ous kernel sizes top of the sentence embeddings from pre-trained BERT as features aggregate the information from posting list. Symp (Zhang et al., 2022b) the same CNN backbone, but further replace the BERT embedding features with symptoms features, and it establishes strong base- line. HAN-GRU reimplements the hicrarchical at- tention network for MDD proposed in Sekulic and Strube (2019), which utilizes Bidirectional GRU as encoders. Since large language model shows superior performance in various NLP tasks (Ye et al., 2023), we also incorporate such systems for comparison, and uses ChatGPT predict user diseases. More details like hyperparameter settings, prompts of the baseline experiments can be found in Appendix C.
The final dataset consists of 5,624 diagnosed users and 17,209 control users. Each diagnosed user can have one or more disease labels, so we provide the distribution of the user's disease count in Table 8 (Appendix B). The statistics show that 57% users in the dataset suffer from two more kinds of mental disorders, and this comorbidity scenario is challenging for disease detection mod- els. Moreover, due to the uneven distribution of different mental disorders in reality, the dataset is naturally unbalanced, with more users suffering from Depression and Anxiety, while fewer users with OCD and PTSD7. Consequently, the detec- tion of rarer discases can be even more difficult to resolve.
3.3 Experimental settings
For PsyEx model, we select K 16 high-risk posts during the screening process to form the risky post stream". We utilize pre-trained bert-tiny (in binary setting) or mental-bert!! (in multi-label set- ting) the basis of the post encoder. The user encoder is 4-layer 8-head transformer encoder. We train with batch size of 32 and set learning
3.2 Methods of Comparison
For multiple MDD task, we mainly compared the proposed methods with types of baselines: TF- IDF+LR (Cohan et al., 2018) is a representative traditional machine learning method which utilizes TF-IDF to extract textual features, followed by
"We further explore the impact of post number the detection results in Figure (Appendix D).
10 /htttps://hugginggacc wtwps//naaaa...oaaaaaaaee/ee httpg/innnaacc.cc/enaalalall
"Se Table 7 in Appendix B for the exact numher of users suffering from each discase.
"https: http://hugggggaaaa co/mental/

---

rate at le-5. We also employ early-stopping with a patience of 4 epochs according validation per- formance to prevent overfitting.
with ablation tests in binary setting. Results are shown in Table 3.
Method
Avg.
Two-Stream PsyEx
86.12
3.4 Experiment Results
wo symp-stream
84.67
To be consistent with previous works (Cohan et al.. 2018), we train and evaluate the models in both binary and multi-label setting.
w/ multi-attn
83.68
we multi-task 85.40
Table 3: Ablation tests for the design choices of PsyEx, reporting average Fl across diseases on Reddit dataset. Results of each disease are in Table 9 (Appendix D).
3.4.1 Binary Setting Results For the binary task, we only need to decide whether the user is suffering from a certain mental disease, so only users with this mental disorder plus all control users are selected to train and evaluate. The results in binary setting are shown in Table 1. We see that our proposed PsyEx outperforms all the baseline methods including CharGPT, suggesting the advantage of our symptom-base bbsed risky post screening and two-stream psychiatric expert model. Further, owing to the multi-task learning and the shared knowledge between diseases, the detection effect of rarer classes (i.c., cating disorder, OCD, PTSD) is largely improved.
First, we implement w/o symp-stream model which only preserves the risky post stream part exhibit the effectiveness of symp- toms. As shown in Table 3, the detection perfor- mance drops without symptom stream, indicating that symptoms can not only disentangle multiple diseases better with embedded domain knowledge, but also provide a global view of users' entire post- ing history.
Then, examine the D disease- specific atten- tion layers by implementing model w/o multi- attn, in which all the discases share single atten- tion head but are still trained simultancously with multi-task learning and both streams are preserved as well. We can see that the performance is greatly harmed without multiple attention heads, illustrat- ing the effectiveness of disease-specins "experts" to properly capture the characteristics of different mental discases.
3.4.2 Multi-label Setting Results
In multi-label setting, we have to determine if and which mental diseases the user was diagnosed with, that is, the user have one or more diseases, and all data is used for both training and evalua- tion. We show the multi-label results in Table 2, in which we evaluate these classifiers with a strict metric, exact match ratio, together with macro and micro FI score which take partially correct into consideration.
Moreover, the w/o multi-task method further trains w/o multi-atm model for each disease sepa- rately without multi-task learning. We can notice a slight decrease on the detection performance even with much more model parameters, as we need to train 7 independent models for w/o multi-task method.
This setting is challenging and underexplored in previous works mainly due to the complexity brought by comorbidity,
well the various overlapped manifestations among different men- tal disorders. Our PsyEx model shows significant advantage over other strong baselines, especially on the rarer classes, in which some classifiers can not even find a true positive sample. It is worth not- ing that ChatGPT also exhibits acceptable perfor- mance on rare classes, benefiting from its training extensive amount of data, which enhances its robustness and ability to generalize to infrequent cases. However, its practical usage in the mental health domain limited by its slow speed and high resource requirements.
Screening Method
Avg. Fl
Symptom- based (PsyFx)
86.12
Similarity (Zhang al., 2022a)
82.47
K-Means (Zogan al., 2021)
74.76
Last
56.74
Table 4: Ablation test for different risky post screening methods, reporting Fl score averaged across diseases. Results of each disease are in Table 10 (Appendix D).
We also compare our symptom-based risky post screening method with other approaches the liter- ature. Similarity and K-Means both utilize pre- trained Sentence BERT (Reimers and Gurevych,
3.4.3 Ablation Study
We examined the effectiveness of various design choices of the proposed Two-Stream PsyEx model
13 "woo"" means without, so symptom stream is removed in this model.

---

Method
Depression
Anxicty
ADHD
Ripolar
OCD
PTSD
Avg. F1
1I-idf+LR
36.5
37.33
61.19
56.01
33.89
BERT
56.3
53.43
57.86
62.38
44.24
45.57
53.30
Symp
41.49
37.93
31.24
41.52
32.49
22.46
34.52
HAN-GRU
72.24
71.21
65.17
75.44
58.58
60.24
67.15
PsyEx
76.51
77.31
67.15
78.36
67.35
65.64
72.05
Table 5: Mental Disease Detection Results across 6 diseases Twitter dataset, reporting Fl scores in binary setting.
2019) to obtain sentence representations. The for- mer one extracts key posts according to the cosine similarity between post and mental disease descrip- tions, and select K posts with the highest similarity score. The later gorithm the post embeddings and gets the K posts nearest to the cluster center as representative posts. Last simply selects the last K post as risky posts. Except for the different screening methods, we use the same model structure (i.e., our proposed PsyEx), and experiment results are listed in Ta- ble 4. It can be seen that symptom based risky post screening outperforms all the heuristic approaches, especially K-Means and Last, highlighting the im- portance of precise screening method, there can be large amount of posts irrelevant to mental disorders in the users' posting history.
3.6 User-level Interpretability Analysis
The interpretability of our proposed PsyEx model primarily stems from two key factors: (1) the usage of symptoms in risky posts screening and direct input for model prediction; (2) the incorporation of disease-specific "experts" through the attention mechanism, which allows us to assess the impor- tance of each post and symptom in determining the final prediction, providing insights into which posts and symptoms have the most significant influence on the model's 's decision.
runs K-means clustering al-
Consequently, the interpretability of PsyEx can be manifested in two aspects: Difference, different symptoms can contribute differently to predicting different diseases; Reasonableness, the contribu- tion proportion of symptoms to different discases in the model is reasonable (i.e., is roughly consistent with the authoritative DSM-5 criteria).
3.5 Generalizability
Therefore, we provide concrete user-level ex- ample to illustrate the decision-making process of PsyEx. The selected user suffers from three men- tal disorders, including anxiery, bipolar and de- pression. We apply the trained PsyEx model his/her posting history, and obtain the attention score matrix of symptom stream, which is D x N, since there're re disease-specific attention heads. To figure out which symptoms are more critical for diagnosing a certain disease, we measure the contribution of each symptom to the detection of different diseases, with the help of attention score matrix.
To evaluate the models' generalizability, we con- duct experiments using another dataset (Suhavi et al., 2022a) sourced from Twitter. Despite dif- ferences in language style and topics, Twitter posts generally feature more shorter content and higher frequency compared Reddit posts.
This Twitter dataset includes users with eight distinct mental disorders, some of whom have co- morbidity. Notably, it has a 60% hand-annotation rate, ensuring the precision of disorder labels. We worked with a subset of this dataset, and for de- tailed data statistics, please refer to Appendix B.
For each diagnosed disease d, we calculate the symptom contribution vector Cj as follows. First, we select eight symptom probability vectors S = {(81 $2, with the highest attention score among the symptom stream. Next, for each 38-dimensional vector 8,. we only preserve the value of three symptoms with highest probability. So we set the probability of rest symptoms to and obtain š, Finally, we can get the user's symptom contribution vector
We conduct experiments on this dataset in bi- nary setting. Given the shorter yet more frequent nature of tweets, we select K = 128 tweets as risky posts, and the other hyper-parameter settings (Re- fer to Sec. 3.3 and Appendix C for details) remain consistent with the Reddit dataset without further tuning. The experimental outcomes are in Table 5. These results demonstrate the effectiveness of our proposed PsyEx model on Twitter data, surpassing the performance of all baselines, which illustrates the model's capability to generalize across various social media platforms.
Ca
(4)

---

5
Symplom Contriutionn Volor
3.75
2.5
1,25
14
20
26
33
Symptom ID
Amxiety
Bipolar
Depression
Figure 3: User-level symptom contribution values according to the attention scores obtain from symptom stream model. user diagnosed with anxiety, hipolar and depression. provide the corresponding symptom names with symptom ID in Table 12 (Appendix E), and omit the symptoms whose contribution values of anxiety, bipolar and depression are all O.
10
adhd
50
0.0
1.0
amsiey
014
0.22
035 0.25 0.6 0:53
0.14 OLL 018 0.75 624 3:5 0.11
312
25
bppoler
2.02 06 3.6
0.00
0.27 055 0:11 0.2 02T 05 033 0.0 33 0.13 01 0.30 0.88 8.05 052 032
05 0.15 0333 01 05 0.3 0.6 051 003 D.00 039 300 107
2.0
desression
6.05 0.0 385
000 0.2 16.00000 033 6.7 083 032 00L 0.1 0.3 05 0.33 646
023 233 058 0.9 532 000 0.21 043 0.2 01 334
UN2 0. 1 DIN 5//
0 1.S 7. oCR 11 997
10
-1.9 037
238 048 1E3 DIG 038 0.8
0.00
03 004 912 502 055
005
05
055
25
26
36
Figure 4: Global symptom contribution vectors of all users in the test set. provide the corresponding symptom names with symptom id in Table 12, Appendix E.
which is demonstrated in Figure 3.
fore, we claim that the explicit usage of symptoms, as well as disease-specific attention layers, can ex- actly improve the interpretability of our neural net- work model.
The symptom contribution vectors of these three mental diseases quite different from each other, satisfying the first aspect of interpretability (i.e., Difference), which means that our proposed model, some extent, have learned how to "diagnose" certain disease with corresponding symptoms just like psychiatrists.
3.7 Global Symptom Contribution Analysis
To provide global view of nigghcoontrbbttoon symp- toms for all the 7 diseases, we plot heatmap (See Figure 4) symptom contribution vectors based on all the users in the test set. Here, we group the users according to his/her diagnosed discases, and obtain 7 disease-specifi user subsets 15 For each disease subset, we calculate all the user-level symp- tom contribution vectors Eq. (4) based on the attention score, and aggregate them by averaging get the global contribution vector of each disease.
But do these learned high-contribution symp- toms truly make sense for the diagnosis? We pare our symptom contribution vectors with the standard criteria DSM-5 to validate the second as- pect of interpretability (i.e., Reasonableness). From Figure 3, we can find out that typical symptoms "6: depressed mo 113 and "37: weight and appetite change" of depression, "1: anxious mood" and "30: panic fear" of anxiety do contribute more to the detection of them than other diseases, which is in line with DSM-5 Moreover, we can also notice many shared high-contribution symptoms between bipolar disorder and depression, which is reason- able because bipolar disorder contains both depres- sive episodes and hypomanic episodes There-
Apart from a lot of agreement reached between our global contribution vector and the authorita- tive DSM-5 (c.g.. "21: drastical shift in mood and energy" for bipolar disorder), we can find several interesting inconsistencies. For example, eating dis- order patients are "38: more irritable" and tend to "20: do things that casily get painful consequences", both of which are typical symptoms for bipolar disorder patients in the manic episodes, probably
18 We present the symptoms "ID: name" format. "https:/ /ttp:///whhh..///t//..tttt../htt nih. www..ni......e.eeeee http:/hhalttttttttttttehh co/cppccc/ bi polar- dorder
15 user with multiple mental discases in multiple subsets

---

due to the high comorbidity of these two disorders proved by previous researches (Lunde et al., 2009; Ruiz and Gutiérrez-Rojas, 2015). Further, almost all the diseases treat "1: anxious mood" and "6: depressed mood" as the most important symptoms in PsyEx, while in DSM-5, they're not the typical symptoms for diagnosing many mental diseases such as ADHD. We try to explain this as follows. As a diagnostic criteria, DSM-5 tend to focus on the distinctive symptoms each disease rather than the common ones. However, these common symp- toms can be generally meaningful to distinguish patients of most disorders from control users, so they are highly weighted by our PsyEx model.
that simultaneously detects multiple mental dis- orders or for mental health symplom extraction. While we acknowledge LLM's potential in mental health application, needs further exploration and investigation as our preliminary results indicated that this remains a challenging task.
Conclusions
In this work, we tackle the challenge of detect- ing multiple mental diseases simultaneously in the scenario of comorbidity, and propose a multiple MDD framework achieving SOTA performance in both binary and multi- settings. We first apply risky post screening based on symptoms, providing reliable diagnostic basis for further discase detec- tion. Then, we propose two-stream Psychiatric Experts Model with shared hierarchical encoder and disease-specifts attention layers, which simul- taneously process the symptom and text features to combine the advantage of both modality. We also explore the interpretability of PsyEx by provid- ing user-level analysis, and measuring the global symptom contribution to the detection of different diseases.
4 Related Work
In the literature, substantial efforts have been made detect certain mental disease. Some of these works focus on leveraging features like TF-IDI, LIWC (Pennebaker al., 2001), and posting pat- terns (Trotzek et al., 2020; Losada and Crestani, 2016) for MDD. Others apply various deep learn- ing methods (Yates et al., 2017; Gui et al., 2019), well as the contextualized embedding (Ji et al., 2022; Jiang et al., 2020) to improve the perfor- mance of classifiers. However, these methods often fail generalize well (Harrigian et al., 2020) and cannot provide explainable results due to lack of knowledge in the psychiatric domain. To tackle these issucs, some works (Lee et al., 2021; Nguyen et al., 2022) began to utilize symptom features, but they extract symptom features with unsuper- vised/weakly supervised methods, which isn't so reliable for the downstream MDD task.
6 Ethical Statement
We rely on publicly available Reddit posts 16 our work and we make every possible effort to mini- mize the risk of leaking privacy of individuals in the data collection process. We made no attempt to contact users link users other social media accounts following the previous best practices (Co- han al., 2018; Losada al., 2017: Suhavi et al., 2022b). We also replace usernames with random identifiers to prevent users' identities from being known without the use of external information. For the usage of symptom identification dataset, we sign and comply with the data usage agreement to prevent the invasion of privacy or other potential misuses. What's more, we carefully consider the application of mental disease detection from social media. The purpose of this work are not to replace psychiatrists. Instead, we expect our model to be used as an effective auxiliary tool by experienced psychiatrists in the future.
Recent years, some works start to detect multiple mental disorders. Cohan et al. (2018) proposed a massive Reddit dataset SMHD containing 9 mental disorders, followed by many subsequent studies based on this dataset, such Sekulic and Strube (2019) and Zhang al. (2022b). However, these works often directly apply a single-disease model the multi-disease data (i.e., train the model for D times to obtain the results of D diseases), over- looking the correlation among multiple discases, and thus fail to perform well on rarer diseases.
With the emergence of large language models (LLMs), there exists few studies utilizing LLMs for tasks like depression detection on social me- dia (Lamichhane, 2023; Qin et al., 2023) or de- veloping chatbots for depression diagnosis (Chen et al., 2023), but we fail to find any existing work
IIn the privacy policy of Reddit, the posts are public and accessible to everyone, and Reddit allows third parties to ac- cess public Reddit content via the Reddit API and other simi- lar technologies. See ht https:/www/ //wwu. //www/reddit com/pol hom/policies/ privacy policyyppyyyyyoooll for more detailed information.

---

7 Limitations
through self-reported diagnoses. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 1-10, Denver, Colorado. As- sociation for Computational Linguistics.
Our work has some limitations that could be ad- dressed in future research.
Despite the significant performance boosting over the baseline, our proposed PsyEx model still cannot achieve satisfying performance in multi-label setting, especially on rarer dis- eases like eating disorder (See Table 2). To tackle this issue brought by the imbalanced data, we utilize a commonly-used resampling method, which samples equal amount of users with each disease for each batch. However, find improvement in the detection perfor- mance of these rarer diseases after balanced sampling, indicating that the unsatisfying re- sults aren't just a matter of sparse positive samples. Therefore, we hope to further ad- dress this issue in future studies.
Caitlin Dreisbach, Theresa A. Koleck, Philip E. Bourne, and Suzanne Bakken. 2019. systematic review of natural language processing and text mining of symptoms from electronie patient-authored text data. International Journal Medical Informatics. 125:37
Eduardo Fonseca, Shawn Hershey, Manoj Plakal, Daniel PW Ellis, Aren Jansen, and R Channing Moore. 2020. Addressing missing labels in large- scale sound event recognition using a teacher- student framework with loss masking. IEEE Signal Processing Letters, 27:1235-12399
Tao Gui, Liang Zhu, Qi Zhang, Minlong Peng, Xu Zhou, Keyu Ding. and Zhigang Chen. 2019. Cooperative multimodal approach to depression detection in twitter. In Proceedings of the Thirty-Third AAAI Conference and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAT Symposium on Educational Advances in Artificial Intelligence, AAAI Press.
Artificial Intelligence
Multiple MDD task is still under-explored cur- rently. Many previous works (e.g.. Sekulic and Strube (2019)) only conduct experiments on binary setting (i.e., separately train D mod- els for detecting D mental diseases). There- fore, for comparison under multi-label setting, can only adopt their hyperparameters the binary setting, which may not be optimal in some cases.
Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2020. Do models of mental health based on so- cial media data generalize? In Findings of the Association for Computational Linguistic;: EMNLP 2020, pages 3774-3788, Online, Association for Computational Linguistics.
Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Men- talBERT: Publicly available pretrained language models for mental healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7184-7190, Marseille, France. European Language Resources Association.
References
David Adam. 2013. Mental health: On the spectrum. Nature, 496:416-418.
DS APA, American Psychiatric Association, al. 2013. Diagnostic and statistical manual of mental disorders: DSM-5, volume 5. American psychiatric association Washington, DC.
Zhengping Jiang, Sarah Ita Levitan, Jonathan Zomick. and Julia Hirsehberg. 2020. Detection of mental health from Reddit via deep contextualized repre- sentations. In Proceedings of the 1 11th International Workshop on Health Text Mining and Information Analysis, pages 147-156, Online. Association for Computational Linguistics.
Siyuan Chen, Mengyue Wu, Kenny Q. Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023. LLM- empowered chatbots for psychiatrist and patient sim- ulation: Application and evaluation.
Arman Cohan, Bart Desmet, Andrew Yates, Luca Sol- daini, Sean MacAvaney, and Nazli Goharian. 2018. Smhd: a Large-scale resource for exploring online lan- guage usage for multiple mental health conditions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1485-1497, Santa Fe, New Mexico, USA. Association for Computa- tional Linguisties.
Femke Lamers, Patricia van Oppen, Hannie C. Comijs, Johannes H. Smit, Philip Spinhoven, Anton J. L. M. van Balkom, Willem A. Nolen, Frans G. Zitman, Aartjan T. F. Beekman, and Brenda Penninx. 201 Comorbidity patterns of anxicty and depressive disor- ders in large cohort study: The netherlands study of depression and anxiety (nesda). Journal of Clinical Psychiatry, 72(3):341-34
Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015. From adhd to sad: An- alyzing the language of mental health on twitter
Bishal Lamichhane. 2023, Evaluation of chatgpt for nlp-based mental health applications.

---

Andrew Lee, Jonathan K Kummerfeld, Larry An, and Rada Mihalcea. 2021. Micromodels for efficient, explainable, and reusable systems: A case study on mental health. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4257-4272, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguisties.
Eva Ruiz and Luis Gutiérrez-Rojas 2015. Comorbidity of bipolar disorder and eating disorders. Revista de psiquiatria y salud mental, 844:2332-441
Ivan Sekulie and Michael Strube. 2019. Adapting deep learning methods for mental health prediction on social media. In Proceedings of the 5th Workshop Noisy User-generated Text (W-NUT 2019), pages +22-327, Hong Kong, China. Association for Com- putational Linguistics.
Shikun Liu, Edward Johns, and Andrew J. Davison. 2019, End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1881-
Suhavi, Asmit Kumar Singh, Udit Arora, Somyadeep Shrivastava, Aryaveer Singh, Rajiv Rain Shah, and Ponnurangam Kumaraguru. 2022a. Twitter- stmhd: An extensive user-level database of mul- tiple mental health disorders. Proceedings of the International AAAT Conference on Web and Social Media, 16(1):1182-11) 1191.
David Losada and Fabio Crestani. 2016. A test col- lection for research on depression and language use. volume 9822, pages 28-39.
David Losada. Fabio Crestani, and Javier Parapar. 2017. erisk 2017: Clef lab early risk prediction the internet: Experimental foundations. pages 346-360.
Suhavi, Asmit Kumar Singh, Udit Arora, Somyadeep Shrivastava, Aryaveer Singh, Rajiv Ratn Shah, and Ponnurangam Kumaraguru. 2022b. Twitter-stmhd: An extensive user-level database of multiple men- tal health disorders. In International Conference on Web and Social Media.
Anna V. Lunde, Ole B. Fasmer, Kareen K. Akiskal, Hagop S. Akiskal, and Ketil J. Oedeguard. 2009. The relationship of bulimia and anorexia nervosa with bipolar disorder and its temperamental foundations. Journal of Affective Disorders, 1533)330--314
Marcel Trotzek, Sven Koitka, and Christoph M. Friedrich. 2020. Utilizing neural networks and lin- guistic metadata for early detection of depression indications in text sequences. IEEE Transactions on Knowledge and Data Engincering, 32(3):588-600
Danielle Mowery, Hilary Smith, Tyler Cheney, Greg Stoddard, Glen Coppersmith, Craig Bryan, and Mike Conway. 2017. Understanding depressive symptoms and psychosocial stressors on twitter: a corpus-based study. Journal of medical Internet research, 19:e48.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Ilaia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022. Improving the generalizability of depression detection by lever- In Proceedings
aging clinical questionnaires. of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8446-8459, Dublin, Ireland. Asso- ciation for Computational Linguistics.
Zichao Yang, Diyi Yang, Chris Dyer. Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierar- chical attention networks for document classifica- tion. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, Cali- fornia. Association for Computational Linguistics.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates.
Wei Qin, Zetong Chen, Lei Wang, Yunshi Lan, Weijiey- ing Ren, and Richang Hong. 2023. Read, diagnose and chat: Towards explainable and interactive LLMs- augmented depression detection in social media.
Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages putational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNP), pages 3982-39229 29922 Hong Kong, China. Association for Computational Linguistics.
Association for Com-
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. comprehen- sive capability analysis of gpt-3 and gpt-3.5 series models. ArXiv, abs/2303. 10420.
M. Roca, M. Gili, M. Garcia- Garcia, Salva, M. Vives, J. Garcia Campayo, and A. Comas. 2009. Preva- Ience and comorbidity of common mental disorders primary care. Journal of Affective Disorders, 1 119( 1)552-8.
Zhiling Zhang, Siyuan Chen, Mengyue Wu, and Kenny Q. Zhu. 2022a. Psychiatric scale guided risky

---

post screening for early detection of depression. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, pages 5220-5226. International Joint Conferences on Artificial Intelligence Organization. AI for Good.
Disease
Users (Reddit)
# Users (Twitter)
Depression
3105
1448
Anxiety
2239
ADHD
2374
1370
Bipolar Disorder
1366
2011
OCD
PTSD
391
588
Zhiling Zhang. Siyuan Chen, Mengyue Wu, and Kenny Zhu. 2022b. Symptom identification for interpretable detection of multiple mental dis- orders on social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, page 9970-9985, Association for Computational Linguistics..
Eating Disorder
138
'able 7: Number of users suffering from each disease in the multiple MDD datasets (Reddit and Twitter).
than one mental disorders in the Reddit dataset, and PsyEx can achieve superior performance for its specific design targeting these comorbidity sce- narios.
Hamad Zogan, Imran Razzak, Shoaib Jameel, and Guan- dong Xu. 2021. Depressionnet: novel summariza- tion boosted deep framework for depression detection social media. arXiv preprint arXiv:2105.1088
Disease
Users (Reddit)
Users (Twitter)
A Introduction 7 Mental Disorders
2326
2842
1738
1716
Here, we briefly introduce 7 mental disorders stud- ied in paper, listing their typical manifestations summarized from DSM-5 in Table 6 for better un- derstanding of these mental disorders.
134
407
152
7
14
Discase
Typical Symptoms
Table 8: Distribution of user's disease counts. For
inattention; hyperactivity and impulsivity.
example, there are 1738 users suffering from two mental diseases the Reddit dataset.
ADHD
excessive fear and worry; panic attacks; anxious mood.
Anxicty
drastic shifl in mood and energy: experience periods of mania and depression.
Bipolar Disorder
C Detailed Experimental Settings of Baselines
depressed mood; loss of plea- sure interest; poor concen- tration; guilty feelings; suicidal ideas.
Depression
For the CNN backbone of BERT and Symp method, the model structure et al. (2022). We train both model with batch size=64, but we set the learning rate as 0.01 when using symptom features, and as 0.003 when using BERT embedding. The HAN-GRU model is trained with batch size=32 and learning rato=0.0001. The posting list will be truncated to preserve the earliest 256 posts at most. The average posting number of users the dataset 115.2, so 256 is safe enough to preserve almost all the posts of a user.
intense fear of gaining weight; binge and purge; rumination; weight and appetite change.
same as that of Nguyen
Eating Disorder
OCD
obsession; compulsion. often develops after shocking, dangerous event; flashbacks; bad dreams.
PTSD
Table 6: 7 mental disorders detected in our work with their typical symptoms.
B Detailed Data Statistics
In the ablation test of different risky post screening methods, the sentence BERT model utilized for Similarity and K-Means is What's more, the Similarity (Zhang et al., 2022a) method extracts key posts according to the cosine similarity be- tween post and mental disease descriptions, which are the description of 38 symptoms (see Table 12) manually summarized from DSM-5.
For the multiple MDD datasets (Section 3.1), we show the number of users suffering from each dis- ease in Table 7. The distribution of the 7 diseases are similar to SMHD (Cohan et al., 201 8), and the training/validaatio/ntsst
paraphrase-iiniLM-G-W
set is 8:1:1.
In addition, the users in these two datasets can suffer from multiple mental disorders simultane- ously. So we also provide the distribution of the number of diseases on single user in Table 8. The statistic shows that 57% users suffer from more
Further, to infer the symptoms mentioned in the posts, we perform some pre- propescggsingg steps. First,

---

Model
Depression
Anxicty
ADHD
Bipolar
OCD
PTSD
Fating
Avg. F1
Two-Stream PsyEx
87.89
89.84
91.58
81.69
81.69
85.71
86.12
w/ symp-stream
88.12
89,09
84.55
89.33
76.71
81.58
83.33
84.67
w/ multi-attn
87.64
89.7
76.39
79.49
83.68
w/o multi-task
87.69
88.89
84.21
91.16
75
84.21
86.67
85.40
Table 9: Ablation tests for the design choices of PsyEx, reporting F1 score of cach discase (detailed results of Table 3).
Screen method
Depression
Anxicty
ADHD
Bipolar
OCD
PTSD
Eating
Avg. Fi
Symptom-basee based (PsyEx)
87.89
89.84
84,4
91.58
81.69
81.69
85.71
86.12
Similarity
86.84
87.7
83.03
87.84
78.26
77.14
76.47
82.47
K-Means
79.34
81.15
68.66
82.0
70.34
73.42
68.42
74.76
Last
69.27
67.89
55.31
67.6
45.8
46.15
45.16
56.74
Table 10: Ablation tests different risky post screening methods, reporting score of each disease (detailed results of Table
we use blingfire' to split a post into sentences. Then, we use regular expressions to filter out the hyperlink format like "[anchor text] (web url)" and preserve the anchor text. Finally, we remove sen- tences like "[removed]".
Multi-label setting
Please predict the user's mental disorders accord- ing to his/her posts on reddit. The mental dis- oders should be chosen from: 'depression' "anxi- cty', 'bipolar", 'cating disorder', 'ADHD', "PTSD', 'OCD'. The user can have multiple disorders, or have no disorder. Your answer format should be: 'mental disorders:
C.1 ChatGPT baseline
(MENTAL DISORDERS).
Due the input length limitation of ChatGPT, utilize the screening method described in Sec. 1, which selects 16 posts from the user history instead using all of them.
D Detailed Ablation Results
We show the detailed results of ablation tests in Table 9 and Table 10. Without symptom stream (i.e., w/o symp-stream) disease-specific attention layer (i.e., w/o multi-attn), the F1 score on nearly all the diseases dropped, especially the rarer dis- eases like eating disorder and OCD, What's 's more, we can notice a significant drop in the performance without symptom-bass based risky post screening in all the diseases, suggesting the importance of precise screening method to filter out the noisy data.
To maximize the effectiveness of ChatGPT, we made substantial efforts in prompt engineering. Through this process, observed that directly requesting prediction results yielded unsatisfactory performance in binary setting. To address this, we introduced the requirement for ChatGPT to provide explanations alongside its predictions, leading to a significant improvement in accuracy, especially eating disorder (See Table 11). We also attempted to include explanation instructions in multi-labe setting: however, the outcome was even worse than without explanations. This suggests that there is still long way to go in effectively utilizing Chat- GPT for this task, and we hope to address this in the future work.
D.1 Impact of the Number Selected Posts
Here we study the impact of the number of posts selected in risky posts screening (see Figure 5). We can observe that 16 posts have the best performance for nearly all the diseases. To find out the reason, we calculate the average symptom probability the selected K posts sorted by its highest symptom probability, which is 0.25 for the posts ranked 16, meaning that 16 posts is enough to include most of the symptomatic posts and adding more posts can easily introduce some noisy data.
We present the final prompts for both binary and multi-label settings below.
Binary setting
Please predict whether the user has mental disor- ders, such as DISEASE, from the clues showed in his/her posts on reddit. Your answer should be a single yees" or 'no', followed by an explanation the predicted result.
E Psychiatric Symptoms
We use serial numbers to represent symptoms Figure and Figure 4, so we provide the corre-

---

Method Depression
Anxiety
ADHD
Bipolar
OCD
PTSD
Fating
Avg.
direct
70.16
59.36
67.95
39.22
53.13
8.7
52.21
explanation
70.12
73.09
64.08
67.12
52.98
67.61
29.73
60.68
Table 11: Mental disease detection results across 7 diseases utilizing ChatGPT in binary setting
87.5
05.0
02.5
Symptom
deprwoith
Ippller
1
Anxious Mood
2
Autonomic symptoms
3
Cardiovascular symptoms
Catatonic behavior
5
Decreased energy tiredness fatigue
6
Depressed Mood
Figure 5: Impact of the number of selected posts on each disease and the mean Fl.
7
Gastrointestinal symptoms
8
Genitourinary symptoms
9
Hyperactivity agitation
10
Impulsivity
sponding symptom names Table 12 for refer- ence.
11
Inattention
12
Indecisiveness
13
Respiratory symptoms
These symptoms are defined in the symptom identification dataset proposed by Zhang et al. (2022b). This dataset contains 8,554 sentences extracted from the Reddit corpus. All the sentences are annotated with symptoms that can be identified
14
Suicidal idcas
15
Worthlessness and guilty
16
Avoidance of stimuli
17
Compensatory behaviors to prevent weight gain
18
Compulsions
19
Diminished emotional expression
20
Do things easily painful consequences
from the sentence. The average Fleiss's for all the 38 symptoms 0.7708.
21
Drastic shift in mood and energy
22
Fear about social situations
23
Fear of guining weight
24
Fears of being negatively evaluated
25
Flight of idcas
26
Intrusion symptoms
27
Loss of interest motivation
28
More talkative
29
Obsession
30
Panic fear
31
Pessimism
32
Poor memory
Sleep disturbance
Somatic muscle
Somatic symptoms others
Somatic symptoms sensory
Weight and appetite change
Anger Irritability
Table 12: Id and its corresponding symptoms


Reviews:
[{"review_id": "GbXeqoGBS3", "reviewer": "Reviewer_hhgj", "summary": "", "questions": "How the task fine-tuned models perform in this scenario? Do they outperform your approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** How the task fine-tuned models (Mental-BERT, Depression-RoBERTa) perform in this scenario? Do they outperform your approach?\n\n**Ans:** Thank you for raising this point. In our PsyEx model, we have already incorporated Mental-BERT as the post encoder. Directly finetuning Mental-BERT[1] or Depression-RoBERTa[2] is virtually the ablation of the proposed method. \n\nThe outcomes from direct task fine-tuned models fall behind PsyEx, as indicated in the table below. This can be attributed to these models failed to capture the sequential structure of the posts. Therefore, design choices like the Two-Stream Psychiatric Experts Model (Sec. 2.2) in the proposed PsyEx model are crucial to the model's significantly improved performance.\n\n\n|                           | depression | anxiety | adhd  | bipolar | ocd   | ptsd  | eating | Mean F1 |\n|---------------------------|------------|---------|-------|---------|-------|-------|--------|---------|\n| MentalBERT(base)          | 65.05      | 61.68   | 54.54 | 62.54   | 45.07 | 47.5  | 37.84  | 53.46   |\n| Depression-RoBERTa(large) | 63.46      | 62.3    | 53.55 | 63.49   | 48.73 | 52.78 | 53.85  | 56.88   |\n| PsyEx                     | 87.89      | 89.84   | 84.4  | 91.58   | 81.69 | 81.69 | 85.71  | 86.12   |\n\n\n[1] Ji et al. \"MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare\" LREC (2022).\n\n[2]Poswiata et al. “OPI@LT-EDI-ACL2022: Detecting Signs of Depression from Social Media Text using RoBERTa Pre-trained Language Models.” LTEDI (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach using two stream psychiatric experts is novel. A detailed experimental analysis was presented. The results seem to improve upon the SOTA models.", "reasons_to_reject": "Multiple SOTA task fine-tuned models were not included in the experiments (i.e. Mental-BERT, Depression-RoBERTa).", "questions_to_authors": "How the task fine-tuned models perform in this scenario? Do they outperform your approach?", "paper_topic_and_main_contributions": "The authors presented a new approach towards the Mental Disorder Detection task. They presented a thorough experimental analysis and showed the superiority of their own approach.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Bj4jSBhmz", "reviewer": "Reviewer_qJms", "summary": "", "questions": "Is the dataset going to be public? Making it public (together with symptoms and disease labels) would bring a lot to mental health research.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. It is not clear if this can be applied to shorter comments or different social media (twitter, facebook)\n\n**Ans:** Thank you for the insightful question. Obtaining suitable datasets for the detection of multiple mental disorders is quite challenging, as most datasets tend to focus on the detection of single disorders (e.g., depression), or doesn't have open access. Fortunately, we came across a recent Twitter dataset[1] encompassing users with eight distinct mental disorders. A notable aspect of this dataset is that 60% of it is hand-annotated, resulting in precise disorder labels. We worked with a subset of this Twitter dataset, and the data statistics are as follows:\n\n|            | depression | anxiety | adhd | bipolar | ocd | ptsd | control |\n|------------|------------|---------|------|---------|-----|------|---------|\n| # of users | 1448       | 933     | 1370 | 2011    | 368 | 588  | 4804    |\n\n|            | 1 disorder | 2 disorders | 3 disorders | 4 disorders | 5 disorders |\n|------------|------------|-------------|-------------|-------------|-------------|\n| # of users | 2842       | 1716        | 134         | 8           | 1           |\n\nIn this dataset, each user averages 8821.1 tweets, with each tweet containing an average of 13.59 words. This indicates a higher frequency of shorter content compared to Reddit posts.\n\nWe conduct experiments on this twitter dataset in binary setting. We directly apply the hyper-parameter settings (e.g., batch size, learning rate, etc.) of each model from the SMHD dataset without further tuning (Refer to Section 3.3 and Appendix C for details). As each user's tweets are shorter yet more frequent than Reddit posts, we select K=64 tweets as risky posts for PsyEx (K=16 in the SMHD dataset). The experimental outcomes on the Twitter dataset are as follows:\n\n|                    | depression | anxiety   | adhd      | bipolar   | ocd       | ptsd      | Mean F1   |\n|--------------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|\n| tf-idf+LR          | 36.5       | 37.33     | 61.19     | 56.01     | 3.8       | 8.5       | 33.89     |\n| Symp               | 41.49      | 37.93     | 31.24     | 41.52     | 32.49     | 22.46     | 34.52     |\n| BERT               | 56.3       | 53.43     | 57.86     | 62.38     | 44.24     | 45.57     | 53.30     |\n| HAN-GRU            | 72.24      | 71.21     | 65.17     | 75.44     | 58.58     | 60.24     | 67.15     |\n| PsyEx              | **76.51**  | **77.31** | **67.15** | **78.36** | **67.35** | **65.64** | **72.05** |\n\nThe results demonstrate the effectiveness of our proposed PsyEx model on Twitter data, surpassing the performance of all baselines. This illustrates the model's capability to generalize across various data types.\n\n[1] Suhavi et al. “Twitter-STMHD: An Extensive User-Level Database of Multiple Mental Health Disorders.” International Conference on Web and Social Media (2022).\n\n\n2. To use or extend the model, psychiatric knowledge is required within the team in order to be able to understand all symptoms and how they are related to dieseases.\n\n\t**Ans:** We agree that psychiatric knowledge is vital to our approach, endowing PsyEx with high precision and interpretability based on a relatively higher bar of usage. However, we expect that organizations who would deploy such disease detection system for social good should at least have some members with psychiatric knowledge, so that they can audit the model predictions and provide further psychiatric aid. Consequently, the requirement of psychiatric knowledge would not be a real obstacle. \n\n3. Is the dataset going to be public? Making it public (together with symptoms and disease labels) would bring a lot to mental health research.\n\n\t**Ans:** Yes, we will make the dataset and code be public.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The application of the paper is relevant and very important.\n2. Authors have developed method to automatically extract symptoms, labels and understand persons diseases from reddit posts\n3. Proposed model performance is better than strong baselines and something that is generally hard to achieve on mental health data given the skewed datasets. This is especially true for multi-label settings.", "reasons_to_reject": "1. It is not clear if this can be applied to shorter comments or different social media (twitter, facebook)\n2. To use or extend the model, psychiatric knowledge is required within the team in order to be able to understand all symptoms and how are they related to dieseases.", "questions_to_authors": "Is the dataset going to be public? Making it public (together with symptoms and disease labels) would bring a lot to mental health research.", "paper_topic_and_main_contributions": "Authors propose a model to detect multiple mental disorders from social media (reddit) posts. They use two streams of learning that should understand if the post is involving a symptom related to one of the diseases and if the post is risky (ie. relevant to the mental health) or not. Mental health comorbidities are common and there is not much work on detecting multiple diseases that person can have in the same time, while that is pretty common in practice. Authors use hierarchical network to learn post embedding in the lower level and then user embedding given all relevant user posts. According to evaluation, the proposed model performs better than strong existing baselines, including ChatGPT.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRLTKpo5MV", "reviewer": "Reviewer_CNuF", "summary": "", "questions": "The authors need to verify this on more data with more state-of-the-art methods.\n1.Limited scope of population and Lack of clinical real cohort: The proposed framework is evaluated on a single dataset, which may not be representative of the general population.  \n\n2. Lack of explainability. The paper does not provide a detailed discussion of the interpretability of the model. Although the model is excellent at detecting mental disorders, its inner workings are still not well understood. Therefore, the paper may need to discuss in more detail how to improve the explainability of the model and explain how the model makes predictions.\n\n3.Lack of parameter details: The article provides a detailed theoretical description of the proposed approach, but does not provide much information on parameters in practice.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The authors need to verify this on more data with more state-of-the-art methods. Limited scope of population and Lack of clinical real cohort: The proposed framework is evaluated on a single dataset, which may not be representative of the general population.\n\n**Ans:** In our paper, we utilized the SMHD dataset, which includes real social media posts from 5624 diagnosed users and 17209 control users. This dataset covers 7 common mental disorders, making it one of the most comprehensive datasets in terms of mental disorder detection task.\n\nFurther, to address your concern, we observed the potential to extend our method to different types of social media platforms, such as Twitter. Therefore, we conducted supplementary experiments on a Twitter dataset[1], which encompasses eight distinct mental disorders. Importantly, 60% of it is hand-annotated, resulting in precise disorder labels. We worked with a subset of this Twitter dataset, and the corresponding data statistics are outlined below:\n\n\n|            | depression | anxiety | adhd | bipolar | ocd | ptsd | control |\n|------------|------------|---------|------|---------|-----|------|---------|\n| # of users | 1448       | 933     | 1370 | 2011    | 368 | 588  | 4804    |\n\n|            | 1 disorder | 2 disorders | 3 disorders | 4 disorders | 5 disorders |\n|------------|------------|-------------|-------------|-------------|-------------|\n| # of users | 2842       | 1716        | 134         | 8           | 1           |\n\nIn this twitter dataset, each user averages 8821.1 tweets, with each tweet containing an average of 13.59 words. This indicates a higher frequency of shorter content compared to Reddit posts.\n\nWe also introduce an additional baseline, referred to as \"Depression-RoBERTa\"[2], to demonstrate the performance of fine-tuning a pre-trained RoBERTa model in this multiple disorder detection task.\n\nWe directly apply the hyper-parameter settings (e.g., batch size, learning rate, etc.) of each model from the SMHD dataset without further tuning (Refer to Section 3.3 and Appendix C for details). As each user's tweets are shorter yet more frequent than Reddit posts, we select K=64 tweets as risky posts for PsyEx (K=16 in the SMHD dataset). The experimental outcomes on the Twitter dataset are as follows:\n\n|                    | depression | anxiety   | adhd      | bipolar   | ocd       | ptsd      | Mean F1   |\n|--------------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|\n| tf-idf+LR          | 36.5       | 37.33     | 61.19     | 56.01     | 3.8       | 8.5       | 33.89     |\n| Symp               | 41.49      | 37.93     | 31.24     | 41.52     | 32.49     | 22.46     | 34.52     |\n| BERT               | 56.3       | 53.43     | 57.86     | 62.38     | 44.24     | 45.57     | 53.30     |\n| Depression-RoBERTa | 58.29      | 53.83     | 61.32     | 62.78     | 46.69     | 49.38     | 55.38     |\n| HAN-GRU            | 72.24      | 71.21     | 65.17     | 75.44     | 58.58     | 60.24     | 67.15     |\n| PsyEx              | **76.51**  | **77.31** | **67.15** | **78.36** | **67.35** | **65.64** | **72.05** |\n\nThe results demonstrate the effectiveness of our proposed PsyEx model on Twitter data, surpassing the performance of all baselines. This illustrates the model's capability to generalize across various data types.\n\n\n[1] Suhavi et al. “Twitter-STMHD: An Extensive User-Level Database of Multiple Mental Health Disorders.” International Conference on Web and Social Media (2022).\n\n[2] Poswiata et al. “OPI@LT-EDI-ACL2022: Detecting Signs of Depression from Social Media Text using RoBERTa Pre-trained Language Models.” LTEDI (2022).\n\n\n2. Lack of explainability. The paper does not provide a detailed discussion of the interpretability of the model. Although the model is excellent at detecting mental disorders, its inner workings are still not well understood. Therefore, the paper may need to discuss in more detail how to improve the explainability of the model and explain how the model makes predictions.\n\n**Ans:** The explainability in our method can be mainly derived from two components: the symptom features and the multiple attention layers, which function as dedicated \"expert\" for each disorder.\n\nFirst, in Symptom-based Risky Posts Screening (Sec 2.1), we predicted the probabilities of any symptoms reflected in one post, so that we can detect the symptoms that the user may be suffering, and find highly risky posts that shows signals of certain symptom. The symptoms then become direct features for model prediction. \n\nSecond, the attention mechanism used in our model (Eq. 2,3) allows us the check the importance(alpha) of each post/symptom in deciding the final prediction, so that we know which posts/symptoms have major effects on the model prediction. \n\nIn Section 3.5, we further combine the 2 aspects and calculate symptom contribution vector. This vector further helps us visualize the contribution of each symptom to the prediction of each disease according to the model (Figure 3).\n\n3. Lack of parameter details: The article provides a detailed theoretical description of the proposed approach, but does not provide much information on parameters in practice.\n\n**Ans:** Thank you for pointing this out. We have provided descriptions of the experimental parameter settings in Section 3.2 and Appendix C, such as learning rate and batch size. Here we further supplement the specification of the model's parameter. We don't know ChatGPT's exact number of parameter, so we approximate it with its ancestor GPT-3.\n\n| Model                          | # of Parameter |\n|--------------------------------|----------------|\n| Symp                           | 38.5K          |\n| BERT                           | 110M           |\n| **PsyEx(bert-tiny)**           | **4.8M**       |\n| **PsyEx(mental-bert-uncased)** | **123M**       |\n| GPT-3                          | 175B           |\n\nAs shown in the table above, the parameter of PsyEx(mental-bert-uncased) is similar to the commonly used BERT model, and PsyEx(bert-tiny) is much more smaller. They are also significantly smaller than recent LLMs like GPT-3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel approach: This article presents a novel approach to mental disease detection that considers the possibility of multiple disorders occurring simultaneously and incorporates domain knowledge for more interpretable results.\n\n2. Improved detection performance: The two-stream architecture of this approach improves detection performance by complementing symptom-based screening with psychiatric knowledge explicitly and implicitly embedded into the risky post stream.\n\n3. Applicability to rare classes: This approach brings significant improvement to rarer diseases on which the baselines struggled heavily, addressing the issue of rare classes in mental disease detection.", "reasons_to_reject": "1.Limited scope of population and Lack of clinical real cohort: The proposed framework is evaluated on a single dataset, which may not be representative of the general population.  \n\n2. Lack of explainability. The paper does not provide a detailed discussion of the interpretability of the model. Although the model is excellent at detecting mental disorders, its inner workings are still not well understood. Therefore, the paper may need to discuss in more detail how to improve the explainability of the model and explain how the model makes predictions.\n\n3.Lack of parameter details: The article provides a detailed theoretical description of the proposed approach, but does not provide much information on parameters in practice.", "questions_to_authors": "The authors need to verify this on more data with more state-of-the-art methods.\n1.Limited scope of population and Lack of clinical real cohort: The proposed framework is evaluated on a single dataset, which may not be representative of the general population.  \n\n2. Lack of explainability. The paper does not provide a detailed discussion of the interpretability of the model. Although the model is excellent at detecting mental disorders, its inner workings are still not well understood. Therefore, the paper may need to discuss in more detail how to improve the explainability of the model and explain how the model makes predictions.\n\n3.Lack of parameter details: The article provides a detailed theoretical description of the proposed approach, but does not provide much information on parameters in practice.", "paper_topic_and_main_contributions": "The article proposes a novel framework called PsyEx, which combines symptom-based screening with psychiatric knowledge explicitly and implicitly embedded into the risky post stream. The proposed framework is evaluated on a large-scale dataset of social media posts, and the results show that it outperforms several baseline models in detecting multiple mental disorders. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.add a paragraph to describe the experimental design \n2.it is recommended that Figure 2 should be added more technical details of the model\n\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of detecting multiple mental health disorders from social media posts, which is experimented with Reddit posts. The proposed method leverages two streams of learning to enable this detection.

The contribution is novel, achieving remarkable improvements over competitive baseline models on a challenging and important task.

I appreciate the authors for the effort in preparing a solid rebuttal and I would likewise ask them to incorporate the new results reported in the rebuttals in a revised version of the paper.
Decision: Accept-Main