SERCNN: Stacked Embedding Recurrent Convolutional Neural Network in Depression Detection on Twitter
Authors: Heng Ee Tay
Mei Kuan Lim
Chun Yong Chong
Venue: ICLR 2022 Submitted (2022)
Keywords: Social Media
Twitter
NLP
Depression
Mental Health

Abstract: Conventional approach of self-reporting-based screening for depression is not scalable, expensive, and requires one to be fully aware of their mental health. Motivated by previous studies that demonstrated great potentials for using social media posts to monitor and predict one's mental health status, this study utilizes natural language processing and machine learning techniques on social media data to predict one's risk of depression. Most existing works utilize handcrafted features, and the adoption of deep learning in this domain is still lacking. Social media texts are often unstructured, ill-formed, and contain typos, making handcrafted features and conventional feature extraction methods inefficient. Moreover, prediction models built on these features often require a high number of posts per individual for accurate predictions. Therefore, this study proposes a Stacked Embedding Recurrent Convolutional Neural Network (SERCNN) for a more optimized prediction that has a better trade-off between the number of posts and accuracy. Feature vectors of two widely available pretrained embeddings trained on two distinct datasets are stacked, forming a meta-embedding vector that has a more robust and richer representation for any given word. We adapt Lai et al. (2015) RCNN approach that incorporates both the embedding vector and context learned from the neural network to form the final user representation before performing classification. We conducted our experiments on the Shen et al. (2017) depression Twitter dataset, the largest ground truth dataset used in this domain. Using SERCNN, our proposed model achieved a prediction accuracy of 78% when using only ten posts from each user, and the accuracy increases to 90% with an F1-measure of 0.89 when five hundred posts are analyzed.

Under review as a conference paper at ICLR 2022
SERCNN:
STACKED EMBEDDING RECURRENT
CONVOLUTIONAL NEURAL NETWORK IN DEPRESSION DETECTION ON TWITTER
Anonymous authors Paper under double-bling brind review
ABSTRACT
Conventional approach of self-reporting- screening for depression is not scalable, expensive, and requires one to be fully aware their mental health. Motivated by previous studies that demonstrated great potentials for using social media posts to monitor and predict one's mental health status, this study utilizes natural language processing and machine learning techniques social media data to predict one's risk of depression. Most existing works utilize handcrafted fea- tures, and the adoption of deep learning in this domain is still lacking. Social media texts are often unstructured, ill-formed, and contain typos, making hand- crafted features and conventional feature extraction methods inefficient. More- over, prediction models built on these features often require a high number of posts per individual for accurate predictions. Therefore, this study proposes a Stacked Embedding Recurrent Convolutional Neural Network (SERCNN) for a more optimized prediction that has better trade off between the number of posts and accuracy. Feature vectors of two widely available pretrained embeddings trained on two distinct datasets are stacked, forming a meta-embedding vector that has more robust and richer representation for any given word. We adapt Lai et al. (2055) RCNN approach that incorporates both the embedding vector and context learned from the neural network form the final user representation be- fore performing classification. We conducted our experiments on the Shen et al.] (2017) depression Twitter dataset, the largest ground truth dataset used in this do- main. Using SERCNN, our proposed model achieved a prediction accuracy of 78% when using only ten posts from each user, and the accuracy increases to 90% with Fl-measure of 0.89 when five hundred posts are analyzed.
INTRODUCTION
Depression is a serious yet common mental disorder that affects more than 264 million people world- wide. The number is projected to grow amid the war against the pandemic of COVID-19. Unlike the usual mood fluctuations and emotional responses, the long-lasting sadness, emptiness, irrita- tion one's day-to-day life, accompanied by somatic and cognitive changes that heavily disrupt an individual's i capacity to function normally. Hence, depression is often associated with suicide its worst. In the latest Word Health Statistics 2021 (World Health Organization] [021]) 28% increases the suicide rate in the United States in this period, which may result from the increasing pressure on the loss of loved ones and incomes, city lockdowns, social distancing, and the reduction of human interactions.
Existing screening methods, such psychometric self-reporting questionnaires and clinical inter- views, are expensive, not scalable, not reachable to many, and have limitations. For example, the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Associa- tion et al. 2013) requires patients to be involved and disclose truthful information during clinical interviews. Besides that, people who suffer from depression tend to be unaware of their condition because there is no specific feedback from the body like common physical injuries do. To get di- agnosed with depression, global provision and service for identifying, supporting, and treating mental health issues are insuffi- cient despite the disruption of essential health services, community mistrust, and fears of COVID-19
must first consult doctor mental health professional. Up to date,

---

Under review as a conference paper at ICLR 2022
infections. The lack of mental health awareness in the Southeast Asia Region is another contributing factor in the sky-high figure of depression.
Social media has become part and parcel of everyday life. It has become a valuable source of information that led to a new research direction for detecting depression. Here, social media data serve a lifelog that records users' activities in the text, image, audio, and video. The enormous amount of time series first-person narrative posts can provide insights about one's ' thoughts, feelings, behavior, or mood for some time, allowing an unintrusive way to study patients' conditions before an actual clinical diagnosis is made.
Motivated by previous studies that demonstrated great potentials for using social media posts to monitor and predict one's 's mental health status, this paper utilizes natural language processing and machine learning techniques on social media data predict one's risk of depression. We propose Stacked Embedding Recurrent Neural Network (SERCNN). which can effectively learn the rep- resentation from dirty and unstructured social media text without the need to pursue multimodal learning. Instead of using the recent transformer model, we suggest meta-embedding approach for our feature extraction. We utilize two widely available yet low dimensional pretrained GloVe em- beddings trained on different domain datasets provide a more robust and accurate representation of the dirty and unstructured social media text that often contains typos. Our SERCNN can achieve ofate-of-thh-ar performance without using all the social media posts in the user posting history, providing a perspective on the number of posts required to make a reliable prediction. It is worth pointing out that there is still room in social media text that is worth exploring.
The contributions of this paper can be summarized as below.
1. SERCNN can retrieve a more robust and richer representation of the social media text by stacking multiple low dimensional pretrained embeddings.
2. Provides insights on how the different number of posts influence the prediction model's accuracy.
2 RELATED WORK
In this section, we provide an overview of the current research landscape and how previous works convert information extracted from social media posts into more representative features that can be used to identify depressed individuals.
Depression detection on social media Evidence from various published works has shown the via- bility of using social media data to predict depression and other mental disorders. The seminal work by |Holleran 2010) has inspired natural language processing (NLP) researchers to identify potential markers of mental disorder from social media posts. In contrast to the 14 days of observation stated DSM-5, Hu et al. (2015) and Tsugawa et al. (2015). empirically suggested that features extracted over two months are sufficiently be used to identify one's 's depression condition. Coppersmith et al. have presented a. novel method to collect ground truth on social media related to depression and PTSD. The dataset is then being used for the shared task of The Workshop on Computational Lin- guistics and Clinical Psychology (CLPsych) (Coppersmith et al.] 2015). A similar workshop. CLEF eRisk (Losada et 2017). focuses on the early depression symptoms discoveries on social media. Workshop tasks such as depression detection (CLPsych 2015 (Coppersmith et al.) (0115) ) and early detection of signs of depression (CLEF eRisk 2017 (Losada et al.) 2017)) have generated significant numbers of novel approaches in identifying depression on social media.
Feature representation for depression classification Modeling feature representations a crucial task in machine learning; features that are not discriminative and representative will result in poor and faulty model performance. Hence, earlier research works are mainly focused on feature extrac- tion techniques. Choudhury et al. (2013); Tuugaaa et al. (2015) have found that depressed users tend to be emotional, Wang et al. (2013) found that using sentiment analysis in depression detection can achieve about 80% accuracy. Tsugawa et al. (2015); Resnik et al. (2015) extracted topics distribu- tion with Latent Dirichlet Allocation (LDA) Blei et al. (003) to differentiate depressed individuals from the healthy controls. Researchers also extracted features based on criteria stated in the industry- standard Diagnostic and Statistical Manual of Mental Disorders (DSM), such as the insomnia index derived from the user posting time. Linguistic Inquiry and Word Count (LIWC) Tausezik & Pen-
2

---

Under review as a conference paper at ICLR 2022
nebaker 2010) is widely used word matching baased feature extraction tool that builds on top of Pennebaker et al.' findings (Rude et al. 20044 Gortner et al. 2006) decades ago. (Choudhury et al, 2013 Shen et al. (0117) show that depressed users tend to have high self-attentional focus, increased medicinal concern,, and increased expression of religious thoughts. These findings are aligned with the Rude et al. (2004) work where depressive indicators be found on the human-generated
tent.
However, LIWC and word matching-based sentiment analysis approaches are often ineffective. The nature of social media text being dirty and unstructured often required extensive data cleaning and preprocessing for the tools to act as intended. Recent works (Gui al., 2019a b Rao et al. 2020) have shown that using the word embedding and deep learning models are effective than the hand- crafted features with minimal effort of text preprocessing. We observed that most works that em- ployed deep learning models tend to use hierarchical document modeling to generate their input representation. This raises the question of whether depression classification using social media data actually takes advantage of the hierarchical document modeling.
STACKED EMBEDDING RECURRENT CONVOLUTIONAL NEURAL NETWORK (SERCNN)
Overview Our goal is improve the performance of depression classification on social media by learning a much robust user representation. Given that our dataset, d, consists of N1 number of social media users, u, where each user's N2 number social media posts, p. within month were collected, and each post has N3 words, w. we denote the dataset as d = {ui, UN, ). social media user as u = (pi...., PN.) }. and the post as = {W1,... W
We propose Stacked Embedding Recurrent Convolutional Neural Network (SERCNN) which is made up of a Stacked Embedding (SE) and Lai et al. (2015) Recurrent Convolutional Neural Net- work (RCNN). The overall architecture of SERCNN is simple, consisting SE layer, single direc- tional LSTM, a max-pooling layer and an output layer which is fully connected layer with sigmoid function, as visualized in Figure
Date
Tweet
Aug
How could you
don't even love So hard move on, from person and from place.
Sep
Sep
Feeling weird
concatenation preprocessing
how you lov to move ony, from a person and from a pace. foeling weird
me.
Stacked Embedding
Recurrent Convolutional
Layer
Max-Pooling Layer
Fully Connected Layer
depressed?
Figure 1: The overall architecture of the proposed SERCNN.
3

---

Under review as a conference paper at ICLR 2022
3.1 FEATURE EXTRACTION WITH STACKED EMBEDDING
Unlike previous works, model our user representation by firstly concatenating N₂ social media posts chronological order. For random user Uj, the concatenated text representation can be formulated into:
u; = {pr + + PNN2
04 = [w] + + wNss +... + Ng
(2)
This concatenated post can be viewed as single "monthly diary". journal or large document that characterizes the user.
Then, we extract the distributed text representation of each word with the SE, which an ensemble embedding, commonly known as the Meta-embedding technique. The concept of Meta-embedding was first introduced by Yin & Schütze (2016) to utilize and learn the meta of existing well-trained pretrained embeddings and extend the vocabulary. Since different pretrained embeddings were trained on different datasets, each embedding can now complement each other allowing an improved vocabulary coverage and reducing out-of- offvoobbullary words. As the name suggests, SE is formed by stacking the collection dense vectors (pretrained weights) = {Ei.... EN,} extracted from Na number of pretrained embeddings included:
ESg = {E1 +... + EN,
(3)
where the embedding vector, æ, of given word, w can be obtained via:
EsE(w)
(4)
The vocabulary of SE, VSE, is now considered as the vocabulary union of the N₂ pretrained embed- dings, resulting in more extensive vocabulary than a single embedding:
VSE U
Vn
(5)
n=1
love
Embedding Wiki
Embedding Twitter
100
Embeddingse
125
Vector
Figure 2: Illustration of st
In this study, our SE is the simple stacked vector made up of two pretrained GloVe embeddings (Pennington et al. 2014) trained on Twitter and Wikipedia 2014 + Gigaword 5 datasets, respectively:
1. GloVe Twitter (25 dimensions) trained using global word co-occurrences information by Pennington et al. (2014) under an uncased setting, using billion tweets, with 27 billion tokens. The resulting vector consists of 1.2 million vocabularies learned from the corpus.
2. GloVe Wikipedia 2014 Gigaword (100 dimensions) (Pennington et al. 2014), similar to the GloVe Twitter embedding, this embedding is trained using global word co-occurrences information but using a different corpus, which is the combination of Wikipedia 2014 and Gigaword datasets. There approximately 400 thousand words in the vocabulary.
4

---

Under review as a conference paper at ICLR 2022
3.2 REPRESENTATION LEARNING AND DEPRESSION CLASSIFICATION WITH RCNN
Recurrent neural network (RNN) is capable of capturing contextual information over a long se- quence. However, the RNN model favors the later words than words the earlier sequence. In depression detection on social media, where posts are collected over a time interval, we are inter- ested in identifying words throughout that period, rather than just words that occurred later. Lai al. (2015) Recurrent Convolutional Neural Network (RCNN) overcomes the limitation of the existing RNN model by incorporating a pooling layer to extract key features from both the embedding features and the context learned. The max-pooling layer reconsiders embedding features instead of just the context learned the conventional RNN-basee setting, selecting the important features for the classification task.
recurrent convolutional layer
max-pooling layer
left context
stacked embedding
could (197
O0O
you (eSTu(Ug)
000
Y
love (L5
000
d 000
O
Figure 3: Recurrent Convolutional Neural Network (RCNN) using single Forward LSTM
The overall architecture of our RCNN model is visualized in Figure Instead of using bidirectional Long Short Term Memory (LSTM) as described in Lai et al. (2015), we use generic single Forward LSTM to learn the context, c, from the embedding vector, With refers to the jth social media post and i refers to the thth number of words in the post, we can formulate the context for a given word:
(c(w)), hww?)) LSTM (oWW)-). o(w())
(6)
where |c| E IR and h is the output vector.
The output of the LSTM then concatenated with the embedding features, forming an extended context vector:
y = (qq
A max-pooling layer is applied after the RCNN representation of the user is computed:
máx yı
i=1
where the total number of words, L, can be calculated by multiplying the total number of social media posts, N2, by the number of words in each post,N3.
fully connected layer is then used to discriminate the max-pooled context with a sigmoid function, presenting the classification output, ÿ. probabilities:
1
g = PD u:) = 1 eap--ep--aee (Wontbb) E [0,
4 EXPERIMENTS
Benchmark dataset and data preprocessing We employed Shen et al. (2017) Depression Twitter dataset, which is by far the largest benchmark dataset available at the time of submission. The ground truth is collected based on the user's self-disclosure of their depression diagnosis, which
5

---

Under review as a conference paper at ICLR 2022
satisfied the strict pattern "(I'm/ I was/ I am/ I've been) diagnosed depression". Users who never mention the character string "depress" their posting history were included as the healthy controls.
To ensure fair comparison against previous papers, do not expand the dataset and use the data included the dataset only. For data preprocessing, we performed:
1. ground truth (anchor tweet) removal, to avoid the model being biased towards the string character "depress".
2. retweet removal, to focus on the genuine content generated by the user.
3. url removal.
4. non-English healthy controls removal. reduce noise in the training data.
5. lowercasing, we are using the pretrained GloVe uncased embeddings.
6. splitting the symbol "@' from user mentioned, to better characterize subject entity, instead of out-of-vocabulary word. For example, "@ICLR "@ICLR2022" becomes "@" and "ICLR2022
After performing data preprocessing, we realized that there is of the ground truths consists only of retweets. Therefore, we removed that user and randomly select one from the non non-depression dataset form balanced dataset. The statistic of the dataset shown below:
Table I: Dataset statistics
Dataset
Number users Number posts
Average posts per user
D1
Depression Non-depression
1,401
231,494
165
D2
1, 1,401
1.1 19,080
799
Total
2,802
1,350,574
482
Experiment settings We trained our model using PyTorch 1.6 and torchtext 0.7 libraries with CUDA on an Nvidia P5000. All models have trained with Adam (Kingma & Ba 2015) optimizer and cross entropy function as the loss function. We perform grid search to find the optimized hyperparameters for both SERCNN and baselines; the hyperparameters include learning rate, hidden layer dimension, number of vocabulary, dropout, training epochs, early stopping criteria, and batch size are reported Table
Table 2: Hyperparameter settings Search range {0.0005, 0.001, 0.002, 0.01)
Hyperparameter learning rate hidden layer dimension
Optimized
0.001
{100, 200, ... 1000)
500
number of vocabulary dropout
100,000
0.5
training epochs early stopping criteria batch size
100
20
For the choice embeddings for SE, we select GloVe Twitter with 25 dimensions and GloVe Wikipedia with 100 dimensions, empirically, described in Section 3.1
5 RESULTS AND ANALYSIS
We benchmarked our SERCNN with the best models reported by Shen et al. 2017); Gui et al. b2019a that are trained on the same dataset. Specifically, we compare the Shen et al. (2017) Multimodal Dictionary Learning (MDL) and Multiple Social Networking Learning (MSNL), which is trained with handcrafted features, Gui et al. (2019a) CNN and LSTM with Policy Gradient Agent (PGA), and Gui al. (2019b) Gated Recurrent Unit (GRU) VGG-Net Cooperative Misoperation aud
Ageett (COMMA) policy gradients. addition, we trained an LSTM model that used the concatenation representation proposed in Section as well Yang et al. (016) Hierarchical

---

Under review as a conference paper at ICLR 2022
Table 3: Performance comparison against baselines
Model
Training data
Acc
Pre
Rec
F1
Shen et al.
(2017)
MSNL
Handcrafted
0.818
0.818
0.818
0.818
Shen et al.
2017
MDL
features
0.848
0.848
0.85
0.849
Gui et al.
2019a
CNN
Text
0.843
0.843
0.843
0.844
Gui et
2019a
CNN + PGA
(Hierarchical)
0.871
0.871
0.871
0.871
Gui et al.
2019a
LSTM
0.828
0,830
0.828
0.828
Gul et
20119a
LSTM + PGA
0.870
0.872
0.870
0.871
Gui ct al.
(2019b)
GRU VGG COMMA
Text image
0.900
0.900
0.901
0.900
IEHAN
Text
0,906
0,920
0.906
0.913
SEHAN
(Hierarchical)
0.931
0.978
0.931
0.954
JELSTM
Text
0.900
0.921
0.900
0.910
SELSTM
(Concatenated)
0.920
0.936
0.920
0.928
IERCNN
0.893
1.0
0.895
0.944
SERCNN (10 posts)
0.784
0.809
0.764
0.770
SERCNN (500 posts)
0.899
0.879
0.915
0.893
SERCNN (all posts)
0.914
1.0
0.905
0.949
Attention Network (HAN) that used the hierarchical document modeling approach both single embedding (prefix of IE) and Stacked Embedding (prefix of SE) settings. 1E models are trained with pretrained Glo Ve Wikipedia 300 dimensions. Performances of the models are evaluated with accuracy, macro- averrged precision, macro-averaged recall, and macro-averaged F1-measure.
From Table 3 we can observe that the SERCNN and the baselines we trained outperformed previous works Shen et al. (2017); Gui al. (2019a b). This significant performance gap is due to these aspects we observed:
Posts included in the study We exclude all non non-genuine posts (retweets) created by other users because these posts do not reflect the user writing style. In similar action, Gui ct al. (2019a) implemented policy gradient model to selectively exclude irrelevant social media posts in generating their user representation and increase F1-measure by 4.3% for the LSTM and 2.7% for the CNN models. However, having a policy gradient model to control the imput might yield bias and leave out some important information not previously identified. Therefore, we suggest including only the genuine user-generated content by the users is sufficient determine one's depression condition.
Richer representation The combination of SE and RCNN aims to enrich existing repre- sentations. Our reported results show that stacked embedding yields a more robust repre- sentation than high dimensional pretrained embedding, where all models implemented (with the prefix SE-) have improved performance, with HAN gains a 4.1 % increment, and LSTM gains 1.8 % in Fl-measure. Even though there is minimal improvement using SE over IE on RCNN, IERCNN achieved better FI- -eaaurre performance than all other models trained using single embedding and comparable performance with SEHAN and SELSTM. This empirical result suggested that the LSTM might neglect or miss some cru- cial information, as the main difference between the RCNN and LSTM the additional step in concatenating the word embedding features with the LSTM context. The reintro- duction of embedding features allows the model to have a second chance to understand the context better.
Concatenated post A good feature representation is vital to the performance of any clas- sification model. Representing our input as single concatenated text sequence allows IELSTM and SELSTM to have competitive performance over the 1EHAN and SEHAN, which are more sophisticated models. The hierarchical modeling approach learns the con- text of each social media posts independently first before forming the global context. As stated in the previous paragraph, the context learned is not necessarily the most discrimi- native. Consequently, the underlying relationships across different posts might not be cap- tured correctly. Earlier works (Shen et al. 2017) have suggested that first-person pronouns are solid features, but they are often treated stop words and removed from the corpus.
7

---

Under review as a conference paper at ICLR 2022
In contrast, presenting the input single text sequence allows the model to learn and exploit words occur posts in different period.
1.00
SERCNN acc
SERCNN pre
SERCNN
0.95
SERCNN
SELSTM acc
SELSTM pre
SELSTM
SELSTM
0.90
Performance
0.85
0.80
0.75
0.70
10
30
50
100
300
500
1000
2000 all posts
Number of Posts
Figure 4: Performance of SERCNN and SELSTM trained on different number of posts
Finding optimal number of posts The performance of our proposed SERCNN is almost identical its single embedding counterpart, these models' performances are topped the Table This observation suggested that RCNN architecture can capture significant embedding features that are neglected by LSTM and relate these features with the context learned by LSTM in discriminating de- pression users and healthy controls. From the identical results, can deduce that the performance reaches the state of plateau when we include all social media posts collected within a month.
We have reconstructed the dataset into eight variants, which these datasets made up of the most recent {10, 30, 50, 100, 300, 500, 1000, 2000) posts, respectively. From the result obtained, found that using minimal as ten posts for each user, SERCNN can achieve high accuracy and F1-measure of 0.793 and 0.781. This result is significant because the observation of just ten posts in month is very appealing compared to the two-month observation window suggested both Hu et al. (2015) and Tsugawa et al.] (2015). Through experiments, the optimal number of maximum 500; without compromising the performance, SERCNN achieves 89.9% accu- racy and an F1-measure of 0.89. The optimal number obtained is closed to the average posts per user included in this study at 482 posts.
posts is found
Table 4: Complexity analysis of SERCNN and baselines
Model
Sequential operation
LSTM
O ((22N5)
RCNN
O (N2
Hierarchical LSTTM
Word level:
(NgNs)
Post level:
(
8

---

Under review as a conference paper at ICLR 2022
Complexity Analysis of SERCNN Table 4 reports the time complexity analysis of SERCNN with LSTM trained with concatenated posts and hierarchical LSTM (hierarchical document modeling), all LSTM trained are single Forward LSTM models. We use the notation we defined in Section where N2 refers to the number of posts and N3 refers to the number of words. Since all three LSTM models run sequentially from the first word to the last word, the base sequential operation can be measured evaluated into (N))N3). However, for the hierarchical LSTM, it incurs a higher com- plexity of (( as it introduced an additional hierarchical structure at post level (OV))) to learn the global context from the post context learned.
CONCLUSION AND FUTURE WORK
Based on this finding, a cost-effective and scalable early intervention solution can be developed to improve global well-being, SERCNN does not require massive computing power and storage. Concatenating social media posts into a single diary (document) allows the RNN-based model to exploit the relationship of words in different time frame, resulting in generalized global context for a user. Stacked Embedding allows various pretrained embeddings from different benchmark datasets to complement each other and reduce the out-of- -ucca--oocabulary words for much robust classi- fication. SERCNN exploits the rich representation from Stacked Embedding, and the context of the user representation learned, allowing to outperform previous works. Besides that, our experiment demonstrates that SERCNN can achieve about 78% accuracy using 10 posts only and 90% accuracy using only 500 posts, without the need to utilized the whole dataset.
For our future work, we would like to extend our work with interpretability and explainability mech- anisms. Providing reasons for the classification decisions is especially crucial with the recent en- forcement of the European GDPR Law, where people have the right to explanation.
REPRODUCIBILITY STATEMENT
The code will be made available once the findings are published.
REFERENCES
American Psychiatric Association et al. Diagnostic and statistical manual of mental disorders (DSM-50). American Psychiatric Pub, 2013.
David M. Blei, Andrew Y. Ng, and Michael Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 30m01):993-1022, March 2003, ISSN 1532-4435.
Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. Predicting depression via social media. In ICWSM, 2013.
Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. Clpsych 2015 shared task: Depression and ptsd on twitter. In Proceedings of the 2nd Workshop Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pp. 31-39,190000 2015.
Eva-Maria Gortner, Stephanie S Rude, and James W Pennebaker. Benefits of expressive writing in lowering rumination and depressive symptoms. Behavior therapy, 37(3):292-303, 2006.
Tao Gui, Qi Zhang, Liang Zhu, Xu Zhou, Minlong Peng, and Xuanjing Huang. Depression de- tection on social media with reinforcement learning. In China National Conference on Chinese Computational L Linguistics, pp. 613-624. Springer, 2019a.
Tao Gui, Liang Zhu, Qi Zhang, Minlong Peng, Xu Zhou, Keyu Ding, and Zhigang Chen. Co- operative multimodal approach to depression detection in twitter. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 110-117, 2019b.
Shannon Holleran. The early detection of depression from social networking sites. 2010.

---

Under review as a conference paper at ICLR 2022
Quan Hu, Ang Li, Fei Heng. Jianpeng Li, and Tingshao Zhu. Predicting depression of social media user on different observation windows. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), volume I, pp. 361-36614430001 2015. doi: N0.1109/W11AT:20055..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. abs/ Abs/1442.6990, 2015.
CoRR,
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. Recurrent convolutional neural networks for text classífication. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI15,, pp. 2267-2273, AAAI Press, 2015. ISBN 026251 1290.
David E Losada, Fabio Crestani, and Javier Parapar. erisk 2017: Clef lab on early risk prediction the internet: experimental foundations. In International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 346-360. Springer, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1543, 2014. htRLtttp:/// www. ac web. org/ http:/aahhoooooo--111111
Guozheng Rao, Yue Zhang. Li Zhang. Qing Cong, and Zhiyong Feng. Mgl-cnn: A hierarchical posts representations model for identifying depressed individuals in online forums. IEEE Access, 8:32395-32400 2020. doi: 10.1 109//C/CSSSSS/.O 202.EES...2...92... 29737777..
Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jor- dan Boyd-Graber. Beyond Ida: exploring supervised topic modeling for depression- lan- guage in twitter. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pp. 99-107, 2015.
Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. Language use of depressed and depression-vulnces vulnerable college students. Cognition & Emotion, 18(8):1221-1133. 2004.
Guangyao Shen, Jia Jia, Liqiang Nie, Fuli Feng, Cunjun Zhang, Tianrui Hu, Tat-Seng Chua, and Wenwu Zhu. Depression detection via harvesting social media: A multimodal dictionary learning solution. In IJCAI, pp. 3838-3844, 2017.
Yla R Tausezik and James W Pennebaker. The psychological meaning of words: Liwe and comput- erized text analysis methods. Journal of language and social psychology, 29(1):24-544 54, 2010.
Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kosuke Nakajima, Yuichi Itoh, and Hiroyuki Ohsaki. Recognizing depression from twitter activity. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, pp. 3187-3196, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781 7788440333456 doi: 10. 10.1145/27021223.270222 URL https::t///ot //t::///oo.... www.ooo/1.............oo 10 1 20255/7021233 2t12222.777222 2702280
Xinyu Wang, Chunhong Zhang, Yang Ji, Li Sun, Leijia Wu, and Zhana Bao. A depression detection model based on sentiment analysis in micro- micro-blog social network. Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 201-213. Springer, 2013.
WHO World Health Organization. World health statistics 2021. World Health Organization, 2021.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong Hc, Alex Smola, and Eduard Hovy. Hicrarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480-1489, San Diego, California, June 2016. Association for Computa- tional Linguistics. doi: 10.18653/w1/16-1111 URL
N1 N16-1174
Wenpeng Yin and Hinrich Schütze. Learning word meta-embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1351-1360, 2016.
10


Reviews:
[{"review_id": "yrPnw8XWj_o", "reviewer": "Reviewer_pXvC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n\n- (S1) SERCNN shows solid improvements over existing solutions on the Twitter depression detection dataset. \n\n\n### Weaknesses\n\n- (W1) Novelty and technical contributions are not significant. \n- (W2) The paper lacks comparisons against pre-trained Transformer-based language models. \n- (W3) No insights about depression are provided.\n\n\n### Major comments\n\n(W1)\nSERCNN is a straightforward combination of existing techniques. Essentially, the paper combines 2x GloVe word embeddings into a single vector (without using any learning mechanism) for each input word and then uses the concatenated vector as input to an RCNN model. \n\n\n(W2)\n\nThe author(s) mention the reason why they prefer RCNN over RNN is to capture context information over a long sequence, which can be addressed by using the self-attention mechanism.\n\nThe paper does not compare with any Transformer-based models, especially pre-trained Transfomer-based language models (e.g., BERT or more recent models,) which are considered more label efficient than conventional word embedding models (e.g., GloVe) + RNNs (technically, this paper uses RCNN.)\n\nIn fact, pre-trained language models were already used in the context of emotional detection from text. Thus, it is not natural to disregard the technique for depression detection. For example, \n\n- [1] Yen-Hao Huang, Ssu-Rui Lee, Mau-Yun Ma, Yi-Hsin Chen, Ya-Wen Yu, Yi-Shin Chen, EmotionX-IDEA: Emotion BERT – an Affectional Model for Conversation, SocialNLP 2019. (https://arxiv.org/abs/1908.06264)\n- [2] Kisu Yang, Dongyub Lee, Taesun Whang, Seolhwa Lee, Heuiseok Lim, EmotionX-KU: BERT-Max based Contextual Emotion Classifier, SocialNLP 2019. (https://arxiv.org/abs/1906.11565)\n\n\n(W3)\n\nI would expect to see new findings of depression detection on Twitter. The paper simply presents a technique (which has issues with respect to novelty and technical significance, as commented above) and claims the contribution by showing the numbers. This point may not be the main scope of ICLR (i.e., which may put more emphasis on technical contributions,) but I believe it is very important for application-oriented papers. \n\n\n### Minor comments\n\n- (Presentation style) Figure 2 is misleading. The figure looks like a linear layer is applied on top of the concatenated vector, which is not (i.e., the concatenation.)\n", "summary_of_the_paper": "This paper develops a SERCNN, which consists of stacked embeddings and Recurrent CNN (RCNN), for depression detection from Twitter text. The key idea of stacked embeddings is to concatenate two embeddings, based on two different word embeddings models (pre-trained on Twitter and Wikipedia copora), into a single embedding vector, which is fed into RCNN. Experimental results on the Twitter Depression dataset show reasonable performance when trained on 10 posts from each user, which was further improved when trained on more data. \n", "summary_of_the_review": "Although the paper tackles an important problem, it does not have a sufficient level of novelty and technical contribution. SERCNN is a straightfoward combination of existing methods. SERCNN relies on pre-trained word embedding models. Pre-trained Transformer-based language models should be compared. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qr39c3jqLIT", "reviewer": "Reviewer_4NmL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+Ves\n+ The depression detection application is valuable, and the method achieves good performance.\n+ The whole structure is nice. \n\n-Concerns\n\n-The key concern about the paper is the lack of novelty. The RCNN method is classic, and the word embedding concatenation is also very common. So, the novelty of this paper is not enough.\n\n-The motivation is not clear. The reason why the authors use RCNN and what problem the authors want to solve should be explained, rather than only for getting better results. \n\n-The paper says “Instead of using the recent transformer model”, Why not use transformer-based methods, like BERT? The authors should explain it and add comparative experiments.\n\n-In the experimental results, why does the EHAN achieve the best results and not the proposed ERCNN? In addition, the paper says that the ERCNN can achieve best performance when using a few posts, but there are no comparative results that contain results of other methods with 10 posts or 100 posts.\n\n-The authors should add ablation studies to show the effectiveness of stack embedding.\n\n-The related works are not sufficient and many latest literature are missing.\n\nMinor comments: \n\n* Different evaluation metrics should not be put in one figure (Fig. 4).\n\n* Some references should be added for background.\n", "summary_of_the_paper": "This paper proposes a stacked embedding recurrent neural network, named SERCNN, to detect depression from Twitter. First, the authors use stacked meta-embedding to gain the stacked word information. Then, the RCNN structure is utilized to capture contextual features. The experimental results show the effectiveness of the proposed model.", "summary_of_the_review": "The novelty is not enough, and many experiments are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DZyCljsdfh", "reviewer": "Reviewer_pWBG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation and the results of the paper are very interesting and definitely will be of importance in a more applied community focused on NLP or ML for mental health. However, the technical novelty for a venue such as ICLR is somewhat limited. All network components used exist already and while the different pre-training embedding models used appear to be effective, they do not teach us anything new about deep learning.\n\nI would suggest to the authors to consider a different venue that appreciates this kind of findings.", "summary_of_the_paper": "The paper proposes a concatenation approach to combine multiple social media texts through a stacked embedding layer and demonstrates its effect in depression prediction based on text.", "summary_of_the_review": "(see above)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "83n6yg8mq1", "reviewer": "Reviewer_XSZ1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points:** \n* In general, the paper is clearly written and easy to follow. The problem of depression detection is well-motivated, the overall presentation is good.\n* The authors discussed adequately different related works to the problem of depression detection. They stated the gap of the existing approaches, which rely on handcraft features to detect depression.\n* The authors describe well the preprocessing steps and create a balanced version of the dataset for training the model. \n\n**Weak Points:**\n\nWhile most of the paper is easy to understand, there are some points in the paper that lack clarity for the novelty and significance.\n\n* The authors propose a flexible concatenation of different embeddings for a robust and rich tweet representation. However, the novelty of this method is inadequate compared with state-of-art contextualized embeddings such as BERT.  For example, the research work [1] addresses the same problem and considers contextualized embeddings (e.g., BERT, Roberta) as state-of-art-embeddings baselines. Here, I suggest the author benchmark their approach against BERT embeddings. For example, the authors can use BERTweet [2] as a baseline to SERCNN . \n* In table 2, the authors show the hyperparameters optimized for the proposed approach and baselines. This is not clear to me that all models are optimized by the same hyperparameter values, given that these models have different architectures.\n* Some details about the experiment's setup are missed. For example, how the authors split the data for training, validation, and testing the models (e.g., train-test split ratio). Is there overfitting during the model’s training? \n* Further, stacking embeddings is also not memory efficient. I find the advantage of stacked embeddings is addressing out-of-vocabulary words. Here, I suggest the authors conduct more experiments to highlight this contribution in their experiments as well.  \n* Since the authors didn’t specify the test data used, It would be better to evaluate their approach on a different dataset (see [3]), which may contain out-vocabulary words, and can show the efficiency of the proposed approach. \n* Also, I wonder if stacking embedding is efficient for this task only? Or It’s a generic, richer representation that can achieve good performances with very little data. \n* In section 5, The authors analyzed the performance of their approach with little training data (e.g., 10, 30, 100, 500, etc.) I wonder how do you sample these tweets from the dataset, randomly?. Here, I recommend obtaining multiple samplings and performing a standard statistical test to benchmark how significant are the outperforming results.\n* Although, this study lacks the explainability of depression detection, and the authors name it as future work. I find some studies [4] in 2020 that address depression detection with an explainable approach. \n\n\n[1] Zhang, Yipeng, et al. \"Monitoring depression trend on Twitter during the COVID-19 pandemic.\" arXiv preprint arXiv:2007.00228 (2020).\n[2] https://github.com/VinAIResearch/BERTweet \n[3] https://github.com/swcwang/depression-detection   \n[4] Zogan, Hamad, et al. \"Explainable Depression Detection with Multi-Modalities Using a Hybrid Deep Learning Model on Social Media.\" arXiv preprint arXiv:2007.02847 (2020).\n\n\n\n", "summary_of_the_paper": "This paper proposes a deep learning approach, called SERCNN, for depression detection from social media data (tweets). The proposed approach is flexible and stacks different embeddings (Glove pretrained model,  and a learned embedding vector from LSTM) for robust and richer tweets representation. Overall, the contributions of this approach are third folds: i) leveraging social media as a valuable source of information for depression detection; ii) the authors employ stacked-embeddings is a good technique to handle out-of-vocabulary words; and iii) Using a few training examples (e.g., 10 tweets per user), the proposed approach showed a remarkable performance (78% accuracy) in depression prediction.  Further, the authors conduct a set of experiments on a public Twitter dataset for depression detection and compare it with different baselines. ", "summary_of_the_review": "Overall, I rate this paper as marginally below the acceptance threshold (5). The problem is interesting, however, the proposed approach lacks a comprehensive evaluation with state-of-the embedding models to ensure the efficiency of the approach. For example, the authors can employ BERT embedding as a baseline to evaluate to which extent SERCNN approach is good. Further, I suggest the authors, clarify how do they perform sampling (e.g., 10 tweets) to assess their approach with little data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles a very important problem of detecting depression on Twitter. As the reviewers expressed in their reviews. this paper will be of interest for the community of researchers applying ML models to mental health domain. It is unfortunate that the authors did not respond to the reviewers' concerns and questions. I strongly encourage the authors to improve the paper based on the authors comments and questions and resubmit to a future venue.
Decision: Reject