SERCNN: Stacked Embedding Recurrent Convolutional Neural Network in Depression Detection on Twitter

1 INTRODUCTION . Depression is a serious yet common mental disorder that affects more than 264 million people worldwide . The number is projected to grow amid the war against the pandemic of COVID-19 . Unlike the usual mood fluctuations and emotional responses , the long-lasting sadness , emptiness , or irritation in one ’ s day-to-day life , accompanied by somatic and cognitive changes that heavily disrupt an individual ’ s capacity to function normally . Hence , depression is often associated with suicide at its worst . In the latest Word Health Statistics 2021 ( World Health Organization , 2021 ) , 28 % increases the suicide rate in the United States in this period , which may result from the increasing pressure on the loss of loved ones and incomes , city lockdowns , social distancing , and the reduction of human interactions . Existing screening methods , such as psychometric self-reporting questionnaires and clinical interviews , are expensive , not scalable , not reachable to many , and have limitations . For example , the Diagnostic and Statistical Manual of Mental Disorders ( DSM-5 ) ( American Psychiatric Association et al. , 2013 ) requires patients to be involved and disclose truthful information during clinical interviews . Besides that , people who suffer from depression tend to be unaware of their condition because there is no specific feedback from the body like common physical injuries do . To get diagnosed with depression , one must first consult a doctor or mental health professional . Up to date , global provision and service for identifying , supporting , and treating mental health issues are insufficient despite the disruption of essential health services , community mistrust , and fears of COVID-19 infections . The lack of mental health awareness in the Southeast Asia Region is another contributing factor in the sky-high figure of depression . Social media has become part and parcel of everyday life . It has become a valuable source of information that led to a new research direction for detecting depression . Here , social media data can serve as a lifelog that records users ’ activities in the text , image , audio , and video . The enormous amount of time series first-person narrative posts can provide insights about one ’ s thoughts , feelings , behavior , or mood for some time , allowing an unintrusive way to study patients ’ conditions before an actual clinical diagnosis is made . Motivated by previous studies that demonstrated great potentials for using social media posts to monitor and predict one ’ s mental health status , this paper utilizes natural language processing and machine learning techniques on social media data to predict one ’ s risk of depression . We propose Stacked Embedding Recurrent Neural Network ( SERCNN ) , which can effectively learn the representation from dirty and unstructured social media text without the need to pursue multimodal learning . Instead of using the recent transformer model , we suggest a meta-embedding approach for our feature extraction . We utilize two widely available yet low dimensional pretrained GloVe embeddings trained on different domain datasets to provide a more robust and accurate representation of the dirty and unstructured social media text that often contains typos . Our SERCNN can achieve a state-of-the-art performance without using all the social media posts in the user posting history , providing a perspective on the number of posts required to make a reliable prediction . It is worth pointing out that there is still room in social media text that is worth exploring . The contributions of this paper can be summarized as below . 1 . SERCNN can retrieve a more robust and richer representation of the social media text by stacking multiple low dimensional pretrained embeddings . 2 . Provides insights on how the different number of posts influence the prediction model ’ s accuracy . 2 RELATED WORK . In this section , we provide an overview of the current research landscape and how previous works convert information extracted from social media posts into more representative features that can be used to identify depressed individuals . Depression detection on social media Evidence from various published works has shown the viability of using social media data to predict depression and other mental disorders . The seminal work by Holleran ( 2010 ) has inspired natural language processing ( NLP ) researchers to identify potential markers of mental disorder from social media posts . In contrast to the 14 days of observation stated in DSM-5 , Hu et al . ( 2015 ) and Tsugawa et al . ( 2015 ) . empirically suggested that features extracted over two months are sufficiently be used to identify one ’ s depression condition . Coppersmith et al . have presented a. novel method to collect ground truth on social media related to depression and PTSD . The dataset is then being used for the shared task of The Workshop on Computational Linguistics and Clinical Psychology ( CLPsych ) ( Coppersmith et al. , 2015 ) . A similar workshop , CLEF eRisk ( Losada et al. , 2017 ) , focuses on the early depression symptoms discoveries on social media . Workshop tasks such as depression detection ( CLPsych 2015 ( Coppersmith et al. , 2015 ) ) and early detection of signs of depression ( CLEF eRisk 2017 ( Losada et al. , 2017 ) ) have generated significant numbers of novel approaches in identifying depression on social media . Feature representation for depression classification Modeling feature representations is a crucial task in machine learning ; features that are not discriminative and representative will result in poor and faulty model performance . Hence , earlier research works are mainly focused on feature extraction techniques . Choudhury et al . ( 2013 ) ; Tsugawa et al . ( 2015 ) have found that depressed users tend to be emotional . Wang et al . ( 2013 ) found that using sentiment analysis in depression detection can achieve about 80 % accuracy . Tsugawa et al . ( 2015 ) ; Resnik et al . ( 2015 ) extracted topics distribution with Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) to differentiate depressed individuals from the healthy controls . Researchers also extracted features based on criteria stated in the industrystandard - Diagnostic and Statistical Manual of Mental Disorders ( DSM ) , such as the insomnia index derived from the user posting time . Linguistic Inquiry and Word Count ( LIWC ) ( Tausczik & Pen- nebaker , 2010 ) is a widely used word matching-based feature extraction tool that builds on top of Pennebaker et al. ’ s findings ( Rude et al. , 2004 ; Gortner et al. , 2006 ) decades ago . ( Choudhury et al. , 2013 ; Shen et al. , 2017 ) show that depressed users tend to have high self-attentional focus , increased medicinal concerns , and increased expression of religious thoughts . These findings are aligned with the Rude et al . ( 2004 ) work where depressive indicators can be found on the human-generated content . However , LIWC and word matching-based sentiment analysis approaches are often ineffective . The nature of social media text being dirty and unstructured often required extensive data cleaning and preprocessing for the tools to act as intended . Recent works ( Gui et al. , 2019a ; b ; Rao et al. , 2020 ) have shown that using the word embedding and deep learning models are effective than the handcrafted features with minimal effort of text preprocessing . We observed that most works that employed deep learning models tend to use hierarchical document modeling to generate their input representation . This raises the question of whether depression classification using social media data actually takes advantage of the hierarchical document modeling . 3 STACKED EMBEDDING RECURRENT CONVOLUTIONAL NEURAL . NETWORK ( SERCNN ) Overview Our goal is to improve the performance of depression classification on social media by learning a much robust user representation . Given that our dataset , d , consists of N1 number of social media users , u , where each user ’ s N2 number of social media posts , p , within a month were collected , and each post has N3 words , w , we denote the dataset as d = { u1 , ... , uN1 } , social media user as u = { p1 , ... , pN2 } , and the post as p = { w1 , ... , wN3 } . We propose Stacked Embedding Recurrent Convolutional Neural Network ( SERCNN ) which is made up of a Stacked Embedding ( SE ) and Lai et al . ( 2015 ) Recurrent Convolutional Neural Network ( RCNN ) . The overall architecture of SERCNN is simple , consisting a SE layer , single directional LSTM , a max-pooling layer and an output layer which is a fully connected layer with sigmoid function , as visualized in Figure 1 . 3.1 FEATURE EXTRACTION WITH STACKED EMBEDDING . Unlike previous works , we model our user representation by firstly concatenating N2 social media posts in chronological order . For a random user ui , the concatenated text representation can be formulated into : ui = { p1 + ... + pN2 } ( 1 ) ui = { w11 + ... + w1N3 + ... + w N2 N3 } ( 2 ) This concatenated post can be viewed as a single ” monthly diary ” , journal or large document that characterizes the user . Then , we extract the distributed text representation of each word with the SE , which is an ensemble embedding , commonly known as the Meta-embedding technique . The concept of Meta-embedding was first introduced by Yin & Schütze ( 2016 ) to utilize and learn the meta of existing well-trained pretrained embeddings and extend the vocabulary . Since different pretrained embeddings were trained on different datasets , each embedding can now complement each other allowing an improved vocabulary coverage and reducing out-of-vocabulary words . As the name suggests , SE is formed by stacking the collection of dense vectors ( pretrained weights ) E = { E1 , ... , EN4 } extracted from N4 number of pretrained embeddings included : ESE = { E1 + ... +EN4 } ( 3 ) where the embedding vector , x , of a given word , w can be obtained via : x = ESE ( w ) ( 4 ) The vocabulary of SE , VSE , is now considered as the vocabulary union of the N4 pretrained embeddings , resulting in a more extensive vocabulary than a single embedding : VSE = N4⋃ n=1 Vn ( 5 ) In this study , our SE is the simple stacked vector made up of two pretrained GloVe embeddings ( Pennington et al. , 2014 ) trained on Twitter and Wikipedia 2014 + Gigaword 5 datasets , respectively : 1 . GloVe Twitter ( 25 dimensions ) trained using global word co-occurrences information by Pennington et al . ( 2014 ) under an uncased setting , using 2 billion tweets , with 27 billion tokens . The resulting vector consists of 1.2 million vocabularies learned from the corpus . 2 . GloVe Wikipedia 2014 + Gigaword 5 ( 100 dimensions ) ( Pennington et al. , 2014 ) , similar to the GloVe Twitter embedding , this embedding is trained using global word co-occurrences information but using a different corpus , which is the combination of Wikipedia 2014 and Gigaword 5 datasets . There are approximately 400 thousand words in the vocabulary . 3.2 REPRESENTATION LEARNING AND DEPRESSION CLASSIFICATION WITH RCNN . Recurrent neural network ( RNN ) is capable of capturing contextual information over a long sequence . However , the RNN model favors the later words than words in the earlier sequence . In depression detection on social media , where posts are collected over a time interval , we are interested in identifying words throughout that period , rather than just words that occurred later . Lai et al . ( 2015 ) Recurrent Convolutional Neural Network ( RCNN ) overcomes the limitation of the existing RNN model by incorporating a max-pooling layer to extract key features from both the embedding features and the context learned . The max-pooling layer reconsiders embedding features instead of just the context learned in the conventional RNN-based setting , selecting the important features for the classification task . The overall architecture of our RCNN model is visualized in Figure 3 . Instead of using bidirectional Long Short Term Memory ( LSTM ) as described in Lai et al . ( 2015 ) , we use a generic single Forward LSTM to learn the context , c , from the embedding vector , v. With j refers to the jth social media post and k refers to the kth number of words in the post , we can formulate the context for a given word : ( c ( wjk ) , h ( w j k ) ) = LSTM ( c ( wjk−1 ) , h ( w j k−1 ) , v ( w j k ) ) ( 6 ) where |c| ∈ R and h is the output vector . The output of the LSTM is then concatenated with the embedding features , forming an extended context vector : yjk = [ c ( w j k ) ; x j k ] ( 7 ) A max-pooling layer is applied after the RCNN representation of the user is computed : ỹ = L max l=1 yl ( 8 ) , where the total number of words , L , can be calculated by multiplying the total number of social media posts , N2 , by the number of words in each post , N3 . A fully connected layer is then used to discriminate the max-pooled context with a sigmoid function , presenting the classification output , ŷ , as probabilities : ŷi = p ( ỹ | ui ) = 1 1 + exp− ( Wcui+bc ) ∈ [ 0 , 1 ] ( 9 )

Summary: This paper develops a SERCNN, which consists of stacked embeddings and Recurrent CNN (RCNN), for depression detection from Twitter text. The key idea of stacked embeddings is to concatenate two embeddings, based on two different word embeddings models (pre-trained on Twitter and Wikipedia copora), into a single embedding vector, which is fed into RCNN. Experimental results on the Twitter Depression dataset show reasonable performance when trained on 10 posts from each user, which was further improved when trained on more data.