Sentiment-guided Transformer with Severity-aware Contrastive Learning for Depression Detection on Social Media
Year: 2023

Abstract:
Early identification of depression is beneficial to public health surveillance and disease treatment. There are many models that mainly treat the detection as a binary classification task, such as detecting whether a user is depressed. However, identifying users’ depression severity levels from posts on social media is more clinically useful for future prevention and treatment. Existing severity detection methods mainly model the semantic information of posts while ignoring the relevant sentiment information, which can reflect the user’s state of mind and could be helpful for severity detection. In addition, they treat all severity levels equally, making the model difficult to distinguish between closely-labeled categories. We propose a sentiment-guided Transformer model, which efficiently fuses social media posts’ semantic information with sentiment information. Furthermore, we also utilize a supervised severity-aware contrastive learning framework to enable the model to better distinguish between different severity levels. The experimental results show that our model achieves superior performance on two public datasets, while further analysis proves the effectiveness of all proposed modules.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 114–126 July 13, 2023 ©2023 Association for Computational Linguistics
Early identification of depression is beneficial to public health surveillance and disease treatment. There are many models that mainly treat the detection as a binary classification task, such as detecting whether a user is depressed. However, identifying users’ depression severity levels from posts on social media is more clinically useful for future prevention and treatment. Existing severity detection methods mainly model the semantic information of posts while ignoring the relevant sentiment information, which can reflect the user’s state of mind and could be helpful for severity detection. In addition, they treat all severity levels equally, making the model difficult to distinguish between closely-labeled categories. We propose a sentiment-guided Transformer model, which efficiently fuses social media posts’ semantic information with sentiment information. Furthermore, we also utilize a supervised severity-aware contrastive learning framework to enable the model to better distinguish between different severity levels. The experimental results show that our model achieves superior performance on two public datasets, while further analysis proves the effectiveness of all proposed modules.

heading: 1 Introduction
text: According to the World Health Organization (WHO)1, depression is a common mental disorder and a leading cause of disability worldwide. It is estimated that around 350 million people suffer from depression and over 70% of people do not receive timely treatment (Olfson et al., 2016). Therefore, if depression is detected at an early stage, it could be useful for disease intervention and treatment. In addition, with social media becoming increasingly popular, people often share their feelings and mental states through social media platforms such as
1https://www.who.int/health-topics/depression
Twitter and Reddit, making social media posts critical for mental illness detection, including depression detection (De Choudhury et al., 2013; Skaik and Inkpen, 2020; Chiong et al., 2021).
Recent advances in natural language processing (NLP) have played an increasingly essential role in supporting the analysis of user-generated contents from social media, including mental illness surveillance (Skaik and Inkpen, 2020; Chancellor and De Choudhury, 2020; Ríssola et al., 2021). There are many methods leveraging NLP technologies for automated depression detection. However, existing depression detection methods mainly treat detection as a binary classification task due to the limitation of existing annotated datasets. According to the Beck’s Depression Inventory (BDI) (Beck et al., 1961), not all cases of depression are the same; depression could be classified as: minimal, mild, moderate and severe (see Figure 1). For our methods to be translational, detecting the severity level of depression is expected to be more important for further prevention and treatment (Hollon et al., 2002). For instance, evidence-based psychological therapies could be appropriate for mild or moderate depression (Naseem et al., 2022), while users with severe level might be provided more resource-intensive interventions (Desrochers and Houck, 2014).
Previous psychological studies (Rude et al., 2004; Molendijk et al., 2010) have examined the correlation between sentiment and depression, illustrating that sentiment can reflect the user’s state of mind and help to identify depressed individuals (Babu and Kanaga, 2022; Zhang et al., 2023). For example, a depressed user may post a comment "Today was a really bad day. I had no energy and thought about suicide all day" with a negative sentiment, while non-depressed individuals tend to express a positive or neutral attitude, such as “idk why I’m doing this. I guess I just am. Maybe it will help someone else”. However, existing sever-
114
ity detection methods mainly focus on modeling the semantic information of posts while ignoring the relevant sentiment information. Additionally, traditional classification models with cross-entropy loss treat all severity levels equally, making it challenging to distinguish between closely-labeled categories. For example, ‘moderate’ and ‘severe’ are semantically closer in severity level than ‘mild’ and ‘severe’, making them more difficult to distinguish. To tackle the above challenges, we propose a sentiment-guided Transformer model with severity-aware contrastive learning for depression severity detection. We first leverage post encoders MentalRoBERTa and SentiLARE (Ke et al., 2020) to obtain the semantic as well as sentiment hidden features and then design a sentiment-guided Transformer model to better fuse these two features. For the second challenge, we implement a supervised severity-aware contrastive learning framework, which helps the model adaptively differentiate the weights between different negative samples based on the label information. We validate our method on two depression severity detection datasets. The experimental results show that our model consistently outperforms other baselines, such as DepressionGCN (Naseem et al., 2022) and MentalRoBERTa (Ji et al., 2022). Our main contributions can be summarized as follows:
(1) We propose a sentiment-guided Transformer fusion model to fuse sentiment information into the depression severity detection task. This network
can better combine semantic information and sentiment information compared with existing PLMsbased models.
(2) We implement a supervised severity-aware contrastive learning framework to enable our model to better capture class-specific features and distinguish between different severity levels.
(3) We conduct experiments and further analysis on mainstream public datasets. The experimental results illustrate that our model achieves competitive performance compared with others and each of the proposed modules is effective in depression detection.

heading: 2 Related Work
text: With the fast-growing numbers of social media users, using user-generated posts to detect depression manually is no longer practical. Automated NLP and text mining technologies provide new opportunities for mental health analysis (Ríssola et al., 2021; Zhang et al., 2022).
Early researchers (Islam et al., 2018; Trifan et al., 2020) extracted linguistic features (e.g., Linguistic Inquiry and Word Count (LIWC), Bagof-Words) and statistical features (e.g., n-gram, Term Frequency-Inverse Document Frequency (TF-IDF)) from posts, then employed them in traditional machine learning methods such as SVM (Tadesse et al., 2019), Logistic Regression (Martínez-Castaño et al., 2018) and Random Forest (Cacheda et al., 2019) to detect depression. Recently, deep learning-based models have been used for depression detection, allowing models to automatically capture valuable features without feature engineering. Some works have shown that CNN (Orabi et al., 2018) and RNN (Wu et al., 2020), including Long Short Term Memory (LSTM) (Ghosh and Anwar, 2021) and Gated Recurrent Unit (GRU) (Sekulic and Strube, 2019) are effective. Furthermore, PLMs such as BERT (Owen et al., 2020), RoBERTa (Dinu and Moldovan, 2021), (Ji et al., 2023) are also widely utilized for many mental illness detection tasks due to strong context modelling ability. In particular, Ji et al. (2022) trained the domain-specific language model MentalRoBERTa for mental healthcare, which introduces the domain-related knowledge and outperforms basic PLMs on several mental health datasets. The depression severity identification task is becoming increasingly attractive due to the practical clinical implications, so some studies have emerged
(Losada et al., 2019; Kayalvizhi et al., 2022). Although the severity level is measured on an ordinal scale, current methods treat it as a traditional multi-classification task. Naseem et al. (2022) first reformulate depression identification as an ordinal classification problem via ordinal loss and employ Graph Convolutional Neural Network (GCN) to capture contextual information, which achieves state-of-the-art performance on depression severity datasets.
Sentiment analysis is one of the key technologies in NLP, which aims to analyse people’s sentiments or opinions expressed in texts (usually classified as positive, negative and neutral). Sentiment analysis is widely applied to reviews, social media, newswires, and medical informatics (Yue et al., 2019; Colón-Ruiz and Segura-Bedmar, 2020; Yang et al., 2023b). Recently, PLMs-based models have achieved competitive performances in sentiment analysis. For example, Yin et al. (2020) propose SentiBERT, a variant of BERT model that effectively captures compositional sentiment semantics. Ke et al. (2020) present SentiLARE with a context sentiment mechanism and knowledge integration, which facilitates sentiment understanding.
Contrastive learning is a recent technique for enhancing the performance of models by learning different representations of contrasting samples. The main purpose is to generate negative pairs using data augmentation and minimise the contrastive loss of the positive pairs. Similar methods are transferred to NLP tasks, like ConSERT (Yan et al., 2021) for better unsupervised sentence representation learning. Supervised contrastive learning can leverage label information for representation learning in a supervised setting (Gao et al., 2021) , which improves the ability of the model to differentiate between samples with different labels. Recent work also uses supervised contrastive learning for depression detection tasks (Yang et al., 2022; Wang et al., 2022), illustrating its effectiveness.

heading: 3 Methodology

heading: 3.1 Task definition
text: We focus on detecting the depression severity level of users by analysing posts on social media. Formally, given a post P = {s1, s2, ..., si}, where si is the i-th sentence of the post, our goal is to classify each post into a corresponding depression severity level label y ∈ Y , where Y is the list of increasing depression severity levels. For example, according
to the definition of different datasets described in Section 4, Y is {minimal, mild, moderate, severe} or {not depressed, moderately depressed, severely depressed}. This task can be formalized as a multiclass text classification problem.

heading: 3.2 Depression severity detection model
text: Our model architecture (Figure 2) consists of a post encoder, a sentiment-guided Transformer and a supervised severity-aware contrastive learning component. We will introduce the details of each component as follows.

heading: 3.2.1 Post encoder
text: With the huge success of PLMs in several NLP tasks, PLMs are widely used as sentence encoder for various tasks, including for mental health detection (Zhang et al., 2022). In particular, Ji et al. (2022) trained a domain-specific language model MentalRoBERTa for mental healthcare, which introduces domain-related knowledge and outperforms basic PLMs on several mental health datasets. We exploited MentalRoBERTa as the semantic encoder. Since we require sentiment information to help us to better understand a user’s state of mind, we also adopt the pre-trained language model SentiLARE (Ke et al., 2020) that achieves good performance on various sentiment analysis tasks, as the sentiment encoder to obtain the sentiment hidden representation of each sentence. Specifically, given a post P = {s1, s2, ..., si}, we have obtained i split sentences and each sentence is s = {[CLS], x1, x2, ..., xj}, where [CLS] is a special token for marking the start of the sentence and xj is the j-th word. Then, we feed each sentence sequence into MentalRoBERTa and SentiLARE to get the two embeddings of the [CLS] as the corresponding sentence embeddings. We denote the process as:
Hm = {Hm1 ,Hm2 , ...,Hmi } = Mental_Encoder({s1, s2, ..., si}) (1)
Hs = {Hs1 ,Hs2 , ...,Hsi } = Senti_Encoder({s1, s2, ..., si})
(2)
where Mental_Encoder and Senti_Encoder denote the MentalRoBERTa encoder and SentiLARE encoder respectively; Hi denotes the i-th sentence embedding; Hm and Hs ∈ Rn×d are encoded input embeddings, n denotes the number of sentences, and d is the hidden dimension of the encoder.

heading: 3.2.2 Sentiment-guided Transformer
text: The architecture of the Sentiment-guided Transformer is illustrated in the red-dotted box of Figure 2, which is composed of a stack of N blocks with their own training parameters. Each block contains a Self_Attention module and a Sentiment-guided Co_Attention module. For one social media post, there are two post embeddings generated as inputs {Hm; Hs} for sentiment-guided Transformer.
Firstly, the two embeddings are fed into Self_Attention module separately. This module is the most important component of Google’s Transformer (Vaswani et al., 2017) that learns the sequence’s representation consisting of a multi-head self-attention mechanism (MHA), a feed-forward network and two layers of normalization (Ba et al., 2016) with residual connection (He et al., 2016). The self-attention mechanism can be described as the relationships between a query and a set of key–value pairs to compute an output, where query, key, value and output are vectors. For semantic embedding, given the matrix Hm, we can get a query matrix Q ∈ Rn×d, a key matrix K ∈ Rn×d and a value matrix V ∈ Rn×d. The self-attention
function is expressed as:
Attention(Q,K, V ) = softmax( QKT√
d )V (3)
where n is the length of the post, d is the embedding hidden dimension, softmax denotes softmax operation, √ d is a scaling factor. Q = K = V = Hm.
To enable the model to jointly learn contextual information from different representation sub-spaces, the three matrices Q, K, V are multiplied with the WQi , W K i , W V i respectively, and the MHA with h heads is used. Then, the results of all attention heads are concatenated together, and a weight matrix WO is used to obtain the final output of the encoder:
MHA(Q,K, V ) = concat(head1, ..., headh)W o
(4)
headi = Attention(QW Q i ,KW K i , V W V i ) (5)
where parameter weight matrices WQi ∈ Rd×dh , WKi ∈ Rd×dh , W Vi ∈ Rd×dh and W o ∈ Rd×d, dh = d/h.
After Self_Attention module, we will get the semantic-level output SAm and the sentimentlevel output SAs. In order to obtain contextual sentiment-guided sentence representation, we leverage the Sentiment-guided Co_Attention module to fuse sentiment information with the corresponding semantic information. In contrast to the Self_Attention, the inputs of Sentiment-guided Co_Attention are from two different embeddings.
We switch the query Qs from the Self_Attention output SAs (the sentiment-level post hidden representation), the key Km and the value V m from SAm (the semantic-level post hidden representation). The calculation process is the same as the multi-head self-attention mechanism:
Co_A(Qs,Km, V m) = softmax( Qs(Km)T√
d )V m
(6)

heading: 3.2.3 Supervised severity-aware contrastive learning
text: After all the Sentiment-guided Transformer blocks are computed, we obtain the final block’s outputs, including the semantic hidden feature representation om = {om1 , om2 , ..., omi } and the sentiment-guided hidden feature representation os = {os1, os2, ..., osi}. Considering the different importance of each sentence to the post, we leverage Soft_Attention mechanism (Yang et al., 2016) to get semantic-level post embedding pm and sentiment-level post embedding ps:
ui = tanh(Wpoi + bp) (7)
αi = exp(ui Tup)∑ i exp(ui Tup) (8)
p = ∑
i
αioi (9)
Where Wp is a weight matrix, bp is a bias vector, and up is a post context vector which is randomly initialized. Then, we concatenate these two embeddings pf = concat(pm, ps) as the fusion post embedding that retains contextual semantic and sentiment information.
Recent works (Gao et al., 2021; Yang et al., 2022; Wang et al., 2022) utilize supervised contrastive learning to capture the class-specific information, enabling sentences with the same label to be cohesive and those with different labels to be mutually exclusive. In addition, following (Suresh and
Ong, 2021), we design a supervised severity-aware contrastive learning framework to help the model differentiate the more difficult negatives.
To ensure that there are at least two samples of a class in one batch for contrastive learning, according to (Yang et al., 2022), we copy each post embedding to produce augmented samples with the batch size will changing from Nb to 2Nb. The traditional supervised contrastive loss is given by:
LCL =
2Nb∑
i=1
− 1|J | ∑
j∈J log
exp(hfi · h f j /τ)∑
k∈I\i exp(h f i · h f k/τ)
(10)
hfi =L2−Normaliz(Dense_layer(p f i )) (11)
where J = {j : j ∈ I, yj = yi ∧ j = i}, i is the sample index, hfi is the L2-normalized vector of post hidden representation obtained from an encoder (dense layer in Figure 2), and τ indicates the temperature hyper-parameter.
We can see that traditional supervised contrastive learning treats both negative samples (the comparison in Figure 2) the same. However, in the depression severity detection task, the labels contain an inherent ordinal nature. For example, the sample with the label severe is semantically closer to the sample with the label moderate than the one with the label mild in the representation space. Therefore, to distinguish better the closely-labeled examples, we introduce supervised severity-aware contrastive loss for adaptively weighting the samples based on the relationships between the severity labels.
In order to obtain the adaptive weights for different comparison samples, following (Suresh and Ong, 2021), we use a dual-model strategy as shown in Figure 2. The post hidden representations are fed into a weighted layer that is same as the classifier layer, and the output is optimised using crossentropy loss Lw. Then, by using softmax function to obtain prediction probabilities, we can get the confidence scores of the sample that are classified to different classes:
hhfi =weighted_layer(h f i ) (12)
wi = {wi,c}Cc=1 = exp(hhfi,c)∑C c=1 exp(hh f i,c)
(13)
where C is the number of the classes, wi,c denotes the confidence that i-th sample is classified to class c, hhfi,c is the c-th value of hh f i .
The wi is taken as weighting vector to weight the pair-wise similarity values, and the severity-aware contrastive loss is as follows:
Ls =
2Nb∑
i=1
− 1|J | ∑
j∈J log
wi,yi exp(h f i · h f j /τ)∑
k∈I\iwi,yk exp(h f i · h f k/τ)
(14)
Where wi,yi represents the weight between the i-th sample and its corresponding severity label yi. We can find that the loss assign higher weights to confusable negatives based on the model’s confidence scores.
We combine the training of depression severity detection and the supervised severity-aware contrastive learning task in a multi-task learning framework. The framework is jointly optimised using the following training losses: (1) the classification output’s cross-entropy loss Lc; (2) the severity-aware contrastive loss Ls; (3) the weighted layer output’s cross-entropy loss Lw. We formalized the overall training loss as follows:
Loverall = α(Lc + Lw) + (1− α)Ls (15)
where α is a tunable weight parameter.

heading: 4 Experiments and analysis

heading: 4.1 Datasets and Implementation Details
text: To evaluate and compare our method with related works, we conducted experiments on two publicly available depression severity detection datasets: DsD (Naseem et al., 2022) and DepSign (Kayalvizhi and Thenmozhi, 2022). The statistical details and some examples of these two datasets are shown in Table 1 and Table 2.
Depression severity Dataset (DsD): The dataset is collected from the social media platform Reddit and annotated according to the disorder annotation scheme (Mowery et al., 2015), where each post is labeled with one of four severity levels: {minimal, mild, moderate, severe} following the depression rating scale by (Zimmerman et al., 2013).
Depression signs detection Dataset (DepSign): The dataset is from shared task and also collected from Reddit. The posts belong to the following subreddits groups, "r/depression, r/mental health". The data is annotated by two domain experts according
to the signs of depression, and each post is labeled with the severity level of signs of depression: {not depressed, moderate, severe}. Since the shared task does not make the test set publicly available, we merge the train set and valid set as the whole dataset and then construct train/valid/test sets.
We perform pre-processing steps on the datasets to clean the texts. We use the ekphrasis2 library that is a processing tool for texts from social media to correct spelling, remove hashtags and normalize some special words (URLs, emails, digits, user names, elongated words, etc.). We further use emoji3 to replace emoticons and emojis with the associated words, then leverage NLTK4 to split the post.
We use grid search to explore the parameters. The parameter settings are as follows: the number of Sentiment-guided Transformer blocks N ∈ {1, 2, 4, 8}, the number of hidden dimension H ∈ {128, 256, 512, 1024}, the number of heads h ∈ {2, 4, 8, 16}, dropout δ ∈ {0.1, 0.2, 0.3, 0.4}, learning rate lr ∈ {5 × 10−4, 1 × 10−4, 5 × 10−5, 1 × 10−5}, the optimization O ∈ {Adam,AdamW}, the batch size b ∈ {8, 16, 32}, tunable weight parameter α ∈ {0.2, 0.5, 0.8}. The optimal hyperparameters used for both datasets are N = 4, H = 1024, h = 8, δ = 0.2, lr = 5 × 10−5, O = AdamW, b = 16, α = 0.5. We train the model on an Nvidia Tesla V100 GPU with 16GB of memory for 100 epochs and apply early stopping with patience of 20 epochs. We use Huggingface’s Transformers library5 as pre-trained language models. For consistency, we use the same experimental settings and 10-fold cross-validation for all models. Our reported results are averaged across all folds on both datasets.

heading: 4.2 Evaluation metrics
text: To better evaluate our model from the point of view of depression severity assessment, we choose the evaluation metrics Graded Precision (GP), Graded Recall (GR) and FScore used by (Naseem et al., 2022; Gaur et al., 2019). They altered the formulation of False Positives (FP) and False Negatives (FN). FP is changed as the proportion of the number of times the predicted severity level (lp) is higher than the actual severity level (la). FN is the propor-
2https://pypi.org/project/ekphrasis/ 3https://pypi.org/project/emoji/ 4https://www.nltk.org/ 5https://huggingface.co/models
tion of the number of times the predicted severity level lp is lower than the actual severity level la.
FP =
∑NT i=1 I(l p i > l a i )
NT , FN =
∑NT i=1 I(l a i > l p i )
NT (16)
where Δ(lpi , l a i ) denotes the difference between predicted lp and actual la depression severity level for post Pi, NT is the size of test set.
The precision and recall are reformulated as GP and GR since FP and FN contain the comparison between lp and la:
GP = TP
TP + FP ,GR =
TP
TP + FN (17)
FScore = 2GP ·GR GP +GR
(18)
where TP is the True Positives, FScore is the F1score calculated by GP and GR.

heading: 4.3 Overall results
text: We compare our model with recent deep learningbased methods to show the overall performance. Table 3 shows the experimental results of our model and the baselines. The comparative approaches are as follows:
• DepressionNet (Hamad et al., 2021) : a text summarization model for depression detection.
• DepressionGCN (Naseem et al., 2022) : a recently proposed depression severity detection method combining GCN and Bi-LSTM structures.
• BERT (Devlin et al., 2019) : a widely used basic pre-trained language model.
• MentalBERT and MentalRoBERTa (Ji et al., 2022) : two domain specific PLMs pre-trained on mental health corpus. Moreover, MentalRoBERTa is the current state-of-the-art model on this task.
As shown in Table 3, we notice that PLMs-based models, especially mental health-related PLMs, offer a general advantage with over 85% performance and outperform DepressionGCN on both datasets, which shows the advantage of context modelling for PLMs. Compared with these baselines, our model achieves a new state-of-the-art on both datasets (87.5% / 89.1% of FScore), with 0.8% and 1.9% improvements over the MentalRoBERTa.
In summary, the results prove that the proposed model achieves better performance on the depression severity detection task.

heading: 4.4 Ablation studies
text: In this section, we perform ablation studies of our model on both DsD and DepSign datasets to illustrate the effectiveness of each module of our proposed model. Specifically, we conduct module ablation experiment, sentiment fusion ablation experiment and contrastive ablation experiment.

heading: 4.4.1 Module ablation
text: To evaluate the contribution of different modules in our model, we perform a module ablation study. ‘w/o contrastive’ denotes the model without severity-aware contrastive learning module. ‘w/o SGT’ denotes the model without sentiment-guided Transformer module. ‘w/o all’ removes all modules while keeping MentalRoBERTa encoder. The results are shown in Table 4. We note that our complete model achieves the best performance while the performance degrades when components are removed. In detail, e.g., with DepSign, the performance of FScore decreases by 0.6% when contrastive learning is removed and the FScore drops by 1.5% when SGT is removed. The model without SGT consistently performs worse than the model without contrastive, which indicates the importance of sentiment information in depression detection. If the two modules are both removed, the model degrades to the basic MentalRoBERTa encoder and the FScore drops by 1.9%. The results above demonstrate that each proposed module works effectively and the performance is improved when these components are combined together.

heading: 4.4.2 Sentiment fusion ablation
text: In order to demonstrate the effectiveness of our proposed sentiment fusion strategy (sentiment-guided Transformer), we compare it with other methods. ‘SA(m)’ means only using hidden representations (m) from MentalRoBERTa encoder combining Self_Attention module. ‘SA(m)+s’ denotes the
concatenation that the representations from SA(m) are concatenated with a sentiment label, which is a common fusion strategy to add external features. We leverage SenticNet 6 (Cambria et al., 2020) to obtain the sentiment label of each sentence and the concatenation is implemented at the classification layer. ‘SA(m)+SA(s)’ denotes the concatenation of semantic representations from SA(m) and sentiment representations from SA(s). ‘SGT(m+s)’ denotes our sentiment-guided Transformer. In addition, we also explore whether emotion information is valuable for depression severity detection, thus we fuse semantic information and emotion information using the emotion-guided Transformer module (EGT). We use the pre-trained model j-hartmannroberta 6 as the post emotion encoder, which is trained on various emotion classification datasets and achieves good performances.
The results are shown in Table 5. We notice that adding sentiment information improves the performance of the model regardless of the fusion strategy used. In particular, our proposed sentiment-guided Transformer performs better than others since the fusion structure can better integrate semantic and sentiment hidden information, while only using label information is not sufficient. We observe that the emotion-guided Transformer also improves the performance on two datasets, by only 0.1% FScore on DsD, and the improvements are both smaller than the sentiment-guided Transformer. A possible reason is that emotion classification is a more difficult task than sentiment classification, resulting in lower accuracy of sentiment classification (66% F1-score on emotion task than 80%-90% F1-score on sentiment task), which will bring the noise to the depression detection.

heading: 4.4.3 Contrastive learning ablation
text: We also conducted a contrastive learning ablation experiment. ‘Base’ means our proposed model
6https://huggingface.co/j-hartmann/emotion-englishdistilroberta-base
without a contrastive learning module. ‘+CL’ denotes the base model with supervised contrastive learning. ‘+SCL’ is the base model with our supervised severity-aware contrastive learning. Table 6 shows the performance comparison of the case study. The results demonstrate that adding supervised contrastive learning improves detection performance. In detail, on both datasets, the FScore of using severity-aware contrastive learning is 0.3% higher than that of traditional supervised learning, which proves the effectiveness of our designed severity-aware contrastive learning.

heading: 4.5 Case study
text: To validate the effectiveness of our model, we provide three cases from the test results of MentalRoBERTa and our model in Appendix. A for further investigation. In the first case, MentalRoBERTa fails to detect the depression severity, and classifies it as mild. Meanwhile, our model successfully predicts the label due to the use of sentiment information (positive). In the second case, both models are able to make accurate predictions, indicating semantic information’s importance, and sentiment information does not bring additional errors. In the third case, although our model fails to detect the severity, the predicted label (moderate) is closer to the true label (severe) than the prediction of MentalRoBERTa (mild), which also reveals that the negative sentiment in this post is helpful for predicting depression severity.

heading: 4.6 Ethical considerations
text: Although the datasets are publicly available, we also follow strict ethical protocols (Benton et al., 2017) of sharing datasets. To prevent misuse and protect user privacy, all examples shown in this paper have been obfuscated and paraphrased according to the moderate disguise scheme (Bruckman, 2002).

heading: 5 Conclusion
text: In this paper, we propose a depression severity detection model based on sentiment-guided
Transformer and severity-aware contrastive learning. The sentiment-guided Transformer can efficiently fuse semantic and sentiment information from user’s posts. Then we use a severity-aware contrastive learning framework, which can fully leverage label information for capturing severityspecific features and helping the model distinguish closely-labeled categories. The experimental results show that our model performs better than recent models and achieves superior performance on two publicly available datasets. Further analysis determines the effectiveness of all proposed modules.
Limitations
There are also several potential limitations. First, the severity of the identification of depression is subjective, which inevitably leads to annotation bias, and we cannot verify the actual diagnosis. Thus, our model is not intended to be used as a psychiatric diagnosis tool but an estimate of depression level for users, which can be utilized to direct intervention and treatment for non-clinical use. Second, the datasets used are collected from a single social media platform (Reddit) and are imbalanced. To train a more robust and effective model, future works should collect more precise data from other social media and increase collaboration with clinicians to ensure the quality of the data. Despite these limitations, we believe that our work will facilitate depression severity detection. In future work, we would like to explore other fusion strategies to better integrate various information and assess the performance of ChatGPT (Yang et al., 2023a) on the depression severity detection task.

heading: Acknowledgements
text: This work is supported in part by the Alan Turing Institute and funds from MRC MR/R022461/1 and the project JPNP20006 from New Energy and Industrial Technology Development Organization (NEDO).

heading: A Case study
text: The three cases from the test results of MentalRoBERTa and our mode are shown in Table 7.

heading: Future Work
text: To train a more robust and effective model, future works should collect more precise data from other social media and increase collaboration with clinicians to ensure the quality of the data. Despite these limitations, we believe that our work will facilitate depression severity detection. In future work, we would like to explore other fusion strategies to better integrate various information and assess the performance of ChatGPT (Yang et al. , 2023a) on the depression severity detection task