Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis (2024)

Abstract: In the current environment, psychological issues are prevalent and widespread, with social media serving as a key outlet for individuals to share their feelings. This results in the generation of vast quantities of data daily, where negative emotions have the potential to precipitate crisis situations. There is a recognized need for models capable of efficient analysis. While pre-trained language models have demonstrated their effectiveness broadly, there’s a noticeable gap in pre-trained models tailored for specialized domains like psychology. To address this, we have collected a huge dataset from Chinese social media platforms and enriched it with publicly available datasets to create a comprehensive database encompassing 3.36 million text entries. To enhance the model’s applicability to psychological text analysis, we integrated psychological lexicons into the pre-training masking mechanism. Building on an existing Chinese language model, we performed adaptive training to develop a model specialized for the psychological domain. We evaluated our model’s performance across six public datasets, where it demonstrated improvements compared to eight other models. Additionally, in the qualitative comparison experiment, our model provided psychologically relevant predictions given the masked sentences. Due to concerns regarding data privacy, the dataset will not be made publicly available. However, we have made the pre-trained models and codes publicly accessible to the community via: https://github. com/zwzzzQAQ/Chinese-MentalBERT.

heading: 1 Introduction
text: Mental illnesses, particularly depression, impose a considerable strain on global societies. The World Health Organization reports that approximately 3.8% of the global population suffers from depression (Organization et al., 2023). Notably, the incidence of depression in China accounts for as high
∗*Corresponding author: guanghui.fu@inria.fr
as 6.9% of the prevalence (Huang et al., 2019). Individuals experiencing emotional distress often resort to passive coping mechanisms and seldom seek professional help (Rüsch et al., 2005). Traditional channels for emotional crisis intervention, such as hotlines and psychological clinics, are not designed to proactively identify individuals facing emotional challenges (Organization et al., 2014). Moreover, the resources for such interventions are frequently inadequate. The stigma associated with mental illness has led many to use social networks as a primary outlet for expressing their emotional struggles (Primack et al., 2017). Platforms like X (Twitter), and Sina Weibo in China serve as venues for individuals to share their feelings and opinions in real time, with posts often providing immediate insights into one’s daily experiences and emotional states (De Choudhury et al., 2013). Within specific topics or hashtags on social media, there is a pronounced focus on the expression of negative emotions, with some users displaying evident suicidal tendencies (Robinson et al., 2016). This situation underscores the critical necessity to develop tools aimed at enabling the early detection of such distress signals and implementing timely intervention strategies (Coppersmith et al., 2018).
Pre-trained language models (PLMs), such as BERT (Devlin et al., 2018), have demonstrated remarkable success across a variety of language tasks and have seen extensive application (Koroteev, 2021). Recently, the development of large language model (LLM) technology has garnered global interest, notably within the psychology sector, where a plenty of exploratory applications have been initiated (He et al., 2023). However, according to research by Qi et al. (2023), in comparison to supervised learning methods, LLMs are yet to fully addressed the complexity of psychological tasks. Thus, the development of PLMs specifically targeting for supervised learning still crucial, particularly for the specific domains.
10574
To address the lack of large-scale PLMs tailored for specific applications in Chinese community mental health, we introduce the Chinese MentalBERT, a pre-trained language model specifically designed for psychological tasks. To the best of our knowledge, it is the first PLM developed specifically for the mental health field in Chinese. In this study, we employ a domain-adaptive pre-training model (Cui et al., 2021), and introduce a novel lexicon guided masking mechanism strategy based on the Chinese depression lexicon. We conduct four Chinese mental health datasets from social media and public dataset, including over 3.36 million data items for domain-adaptive pre-training. This lexion guided masking mechanism strategically biases the learning process towards vocabulary crucial for the intended application, enhancing the model’s relevance and effectiveness in its target domain. We evaluated the performance of our model on six mental health-related public datasets, including: two cognitive distortion classifications, one suicide risk classification, and three sentiment analysis tasks. The results demonstrate that our model outperforms eight comparison methods across six public datasets. Due to privacy issues, we cannot open the data, we have made all the training code and models publicly available to support research in Chinese mental health.

heading: 2 Related work
text: Social media has been used to identify signs of needing medical or psychological support (Keles et al., 2020). Recent studies show that mental health analyses on social media are using NLP technologies to capture users’ mental health states (Calvo et al., 2017). With the advent of BERT (Devlin et al., 2018), studies have been carried out on using this technology to assess suicidal tendencies and identify depressive tendencies (Wang et al., 2019; Ambalavanan et al., 2019; Matero et al., 2019). Yang et al. (2022) leveraged knowledge graph method to screen high-suicide risk comments within online forums and explored various attributes such as time, content, and suicidal behavior patterns by analyzing these comments. Fu et al. (2021) proposed a distant supervision method to develop an automated method capable of categorizing users into high or low suicide risk categories based on their social media comments. This model serves as an early warning system to aid volunteers in preventing potential suicides among
social media users. On the other hand, many studies focus on domain-adaptive pretraining in specific fields to pursue better domain performance. Chalkidis et al. (2020) systematically examined methods for adapting BERT to the legal field. They gathered 12 GB of varied English legal text from public sources and achieved improved performance compared to the baseline on three end-tasks. Lee et al. (2020) pretrained the BERT model with a domain-specific regimen on extensive biomedical corpora, leading to improved performance compared to the original BERT across various text mining tasks in the biomedical field.
The two most closely related works to our study are the research in the domain of mental health data analysis. Ji et al. (2022) implemented BERT within the mental health sector by creating a targeted dataset from Reddit, resulting in the development of a model known as MentalBERT. Subsequently, Aragon et al. (2023) introduced a dualdomain adaptation process for language models pretraining, which involves firstly adapting the model to the social media text and then to the mental health domain. Throughout both stages, the integration of lexical resources played a critical role in directing the language model’s masking procedure, thereby ensuring a heightened focus on vocabulary associated with mental disorders. Currently, there is no PLM customized for the Chinese mental health domain, and unique challenges exist within the Chinese field. To bridge this gap, we collected around 4 million data from social media for domain adaptive pre-training in the Chinese mental health domain. The guided masking mechanism based on the domain lexicon can help the model focus on the domain-related context.

heading: 3 Pretraining Corpus
text: • Comment from “Zoufan” Weibo treehole (ZouFan, 2023): “Zoufan” Weibo accounts are often likened to digital confession booths or “tree holes”, where users predominantly express negative emotions. Some users have even expressed suicidal intentions on it. By 2020, we had collected roughly 2.34 million comments from 351,069 users. However, from 2021 onwards, data collection has been halted due to platform restrictions.
• Weibo Depression “Chaohua” (super topic on Weibo) (Depression, 2024): it functions as a
specialized sub-forum intended for communication and information exchange among individuals dealing with depression or those interested in learning more about it. This digital space enables users to share personal experiences, discuss treatment options, provide mental health resources, and offer insights and understandings related to depression. In total, we collected 504,072 posts from 69,102 users.
• Sina Weibo Depression Dataset (SWDD) (Cai et al., 2023)1: this dataset represents a comprehensive collection of depression-related data from Sina Weibo, including complete user posting histories up to January 1, 2021. It encompasses user profiles and their entire post history, focusing exclusively on depressionrelated content. Our analysis leveraged only depression-related user and posts and included data from 3,711 users and a total of 785,689 posts.
• Weibo User Depression Detection Dataset (WU3D) (Wang et al., 2020)2: WU3D contains enriched information fields, including posts, the posting time, posted pictures, the user gender, etc. This dataset is labeled and further reviewed by professionals. We use the depressed user’s posts data including 10,325 users and 408,797 posts.
Ultimately, our corpus comprised over 4.04 million posts/comments from 434,207 users. After
1https://github.com/ethan-nicholas-tsai/ DepressionDetection
2https://github.com/aidenwang9867/ Weibo-User-Depression-Detection-Dataset
thoroughly cleaning the dataset by removing excessively short and meaningless sentences, we compiled 3,360,273 data. This cleaned dataset served as the pre-training set for our model.

heading: 4 Methods

heading: 4.1 Basic pre-training language model: Chinese-BERT-wwm-ext
text: The proposed Chinese MentalBERT is a domain adaptive pretrained version of Chinese-BERTwwm-ext (Cui et al., 2021). Domain-adaptive pretraining has been proven to be effective (Gururangan et al., 2020). The continual pretraining can benifit for the targeted downstream domain (Jin et al., 2022). The original pretrained model acquires knowledge from general domains. Our adaptive pre-training process enhances the model’s ability to perform tasks specifically within the Chinese comunity mental health domain. ChineseBERT-wwm-ext integrates a “Whole Word Masking” (WWM) strategy in its pre-training phase (Cui et al., 2021). Unlike the original BERT (Devlin et al., 2018), which typically masks only a single English word, Chinese-BERT-wwm-ext masks to
mask entire Chinese words. This approach is due to the lexical differences between Chinese and English: English uses single words to convey meanings, while Chinese relies on compound words composed of multiple characters for complete concepts. Masking single characters in Chinese leads to incomplete meanings, obstructing the model’s learning of the language’s structure. The pretraining data is comprised of Chinese Wikipedia dump3, alongside an extensive collection of additional data including encyclopedic content, news articles, and question-answering websites, encompassing a total of 5.4 billion words.

heading: 4.2 Masking mechanism guided by depression lexicon
text: As previously discussed, the masking mechanism plays a pivotal role in the pre-training of language models. The pretrained model in general domain such as BERT and Chinese-BERT-wwm-ext, typically employ random masking. This technique involves selecting random words within a sentence to be masked, challenging the model to predict these hidden words based on their context. Although random masking is a proven method for enhancing a model’s contextual understanding, the development of knowledge-guided masking strategies represents an advanced step towards crafting domain-specific language models (Tian et al., 2020; Shamshiri et al., 2024).
To better tailor our model to the specific needs of the mental health domain, we implemented a guided masking strategy utilizing a depression lexicon. This approach begins by identifying whether the pre-training text contains lexicon words; if so, these words are masked for prediction training. Should the proportion of text occluded fall below 20%, we augment the masked selection with additional, randomly chosen words. It’s important to note the distinct strategies required for word guidance and masking in English versus Chinese texts. While masking in word-level suffices in English, Chinese requires word segmentation to mask compound words accurately, ensuring complete concepts are expressed and understood by the model.
Our research investigates a lexicon-guided masking mechanism. The depression lexicon is developed by Li et al. (2020). This lexicon is derived from a labeled dataset of depression-related posts on Weibo, comprising 111,052 posts from 1,868
3https://dumps.wikimedia.org/zhwiki/latest
users, including both depression-related and nondepression-related content. The lexicon construction employs 80 seed words to establish semantic associations between these seeds and potential candidate words, forming a semantic association graph. The label propagation algorithm (LPA) is then applied to automatically assign labels to new words within this graph. This enriched dictionary serves as a input for machine learning algorithms, improving the performance to detect a test subject’s depressive state.

heading: 5 Experimental Settings

heading: 5.1 Dataset
text: The proposed pretrained model underwent evaluation on six public datasets in the mental health domain, including sentiment analysis, cognitive distortion identification and suicide detection. The distribution of experimental datasets can be seen in Table 2.

heading: 5.1.1 Sentiment analysis tasks
text: We conducted experiments on three sentiment analysis tasks, using two datasets from SMP2020EWECT4 for social media emotional classification and Waimai_10k dataset5 for positive and negative sentiment classification on delivery platforms. The SMP2020-EWECT includes two datasets: the first consists of posts randomly collected from Weibo social media platform on various topics; the second consists of COVID epidemic-related posts from Weibo. The objective of the two SMP2020EWECT tasks are to identify the six categories of emotions in the posts: positive, angry, sad, fearful, surprised, and no emotion. The Waimai_10k dataset is derived from user reviews collected on a food takeout platform, and the task is to identify positive and negative emotions.

heading: 5.1.2 Cognitive distortion classification tasks
text: In the cognitive distortion classification task, we conducted experiments on two datasets: “Cognitive” (Qi et al., 2023) and “C2D2” (Wang et al., 2023).
• “Cognitive” dataset: is from Weibo, sourced from the “Zoufan” tree hole. The cognitive distortion task centers on the categories defined by Burns (1981). Data were obtained by
4https://github.com/BrownSweater/BERT_ SMP2020-EWECT
5https://github.com/SophonPlus/ ChineseNlpCorpus/tree/master/datasets/waimai_10k
crawling comments from the “Zoufan” blog on the Weibo social platform. Subsequently, a team of qualified psychologists was recruited to annotate the data. The experts define it as a multi-label classification task since each post may reflect multiple cognitive distortions. The classification labels in the cognitive distortion dataset include: all-or-nothing thinking, overgeneralization, mental filter, disqualifying the positive, mind reading, the fortune teller error, magnification, emotional reasoning, should statements, labeling and mislabeling, blaming oneself and blaming others. Given that the data are publicly accessible, privacy concerns are not applicable.
• “C2D2” dataset: it is a Chinese cognitive distortion dataset, containing 7,500 entries of cognitively distorted thoughts from daily life scenarios. The data annotation process is conducted through a collaborative effort between carefully selected, specially trained volunteers and domain experts. They treat it as a multiclass classification task because it simplifies the annotating process. The classification labels in the cognitive distortion dataset include: black-and-white thinking, emotional reasoning, fortune-telling, labeling, mindreading, overgeneralization, personalization, and non-distorted.

heading: 5.1.3 Suicide intention classification task
text: This dataset is also from Weibo, specifically collected from the “Zoufan” tree hole, as detailed in the study by Qi et al. (2023). The suicide risk task aims to differentiate between high and low suicide risk. For the suicide detection data, the dataset contained 645 records with low suicide risk and 601 records with high suicide risk.

heading: 5.2 Implementation Details

heading: 5.2.1 Domain adaptive pre-training
text: We conducted text preprocessing to eliminate irrelevant information, which included URLs, user tags (e.g., @username), topic tags (e.g., #topic#), and we also removed special symbols, emoticons, and unstructured characters. Following this preprocessing, we concatenated all samples and segmented the entire corpus into equal-sized chunks, each consisting of 128 words.
The domain adaptive pretrained model is tasked with predicting these masked words in the training process. The words appeared in the depression lexion are preferentially masked within the sample. If the proportion of masked words in the original text less than the required 20%, additional random words are incorporated into the mask to meet this threshold.
The pretraining was performed on an NVIDIA Tesla V100 32GB SXM2 GPU. Building upon the foundational pretrained model (Chinese-BERTwwm-ext), we continued training three epochs, utilizing a batch size of 128 and a learning rate of 5e−5.

heading: 5.2.2 Finetuning for downstream tasks
text: We followed the same pre-training text preprocessing step, removing extraneous information such as URLs, user tags, and topic tags to ensure the cleanliness and relevance of the dataset. Both the supervised learning models and large language models are fine-tuned for downstream tasks.
• Supervised learning models: These models were fine-tuned for 10 epochs on the training set for all these tasks. We employed a batch size of 16, utilized the Adam optimizer (Kingma and Ba, 2014), set the learning rate to 2e−5, and used cross-entropy as the
loss function. The model was trained using an NVIDIA GeForce RTX 4090 24GB GPU.
• Large language model: The LLMs were finetuned for 5 epochs using the LoRA technique (Hu et al., 2021) on the training set. LoRA is recognized as one of the most effective methods for fine-tuning LLMs. It enhances fine-tuning efficiency by reducing the number of trainable parameters while achieving performance comparable to that of fully fine-tuned models. For the model training, we employed a batch size of 8 and set the learning rate to 5e−5. The model was trained using an NVIDIA Tesla V100 32GB SXM2 GPU.
The model that showed the best performance on the validation set was then selected for further evaluation on the test set. For the cognitive distortion and suicide classification tasks, we utilized a five-fold cross-validation approach. We utilize the precision, recall, and F1-score as evaluation metrics.
For both model adaptive pretraining and finetuning stages, we use PyTorch (Paszke et al., 2019) as our implementation framework. All the pretrained models were employed from HuggingFace v4.28.1 (Wolf et al., 2020). Detailed configurations, source codes, and our pretrained language model are made public available via: https://github. com/zwzzzQAQ/Chinese-MentalBERT.

heading: 5.3 Comparison methods
text: We selected some representative models for performance comparison. The methods we compared can be broadly categorized into three main groups: Word2Vec-based methods (Word2Vec-BiLSTM, Word2Vec-CNN), pre-trained language models (BERT, Chinese-BERT-wwm-ext, DKPLMfinancial, and DKPLM-medical), and LLMs (Llama2-Chinese-Chat and Chinese-Alpaca-2). Although LLMs are also PLMs, we list them separately for discussion because they refer to largerscale PLMs. Our method achieved the best performance across all six datasets.
Word2Vec based DNNs In our experiments, we built two deep neural networks using word2vec: Word2vec-CNN and Word2vec-BiLSTM. We utilized 300-dimensional word2vec word embeddings to represent the input text.
• Word2vec-CNN: this model employs two layers of 1D-CNN for text feature extraction. The
first layer transforms the 300-dimensional input into a 100-dimensional output with a kernel size of 1, while the second layer uses a kernel size of 2 to capture features at different scales. The process concludes with a fully connected layer for classification.
• Word2vec-BiLSTM: this model uses a singlelayer BiLSTM to capture both forward and backward dependencies in the text, with input and hidden state sizes of 300 and 100, respectively. The output from the LSTM layer is averaged pooled to extract the overall features of the sequence, followed by a fully connected layer for classification.
BERT and whole word masking BERT BERT is a transformer-based pretrained language model (Devlin et al., 2018). We utilize the Chinese pretrained BERT with a fully connected layer as a classifier. The Chinese-BERT-wwm-ext model (Cui et al., 2021), designed for processing Chinese text, enhances the understanding of the nuanced aspects of the Chinese language through Whole Word Masking (WWM) implementation.
DKPLMs DKPLM is a knowledge-enhanced PLM (Zhang et al., 2022). We employed two variant models for experiments, specifically DKPLMfinancial and DKPLM-medical. The DKPLMfinancial model is pre-trained in the financial domain and exhibits adeptness in deciphering intricate financial terms and context. The model excels in applications like sentiment analysis and market trend prediction within the financial sector. DKPLM-medical is pre-trained on an extensive collection of medical texts and proficiently comprehends medical terminology and patient narratives. The demonstrated proficiency guarantees enhanced performance in tasks such as clinical information extraction and medical literature analysis.
Chinese LLMs We selected two large-scale Chinese language models for comparison: Llama2Chinese-Chat6 and Chinese-Alpaca-27, each with a model size of 7 billion parameters. Both models, derived from existing large-scale frameworks, have undergone additional fine-tuning and training to enhance their processing capabilities for Chinese.
6https://huggingface.co/FlagAlpha/ Llama2-Chinese-7b-Chat
7https://huggingface.co/hfl/ chinese-alpaca-2-7b
• Llama2-Chinese-Chat: this model is an adaptation of the Llama2-Chat, specifically optimized for the Chinese language. Llama2Chat was developed through supervised finetuning of the Llama2 model (Touvron et al., 2023) and boasts significant dialogue processing capabilities. This model was further enhanced by fine-tuning with a Chinese instruction dataset, significantly boosting its performance in Chinese conversational contexts.
• Chinese-Alpaca-2: this model was built on the LlaMA-2 framework, utilizes a 120GB Chinese corpus for incremental training to improve understanding and generation of Chinese text (Cui et al., 2023). Furthermore, em-
ploying 5 million units of Chinese instructional data for precise adjustments culminated in the more refined Chinese-Alpaca-27B model.

heading: 6 Results
text: We compared the performance of our proposed method, which includes two masking mechanisms (random masking and guided masking), with eight other models across six open datasets. The experimental results can be seen in Tables 3 and 4.
For a detailed analysis, Word2Vec-CNN outperformed Word2Vec-BiLSTM in all tasks, although Word2Vec-based methods were generally have lower performance on all six datasets. Es-
pecially, On the “C2D2” data, their performance was 7.81% lower in F1-score compared to our model, highlighting their limitations. Among the four pre-trained model-based methods (BERT, Chinese-BERT-wwm-ext, DKPLM-financial and DKPLM-medical), performance differences across the six tasks were not great. In the best-performing “Cognitive” task, the best model BERT still had an 1.69% point lower F1-score than our model. Also, in the “ESWECT-epidemic” task, the bestperforming BERT model had an F1-score 3.65% lower than our model. For LLMs, Chinese-Alpaca2 outperformed Llama2-Chinese-Chat in all tasks. However, LLMs generally fell short compared to pre-trained model-based methods, except for the “ESWECT-usual” task, where Chinese-Alpaca-2 outperformed Chinese-BERT-wwm-ext by 1.19% points in F1-score. This indicates that traditional deep learning solutions still have advantages in domain-specific tasks. Also fine-tuning LLMs can be challenging due to their large model sizes for model training. The superior performance of our models in all tasks is attributed to our domainadaptive training and guided masking mechanism in mental health domain. The performance of our model with random masking was lower than with the guided masking training approach in all tasks, especially in the Waimai_10k task, where there was a 1.76% point F1-score gap. This shows that the good performance of our method is not only due to the collection of large datasets, but also benefits from the training mechanism based on lexicon guided masking.
Among the six tasks, we consider “Cognitive” and “Suicide” to be the best for validating model performance due to having the highest inter-method standard deviations (SD), which are 4.72 and 4.66, respectively. The high SDs indicate that the differences between methods are meaningful, suggesting that performance on these tasks does not saturate too quickly and that the model’s performance can effectively highlight the differences between methods (Isensee et al., 2024). The best performance achieved by our model in these two tasks reconfirms the reliability of our approach.

heading: 7 Qualitative comparison
text: We conducted a qualitative analysis to explore the behavior and tendencies of language models by predicting masked words, using questions from the Symptom Checklist-90 (SCL-90 scale) (LR
et al., 1973) as our experimental basis. The SCL-90 scale, a widely recognized 90-item tool for evaluating mental health, assesses nine primary psychiatric symptoms and psychological distress. In this study, specific keywords in each question of the SCL-90 scale were obscured, and the predictions made by Chinese-BERT-wwm-ext and Chinese MentalBERT—models trained through random or guided masking mechanisms were analyzed. Table 5 presents examples of these sentences and the corresponding predictions for the masked words.
The examples shown in the table reveal that the Chinese-BERT-wwm-ext model typically generates more general and less emotionally charged predictions compared to the proposed models. For instance, when faced with sentences expressing self-blame or difficulty breathing, the basic model opts for neutral words like “告诉” (tell) and “急性” (acute), which lack the emotional depth present in the context. In contrast, the proposed models, developed with a focus on psychological or emotional states, consistently select words that better capture the negative emotions or psychological nuances implied in the sentences, such as “折磨” (torture) and “困难” (trouble).
Comparing the two masking mechanisms (Random and Guided), highlights their different tendencies to word prediction. While both are inclined towards psychological and emotional expressions, the model trained with guided mechanism exhibits a clearer focus on accurately capturing the emotional context of the sentences. For example, in predicting masked words related to thinking about death, the guided model predicted words that directly relate to emotional states like “难过” (sadness) and “伤心” (grief), demonstrating its enhanced sensitivity towards psychological vocabularies. This suggests that the guided training mechanism employed in the model improves its ability to predict emotionally relevant context.

heading: 8 Conclusion
text: In this paper, we present Chinese MentalBERT, the first adaptive pre-trained language model for the Chinese mental health domain to the best of our knowledge. The model features a simple yet effective domain adaptive framework, and experiments have shown its strong performance in Chinese psychology-related tasks. The domain lexiconguided masking mechanism used in this study can adjust the model’s tendency to enhance the perfor-
mance of downstream tasks. Our pre-trained model is publicly available to support the advancement of this field.
In future research, we plan to validate our model across a broader range of data types and tasks, including analyzing mental health interview data and summarizing psychology-related content. Additionally, we aim to explore the use of diverse lexical resources tailored to specific tasks and employ clinical data to develop more specialized language models.
Limitations
In qualitative comparison, Chinese MentalBERT shows a greater inclination to predict negative emotional words, whereas Chinese-BERT-wwm-ext in the general field produces more random predictions. We hypothesize this may be attributed to the guided masking mechanism, indicating its effectiveness in adjusting the model’s tendencies. These tendencies can influence the model’s attention on data to benefit specific tasks. The relationship between pretrained model tendencies and downstream task performance still needs to be explored.
Access to clinical textual data is constrained by the sensitivity of personal information and the stringent confidentiality requirements, leading to data scarcity in this field. Considering this challenge, this study utilized the Weibo corpus as the primary domain adaptation training resource. However, this approach may predispose the model to learn general Weibo language features over those specific to clinical reports, potentially limiting its applica-
bility and effectiveness in clinical contexts. This emphasizes the importance of tailoring training resources to match the specific needs of the clinical environment. A limitation of the current study is the absence of a specific dataset for the clinical psychiatric domain, which lacks a comprehensive evaluation of the Chinese MentalBERT model’s performance in clinical psychiatric diagnosis tasks. This underscores the necessity for future research to acquire datasets in pertinent fields and investigate the model’s potential in particular clinical scenarios.
Ethics statement
In order to mitigate the risk of disclosing personal information, we anonymize and de-identify the data to the greatest extent possible during processing and analysis. We guarantee that the research outcomes do not include any information that could directly or indirectly identify individuals. Due to data privacy concerns, the pretraining corpus will not be made available to the public. However, to further the development of the mental health field in China, we have made the pre-trained model and code public accessible to the community. It is worth noting that datasets for downstream tasks might contain biases from social media data, including gender, age, or sexual orientation profiles, potentially leading to the incorrect labeling of individuals as having a mental disorder. We emphasize that the experimentation with and utilization of these data are strictly confined to research and analysis purposes, and any misuse or mishandling of the information
is expressly forbidden.

heading: Acknowledgements
text: This work was supported by grants from the National Natural Science Foundation of China (grant numbers: 72174152 and 82071546), Fundamental Research Funds for the Central Universities (grant numbers: 2042022kf1218; 2042022kf1037), the Young Top-notch Talent Cultivation Program of Hubei Province. Guanghui Fu is supported by a Chinese Government Scholarship provided by the China Scholarship Council (CSC).

heading: Appendix Error analysis
text: We conducted a detailed analysis of model performance across each major category of cognitive distortions in the “Cognitive” dataset. Given the limited number of cognitive distortion types in four of the categories within the dataset, our discussion primarily focuses on the experimental results of the other eight categories. The experimental results are detailed in Table S1. The results indicate that the proposed Chinese MentalBERT outperforms other models in seven of the eight primary categories of the cognitive distortion multi-label classification task, while it exhibits slightly inferior performance in one category. Addressing this discrepancy will be the primary focus of our future research endeavors.
Method C1 C2 C3 C4 BERT 0.00 55.56 23.53 70.39
Chinese-BERT-wwm-ext 0.00 55.17 30.00 70.54 Chinese MentalBERT 6.25 55.07 40.91 72.58
Method C5 C6 C7 C8 BERT 31.58 62.07 92.50 65.71
Chinese-BERT-wwm-ext 18.18 57.14 92.12 48.28 Chinese MentalBERT 34.19 72.00 92.67 66.67
Table S1: Performance (F1-score) comparison of three models across the eight cognitive distortion categories within the “Cognitive” dataset. C1: Over-generalization, C2: Mental Filter, C3: Mind Reading, C4: The Fortune Teller Error, C5: Magnification, C6: Should Statements, C7: Labeling and Mislabeling, C8: Blaming Oneself.

heading: Future Work
text: In future research, we plan to validate our model across a broader range of data types and tasks, including analyzing mental health interview data and summarizing psychology-related content. Additionally, we aim to explore the use of diverse lexical resources tailored to specific tasks and employ clinical data to develop more specialized language models. Limitations
In qualitative comparison, Chinese MentalBERT shows a greater inclination to predict negative emotional words, whereas Chinese-BERT-wwm-ext in the general field produces more random predictions. We hypothesize this may be attributed to the guided masking mechanism, indicating its effectiveness in adjusting the model’s tendencies. These tendencies can influence the model’s attention on data to benefit specific tasks. The relationship between pretrained model tendencies and downstream task performance still needs to be explored. Access to clinical textual data is constrained by the sensitivity of personal information and the stringent confidentiality requirements, leading to data scarcity in this field. Considering this challenge, this study utilized the Weibo corpus as the primary domain adaptation training resource. However, this approach may predispose the model to learn general Weibo language features over those specific to clinical reports, potentially limiting its applica-
bility and effectiveness in clinical contexts. This emphasizes the importance of tailoring training resources to match the specific needs of the clinical environment. A limitation of the current study is the absence of a specific dataset for the clinical psychiatric domain, which lacks a comprehensive evaluation of the Chinese MentalBERT model’s performance in clinical psychiatric diagnosis tasks. This underscores the necessity for future research to acquire datasets in pertinent fields and investigate the model’s potential in particular clinical scenarios