C2D2 Dataset: A Resource for the Cognitive Distortion Analysis and Its Impact on Mental Health
Authors: BiChen Wang
PengFei Deng
Yanyan Zhao
Bing Qin
Venue: EMNLP 2023 Findings (2023)
Keywords: cognitive distortion
mental health
text analysis
TL;DR: We construct a cognitive distortion dataset and explore the effects of cognitive distortion on mental health.

Abstract: Cognitive distortions refer to patterns of irrational thinking that can lead to distorted perceptions of reality and mental health problems in individuals. Despite previous attempts to detect cognitive distortion through language, progress has been slow due to the lack of appropriate data. In this paper, we present the C2D2 dataset, the first expert-supervised \textbf{C}hinese \textbf{C}ognitive \textbf{D}istortion \textbf{D}ataset, which contains 7,500 cognitive distortion thoughts in everyday life scenes. Additionally, we examine the presence of cognitive distortions in social media texts shared by individuals diagnosed with mental disorders, providing insights into the association between cognitive distortions and mental health conditions. We propose that incorporating information about users' cognitive distortions can enhance the performance of existing models mental disorder detection. We contribute to a better understanding of how cognitive distortions appear in individuals' language and their impact on mental health.

C2D2 Dataset: A Resource for Analyzing Cognitive Distortions and Its Impact on Mental Health
Bichen Wang, Pengfei Deng, Yanyan Zhao; Bing Qin
Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, Heilongjiang, China (bichenwang, pfdeng, yyzhao, qinb}@ir. wiw.eei... eet.edd.ccm cn
Abstract
Situation
[Whot really hegpene7) ewen is approssiiiyg
Cognitive distortions refer to patterns of irra- tional thinking that can lead to distorted percep- tions of reality and mental health problems in individuals. Despite previous attempts detect cognitive distortion through language, progress has been slow due to the lack of appropriate data. In this paper, present the C2D2 dataset, the first expert-supervised Chinese Cognitive Distortion Dataset, which contains 7,500 cogni- tive distortion thoughts in everyday life scenes. Additionally, we examine the presence of cog- nitive distortions in social media texts shared by individuals diagnosed with mental disorders, providing insights into the association between cognitive distortions and mental health condi- tions. We propose that incorporating informa- tion about users' cogmitive distortions can en- hance the performance of existing models men- tal disorder detection. We contribute to a better understanding how cognitive distortions ap- pear in individuals' language and their impact on mental health.
Results
Trigger
Strengthetr
Really. foll heppen?)
Cognitive Distortion
Reinforcing Loop
(What you think?) I'm sure I will fail.
Couse
Actions (Whect you do?)
Induce
Aeasi shplaappa crafus
Depression
PTSD
Suicide
Mental Disorders
Figure 1: Cognitive distortions' impact on individuals. Cognitive distortions are constantly strengthened within this reinforcing loop, contributing to the development of mental disorders such as depression, PTSD, and anx- iety (Burns, 1981).
traumatic stress disorder (PTSD) (Marton et al., 1993; Muris and Field, 2008; Abel et al., 1989; Strohmeier et al., 2016; Hammen, 1978). Hence, the development of automated tools for detecting cognitive distortions is important, as it assists re- searchers in carly detection cognitive distortions, facilitating timely intervention strategies aimed at enhancing individuals' psychological well-being and happiness.
1 Introduction
Cognitive distortions are irrational thinking pat- terns that can lead to distorted perceptions of real- ity (Beck, 1970). For example, the thought "/ lost my puppy, and my future will be perpetually filled with sadness and loneliness" exemplifies cogni- tive distortion known "Overgeneraliizatoo." "In this case, the conclusion implies an excessively broad and permanent state of unhappiness. more precise and less distorted thought for this situation would be: "The sudden loss of my puppy was in- credibly painful, and struggled to come to terms with it."
Previous research demonstrates that computa- tional techniques can effectively detect cognitive distortions from language (Shickel et al., 2020; Simms et al., 2017: Bathina et al., 2021). However, these studies have been conducted using private datasets or datasets with limited and low-quality annotations (Alhaj et al., 2022; Ziems et al., 2022). Despite cognitive psychology acknowledging an association between mental disorders and cognitive distortions, the absence of reliable and publicly accessible dataset has obstructed the establishment of a credible benchmark and further advancements in this field.
As depicted in Figure 1, cognitive distortions sig- nificantly hinder individuals" perception, trapping them further through continuous relf-reinforeementt and contributing to the development of mental dis- orders, including depression, anxiety, and post-
To tackle these challenges, we introduce our C2D2 dataset, a Chinese dataset created to address
Corresponding author

---

the shortage of research resources regarding cogni- tive distortions. is a publicly accessible resource within this domain that includes 7,500 instances cognitive distortion thoughts from 450 different scenes. The research of our C2D2 can promote re- search on cognitive distortion and provide insights into users' mental state. We believe this resource will contribute to mental health research in China, developing country, particularly in light of increas- ing social pressure and inadequate support for the mental healthcare system.
users' cognitive distortions into mental disor- der detection models, emphasizing the signif- icance of including cognitive distortion com- putations.
Related Work
In the following sections, we review studies per- taining to cognitive distortions, covering both psy- chological research and computational techniques.
2.1 Cognitive Distortion in Psychology
Moreover, we conduct various experiments, in- cluding the cogmitive distortion detection and ex- ploring the relationship between cognitive distor- tions and mental disorders. We compare the per- formance of finetuning different pretrained models and large language models using in in-context learn- ing on C2D2 dataset. We demonstrate the cur- rent models' ability to detect cognitive distortions. Furthermore, our research is not limited to cogni- tive distortions alone. Inspired by psychology, we use computational methods to investigate the asso- ciation between cogmitive distortions and mental health. We use several datasets on mental health to detect and analyze present cognitive distortions in the online texts of individuals with different disor- ders using our model. We discover some interesting phenomena and conclusions that have not been pre- viously considered, but could potentially validate the underlying mechanisms of certain disorders. Fi- nally, we develop a simple method to enhance the performance of detecting disorders based on cogni- tive distortions. The creation of the C2D2 dataset contributes to better understanding of cognitive distortions and the impact of cognitive distortions on mental health.
Since the emergence of cognitive-behavioral the- ory, cognitive distortions have been the subject of extensive research (Beck, 1970). This line of study, enriched by various researchers, has cultivated a comprehensive theoretical framework (Beck, 2020, 1979; Burns, 1981).
Although originating in depression research, cog- nitive distortions have also been linked variety of issues such as pathological gambling, anxiety, suicide, and anorexia (Marton et al., 1993; Muris and Field, 2008; Abel et al., 1989; Strohmeier et al., 2016; Fortume and Goodie, 2012). From a sociolog- ical perspective, they are hypothesized to correlate with juvenile delinquency and antisocial personal- ity (Nas et al., 2005: Wallinius et al., 201 1: Gannon and Polaschek, 2006; Feldman, 2007). Further- more, another branch of rescarch aims to guide psychologists in detecting and rectifying these dis- tortions, thereby studying their impact on treat- ment (McClenahan, 2005; Yurica and DiTomasso. 2005).
2.2 Cognitive Distortion Detection and Application
In the study of mental health computing, the main focus includes emphasis on emotion analysis and symptom identification (Gkotsis et al., 2016; Shickel al., 2016). Some researchers conduct search from the perspective of cognition and emo- tion (Uban et al., 2021). However, the applica- tion of machine learning methods from a cognitive- behavioral perspective, however, has garnered less attention. Prior work has explored the use of ma- chine learning to detection cognitive distortions in mental health texts et al., 2020; Simms et al., 2017; Wang et al., 2022), as well as within medical dialogues between physi- cians and patients (Shreevastava and Foltz, 2021; Tauscher al., 2023).
We have developed a publicly accessible Chi- nese Cognitive Distortion D for the first time, aiming to facilitate the analysis of cog- nitive distortions and promote the integration of psychology and computational technology.
We explore the association between cognitive distortions and various mental disorders on social media platforms. To our knowledge, this is the first work that utilizes cognitive distortion assess users' mental states.
social media (Shickel
We attempt to incorporate information about
'Our dataset
will be available in
The limited exploration in this field can be at- tributed to the lack of publicly available, well-
Distortion.git git

---

Cognitive Distortion
Definition
Example
Black and white thinking
Oversimplification of complex situations into
"I[ I don't get A this test, have pletely failed."
binary categories.
Emotional reasoning
Belief that what we feel must be true, regard-
"I feel like bad, so must be."
less of the evidence.
Fortune-telling
Predicting negative outcomes without basis.
know I will do poorly in the presentation tomorrow"
Labeling
Attaching arbitrary labels to oneself others.
made mistake, I am loser""
Mindreading
Belief of accurately disceming others' think- ings without direct communication.
"I can tell they're laughing me hchind my back."
Overgeneralization
Making sweeping negative conclusions based on limited instances.
"I failed one exam, I'm going to fail the whole course."
Personalization
Attributing the negative behaviors others to oneself without plausihle connections.
"My boss upset, it must be something did wrong."
Table 1: Definitions and Examples of Cognitive Distortions. Our data contains seven common cognitive distortions.
annotated datasets. Previous studies compu- tational mental health primarily focus on emotions and symptoms. However, we believe that the im- pact of cognitive distortions and thinking patterns on mental health is also significant.
tortion thoughts based the given scenes. An example presented in Table it simply displays the translated content of our dataset. It should be noted that different types of cognitive distortions do not strictly appear independently. However, for the purpose of simplifying the annotation process, we have treated it as single-label task. The volun- teers' goal is to generate instances that represent single type of distortion. In situations where multi- ple cognitive distortions occur simultaneously, we ask volunteers to select the label of the dominant cognitive distortion.
3 Dataset Construction
Previous researchers have collected posts from so- cial media platforms gathered data related to cognitive distortions through crowdsourced writ- ing (Shickel et al., 2016; Alhaj al., 2022). We have chosen not to directly annotate social media content due to the inability to effectively control its quality and accuracy. Instead, we have adopted a specially designed task to collect cognitive dis- tortion thoughts Unlike previous work that recruit volunteers widely from the internet, our data anno- tation process is executed through collaborative effort between carefully selected and specifically trained volunteers and domain experts. As depicted in Figure 2, we will further describe our task de- sign and the three phases, which include volunteer recruitment and screening, data annotation, and expert evaluation.
We provide the following to volunteers:
Scene: The company's 's project about to be delivered, and you caught a cold this critical juncture
Scene category: Work issues
The content completed by volunteers as follows:
Cognitive Distortion: Cold overtime, bad luck peaks! Why all misfortune on (eeeeegeennerllizttinn)
Cognitive Distortion: Is my illness such critical sign that something will go wrong with the project as well? (Fortune-telling
Non-distorted: Foeling terrible with cold my nose and throat are both sore. pNin distorte))
... least 5 different types cognitive distortions and normal thought.
3.1 Data Annotation Target
Table 2: An instance from C2D2 dataset. This example is translated into English.
Psychologists have identified various categories of cognitive distortions that are often exhibited in individuals' thought (Beck, 1970, 1979; Ellis, 1994). In our work, the C2D2 dataset encom- passes seven typical cognitive distortions, which are shown in Table 1. Our annotation task involves providing volunteers with scenes and requesting them to write down multiple possible cognitive dis-
3.2 Volunteer Recruitment & Screening
Psychology Questionnaire We recruit volunteers to participate our data collection on cognitive dis- tortions. They complete a cognitive distortion ques- tionnaire (Covin et al., 201 1). We select volunteers

---

P1: Voluntear Recouttmnt Screening
Data Annotation
P3: Expert Evaluation
SEBJ:
Category:
Family istues
B Mt Scone:
The Questoomaire
Your pet dog recently passed away.
Cognitive Distortion Questionaire
Question: When negative event occurs, people think that more bad things ure about to happen and see the negutive event as the beginning pattern. How many similar thoughts like this do you have in your life?
183 A 30 43, 31 (
Emotional reasoning: feel so miserable that want die. I shoukdn' have kept any living being as pet. Having pets just distressing.
Expert
Options: (Never) 0 2 (Almest every day) (Question
2O*: S EEET
Questoonaie Score
M10. 8 BA
Mindreading: If there had been choice back then.
Annolated Data
Verified Data
she wouldn't have wanted my pet, down.
Retuined: Questionnaire score is above 15, Which means generate cognitive distortion text.
; 10 -
- BEE
Eliminated: Questionaine score is below IS, Which means harder to generate cognitive distortion text.
Non-Distortion: accept this fact, every wonderful moment I spent with my pel.
Figure 2: The Data Collection Process includes volunteer recruitmem&&screennnng data annotation, and conducting a final expert evaluation. During the volunteer recruitment phase, we carefully select volunteers using a cognitive distortion questionnaire. The data annotation collect cognitive distortions. The expert evaluation guaranteed the reliability of our data.
based their questionnaire results, prioritizing those who show a potential for exhibiting cogni- tive distortions in daily life. This strategy enables us collect more authentic cognitive distortion thoughts
crepancy between ideal and reality (10%) (Lazarus and Folkman, 1984). These categories ious aspects life that can trigger cognitive dis- tortions. By providing these scenes, we encourage the volunteers to imagine themselves in these sit- uations and describe how they would perceive the events from the perspective of someone experienc- ing cognitive distortion thoughts.
var-
Volunteer Training To ensure our volunteers un- derstand cognitive distortions and can recognize them in their daily lives, we provide comprehen- sive psychology training program. This program in- cludes an examination on the identification and un- derstanding of common cognitive distortions. After two days of cognitive psychology training and an introduction to the harms of cognitive distortions, we retain 24 out of the initial 50 volunteers. These volunteers demonstrate an ability to identify cog- nitive distortions and acknowledge their harmful effects. All volunteers agree our data collection and public disclosure requirements.
Data Collection Volunteers are asked adopt the perspective an individual experiencing cogni- tive distortions and generate thoughts related to a specific cognitive distortion that may arise in the given scene. Moreover, volunteers are asked to label thoughts provided by other volunteers. The final cognitive distortion labels for data are determined by majority vote among three volun- teers, reducing the influence of individual bias. Our primary objective cognitive distortion detec- tion; therefore, volunteers are assigned overlapping events for data collection.
3.3 Data Annotation
Scene Preparation As depicted Figure 2. we prepare categories of 450 daily scenes, including work issues (20%), interpersonal issues (20%), eco- nomic issues (10%), random negative events (5%), family issues (25%), physical siress (10%), and dis-
3.4 Expert Evaluation
Our annotated data undergo rigorous expert eval- uate ensure quality. Experts randomly sample from each volunterr-contributed dataset, evaluating each volunteer's work comprchensively. If the data do not meet the standards, volunteers receive feed-
For the protection of volunteers' rights, please refer to Section 9.1 and Appendix A.

---

Dataset (Shickel et al., 2020; Simms et al,, 2017; Shreevastava and Foltz, 2021)
Evaluation Criteria
C2D2(Our Datuset)
MH
Tumblr Dataset
Therapist
Public availability
Multiple cognitive distortion types
/
V
X
/
Training of annotators
/
Consistency in amnotation
/
/
Scale of dataset
7,500
2,769
1,728
3,136
Table 3: Comparison of the key features of work with previous studies.
back, are asked to revise their submissions, and may receive additional training. The evaluation criteria include:
Category
Num
Avg.Len
Black and white thinking
690
29,45
Emotional reasoning
751
30.69
Correctness: Experts examine the assigned labels accurately indicate cognitive distor- tions.
Fortune-telling
682
30.91
Labeling
721
29.11
Mindreading 1,003
30.11
Overgeneralizatioon
894
29.58
Reasonableness:
They evaluate if this cogni- tive distortion aligns with the cognitive distor- tions that may occur in real-life thinking.
Personalization
709
31.36
Non-distorted
2,050
28.14
Sum
7,500
29.68
Emotional Diversity: They verify if the data reflects a broad spectrum of emotions, high- lighting the emotional complexity in human cognition.
Table 4: Dataset Statistics
After the completion data collection, our dataset comprises total of 7,500 thoughts. These thoughts are divided into three sets for training, val- idation, and testing purposes, following 8:1:1 ratio.
3.5
Data Quality Assurance
To ensure the quality of C2D2, we undertake mul- tiple evaluations. Initially, we compute the inter- annotator K Kappa score among our volunteers in re- gard to our labels. The outcomes reflect moderate degree of concordance, with mean K appa score of 0.67, signifying significant consensus among the annotators. Subsequently, we engage experts to examine our complete dataset. These experts are requested to grade the data on a to scale. The assessments return high scores in all categories, presenting mean ratings of roughly 4.7 for Cor- rectness, 4.5 for Emotional Diversity, and 4.1 for Reasonableness. These elevated scores corroborate that dataset is well-aligned with the task instruc- tions and embodies the necessary characteristics train model proficient in detecting cognitive distortions.
4.2 Related Datasets
As shown in Table our C2D2 dataset has many advantages compared to previous works. It is, firstly, a publicly accessible resource within this domain that includes numerous texts recording in- dividuals' thoughts in various scenes. Secondly, we have annotated each thought, not only indicat- ing the presence of cognitive distortion but also categorizing them according to cognitive psychol- ogy (Beck, 1979). Finally, we'v ''ve maintained data authenticity and privacy through comprehensive an- notator training, backed by stringent three-phase data collection process.
The data collection methods of previous works included social media platforms or gathered data re- lated to cognitive distortions through crowdsourced writing. Due the privacy and sensitivity of men- tal health data, collaborating with trained volun- teers and domain experts to construct reliable syn- thetic data is a viable alternative. Our approach en- sures data quality and authenticity while maximiz-
4
Data Characteristics
4.1
Statistics
The provided dataset statistics are displayed in Ta- ble 4, showcasing the overall characteristies of the dataset. It can be seen that the label distribution of our dataset relatively uniform.

---

ing the availability of open datasets for researchers, thereby breaking the barriers of data privacy and low data quality that currently exist.
Model
F1
Rec
Pre
Bert
0.69
0.73
0.68
Roberta
0.73
0.75
0.72
Our data focuses on the Chinese language, par- ticularly in China, a developing country. Previous researches on mental health predominantly conduct in developed regions with advanced psychological resources. However, mental health issues in de- veloping countries often receive limited attention. these countries, there greater need for af- fordable and dependable automated detection tech- nologies compared to the availability of reliable mental health services in developed regions (Patel and Kleinman, 2003; Kohn et al., 2004; Chowdhary et al., 2014).
XLnet
0.72
0.71
0.73
Electra
0.70
0.69
0.71
ChatGPT (Zero-shot)
0.39
0.37
0.41
ChatGPT (Few-shot)
0.51
0.49
0.54
Table 5: Performance of baseline models for the C2D2 tasks. All metrics are calculated using macro-averaging.
models. We believe that tasks such as cognitive distortion detection, which demand specialized an- alytical capabilities and expertise rather than being universally accessible, warrant increased attention in the future.
5 Cognitive Distortion Detection
We develop a model to detect cognitive distortions in text and assign specific categories to these cogni- tive distortions. This detection and categorization can facilitate interventions by psychologists and further analysis for treatment purposes. The task, Cognitive Distortion Detection, involves inputting text X containing cognitive distortions and generat- ing multi-class label y E {0, 7). Here, 0 denotes non-distorted, while the other values represent dis- tinct categories of cognitive distortions.
Cognitive Distortion for Mental Health
After constructing the cognitive distortion detection task, we aim to introduce the importance of cogni- tive distortions in mental health. Previous computa- tional research has concentrated examining the emotions and symptoms exhibited by individuals with mental disorders. In contrast, our objective is to find out the underlying cognitive distortion. This section sims to examine two fundamental questions in order to underscore the significance of cognitive distortion detection.
5.1 Baseline Models and Results
To evaluate the task, we employ pretrained lan- guage models on C2D2 dataset. Table 5 presents the results of the baseline models. Our objective to provide researchers with dataset and establish a benchmark. We finctune the dif- ferent kind of Chinese versions of the pretained language models (Cui et al., 2021, 2020). Addition- ally, we assess the few-shot and zero-shot settings for I.M,, such as ChatGPT (Ouyang et al., 2022), providing three examples for each class in few-shot learning."
Q1: How do thinking patterns differ between individuals diagnosed with mental disorders and the normal group?
In Section 6.2, we employ our cognitive dis- tortion detection model to analyze social media posts from individuals diagnosed with depression and PTSD, as well as normal group without any self-reported mental health issues. From a cogni- tive psychology perspective, aim to identify the cognitive distortions detect differences between these groups.
The pretrained models demonstrate satisfactory performance. However, LLM's in-context learn- ing performance does not match that of the fine- tuned models for this particular psychological task. Nevertheless, significant improvement is observed with few examples in the few-shot learning set- ting. We have built cognitive distortion detection model based on various models. We develop cog- nitive distortion detection model utilizing multiple
Q2: Can these differences be utilized for men- tal disorder detection?
In Section 6.3, we simply integrate the cogni- tive distortions into the method for detecting men- tal disorders, with specific focus depression and PTSD. This approach underscores the poten- tial utility of cognitive distortion improve men- tal disorder detection, particularly when conven- tional research largely concentrates on symptoms and emotional manifestations.
Please refer to Appendix B and Appendix C for more details.

---

Dataset
Depression
PTSD
Normal
Num.subjects
477
396
873
Num.posts
1,083,403
868,428
2,266,740
CLPsych-2015
Avg num. of posts per subject Avg length.of per posts
2,271.5
2,193.4
2,596.5
13.91
13.52
13.88
Num.subjects Num.posts
135
752
49,557
481,837
cRisk-2018
Avg num. of posts per subject Avg length.of per posts
367.1
640.7
45.13
34.11
Table 6: Mental disorder data statistics
6.1 Mental Disorder Dataset
Cognitive Distortion in Different Groups(CIpsych-20155
We utilize two different datasets. Table illustrates the datasets that we have used, which focus on PTSD and depression, leveraging data from Twit- ter and Reddit respectively. The CLPsych-2015 dataset (Coppersmith et al., 2015) consists of Twit- ter data from normal individuals diagnosed with depression, PTSD, and those normal group without mental disorders. On the other hand, the eRisk- 2018 dataset (Losada al., 2018) incorporates Reddit data from individuals diagnosed with de- pression and normal group.
0.30
0.25
8
Dreeq
0.10
0.05
Normal
Depression
PTSD
The original C2D2 dataset is in Chinese. Given that most existing mental health datasets are in English, we have translated the C2D2 into English, producing the C2D2-E (English version of C2D2). We have used machine translation tools for this purpose and performed sampling inspections to ensure quality. The label distribution and content of C2D2-E mirror those of the original C2D2 dataset.
Figure 3: Box plot of the distribution of Pfvey across dif- ferent groups. The vertical axis represents the Pfrey of cognitive distortions, and the horizontal axis represents the mental disorders of the users.
Figure illustrates that the Pfreg of cognitive distortions expressed on social media by individu- als diagnosed with depression is higher than that of both the general population and individuals with PTSD using the CLPsych-2011 dataset. Among individuals diagnosed with depression, PTSD, and the normal group, the average Pfreq is 13% for de- pression, 8% for PTSD patients, and 3% for the normal group. Our results indicate that cognitive distortions are most prominent in patients with depression, followed by those with PTSD, com- pared to individuals without any reported mental disorders. While cognitive distortions exist among the normal group, they are less prevalent compared to individuals diagnosed with mental disorders. By comparing Pfreg across various mental disorders. validate the relationship between cognitive dis- tortions and mental disorders, indicating that an increase in cognitive distortions could potentially
6.2 Cognitive Distortions Analysis on Social Media
In this section, the BERT-based baseline model, trained the C2D2-E dataset, to detect cognitive distortions social media posts from individuals diagnosed with depression and PTSD, as well as normal group. We focus on determining the prevalence of cognitive distortions on social media platforms. We define cognitive distortion prevalence as Prreq. which is computed as follows:
Ned
Pfreg
(1)
Nnor + Ned
where Nwor represents the number of normal post data, and Not represents the number of posts con- taining cognitive distortions. We calculate the Preeg for each user and present the results in a box plot.
*eRisk result is available in Appendix D.

---

serve as an overlooked characteristic for detecting mental disorders.
cognitive distortions. The resulting hidden states are then combined with post popularity in a feed- forward neural network for final classification. The simple formula is shown as follows:
PTSD
Depression
MR:11%
OG: 17%
:R:
8T:8%
FT: 6%
P2: 3%
ER: 315
P2:
hp BIL.TM(Bert(())
(2)
BW:
ER: 7%
BW: 34%
he BiLSTM
(3)
LL 44%
y = FFN : hpp Pfreg))
(4)
Figure 4: Category Differences in Cognitive Distortions among Different Mental Disorders. MR stands for Min- dReading, FT stands for Fortune-Telling, BW stands for Black and White thinking. LL stands for Labeling. OG stands for Overgeneralization, PZ stands for Personal- ization, and ER stands for Emotional Reasoning.
where hp E 12 We incorporate content fea- tures and numerical features related to cognitive distortion in our model. Our experiment was very straightforward, but we aim demonstrate the role of cognitive distortion through this direct approach. Result We conduct a preliminary evaluation of in- corporating cognitive distortion information into the detection of mental disorders. The results show some improvement in detection performance when including users' cognitive distortion information in the model. This approach is effective across dif- ferent platforms and in detecting various mental disorders.
Figure shows our analysis of the average distri- bution of cognitive distortion types related PTSD) and depression using the CLPsych-2015 dataset. Our results reveal distinct cognitive distortion pat- terns across different mental disorders. Specifi- cally, individuals diagnosed with depression show a heightened tendency towards emotional reason- ing and black and white thinking. In contrast, pa- tients diagnosed with PTSD display a significant occurrence of labeling. This unexpected finding suggests that PTSD patients may excessively cate- gorize benign situations as potential threats due to their traumatic experiences (Brewin, 2001; Van der Kolk, 2022). Our study of social media data un- derscores the importance of investigating cognitive distortions, potentially contributing to the improve- ment of therapeutic methods for treating PTSD. Moreover, our results highlight the feasibility of using computational techniques to analyze under- lying psychological phenomena from data.
Datasct
Model
F1
Rec
Pre
Bert-LSTM
0.73
0.75
0.72
CLPsych2005-ddeeesiinn
+CD feature
0.79
0.77
0.82
Bert-LSTM
0.74
0.76
0.73
CLPsych2015-ptsd
+CD feature
0.85
0.87
0.84
Bert-LSTM
0.59
0.62
0.59
2Risk201 8-deeresionn
+CD feature
0.65
0.63
0.65
Table 7: Comparison of performance metries for differ- ent models on multiple datasets. The method we use and the obtained values are shown in bold.
In various mental disorders across different plat- forms, the small modifications we made by incorpo- rating cognitive distortions yielded superior results compared to the original model. We believe that our experiment demonstrates the potential to en- hance the performance of existing mental disorder detection models through modeling cognitive dis- tortions.
6.3 Enhancing Mental Disorder Detection via Cognitive Distortion
After discovering the aforementioned phenomenon, we attempted to incorporate cognitive distortion features into a simple mental disorder detection model. Our approach involves analyzing a user's posting history, denoted as P, and extracting posts that contain cognitive distortions, denoted as C, using a finetuned BERT model. "Overgenrr alization: I failed an exam, so the whole discussion will fail."). By integrating cognitive distortions into the detection process, we aim to improve the performance of detecting mental disorders.
Conclusion
We have introduced the C2D2 dataset, which is the first public Chinese dataset focused on cognitive distortions. This dataset illuminates profound connection between cognitive distortions and men- tal disorders. Addressing these distortions could potentially enhance mental health interventions. By integrating cognitive distortions as an additional
Method Our approach involves utilizing an LSTM to capture the user's 's's historical context and model

---

feature, we have augmented the performance of existing mental disorder detection models. The introduction of the C2D2 dataset represents a valu- able contribution to computational psychology and serves as a catalyst for further research in this emerging field.
ethical considerations for these datasets have been guaranteed by the dataset creators.
9.2 Positive Outcomes
Improved Mental Health Services: Therapists and psychologists could use these techniques identify cognitive distortions quickly, allow- ing them to dedicate more time to therapeutic processes.
8 Limitation
While collecting data, we try our best ensure its quality. However, our data may still have poten- tial biases. Nevertheless, the data's 's inherent value remains unchanged.
Early Detection and Intervention: We could enable early detection of mental health dis- orders, leading to timely interventions. By identifying cognitive distortions social me- dia platforms, professionals might be able reach individuals who need help but haven't sought treatment.
The observed phenomena in mental illness data are validated using two mental health datasets, with the comparison between PTSD and depres- sion based single dataset. These datasets may have biases, but gathering diverse user-level data on various mental illnesses is challenging for us. Therefore, we don't claim our findings as defini- tive in psychology. However, we believe that these phenomena are likely to be widespread. We encour- age future researchers to analyze them further with more data. Recognizing and interpreting the data require additional endorsement from psychology experts, beyond the scope of our work.
Accessible Mental Health Resources: The public availability of the C2D2 dataset encour- ages other researchers continue this impor- tant work. Increased research could lead more accessible mental health resources, fur- ther destigmatizing mental health and support- ing individuals who might not otherwise have access to care.
We believe that there are many areas within our dataset that can be further explored and utilized,
9.3 Negative Outcomes and Mitigation Strategies
and our work only covers a small portion that we consider representative but limited.
Misinterpretation Data: Automated sys- tems can make mistakes, and those mistakes could have serious consequences in the realm of mental health. An incorrect interpretation of person's 's statements could lead to unnec- essary worry intervention, might miss a person who genuinely needs help.
9 Broader Impact and Ethical Considerations
9.1 Ethical Considerations
When dealing with sensitive data such as the psy- chological well-being of human subjects, special care must be taken. In this case, our main goal is to provide dataset for the general public, which makes confidentiality
Over-reliance on Technology: While our work could significantly aid mental health profes- sionals, there's a risk that some may be- come over-reliant on these computational tech- niques. Human judgment and intuition remain essential in mental health services, and bal- ance between technological and human input must be maintained.
more important.
Our rescarch has received approval from the In- stitutional Review Board (IRB) of our institution. All data annotators involved are over 18 years old and have signed informed consent forms agreeing to the public release of the data. We have removed any data that may be personally identifiable to the data subjects. Additionally, during the data an- notation process, had mental health expert monitoring the psychological well-being of the vol- unteers to ensure the well-being of the annotators. Volunteers had the option to withdraw from the process at any time. Finally, the remaining data we used from publicly available datasets, and the
Acknowledgments
We thank the anonymous reviewers for their in- sightful comments and suggestions. This work was supported by the National Key RD Program of China via grant 2021 YFF0901600 and the National Natural Science Foundation of China (NSFC) via grants 62176078 and 62236004.

---

References
Albert Ellis, 1994, Reason and emotion in psychother- apy, revised and updated. Carol Publishing Group.
Gene G Abel, David K Gore, Clarence L Holland, Nancy Camps, Judith V Becker, and Jerry Rathner. 1989. The measurment of the cognitive distortions of child molesters. Annals of sex research, 2:135-152.
Greg Feldman. 2007. Cognitive and behavioral thera- pies for depression: Overview, new directions, and practical recommendations for dissemination. Psy- chiatric Clinics of North America, 30( -)00--00.
Fatima Alhaj, Ali Al-Haj, Ahmad Sharich, and Riad Jabri. 2022. Improving arabic cognitive distortion classification in twitter using bertopic. International Journal of Advanced Computer Science and Applica- tions, 13(1 (3())854-860.
Erica E Fortune and Adam S Goodie. 2012. Cogni- tive distortions as a component and treatment focus of pathological gambling: review. Psychology of addictive behaviors, 26(2):298.
Krishna C Bathina, Marijn Ten Thij, Lorenzo Lorenzo- Luaces, Lauren A Rutter, and Johan Bollen. 2021. Individuals with depression express more distorted thinking on social media. Nature Human Behaviour, 5(4):458-466.
Theresa A Gannon and Devon LLL Polaschek. 2006. Cognitive distortions child molesters: A re- examination of key theories and research. Clinical psychology review, 260$):1000-0199
George Gkotsis, Anika Oelirich, Tim Hubbard, Richard Dobson, Maria Liakata, Sumithra Velupillai, and Rina Dutta. 2016. The language of mental health problems in social media. In Proceedings of the third workshop on computational linguistics and clinical psychology, pages 63-73.
Aaron T Beck. 1970. Cognitive therapy: Nature and relation behavior therapy. Behavior therapy. 1(2):184
Aaron Beck. 1979. Cognitive therapy of depression. Guilford press.
Constance Hammen. 1978. Depression, distortion, and life stress in college students. Cognitive Therupy and Research.
Judith S Beck. 2020. Cognitive behavior therapy: Ba- sics and beyond. Guilford Publications.
Chris R Brewin. 2001. A cognitive neuroscience ac- count of posttraumatic stress disorder and its treat- ment. Behaviour research and therapy, 39(4)3373- 393.
Robert Kohn, Saxena Shekhar, Itzhak Levav, and Benedetto Saraceno. 2004. The treatment gap in mental health carc-bulletin of the world health orga- nization.
David D Burns. 1981. Feeling good. Signet Book.
Richard S Lazarus and Susan Folkman. 1984. Stress, appraisal, and coping. Springer publishing company.
N Chowdhary, AT Jotheeswaran, A Nadkarni, SD Hol- lon, M King, MJD Jordans, A Rahman, H Verdeli, Araya, and V Patel. 2014. The methods and out- comes of cultural adaptations of psychological treat- ments for depressive disorders: systematic review. Psychological medicine, 13(6):1131-11
David E Losada, Fabio Crestani, and Javier Parapar. 2018. Overview of erisk 2018: Early risk prediction on the internet (extended lab overview). In Proceed- ings of the 9th International Conference the CLEF Association, CLEF, pages 1-20.
Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015. Clpsych 2015 shared task: Depression and ptsd on twitter. In Proceedings of the 2nd workshop on computational linguistics and clinical psychology: from linguistic signal to clinical reality, pages 31-39.
Peter Marton, M Churchard, and Stan Kutcher. 1993. Cognitive distortion in depressed adolescents. Jour- nal of Psychiatry and Neuroscience, 18(3): 103.
Eben L McClenahan, 2005.
Cognitive therapy tech- niques: A practitioner's guide. Psychiatric Services, 56(9): 1664-1144.1
Roger Covin, David JA Dozois, Avital Ogniewiez, and Pamela M Seeds. 2011. I. Measuring cognitive errors: Initial development of the cognitive distortions scale (cds). International journal of cognitive therapy, 4(3):297-322.
Peter Muris and Andy Field. 2008. Distorted cogni- tion and pathological anxiety in children and adoles- cents. Cognition and emotion, 3953)2395-421.
Coralijn N Nas. Daniel Brugman, and Willem Koops. 2005. Efffects of the equip programme on the moral judgement, cognitive distortions, and social skills of juvenile delinguents. Psychology, Crime & Law, 11(4):421-434.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 657-668.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 29:3504-3514.

---

Vikram Patel and Arthur Kleinman. 2003. Poverty and common mental disorders in developing countries. Bulletin of the World Health Organization, 81 $1:609- 615.
Carrie Yurica and Robert DiTomasso. 2005. Cogni- tive distortions. Encyclopedia of cognitive behavior therapy, pages 117-122.
Caleb Ziems, Minzhi Li, Anthony Zhang, and Diyi Yang. 2022. Inducing positive perspectives with text reframing. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3682-3700.
Benjamin Shickel, Martin Heesacker, Sherry Benton, Ashkan Ehadi, Paul Nickerson, and Parisa Rashidi. 2016. Self-reflective sentiment analysis. In Pro- ceedings of the Third Workshop Computational Linguistics and Climical Psychology, pages 23-32.
Benjamin Shickel, Scott Siegel, Martin Heesacker, Sherry Benton, and Parisa Rashidi. 2020. Automatic detection and classification cognitive distortions in mental health text. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 275-280. IEEE.
A Ensuring Volunteer Well-being
To prioritize the mental health of our volunteers, we have implemented the following measures through- out the data collection process:
Sagarika Shreevastava and Peter Foltz. 2021. Detecting interac- tions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151-158.
cognitive distortions from
Precautions: Prior to participation, volunteers are provided with detailed information about the task, including potential exposure to chal- lenging situations and cognitive distortions.
Tactem Simms, Clayton Ramstedt, Megan Rich, Michael Richards, Tony Martinez, and Christophe Giraud-Carrier, 2017. Detecting cognitive distortions through machine learning text analyties. In 2017 IEEE international conference on healthcare infor- matics (ICHI), pages 508-512. IEFE
Informed Consent: Volunteers provide in- formed consent before engaging in the data collection process. We emphasize that their participation entirely voluntary, and they have the freedom to withdraw at any time without facing any consequences.
Craig W Strohmeier, Brad Rosenfield, Robert A DIT- omasso, and J Russell Ramsay. 2016. Assessment of the relationship between self-reported cognitive distortions and adult adhd, anxiety, depression, and hopelessness. Psychiatry research, 238: 153-158.
Supportive Environment: We maintain open channels communication with volunteers, encouraging them to share any concerns or dif- ficulties they may encounter during the task.
Justin Tauscher, Kevin Lybarger, Xiruo Ding, Ayesha Chander, William Hudenko, Trevor Cohen, and Dror Ben-Zeev. 2023. Automated detection of cogni- tive distortions in text exchanges between clinicians and people with serious mental illness. Psychiatric services, 74<4):407-410,
Anonymity and Confidentiality: Volunteers are assured that their identities will remain anonymous and their personal information will be kept confidential. This fosters a safe space for open and honest participation.
Ana-Sabina Uban, Berta Chulvi, and Paolo Rosso. 2021. An emotion and cognitive based analysis of men- tal health disorders from social media data. Future Generation Computer Systems, 124:480-494.
Bessel Van der Kolk. 2022. Posttraumatic stress disor- and nature of trauma. Dialogues in clinical neuroscience.
By implementing these measures, we demonstrate our commitment the well-being of our volun- teers and ensure an ethical and responsible data collection process.
Märta Wallinius, Peter Johansson, Martin Larden, and Mats Dernevik. 2011. Self-serving cognitive distor- tions and antisocial behavior among adults and ado- lescents. Criminal justice and behavior. 38(3):286 301.
B Training Details
Bichen Wang. Yanyan Zhao, Xin Lu, and Bing Qin. 2022. Cognitive distortion based explainable depres- sion detection and analysis technologies for the ado- lescent internet users on social media. Frontiers Public Health, 10.
We mention our testing of the large language model and finetuning of the pretrained model. Specifically, our prompt using in LLM and the training details are as follows.

---

Description: Cognitive distortions refer to detrimental patterns of thinking that are prevalent in individuals. Assistance is re- quired to identify these patterns of thinking based on the textual content. The following are the distinct categories of cognitive dis- tortions along with their precise definitions: Definite: Definitions of seven types of cog- nitive distortions
Model
F1
Rec
Pre
Bert
0.72
0.74
0.71
Roberta
0.74
0.75
0.74
ChatGPT (Zero-shot)
0.45
0.42
0.54
ChatGPT (Few-shot)
0.56
0.58
0.55
Table 8: Performance of baseline models for the C2D2- tasks. metrics are calculated using macro- averaging.
Examples: total of twenty-four illustra- tive instances
D Cognitive Distortions Analysis on eRisk-2018
Thought: Input data
For Cognitive Distortion Detection, the learning rate is set to le-5, and the AdamW optimizer is employed. The base version of the model, without any modifications, is used. To determine the best performing model, the learning rate and optimizer are chosen based on empirical observations and pre- vious studies in the field. The model exhibiting the lowest loss on the validation set, which is separated from the training set, is selected for testing. The re- sults averaged over runs to ensure robustness and mitigate the effects of random initialization.
Cognitive Distortion Difterent Groups(eRisk 2018)
0.18
0.7
0.72
0.15
0.13
0.03
Normal
Depression
For Mental Disorder Detection, both non- pretrained models and the pretrained model, in their base versions, are utilized. The learning rate for non-pretrained models is set le-3, while the pretrained model has a learning rate of 1e-6. The LSTM hidden layer state configured with 128 dimensions to capture complex temporal dependen- cies. Similar to the Cognitive Distortion Detection, the AdamW optimizer is employed. The official test set is used for evaluation, and a separate valida- tion set is created from the training set. The model exhibiting the lowest loss the validation set is selected for testing. These hyperparameter settings and model choices are determined through empir- ical evaluation and existing literature on similar tasks
Figure 5: Box plot of the distribution of Pfreg across different groups on eRisk-2018.
As shown in Figure 5, we are presenting here the results obtained on the eRisk-2018 dataset, and without a doubt, the differences in cognitive distor- tions between the depression group and the normal group once again evident. In this dataset, the proportion of normal users exhibiting cognitive dis- tortions is higher than that in the Clpsych-2015 dataset.
C Detailed Cognitive Distortion Detection Results
As shown in Table 8, we also provide some exper- imental results for C2D2-E. Similar to the results for C2D2, but the results for C2D2-E generally higher. Apart from the impact of translation, attribute this the improved ability of the exist- ing model to comprehend English, especially for today's ' LLM.


Reviews:
[{"review_id": "HPXYIRgHgT", "reviewer": "Reviewer_8gtz", "summary": "", "questions": "Will the English version of the dataset also be made publicly available? It might widen the scope of its usefulness in other studies.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your in-depth review of our paper. Your feedback is crucial to us, and we genuinely appreciate the time and effort you've taken to evaluate our work. We are pleased by your acknowledgment of the value of our exploration into cognitive distortions in mental health issues such as depression and PTSD. Conducting analyses traditionally deemed challenging by psychologists using computational techniques is indeed we want to show in our study. \n\nWe genuinely value your constructive criticism and your emphasis on potential methodological considerations and opportunities for enhancement. Please find our responses to the points you raised below:\n\n1.__Impact of English Translation:__\n\nWe acknowledge the issue you've raised. We are indeed cognizant of this matter and due to space constraints, we included the results of the experiment involving English translations  in the appendix (Appendix B). Our experiment reveals that translation does indeed influence performance to some extent. However, the experiments suggest that the performance differences introduced by translation are within an acceptable range. Furthermore, we believe that the varying capabilities of modern models in handling Chinese and English also contribute to this aspect.\n\n__2. Availability of the Dataset:__\n\n__Will the English version of the dataset also be made publicly available?__\n\nYes, we will release our C2D2-E. We are also grateful for your inquiry regarding the availability of the English version of our dataset. In point of fact, we intend to release our CD2D-E to facilitate wider recognition and replication of our research and findings. While our preference is for researchers to engage with the original Chinese version to foster a culturally diverse investigation into mental well-being, the release of C2D2-E is deemed necessary by us.\n\n__3.Remaining Issues:__\n\nWe deeply regret the oversight in spellings and format. Regarding the syntactical and formatting matters you've highlighted, as well as any other oversights, we sincerely apologize. We are committed to rectifying these aspects. Furthermore, we acknowledge and are acting upon your suggestion regarding the addition of pertinent reference literature.\n\nYour commentary is immensely appreciated, and should you have any further suggestions, please do not hesitate to share them with us. We hope our response addresses your concerns and earns your further endorsement. We are truly thankful for your meticulous review of our paper. Your evaluation holds great significance for us, and we deeply value your insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An original topic and the introduction of a new dataset, created through a very thorough process\n- Interesting findings related to the specific cognitive distortions that occur in various mental health disorders (depression and PTSD)", "reasons_to_reject": "- Some potential methodological problems or unclarities in the cross-lingual experiments. It would have been useful to see an analysis of the impact of translation on the dataset, since this might distort the annotations themselves - at least a reiteration of the cognitive distortion detection experiment but on the English version of the data", "questions_to_authors": "Will the English version of the dataset also be made publicly available? It might widen the scope of its usefulness in other studies.", "paper_topic_and_main_contributions": "The paper introduces a new dataset of examples of various cognitive distortions in the Chinese language. The authors then perform machine learning experiments to test whether cognitive distortions are possible to detect automatically from text, and also test their relation to mental disorders, based on some English datasets annotated for depression and PTSD.\n\nThe dataset is created through a complex and rigorous process: by first doing a selection for choosing the set of responders, then asking them to produce examples of cognitive distortions, which are then verified by experts for different criteria. The dataset is made public.\n\nIn the experiments, pretrained LLMs are used to detect cognitive distortions in the proposed dataset and in the English mental disorders dataset, using machine translation to make the datasets compatible (following two objectives: measuring the prevalence of cognitive distortions in mental health disorder data using the trained models, and trying to improve mental disorder detection using cognitive distortion information).\n\nAll the responders used for creating the dataset have agreed to participate in the study, and the ethical section contains a thorough discussion.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Not directly about cognitive distortions, but on the relation between the more general \"cognitive styles\" (including cognitive biases) and mental health disorders:\n\nUban, A. S., Chulvi, B., & Rosso, P. (2021). An emotion and cognitive based analysis of mental health disorders from social media data. Future Generation Computer Systems, 124, 480-494.", "Typos_Grammar_Style_and_Presentation_Improvements": "- line 72: will contributes -> contribute\n- title of Section 4.2: \"Related dataset\" -> \"Related datasets\"?\n- line 324: preferable not to start the paragraph with \"And\"\n- lines 365-367: duplicate sentence (at least in meaning)?"}}, {"review_id": "zmzlWjkI0i", "reviewer": "Reviewer_DcU8", "summary": "", "questions": "1) Will the C2D2 dataset be available in Chinese, English, or both?\n2) What are the volunteer demographics? \n3) Where/how were the volunteers recruited from?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you immensely for the detailed and constructive feedback on our paper. We truly appreciate the time and effort you invested in evaluating our work. It is heartening to know that you found our approach intriguing and see its potential for replication in other languages. Your endorsement is encouraging for us. Please find below our responses to your queries:\n\n\n__1.Availability of the Dataset:__\n\n**Will the C2D2 dataset be available in Chinese, English, or both?**\n\nBoth Chinese and English are available. In point of fact, we intend to release our CD2D-E to facilitate wider recognition and replication of our research and findings. Specifically, the release of the CD2D-E aims to foster a broader understanding and replication of our research. While we want researchers to engage with the original C2D2 to appreciate the cultural nuances and complexities of mental well-being, we deemed the English C2D2-E necessary for global inclusivity.\n\n**2. Addressing Researcher Biases in the Dataset:**  \n\nWe acknowledge your concerns regarding potential researcher biases. It's pertinent to mention that our volunteer selection was thorough, leveraging social media platforms for initial registration followed by a meticulous selection process. This process was carefully calibrated to ensure gender and age balance, thereby mitigating biases. As delineated in our paper, each volunteer had firsthand experience with cognitive distortions and received specialized psychological training by us.\n\nAlthough the inherent intricacies of cognitive distortions did necessitate expert corrections, we ensured multiple layers of checks. Established standardized protocols, encompassing training, labeling, data inspection, and correction, were subjected to review by multiple specialists. This collaborative approach, involving multiple experts in discussions and corrections, was strategized to curtail the subjective inclinations of any single individual. We concede that absolute objectivity is challenging. However, our rigorous methodologies were designed to attenuate any potential biases to the maximum extent.\n\n\n__3.Perspective on Self-disclosure and Potential Self-diagnosis::__\n\nWe concur that relying solely on self-disclosure poses inherent challenges, especially given the nuances of mental health disclosures on social platforms. But using datasets based on self-disclosure has become a common practice due to the inherent challenges in obtaining accurate mental health information on social media platforms. The publishers of the dataset we used say that they have vetted the credibility of the self-disclosed content, hence our decision to utilize it directly. Nevertheless, your emphasis on the tentative nature of such disclosures is well-taken.\n\n__4.Support from the Fields of Psychology, Psychiatry, and Medicine:__\n\nThank you for highlighting the potential relevance of references from psychology, psychiatry, and medicine to our study. We believe that our findings in section 6.2 offer perspectives that traditional psychological research often struggles to access directly. Our analysis, grounded in objective data spanning hundreds of thousands of entries, provides insights from angles seldom explored in conventional psychological studies.\n\n__5.Addressing Typographical and Formatting Issues:__\n\nWe deeply regret the oversight in spellings and format. We assure you of a thorough proofreading to ensure these are rectified.\n\nWe are thankful for your comprehensive review and the clarity of your feedback. We are confident that our responses will elevate the overall quality and depth of our work. Your review has been pivotal in highlighting areas for enhancement, and we are optimistic about addressing your concerns to further solidify our contribution to the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Presents an interesting tool to be used to detect cognitive distortions and has an interesting method that could be replicated to re-create a cognitive bias dataset with another language (e.g., English, French, etc.,).", "reasons_to_reject": "By collecting data from volunteers with prompts and then correcting the volunteer's answers (as discussed in section 3.4), the dataset may include the researcher's biases and be less generalizable or naturalistic. Unclear of potential biases in the demographics of the volunteers who generated the data. The biases and limitations of the dataset due to it being generated by volunteers and then corrected should be discussed. As well, since the authors use social media datasets it's unverifiable if the users are formally diagnosed, thus the findings discussed on 449 to 453 and 465 to 471 as well of the rest of the section should emphasize how these findings are based on self-disclosure and potential self-diagnosis.", "questions_to_authors": "1) Will the C2D2 dataset be available in Chinese, English, or both?\n2) What are the volunteer demographics? \n3) Where/how were the volunteers recruited from?", "paper_topic_and_main_contributions": "This paper discusses the creation of the Chinese Cognitive Distortion Dataset, a dataset based on Chinese tweets which were then translated into English. This dataset can be used to detect cognitive distortions (present or not) and the type of cognitive distortion (e.g., emotional reasoning, overgeneralization, etc) in text. It also examines its use in previously collected datasets with social media data from users with mental illnesses.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2, in the notes missing spaces between the \"&\" in the first line \n\nline 203, missing spaces between the \"&\".\n\nLine 349, extra space before \"Table\""}, "extra_text": {"Missing_References": "The findings discussed in section 6.2 are similar to research on cognitive bias types in psychology although its novel being found on social media. Could support findings with research from psychology, psychiatry and the medical field."}}, {"review_id": "aExCY7CUuj", "reviewer": "Reviewer_jS9Q", "summary": "", "questions": "Do you pay the volunteers and the experts? If so, how do you pay them? This needs to be specified in the Ethics section. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback on our paper. We genuinely appreciate the time and effort you invested in reviewing our work. We are sincerely grateful for your recognition of the novelty of our work and your attention to the field of mental health. Your positive comments are very encouraging, and we are keen to address the concerns you raised.\nPlease find below our responses to your queries:\n\n__1. Human expert performance on our dataset:__\n\nYour point on the absence of human expert performance metrics is valid. To clarify, we did evaluate the performance of two psychology experts on a 500-entry sample from our dataset, which yielded a macro-F1 score of 0.87. and kappa=0.76. Considering the inherent subjectivity of the task, we believe the performance of human experts is reasonable.\n\n__2.Gap between the thoughts from real patients and the composed thoughts:__\n\nWe acknowledge and appreciate your concern. Our approach to having volunteers compose distorted thoughts ensured controlled and diverse data. To ensure the thoughts collected closely mirror reality, we screened volunteers based on scales. Those with personal experiences of cognitive distortions were included to add authenticity to the dataset. This process eliminated half of the potential volunteers. Furthermore, our review standards were stringent, involving multiple experts to ensure data reasonableness. Our goal has been to cooperate with experts and ensure data quality through rigorous processes. \n\n__3.Compensation for volunteers and experts:__\n\n__Do you pay the volunteers and the experts? If so, how do you pay them?__\n\nYes, both volunteers and experts were compensated for their time and contribution. Before initiating the data collection, every participant had signed and confirmed our compensation agreement, which clearly defines the rights and responsibilities of both parties. Upon completion of the dataset annotation, we promptly compensated all participants as per the agreement and ensured that the funds were successfully transferred to their provided bank accounts.\n\nOnce again, we deeply appreciate your invaluable feedback and patience throughout this review process. Your insights offer us a clear direction to refine our paper. We believe that adjustments and additions will enhance the overall quality and depth of our work. We are confident we can address your concerns and further strengthen our contribution to the community. Thank you once again for your invaluable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed new dataset is very valuable, as currently, there is a lack of publically available datasets in the mental health domain. Such a dataset will greatly boost the research for NLP in assisting mental health treatment. \n2. The proposed dataset is the first one for Chinese. This is especially valuable since, in China, the scarcity of mental health professionals and resources is even more severe. There is an increasing demand for building automatic assistance for mental health support. \n3. The dataset shows good quality, verified by experts. \n4. The experiments and analysis demonstrate the great potential of the vast scope of future research that can be extended based on this dataset. ", "reasons_to_reject": "1. No evaluation regarding the human expert performance on this dataset. \n2. The dataset construction process involves asking the volunteers to \"compose\" distorted thoughts but not taking the thoughts from potential patients with mental disorders from online posts. Such a method, theoretically as a kind of proxy to the thoughts of real patients,  is further verified by experts, as stated in the paper. However, I'm wondering what the gap is between the thoughts from real patients and the composed thoughts. ", "questions_to_authors": "Do you pay the volunteers and the experts? If so, how do you pay them? This needs to be specified in the Ethics section. ", "paper_topic_and_main_contributions": "In this paper, the authors propose a cognitive distortion dataset in Chinese, consisting of 7,500 examples with paired scenes, thoughts, and cognitive distortion labels. For the dataset construction, the authors do not take the traditional approach of annotating online posts, but instead recruit volunteers and use interviews to further select qualified volunteers. The volunteers are instructed to compose thoughts based on the given scene, and also label the thoughts composed by other volunteers. The quality of the dataset is further evaluated by experts. \n\nThe authors conduct experiments using various baseline models including fully-supervised training, as well as few-shot and zero-shot using ChatGPT, demonstrating the challenge of this dataset. The author also conducts experiments and comprehensive analysis using the cognitive detection model on datasets from social media, showing the correlations between cognitive distortions and various mental health disorders. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0GKSZYu6FS", "reviewer": "Reviewer_JxAP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for investing the time to rigorously review our manuscript. We sincerely value your insights and commendations, which have played a vital role in elevating the quality of our work. Your feedback not only deepens our understanding but also offers a wider perspective, ensuring that our research stands up to the utmost academic scrutiny. We are genuinely appreciative of your constructive remarks that have considerably enriched our paper. Here are our detailed responses to your highlighted points:\n\n**1. Dataset Construction and Language Annotation:**  \nWe resonate with your concerns. Recognizing the universal relevance of English, we introduced the C2D2-E dataset. Despite the plethora of mental health datasets in English, there's a conspicuous lack of such datasets in the Chinese language. Our aspiration is that multilingual and cross-cultural datasets will spearhead the future endeavors in this sector. \n\n**2. Human Expert Performance on Our Dataset:**  \nYour observation about the omission of human expert performance metrics is precise. For clarification, we subjected two psychology experts to a 500-entry sample from our dataset. Their evaluation reflected a macro-F1 score of 0.87 and kappa=0.76. Considering the inherent subjectivity of the task, we believe the performance of human experts is reasonable. These figures underline the potential for subsequent model refinement. \n\n**3. Clarifications on Experimental Setup:**  \n- **Post-level Model Tests:** We believe that the mental disorder detection at the post-level might sometimes bypass the nuanced cognitive distortions. To illustrate, the statement \"I must be a fool\" exemplifies 'labeling', a cognitive distortion that might not be as prominently captured in some post-level annotations. Conversely, expressions such as \"I have severe insomnia and feel nauseous\" lean more towards symptomatic descriptions. It's also noteworthy that occasional cognitive distortions can be observed even in individuals without diagnosed mental disorders. Therefore, the connection between cognitive distortions and mental disorders should be established from social media through user's long-term social media history rather than a single post.\n\n>> User-level datasets contain this information because it includes the entire history of the patient, including daily thoughts. We acknowledge, however, that the efficacy of our datasets in informing post-level studies remains to be empirically validated. We greatly appreciate your insights, and they could very well shape the trajectory of our subsequent research efforts.\n\n\n- **Presence of Cognitive Distortions not in All Posts:** As shown in our example, It's crucial to highlight that not every post delineating mental disorders exemplifies cognitive distortions. A piece of content that seems immaterial on the surface might, from a psychologist's lens, be of utmost significance.\n\n- **Rationale Behind Extracting Posts with Cognitive Distortions:** Our methodology to detect mental disorders is premised on the identification of enduring cognitive distortion patterns in users. The most rudimentary strategy is to extract all potential posts containing cognitive distortions for modeling, serving as an indicator of long-term cognitive distortion. This approach harmonizes with our agenda of demonstrating the instrumental role of cognitive distortions in discerning mental disorders.\n\n- **Choice of Bert Model:** Your inquiry about the BERT vs. RoBERTa: We embraced BERT due to its universal endorsement as a standard in numerous computational endeavors. We aim to establish a benchmark for successive researchers in mental health detection. Your perspective makes sense; from a performance standpoint, RoBERTa should be the choice for analysis. However, our effort is inclined towards an analytical exploration and a foundational outlook. Therefore, we believe it sufficiently backs our findings and argument..\n\n**4. Issues with $p_{popularity}$:**  \nWe regret the oversight concerning the term $p_{popularity}$ in Equation (4). It was an inadvertent typo and should be read as $p_{freq}$, as delineated in Equation (1). We are thankful for your patience and kindly request any further clarifications on potential ambiguities.\n\n**5. Methodology and Multi-label Tasking:**  \nThe proposition of conceptualizing the problem as a multi-label task when numerous cognitive distortions emerge concurrently is an astute one. However, we believe that our current approach is also reasonable, as psychologists typically prioritize addressing the most evident cognitive distortions. At the same time, we need to take into account our annotation costs.\n\n\nOnce again, we deeply appreciate your invaluable feedback and patience throughout this review process. Your insights offer us a clear direction to refine our paper. We believe that adjustments and additions will enhance the overall quality and depth of our work. We are confident we can address your concerns and further strengthen our contribution to the community. Thank you once again for your invaluable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The C2D2 dataset is valuable for the mental health field. It is a publicly accessible resource within this domain that includes numerous texts recording individuals’ thoughts in various scenes with cognitive distortion. This dataset provides a profound connection between cognitive distortions and mental disorders. Addressing these distortions could potentially enhance mental health interventions. The authors find that integrating cognitive distortions as an additional feature can augment the performance of existing mental disorder detection models. In addition, the overall structure is well-organised.", "reasons_to_reject": "The motivation \"However, mental health issues in developing countries often receive limited attention\" described for not annotating the dataset in English is inaccurate. English has a broader applicability, and the authors could also consider incorporating English annotations. While developing countries are essential to consider, it is also worthwhile to focus on datasets in languages other than Chinese.\n\nFor the test datasets, do the authors consider post-level model tests? Because the C2D2 is also the post-level dataset. Additionally, some other datasets could be added, like SWMH, T-SID, etc. \nIn Sec 6.2, the authors should train the RoBERTa-based model because of the performance shown in Table 5.\nIn Fig.4, do all posts with depression or PTSD have cognitive distortions? Add Non-sd labels?\n\nThere is no human expert performance on this dataset.\n\nMany details need to be expressed clearly. What is the meaning of ppopularity in (4)?\n\nThe reasonable about \"extracting posts that contain cognitive distortions\" should be discussed.  Why not other methods?\n\nIf possible, it is suggested that the problem could be treated as a multi-label task when multiple cognitive distortions occur simultaneously.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about mental health analysis topic. The main contributions are: a publicly accessible Chinese Cognitive Distortion Dataset is constructed for the first time. The authors explore the association between cognitive distortions and various mental disorders on\nsocial media. The authors attempt to incorporate information about users’ cognitive distortions into mental disorder detection models to illustrate the effectiveness of using cognitive distortion.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a Chinese corpus on cognitive distortions including testing some techniques for capturing cognitive distortions and some multi-lingual analysis for detecting depression and PTSD. 

Overall strengths include a new dataset that is likely to be useful in the NLP for mental health space and an approach to create such data that could be useful in other languages while weaknesses include a lack of evaluation on the quality of human experts and a lack of integrating cognitive distortions, the main feature of the Chinese dataset, into the multilingual analyses.
Decision: Accept-Findings