Identifying {Early Maladaptive Schemas} from Mental Health Question Texts
Authors: Sujatha Das Gollapalli
Beng Heng Ang
See-Kiong Ng
Venue: EMNLP 2023 Findings (2023)
Keywords: Mental Health
Schema Therapy
Early Maladaptive Schema
Personality Disorders
Classification
TL;DR: Predicting EMS (Early Maladaptive Schema) labels for mental health QA forum posts

Abstract: In Psychotherapy, {maladaptive schemas}-- negative perceptions that {an 
individual has of the self, others, or the world that endure despite objective reality}--
often lead to  resistance to treatments and  relapse of mental health issues such as depression, anxiety, panic attacks etc. Identification of early maladaptive schemas (EMS) is thus a crucial 
step during Schema Therapy-based counseling sessions, 
where patients 
go through a detailed and lengthy EMS questionnaire. However, such an approach is not practical  in "offline" counseling scenarios, such as community QA forums which are gaining popularity for people seeking mental health support. In this paper, we 
investigate both LLM (Large Language Models) and non-LLM approaches for identifying EMS labels using resources from Schema Therapy. 
Our evaluation indicates that 
recent LLMs can be effective for identifying EMS but their predictions lack explainability and are too sensitive to precise `prompts'. 
Both LLM and non-LLM methods are unable to reliably address the {null} cases, i.e. cases with no EMS labels. However, we posit that the two approaches show complementary properties and together, 
they can be used to further devise techniques
for EMS identification.

Identifying Early Maladaptive Schemas from Mental Health Question Texts
$ Sujatha Das Gollapalli, BBeng Heng A SSe--Kiog Ng
Institute of Data Science, National University of Singapore Integrative Sciences and Engineering Programme, National University of Singapore idssdgênus. edu. sg, bengheng. angêu. nus .eds, seekiongêuus edu. sg
Abstract
increasing adoption in recent counseling practice due its effectiveness (Masley et al., 2012; Bakos et al., 2015; Taylor et al., 2017; Pecters et al., 2021).
Psychotherapy, maladaptive schemas- neg- ative perceptions that individual has of the self, others, or the world that endure despite ob- jective reality- often lead to resistance treat- ments and relapse of mental health issues such as depression, anxiety. panic attacks etc. Iden- tification of early maladaptive schemas (EMS) is thus crucial step during Schema Therapy- based counseling sessions, where patients go through detailed and lengthy EMS question- naire. However, such an approach is not prac- tical 'offline' counseling scenarios, such as community QA forums which are gaining pop- ularity for people secking mental health sup- port. In this paper. we investigate both LLM (Large Language Models) and non-LLM ap- proaches for identifying EMS labels using re- sources from Schema Therapy. Our evalua- tion indicates that recent LLMs can be effective for identifying EMS but their predictions lack explainability and are too sensitive precise pprompts'. Both LLM and non-LLM methods are unable reliably address the null cases, i.e cases with no EMS labels. However, posit that the two approaches show complementary properties and together, they can be used fur- ther devise techniques for EMS identification.
Post: Am going to be alone forever? I feel like I'm trying to convince myself that I'm okay when I'm not. I'm always blocking out the bad things and forgetting. also feel like nobody cares for me and they never will. I feel truly alone.
Expert-assigned EMS Labels:
1. Abandonment/Issaaiiin (AB)
5. Social Coolation/Alienatiio (SI)
Table 1: A cQA forum question is shown with the EMS
labels assigned by professional counselors
In Schema Therapy, early maladaptive schemas refer to "pervasive, unhealthy patterns of thought and behaviour" in individuals that develop from childhood experiences and affect their emotions, cognitions, relationships with others, and overall responses to life events. ST posits that to effec- tively treat mental health symptoms, necessary to identify these underlying maladaptive schemas and alter them with various targeted cognitive, ex- periential, and behavioral strategies (Young and Klosko, 1994; Young et al., 2006). In his pioneer- ing work, Young proposed set of 18 Early Mal- adaptive Schemas (EMS)² that are listed in Table 2. Motivation: Community QA (cQA) forums on various topics including mental health are now ubiquitous (Griffitts et al., 2012), and studies have noted an increase in sharing of problems and seek- ing of both peer and professional support in cQA forums for mental health (De Choudhury and De. 2014; Prescott et al., 2017; Nobles et al., 2020). Due the accessibility and cost effectiveness of cQA forums in providing equitable health support.
1 Introduction
Psychotherapy researchers and prac- titioners have noted that patients with maladap- tive schemas" do not respond fully or resist tra- ditional cognitive-behavioral treatments and tend to relapse (Shea et al., 1990; Sanislow and Mc- Glashan, 1998). Schema Therapy (ST), introduced by Young (2006), addresses this issue and builds on the emdely-employed Cognitive Behavioral Ther- apy (Craske, 2010) and other counseling theories to conceptualize and treat mental health conditions that are due to personality disorders. ST has seen
Background:
[Attorr contributed equally.
Defn. 3 at //tttp:o//iitiionaryy. wwwpaaayyapa.orr.r.o.... orggsc//e
Pnttss//www.scheeaterraayy wttp:/cccmccc..ceeee

---

EMS Labels Names
1. Abandonneend/nstabiiityy (AB), 2. Mistrust/Abuse (MA), 3. Emotional Deprivation (ED)
4. Defectiveness/Shane (DS). 5. Social Isolation/Allenationn (SI), 6. Dependenne/Inncompeaaee (DI)
7. Vulnerability to Harm or Illness (VH), Enmeshmenn/Undeevlopedd Self (EM), 9. Failure to
achieve (FA), 10. EntillementGraaddoostty (ET), 11. Insufficient Seff-Contro//eel--iiciiiinn (IS),
12. Subjugation (SB), 13. Self-Sacrifice (SS), 14. SernivvatSeeingggeeegniiiiooooo SSekinn (AS),
15. Negativity/Pessimism (NP), 16. Emotional Inhibition (EI),
17. Unrelenting Standarbs/Hypercciiicaaaes (US), 18. Pumitiveness (PU)
1. Abandonmenn//Isttabiiiyy (AB): The perceived instability unreliability of those available for support and connection. Involves the sense that significant others will not be able continue providing emotional support. connection, strength.
5. Social Isolation/Alienation (SI): The feeling that one isolated from the rest of the world, different from other people, and/or not part of any group or community.
Table 2: Young's list of 18 EMS labels. Due to space constraints, we only list the label names, and partial descriptions for two EMS labels and refer the reader Table 13 in Appendix well as the website.
numerous cQA websites are now available for men- tal health.
application of sentence similarity and textual en- tailment, respectively, to Young's Schema Ques- tionaire (YSQ), resource employed by Schema Therapists during interactive counseling sessions. (2) In keeping with recent, exciting advancements in large language models capabilities, we employ zero-shot settings with L.MSs for EMS prediction. Our goal is to examine the sophisticated reading and language understanding abilities afforded by these models can be effectively harnessed for EMS labeling via suitable 'prompts'.
CQA forums invite large volumes of diverse questions. Given the limited numbers of profes- sional counselors to cater to this demand, fast and effective "problem solving" support through au- tomated methods for questions triage and mental health issue identification are desirable in such settings (Cohan et al., 2016; Yates al., 2017). Against this background, the question we ask in this paper is: Is it possible to automatically iden- tify EMS labels characterizing negative perceptions from the texts of mental health questions?
(3) We provide evaluation of methods against LLMs small dataset of about thirty counseling questions annotated with EMS labels by experts. This high-quality specialized dataset' con- tains labels and justifications for the labels provided by qualified counselors who use Schema Therapy in practice. It a first expert-curated dataset on this topic. Though admittedly small, it provides rea- sonable benchmark for first investigation of the feasibility of automated approaches for detecting EMS labels from cQA mental health question texts, and comparing non-LLM methods with LLMs.
Consider for instance question shown in Ta- ble 1 from the CounselChat dataset (Bertagnolli, 2020). The EMS labels assigned this text by two trained counselors are shown along with high- lighted text spans based on which they made the label assignments. We included the descriptions of the EMS labels "I. Abandonmen//Instabiiin (AB) and 5. Social Allenntio (SI)" in Table 2 for reference. We posit that effective counseling responses such questions not only answer the overt questions but also account for the negative perceptions about oneself captured the backsto- ries of the inquirers. Towards this objective, we present a first study on the automatic assignment of EMS labels for counseling question texts.
2 Methods
Schema Therapy- based counseling sessions often include the administration of lengthy and detailed "Young's Schema Questionaire" (YSQ) as an initial step. This questionaire comprises 232 statements about oneself that are rated on
Our Contributions: (1) We describe two tech- niques for predicting EMS labels through novel
'Examples https: h/www//www/// 7cups. cwm/qa/ and https://www ment a heal th www.hhhaaatttttueeeeeeeeeeeee wwt.foorruunne/orum///
"Our code and data is available for academic purposes at https:// thtbbb/gtttuu...t cwm/NUS- Con//NU-IDS/eeee IDS/Ims_/mmemeee htts///msmenttllleeeeeeeeeeaaa wet.mmntteeaatttt..

---

scale of 1 ("Completely untrue of myself")-6 ("Describes me perfectly") by the counselee." For example, two statements from YSQ pertain- ing to the EMS labels for the question in Table I are
Vi. For the example in Table I, the statement from YSQ. "In the end, will be alone." " is entailed by the sentence "Am I going to be alone forever?" from the question and therefore, EPM assigns the label "1. Ahandonmen//Istablitty (AB)" to this question.
1. feel alienated from other people. (SI)
2. In the end, will be alone. (AB)
Notice that EMS labels from Schema Therapy² are descriptive and cover range of personal feelings, emotions, outlooks and mental thoughts. As such, with some linguistic understanding. one is able to reasonably assess if a specific EMS label is applicable to a given post using these descriptions. Since recent Large Language Models (LLMs) have shown promising abilities in language reading and understanding, we investigate if these descriptions can be effectively harnessed for predicting EMS labels with LLMs.
The counselor, who is familiar with the as- sociation of YSQ statements with the 18 EMS labels, is able to assess the potential maladaptive schemas, if any, through the scores the coun- selee provides in the questionaire and further confirmatory questions.
The form-filling exercise described above is not practical in the 'offline' scenario of community QA forums, requiring to mimic this process indirectly with automated methods. Towards this goal, we harness NLP associations between the YSQ items and sentences from forum questions to infer how the YSQ form could be probably filled by the counselee. Let {41 ...... }m refer to sentences in the input question q. Y = {ui.... ....... 3 represent the statements from YSQ. and label(y)) = Ey be the EMS label Ej from (EI,.. Eis} } for ys-
Large Language Models: Recent break- through research has shown that LLMs can be trained "to act in accordance with the user's intentions" and consequence be "prompted" to perform range of NLP tasks (Radford et al., 2019; Brown al., 2020; Christiano al., 2017; OpenAI, 2023). Currently, prompt-based fine-tuning is
Similarity-based Voting Predictor (SVP): In the SVP method, we first compute the candidate set of EMS labels for a given question q as the set of label(y;) where simmm (4) 2 0. simmi is similarity according to a sentence transformer model my and 0 is tunable threshold. We employ multiple state-of-the-ane sentence transformers {m1... mm) } and compute our final set of EMS labels for q all labels whose related statements exceed the similarity thresholds according least k out of the available m models, where the value of indicates majority (For example, k = 2 when m 3).
MCPrompt: Consider the label definitions
"I. Abandonmen//nstabiityy (AB): The perceived instability 2. Mistrust/ Abuse::
Question: Which of the above labels are most applicable to the following
context? Context: []
YNPrompt: Here is the definition for the label
"1. A Jandonneen/Instabiity (AB): Is this label applicable the context: [] Answer Yes or No.
Table 3: Outline prompts used EMS prediction.
becoming a leading approach solve problems that require language comprehension and many cases shown to perform on-par with humans or previous supervised approaches (Ouyang et al., 2022; Chung et al., 2022; Touvron et al., 2023). Despite this success, LLMs are also known to suffer from shortcomings such as lack of "reliable techniques to steer the behaviour or interpret the outputs". sensitivity to prompt texts, failure follow or detect instructions, and the ability "make up" facts (Zhao et al., 2023; Bowman, 2023). Following the "prompt" trend, we predict EMS labels in a zero-shot setting by simply providing
Entailment-based Prediction model (EPM): Natural Language Inference (NLI) models are used to compute EMS labels in EPM. Recall that the task of textual NLI involves determining whether a "hypothesis" text true (entailment), false (contradiction), or undetermined (neutral) given a "premise" text (Poliak, 2020; Chen et al., 2021). Thus, label(y.) is included in the set of predictions if there exists qj E that entails the hypothesis,
'ntrss:/ssycholeyy-taiinng com. au/wp- hu///p/-content//pllaa////////11 wetas--uuussstsisiiiiisse
Young- Sring-Schema--eee wwweeeSSeeeeeeesiennneeee

---

Setting
Precision
Recallf
FIt
%0 = 6
%Gg = Po
SVP (Threshold=0.4)
0.276
0.275
0.261
31.81
20.00
EPM (T5-large)
0.328
0.307
0.293
36.36
40.00
MCPr0mpt1-GPT-335
0,419
0.313
0.329
22.72
MCPrompt5-Flan-11
0.481
0.301
0.341
22.72
YNPrompt-GPT-335
0.210
0.519
0.278
22.72
YNPrompt-Flan-T5
0.380
0.272
0.283
54.54
40.00
Table 4: EMS Label Prediction Performance
instructions the LLM for generating our desired output for a given question.
examples were annotated by both independently, and on this subset the Fleiss Kappa (1971) value is 0,412 indicating moderate agreement.
We explore "MCPrompt" where the LLM is re- quired to answer a multiple-choice question re- questing all applicable EMS labels for a given men- tal health question text given their descriptions, and the "YNPrompt" where the LLM is asked separate "Yes/No" questions on the applicability of each la- bel for given forum question. The outline of our prompts are shown Table with more details provided in Appendix A.2.
Recent Psychology-relatee studies report Kappa values indicating a wide range of agreement from low to medium depending on the complexity of the factor that is measured (Zhang et al., 2022a; Pérez et al., 2023). The moderate agreement value for our dataset sets the benchmark for EMS prediction since the annotations were made by professionally- qualified counselors who practise Schema Therapy. We also note however, the possible effect of our annotation guidelines that ask for the top-3 labels for each post. Considering the above aspects, we used the union of labels as the "ground truth" for evaluating our models. Further details the anno- tation process well as the label distribution for our dataset are included in Appendix A.4.
3 Results and Observations
Dataset: For experimental evaluation, we con- sider the dataset from counselchat.com that was curated by Bertagnolli (2020). This representa- tive dataset contains about 3000 questions seek- ing assistance and responses from licensed thera- pists for various mental health topies. From the publicly-released anonymized dataset which was carefully de-identified and curated from the website by Bertagnolli (2020), 6 we considered the subset 310 questions on the topics: anxiety, depres- sion, trauma, self-esteem and anger-management where Schema Therapy is applicable. A random sample of about ten percent from this collection was chosen for expert annotation.
3.1 Experiments
Models and Evaluation: We used state-of-the-art models in our prediction algorithms available on HuggingFace (Wolf et al., 2019). For SVP and EPM, we computed pairwise similarity and entail- ment, respectively, between the sentences in the question and the statements in the Young's 's Schema Questionaire (YSQ). The top-3 transformers based on sentence similarity task performances were in- cluded in SVP, whereas T5-large" was used for EPM. Further details on the models considered and threshold tuning are included in Appendix A.
Annotation Quality: To ensure the ethics, qual- ity, and reliability of the annotations for this spe- cialized task, we followed previous works (Sharma et al., 2020) and selected two counselors with ver- ified credentials and self-disclosed practical expe- rience in Schema Therapy on Upwork.? 7 The two counselors assigned EMS labels with justifications for the question texts for about five examples per hour and were paid between 40 - 50 USD an hour, requested. Between the two annotators, ob- tained 30 questions each annotated with up to three most applicable EMS labels, on average. About 17
For prompt-based approaches, we used the GPT-3.5 API from OpenAI and Flan-T5 from Google." Since EMS prediction multi-label classification task, we employ macro averages the standard classification measures: precision, re- call, and F1 to compare with expert (gold) anno-
hht httttp::/www.sbbrt..ot htteer/tee//cccs/errraiieeeeeeee
html hhttps://hugiingffcee.c/// co/teelargeeeeeeeee
/httpp://githu/ http://tww..eeeeeeetteooo//////e www.upworr.cco
hhttps://opeeai.coom http://ppnni..ooom/ll//////p/aaa
"https://huggingface co/ htt.://gnnnnacc.ooooooooo/aaataa lol/oggglllllllttsxlll

---

tations. Note that when there is insufficient infor- mation in the counseling question, the gold annota- tions set empty and important for the models to correctly predict these null cases. We have about 19% of null cases in our dataset. Thus, we also measure percentage of cases when none of predic- tions match the gold set (overlap O = Φ), and the percentage of null cases correctly predicted by the models (G& Pa).
In summary, the non-LLM and LLM approaches seem complementary in terms of prediction per- formance versus explainability. Considering the overall low prediction performances in this no-data setting and the difference in the explanatory power, we posit that the two types of approaches are best used in combination for generating "noisy" labeled data to speed up annotation efforts towards learning models with higher degree of supervision.
3.2 Comparison of Models
4 Related Work
Prediction Performance: Table 4 lists the perfor- mance of our LLM and non-LLM approaches. Af- ter considering multiple prompts (Appendix A.2), we selected the runs with the best F1 scores. From Table 4, wc see that on all standard performance measures, LLM approaches fare significantly bet- ter than the non-LLM methods. Considering that the high performance was accomplished merely by employing the label descriptions and "instructing" the LLM, it is indeed an impressive result. In par- ticular, Flan-T5 model using MCPrompt obtains significantly high Fl score as well as the smallest %0 = & value compared to the other approaches. However, the MCPrompt method is not good at dis- covering the null cases. The YNPrompts provide more flexibility for handling the varying number of labels and when used with Flan-T5 also discovers large number of the null cases, but the overall performance is significantly lower.
Community QA (cQA) forums are attracting signif- icant NLP research due to their ubiquitious usage the recent age of information sharing (Zhang et al., 2021; Sharma et al., 2020; Sosea and Caragea, 2020). Recent research works on mental health topics address symptom extraction, triage char- acterization as well identification of specific conditions such depression and self-harm be- haviour (Zhang et al., 2022b; Shickel and Rashidi, 2016; Yates et al., 2017; Cohan et al., 2016; Pérez et al., 2023; Zhang et al., 2022a; Parapar et al., 2023; Nguyen et al., 2022). Unlike these previous works, we target the identification of underlying causes behind the manifested symptoms by char- acterizing them using EMS labels from Schema Therapy. We posit that such a theoretical ground- ing using domain-specine frameworks essential for reliably interpreting model predictions men- tal health applications (Fitzpatrick et al., 2017).
Consistency and Explainability: Predictions with the SVP and EPM methods have the nice prop- crty of being linked to specific statements in the Young's s Schema Questionaire, a professional ques- tionaire that is employed in practice by Schema Therapists. We were unable to obtain any expla- nations from the Flan-T5 models despite several attempts prompting for the same. For GPT-3.5 runs, explanations refer to spans in the question (Example in Table 9) requiring further interpreta- tion. In contrast, since non-LLM approaches "ex- plain" on the basis of YSQ, this questionaire can serve as a common ground for comparing across mental health questions.
Conclusions
In this paper, we proposed the novel task of identi- fying Early Maladaptive Schemas in mental health question texts from community QA forums. To this end, we studied prediction techniques using both LLM and non-LLM approaches based on Schema Therapy, examined their strengths and weaknesses for the task. and identified their possible comple- mentary nature regarding performance and expla- nations. For future, we would like to investigate if few-shot and exemplar- approaches (Lo- gan IV et al., 2022; Chung et al., 2022; Touvron et al., 2023) can overcome the drawbacks of LLMs highlighted in our study. We would also like investigate recent research on incorporating human feedback through reinforcement learning (Ouyang et al., 2022) and contrastive learning to further im- prove prediction performance (Chen et al., 2022).
LLMs were also found to be highly sensitive to spe- cific prompt texts (as illustrated in Appendix A.3). For minimal changes in wording and no semantic difference, the output lists of predictions can look vastly different, illustrating one of the known criti- cisms of prompt-appren approachps (Ouyang et al., 2022; Bowman, 2023).

---

Limitations
Nicolas Bertagnolli. 2020. Counsel chat: Bootstrapping high-quality therapy data. https://ggthub com/ nbee
We proposed the novel task of identifying "mal- adaptive schemas" as characterized by Schema Therapy in offline counseling scenarios. standard scenario, this estimation is done by quali- fied counselors based on the YSQ forms filled by the counselee after a round of confirmatory ques- tions (Young et al., 2006). In offline scenarios, we "impute" these values based on what is expressed in the question texts and work backwards. Thus, even with counseling expertise, since the 'client' is missing in the loop, the gold annotations are only the next best alternative. The drawbacks of using LLMs for this specific task despite their impressive prediction performance were highlighted in Sec- tion 3. For sensitive topics such as mental health, incorrect predictions can lead to wrong therapeutic interventions and outcomes. Therefore, consis- tent grounding of the predictions using domain- specific frameworks (such as YSQ in this case) plays key role in determining whether automated approaches can be adopted for final use.
Samuel R. Bowman. 2023. Eight things to know about large language models. CoRR, abs/2304.00612.
the
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901.
Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI models verify QA systems' predictions? In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3841-3854, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Qianben Chen, Richong Zhang, Yaowei Zheng, and Yongyi Mao. 2022. Dual contrastive learning: Text classification via label-aware data augmentation.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Annual Conference on Neural Information Process- ing Systems, pages 4299-4307
Ethics Statement
This research was conducted in accordance with the ACM Code of Ethics. The ethical considerations were fulfilled by hiring qualified counselors for an- notating the mental health questions. Details on the pay. quality, and pace of annotations are included in Section and Appendix A.4 and possible con- siderations automating mental health prediction tasks are discussed under the Limitations section.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami. Siddhartha Brahma, et al. 2022. Scaling eestruction-finetunee language models. arXiv preprint arXiv:2210.114
Arman Cohan, Sydney Young, and Nazli Goharian. 2016. Triaging mental health forum posts. In Pro- ceedings of the Third Workshop Computational Linguistics and Clinical Psychology, pages 143-147, San Diego, CA, USA. Association for Computational Linguistics.
Ackmowledgments
This rescarch/project is supported by the National Research Foundation, Singapore under its Industry Alignment Vund-Preepossttionin (LAF-PP) Fund- ing Initiative and Google South & Southeast Asia Research Award 2022. Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and do not re- flect the views of National Research Foundation, Singapore or Google LLC.
Michelle G Craske. 2010. Cognitive-behaviore ther- apy. American Psychological Association.
Mummun De Choudhury and Sushovan De. 2014. Men- tal health discourse on reddit: Self-disclosure, social support, and anonymity. In Proceedings of the inter- national AAAI conference on web and social media, volume 8, pages 71-80.
Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using fully automated conversational agent (woebot): A randomized controlled trial. JMIR Ment Health, 192)
References
Daniela Schneider Bakos, Alex Eduardo Gallo, and Ri- cardo Wainer. 2015. Systematic review of the clinical effectiveness of schema therapy. In Contemp Behav Health Care, volume I, pages 11-15.
Joseph L Fleiss. 1971. Measuring nominal scale agrec- ment among many raters. Psychological bulletin, 76(5):378.

---

Frances Griffiths. Jonathan Cave, Felicity Boardman, Justin Ren, Teresa Pawlikosska, Robin Ball, Aileen Clarke, and Alan Cohen. 2012. Social networks- the future for health care delivery. Social science & medicine, ((12))2233-2241)
Adam Poliak. 2020. A survey on recognizing textual entailment as an NLP evaluation. In Proceedings of the First Workshop of NLP Systems, pages 92-109, Online. Association for Computational Linguistics.
Evaluation and Comparison
Julie Prescott, Terry Hanley, Katalin Ujhelyi, al. 2017. Peer communication in online mental health forums for young people: directional and nondirectional sup- port. JMIR menial health, 4(3):e6921.
Robert Logan IV. Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Find- ings of the Association for Computational Linguis- tics: ACL 2022. pages 2824-2835, Dublin, Ireland. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Samantha A Masley, David T Gillanders, Susan G Simp- son, and Morag A Taylor. 2012. A systematic review of the evidence base for schema therapy. Cognitive behaviour therapy, 4 1153)218--20.
Charles A Sanislow and Thomas H McGlashan. 1998. Treatment outcome of personality disorders. The Canadian Journal of Psychiatry, 23(3)2237.
Ashish Sharma, Adam Miner, David Atkins, and Tim Al- thoff. 2020. A computational approach to understand- ing empathy expressed text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263-5276, Online. Association for Computational Linguistics.
Thong Nguyen, Andrew Yates, Ayah Zirikly, Bart Desmet, and Arman Cohan. 2022. Improving the generalizability of depression detection by leverag- ing clinical questionnaires. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages (446-8459, Dublin, Ireland. Association for Compu- tational Linguistics.
MT Shea, PA Pilkonis, E Beckham, JF Collins, I Elkin, SM Sotsky, and JP Docherty. 1990. Personality dis- orders and treatment outcome in the nimh treatment of depression collaborative research program. The American journal of psychiatry, 147(6):
Alicia Nobles, Eric Leas. Mark Dredze, and John W Ayers. 2020. Examining peer-to-peer and patient- provider interactions on social media community facilitating ask the doctor services. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 465-
Benjamin Shickel and Parisa Rashidi. 2016. Automatic triage of mental health forum posts. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pages 188-192, San Diego, CA. USA. Association for Computational Linguis- tics.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Tiberiu Sosea and Cornelia Caragea. 2020. Cancer- Emo: A dataset for In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 8892 8904, Online. Association for Computa- tional Linguistics.
emotion detection.
Javier Parapar, Patricia Martin-Rodilla, David E. Losada, and Fabio Crestani. 2023. Overview of crisk 2023: Early risk prediction on the internet. In Exper- imental Meets Multilinguality Multimodality, and Interaction, pages 294-315, Cham. Springer Nature Switzerland.
Christopher DJ Taylor, Penny Bee, and Gillian Haddock. 2017. Does schema therapy change schemas and symptoms? a systematic review across mental health disorders. Psychology and Psychotherapy: Theory, Research and Practice, 9063)4456-479
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Nancy Peeters, Boris van Passel, and Julie Krans. 2021. The effectiveness of schema therapy for patients with anxicty disorders, ocd, or ptsd: A systematic review and research agenda. The British Journal of Clinical Psychology, 61:579 597.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the-ar natural language processing. CoRR. 11./191..03771.
Anxo Pérez, Marcos Fernández-Picher Javier Parapar, and David F. Losada. 2023. DepreSym: A Depres- sion Symptom Annotated Corpus and the Role of LLMS as Assessors of Psychological Markers. arXiv e-prints, page arXiv:2308. 10758

---

Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. self-harm Depression and self- harm risk assessment in online forums. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2968-2978, Copenhagen, Denmark. Associa- tion for Computational Linguisties.
Jeffrey E Young and Janet S Klosko, 1994, Reinventing Your Life: The Breakthough Program to End Negative Behavior... and Feel Great Again. Penguin.
Jeffrey E Young, Janet S Klosko, and Marjorie Weishaar. 2006. Schema therapy: A practitioner's guide. guilford press.
Wenxuan Zhang, Yang Deng, Xin Li, Lidong Bing, and Wai Lam. 2021. Aspect-based sentiment analysis in question answering forums. In Findings of the Asso- ciation for Compulational Linguistics: EMNLP 2021, pages 4582-4591, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhiling Zhang. Siyuan Chen, Mengyue Wu, and Kenny 7hu. 2022a. Symptom identification for interpretable detection of multiple mental disorders on social me- dia. In Proceedings of the 2022 Conference Em- pirical Methods in Natural Language Processing, pages 9970-9985, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zhiling Zhang, Siyuan Chen, Mengyue Wu, and Kenny Q. Zhu. 2022b. Psychiatric scale guided risky post screeming for early detection of depres- sion. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5220-52220 5226. International Joint Conferences on Artificial Intelligence Organization. Al for Good.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yur Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303. 18223.

---

A Appendix
the YNPrompt string shown in Table 12, Flan-T3, Similarly the performance quite sensitive the precise string used for MCPrompts as can seen in Table 11.
A.1 Experiment Settings and List of Models
All experiments performed single GPU of an Nvidia Tesla cluster machine with 32GB RAM. No learning is involved, and depending on the model, predictions on the entire dataset take between a few minutes to hours.
Note: We experimented with more prompts using Flan-T5 models over GPT-3.5 due to their open-source availability. The GPT experiments were performed using the "gpt-3.5-turoo-0001" model with the temperature parameter set to 0.3.
1. Three Sentence Transformers were used in the SVP model based on their performance on the sentence similarity tasks. 12 These are "all-mpnet-base-v2, MIt-MiniLL--11--11 and all-distilrooerta- v1". We use 2/3 votes to de- termine what labels make to the final set and tested similarity thresholds 0.3, 0.4, 0.5, 0.6 (Table 7) for the top-5 similar statements with each EMS label.
Please follow these guidelines while annotating the data. For each stated in the post in column B (counselee post), mark If Schema Therapy is applicable or not (Column C is Yes/No)
2. If not applicable, provide justification in column J (Other comments)
2. We experimented with large, 14 T5-3b, 15 and ElMo-based model from AllenNLP16 and chose T5-large (based on the test performance) for the EPM runs.
T5-
3. If applicable, please select the top-3 most applicable EMS labels (in Columns D, H)
For each selected label, provide justification using either sentences the original post (copy paste) or describe in your own words (in Columns E, G, 1) If less than apply, select NONE applicable in the appropriate column. If more than 3 apply, please list them in the last column (Other comments)
Here is the definition of the label [schema_name) |schema_definiteee definiiion.. Is this label applicable to [contert]? Answer Yes or No.
Table 5: The YNPrompt used with GPT-3.5 and Flan-T5
A.2 Prompt Experiments
Table 6: Guidelines for our annotation task
The general flavor of our prompts is given in Ta- ble 3 with the precise ones used for the results
in Table 4 listed in Tables 5 and 10. In the prompt string, context refers the question text and "schema name" and ischemaa_efinitiiie definition" were taken from https://wwe id73.
A.4 Dataset Annotation Details
On Upwork, we posted an example question and put across our annotation task requesting for cer- tified counselors with background knowledge and expertise in Schema Therapy. From the proposals received, we selected two counselors who available during our data collection period and cor- rectly assigned the labels to the example question. An excel sheet was provided with schema defini- tions and dropdown lists for choosing to three or more EMS labels. The exact instructions are listed in Table 6.
com/
A.3 LLM's Sensitivity to Prompt Texts
LLMs are known to suffer from sensitivity to the exact prompt texts, and failing to follow detect instructions (Ouyang et al., 2022; Bowman, 2023). We observe that these drawbacks were also present for our task. For instance, for simple variations of
12 wwwrt. http://wwwww.stttt..e/////s/sses
We examined the annotations provided by the two annotators and removed three cases where the question texts were about the person asking the question. For example: "My son has violent bursts of temper..." One of the annotators as well as GPT.
html
hhttps://hueee //tlli/s///ugggngacce.e. http:/aac....oggggg//lleeee-eeee
hhttps://huggingee http://ggg.nnn.aa...ooo//aarrggr
1s /https://huggingffae co/t5-3b
"https://demo all www.lnlp..nen..mmmmmmmmmmmmll .rgg

---

Similarity Threshold
Precision"
Recall
Fit
%0 = Φ
%G
Pe. t
0.3
0.251
0.281
0.249
22.72
0
0.4
0.276
0.275
0.261
31.81
20.00
0.5
0.284
0.243
0.252
54.54
60.00
0.6
0.324
0.219
0.240
77.27
80.00
Table 7: EMS Label Prediction Performance for the different similarity thresholds
(AB, 25.9%). (MA, 7, 25.9%), (ED, 9, 33.33%), (DS, 8, 29.6%), (SI, 5, 18.5%), (DI, 3, 11.11%), (VH, 1, 3.7%%, (EM,, 7.4%), (FFA, 0, 0%), (ET,, 0, 0%), (IS, 18.5%). (SB, 5, 18.5%), (SS, 3, 11.11%), (AS, 4, 14.8%), (NP, 10, 37%), (EI, 1, 3.7%), (US, 3, 11.11)), (PU, 1, 3.7%)
Table 8: EMS Label counts and percentage spread in the annotated dataset.
3.5 provides labels for the person mentioned in the question, whereas our second annotator did not. For uniformity, these three cases were disregarded and the combined annotations from both annotators were used as our gold EMS labels for the rest of the cases. The counts for each label in this annotated dataset are shown in Table 8. The labels referring to Negativity/Pessimisn. Emotional Deprivation, Abandoumeen//IIstaaiilyy. Mistrust/Abuse, and De- fectiveness/Shame were noted in 25-37% of the cases seen in our dataset whereas the labels refer- ring to "Fear of Failure" and "Entitlement" were not observed at all.
Prompt
"Does the label schema name defined as schema_definition apply to the text: [contexi]? Answer Yes or No. Explain your answer."
Yes, the label "9. FAILURE TO ACHIEVE (FA)" applies the text about imposter syndrome in graduate school.
The individual experiencing feelings of
inadequacy and fear of being exposed as a fraud, despite evidence of their success and competence.
The second question is not related to the label "9. FAILURE TO ACHIEVE (FA)" and may be better suited for a different label or category. It could potentially fall under social anxiety or avoidance behaviors.
Table 9: Example explanation generated by GPT-3.5 for the context from Table 12 when used with the label "9. FAILURE TO ACHIEVE (FA)"

---

Prompt= Label Definition String from Table 13+
Question: Which of the above labels (maximum 3)
most apply to the following context? If labels are applicable, just return
NONE (NO). Context:
Question: Which of the above labels most apply
to the following context? Select up to three. Context: U
Question: Which of the above labels most apply to the following context?
Select three. Context: []
4 Question: Which of the above labels most apply to the following context?
Select three. If no labels applicable, just return NONE (NO). Context: [
Question: Which of the labels above are most applicable the following context? Select three. Context: []
Table 10: List MCPrompts used Experiments
Setting
Precision
Recallt
FIf
%0 0
%G = Po
GPT-3.5 MCPromptl
0.4198
0.3129
0.3297
22.72
GPT-3.5 MCPrompt2
0.3457
0.3377
0.3192
22.72
GPT-3.5 MCPrompt3
0.3333
0.3019
0.3000
27.27
GPT-3.5 MCPrompt4
0.2962
0.2531
0.2541
27.27
GPT-3.5 MCPrompt5
0.3209
0.3173
0.2973
22.72
Flan-T5 MCPromptl
0.3704
0.1099
0.1557
54.54
Flan-T5 MCPrompt2
0.5185
0.1654
0.2359
36.36
Flan-T5 MCPrompt3
0.4567
0,2117
0.2634
31.81
Flan-T5 MCPrompt4
0.3765
0,2388
0.2641
36.36
Flan-T5 MCPrompt5
0.4814
0.3006
0.3406
22.72
Table 11: EMS Label Prediction Performance the different MCPrompts from Table 10
How do I get over "imposter syndrome"" I'm dealing with imposter syndrome in graduate school.
Context:
know that by all accounts I a phenomenal graduate student, and that I well-published. am well liked by students and faculty alike. And yet cannot shake the feeling that I'm going to found out as a fraud. How can get over this feeling? Prompt Does the label "[schema_nanee the text: [context]? Answer Yes or No.
defined as Dichemmadefinitiion apply to
Yes output for: 9. FAILURE TO ACHIEVE (FA)
Prompt Answer the following yes/no question using the label for aachema_namen
defined [schema definition]. Does the label apply to the text: [context]? Explain the answer.
Yes output for: 4. DEFECTIVENESS SHAME (DS)
Prompt Answer the following yes/no question using the label for accemmanname
defined [schema ] Does the label apply to the text: [context]?
Yes output for: 4. DEFECTIVENESS / SHAME (DS) 15. NEGATIVITY PESSIMISM (NP)
Table 12: Example YNPrompts with Flan-TS T5 illustrating the sensitivity to prompt texts and disregarding of the instruction

---

Label Definitions String
ABANDONMENT INSTABILITY (AB):
The belief that
significant others will not be there for support and connection, leading to fear of abandonment or loss.
2. MISTRUST/ ABUSE (MA): The expectation that others will hurt, abuse, betray you in some way, leading a general mistrust of others.
EMOTIONAL DEPRIVATION (ED): The belief that your emotional needs will not be met by others, resulting sense emptiness or longing for emotional connection. DEFECTIVENESS SHAME (DS) Feeling flawed, inadequate, or unworthy, often accompanied by fear of being exposed as defective and a sense of shame.
5. SOCIAL ISOLATION/ / ALIENATION (SI) Feeling different, disconnected, or isolated from others, resulting in sense of not fitting in or belonging.
6. DEPENDENCE/ INCOMPETENCE (DI) Feeling incapable of handling day-to-day
responsibilities making decisions independently, leading excessive reliance others.
VULNERABILITY TO HARM OR ILLNESS (VH) : Constantly anticipating danger, harm, illness, and feeling vulnerable or unsafe in various situations.
ENMESHMENT/ / UNDEVELOPED SELF (EM) Feeling engulfed, overshadowed, or lacking
a clear sense of self due to excessive emotional fusion with others.
9. FAILURE TO ACHIEVE (FA) The expectation of failure or the fear of not meeting high personal standards, leading to a persistent sense of inadequacy or disappointment.
10. ENTITLEMENT GRANDIOSITY (ET): A belief that you deserve special privileges, recognition, admiration, often accompanied by
of superiority entitlement.
11. INSUFFICIENT SELF-CONTROL. SEL SELF-DISCIPLING (IS) Difficulty in controlling
impulses, sticking plans, delaying gratification, often resulting in self-defcating behaviors.
12. SUBJUGATION (SB) The tendency submit to others' needs and desires while disregarding one's own, leading to a sense being controlled or trapped.
13. SELFFSACRIFICE (SS): Neglecting one's 's own needs and prioritizing others' needs to an excessive degree, often at the expense of personal well-being.
14. APPROVAL-SEEKING RECOGNITION SEEKING (AS) An excessive need for validation,
approval, recognition from others, leading to overemphasis external validation.
15. NEGATIVITY PESSIMISM (NP): An inclination focus the negative
aspects of oneself, others, the world, leading to pessimistic outlook and negative expectations.
16. EMOTIONAL INHIBITION (EI) Restricting or suppressing emotions due to fear of being overwhelmed, losing control, or being rejected or criticized.
17. UNRELENTING STANDARDS / HYPERCRITICALNESS (US) Holding oneself to extremely high standards of performance, often accompanied by self-criticism and constant sense of falling short.
18. PUNITIVENESS (PU) The tendency to be excessively harsh or critical towards oneself or others, often accompanied by desire for punishment revenge.
Table 13: The label definitions string used in MCPrompts


Reviews:
[{"review_id": "OoBRH9iqKM", "reviewer": "Reviewer_CaET", "summary": "", "questions": "Please refer to the weaknesses of the paper above, especially 2, 3, 4. What was the reason for not performing any level of fine-tuning on the dataset.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We apologize that more examples could not be squeezed in due to lack of space.  In addition to the example in the Introduction, one more example is included in the Appendix. \n\n2. We used a subset of the publicly-available dataset from counselchat.com provided by (Nicolas Bertagnolli. 2020. Counsel chat: Bootstrapping high-quality therapy data. https://github.com/nbertagnolli/counsel-chat) for our study. The referenced paper details the de-identification and curation process adopted for releasing this dataset publicly. We will highlight this specifically in our paper.\n\n3. Previous Psychology-related studies report Kappa values indicating a wide range of agreement from low to medium depending on the complexity of the factor that is measured (For example, https://arxiv.org/abs/2308.10758 and https://aclanthology.org/2022.emnlp-main.677.pdf). In our work, we are the first to report the agreement values of predicting EMS by two professional counselors and this sets the benchmark for our task. The moderate agreement between annotators (as indicated by Fleiss Kappa), is also possibly an artifact of our annotation guidelines task where we requested for top-3 labels for a post from each annotator. As such, both the annotators are qualified and practicing counselors who apply a label and provide explanations for each label by highlighting the appropriate spans in the questions. Considering this aspect, the union of labels was used as ground truth. Please find further details in Appendix A.4. \n\n4. The reviewer’s comment on few-shot learning as well as contrastive learning is well-noted for further study. The original dataset as well as our subset of annotations will also be publicly available as mentioned in Footnote 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper highlights a significant application of natural language processing in the realm of mental health.\n2. The authors have curated a valuable dataset for identifying EMS from patient texts or conversations.", "reasons_to_reject": "1. The paper's dataset presentation lacks clarity, making it challenging to follow without specific examples.\n2. The discussion of dataset de-identification, critical for dataset release, is absent.\n3. The inter-annotator agreement is notably low. There's a lack of discussion on the adjudication strategy and revisions that shaped the final test dataset.\n4. Incorporating few-shot learning, especially for null labels, and implementing contrastive learning could enhance the results. Consider making the dataset publicly available, as it could be a valuable resource for the community.\n\nI kindly request the authors to provide a more detailed discussion of the dataset and consider its public availability for the benefit of the community.", "questions_to_authors": "Please refer to the weaknesses of the paper above, especially 2, 3, 4. What was the reason for not performing any level of fine-tuning on the dataset.", "paper_topic_and_main_contributions": "The paper discusses the task of identifying Early Maladaptive Schemas (EMS) in the mental health question texts which are collected from a community QA forum. Two different approaches are also discussed and compared in the paper with varying performance across the dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7mwAHx8Wja", "reviewer": "Reviewer_Gb3g", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Existing mental health datasets do not include EMS labels and we are the first to propose this novel task. Given the lack of annotated data, we presented our study in the zero-shot setting where no learning and fine-tuning is employed and the annotated data collected by us has been used for evaluation alone (of non-LLM versus LLM approaches). \n\n2. We apologize for the confusion on the use of “consistency and explainability”. We are referring to the fact that LLMs provide explanations for each case referring to spans in the text making it impossible to compare across cases. In contrast, non-LLM approaches “explain” on the basis of the Young’s Schema Questionaire, so this is common ground using which two cases can be compared. \n\n3. Ours is the first study that grounds mental health predictions using Schema Therapy, that is practically employed in real-world counseling, will provide the theoretical grounding for development of further mental health applications. We explored the viability of LLMs and non-LLM approaches for EMS labels from Schema Therapy in a zero-shot setting in this paper and find that there is large room for improvement and future work. \n\nWe thank the reviewer for catching the presentation problem on bolding and noted the comment on three decimals. We will also add the problems highlighted in limitations to the Ethics statement as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new idea to apply a concept from schema theory in psychology to NLP texts- the work can be beneficial to other clinical NLP researchers and be impactful for studying triage. \n* The paper is well-written and easy to understand.", "reasons_to_reject": "* Many obvious experiments come to mind given its a first attempt at understanding the concept in language domain -- for example, the models could be finetuned for the methods, or a model trained on mental health datasets specifically could be used. Studies could be done across multiple models to check for inherent knowledge of identifying EMS. The choice of models is not well-motivated either.\n* Consistency and Explainability: although its a section, the discussion on non-LLM is limited to one line and doesn't have any specific experiments designed for either consistency or explainability.\n* The work is small and limited in its applications, I would like to see more experiments and settings for a thorough evaluation. The work in its current form might be better suited for a related workshop.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is a first attempt at understanding a concept called early maladaptive schemas from community question answering forums. The approach involves automating filling up of Young's schema questionnaire using various methods (both LLM and non-LLM based) to obtain labels of schemas (18 classes) for each post.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Please acknowledge the possible impacts of predicting EMS labels in the Ethics section.\n* Table 3-- why is 0.3006 bolded? It might also be better to have results upto 3 decimal places since the point holds with less decimals."}}, {"review_id": "7Sd1Nil4gp", "reviewer": "Reviewer_14Gv", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We respectfully disagree with the reviewer's comment on novelty. The proposed task is different from\nexisting works on depression including symptom extraction and screening/triage works. Borrowing from the clinical field, one way to illustrate this point is the difference between cause, symptom, and condition/disease. Schema Therapy from Psychology/Counseling provides a theoretical framework for understanding underlying causes behind manifested symptoms and is used in practical counseling for a range of mental health conditions such as depression and bipolar disorder.\n\nAs such, the novelty of this paper pertains not to the SVP alone but a study on the novel application of Schema Therapy (unseen in studies so far) via the\nEMS prediction task. We study this problem in a zero-shot setting using sentence similarity (SVP), entailment, as well as \nLLMs which are being harnessed for their high-level of comprehension on a variety of tasks.\n\nWe are happy to highlight this aspect in related work comparison and include the references suggested by the reviewer in related works.\n\n2. In this first study for a short paper, we focused on the zero-shot setting for comparing LLMs with non-LLM approaches. Further details on EPM as well as annotation are included in the Appendix (A.1 and A.4)  due to space constraints. We will include the other finer details requested by the reviewer. \n\n3.Previous Psychology-related studies report Kappa values indicating a wide range of agreement from low to medium depending on the complexity of the factor that is measured (For example, https://arxiv.org/abs/2308.10758 and https://aclanthology.org/2022.emnlp-main.677.pdf). In our work, we are the first to report the agreement values of predicting EMS by two professional counselors and this sets the benchmark for our task. The moderate agreement between annotators (as indicated by Fleiss Kappa), is also possibly an artifact of our annotation guidelines task where we requested for top-3 labels for a post from each annotator. As such, both the annotators are qualified and practicing counselors who apply a label and provide explanations for each label by highlighting the appropriate spans in the questions. Considering this aspect, the union of labels was used as ground truth. Please find further details in Appendix A.4. \n\nTypo: We are not sure what this comment means.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": ">Describing two methods for predicting EMS labels through a novel application of sentence similarity and textual entailment.\n>Providing an evaluation of the methods using LLMs and non-LLMs \n>Constructing a small EMS dataset.", "reasons_to_reject": ">The overall novelty of this paper is limited. The proposed SVP method exists. In addition, the EMS prediction task is similar to depressive symptoms detection tasks. Therefore, there are many related methods leveraging semantic information, such as:\nZhang, Z., Chen, S., Wu, M., & Zhu, K. Q. (2022a). Psychiatric scale guided risky post screening for early detection of depression. arXiv preprint arXiv:2205.09497. Zhang, Z., Chen, S., Wu, M., & Zhu, K. (2022b). Symptom identification for interpretable detection of multiple mental disorders on social media. In Proceedings of the 2022 conference on empirical methods in natural language processing (pp. 9970–9985).\nIn addition, although the LLMs' method and evaluation are useful for the EMS prediction task, there are no interesting findings, and the results can not help justify the lack of interpretability. The authors could add a chain of thought to learn more about the performance of LLMs.\n\n>Some contents need to be included and need to be more clearly described: The specific structure and usage of the EPM method.  Which F1 (micro, macro, average) has been used?  The label distribution of the dataset (imbalanced or balanced)?\n\n>Some experiments should be added, such as ablation studies about non-LLMs-based models.  \n\n>The Fleiss Kappa is low. What is the reason, are two annotators enough, and is the guideline set reasonable? \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about mental health analysis. The paper proposes a task of identifying Early Maladaptive Schemas in mental health question texts from community QA forums. In addition, this paper tests the performance of LLMs and non-LLMs methods on the task. There are some findings about EMS detection.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 162, the statement from YSQ should be added to the table."}, "extra_text": {}}, {"review_id": "WFCmKs0LUI", "reviewer": "Reviewer_ikMh", "summary": "", "questions": "A. The exact process of computing candidate EMS labels based on sentence similarity using multiple sentence transformer models is unclear. More details would be beneficial like which similarity score was computed.\nB. Can you elaborate on how entailment is determined between YSQ statements and question sentences?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We apologize that some clarity was lost due to space constraints of short paper. Further details on these two methods are included in Appendix A.1.\n\n1. We computed pairwise similarity (or alternatively, entailment) between the sentences in the question and the statements in the Young’s Schema Questionaire (YSQ) resource available from Schema Therapy. The YSQ statements are grouped into EMS categories/labels.\n\nFor SVP, we applied three state-of-the-art sentence transformer models to obtain sentence embeddings and compute cosine similarity. An EMS label (attached to the corresponding YSQ statement) is only applied if at least two of the transformers provide a similarity score above a threshold. The effect of these thresholds are illustrated in Table-7 of the appendix.\n\n2. For the EPM method, we used an off-the-shelf T5-large entailment prediction model \n(T5 is already fine-tuned on the entailment task: JMLR 2020 \"Exploring the Limits of Transfer Learning with a Unified\nText-to-Text Transformer\"). We choose EMS labels corresponding to the YSQ statement for which entailment holds. For example (lines 161-163), the statement in YSQ \"In the end, I will be alone\" is entailed by the sentence in the post from Table-1 \"Am I going to be alone forever?\" according to the T5 model prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces an interesting problem of Early Maladaptive Schema Identification in mental health texts. \nEvaluated various non-LLMs and LLM-based methods and offered a thorough analysis of the potential and limitations of different techniques for EMS label prediction.\n", "reasons_to_reject": "The paper maintains a formal tone and technical language appropriately. However, some sections could be further polished for clarity, particularly in conveying the details of complex methodologies.  The methodology section introduces three distinct approaches logically, but it lacks detailed explanations for certain techniques. The methodology section of the paper introduces various approaches for predicting early maladaptive schemas (EMS) labels but falls short in several critical aspects. \n\n1. The description of the Similarity-based Voting Predictor (SVP) is insufficiently detailed. The exact process of computing candidate EMS labels based on sentence similarity using multiple sentence transformer models is unclear. \n\n2. The Entailment-based Prediction model (EPM) is introduced briefly but lacks clarity in its application. The section does not elaborate on how entailment is determined between YSQ statements and question sentences. \n", "questions_to_authors": "A. The exact process of computing candidate EMS labels based on sentence similarity using multiple sentence transformer models is unclear. More details would be beneficial like which similarity score was computed.\nB. Can you elaborate on how entailment is determined between YSQ statements and question sentences?\n", "paper_topic_and_main_contributions": "This paper explores novel approaches, including Large Language Models (LLMs), to predict early maladaptive schema (EMS) labels in mental health cQA forums. The paper introduces the first expert-curated dataset on this topic, which can provide a reasonable ground for investigating the feasibility of automated approaches for detecting EMS from cQA mental health question texts.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the task of identifying Early Maladaptive Schemas (EMS) in mental health texts obtained from community question-answering forums.  While the work makes valuable contributions, particularly in dataset curation and initial analysis, it also suffers from a series of shortcomings that affect its overall impact. All reviewers acknowledge the significance of applying NLP in the mental health domain. he dataset generated is of noteworthy value, especially since it focuses on Early Maladaptive Schemas, a topic not previously covered in available resources. Multiple reviewers noted that the dataset presentation could be more precise, with more context and examples for easier comprehension. Given the sensitive nature of mental health data, reviewers have expressed concerns over the lack of discussion regarding dataset de-identification. The reviewers found the paper’s contributions to be incremental. They recommend further experiments for a more thorough evaluation. Given the strengths and weaknesses pointed out by the reviewers, the paper is recommended accepted into the Findings. The authors are encouraged to consider the detailed feedback from the reviewers to enhance the quality and impact of their work.
Decision: Accept-Findings