Using Large Language Models to Detect Outcomes of Qualitative Studies on Adolescent Depression

Authors: alison w xin
dylan m nielson
karolin  krause
guilherme  fiorini
nick  midgley
francisco  pereira
juan antonio lossio-ventura
juan antonio lossio
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-06-04 07:02:50

Abstract: Depression treatment studies often focus exclusively on changes in depressive symptoms, such as low mood, anhedonia, or sleep disruption. However, incorporating other outcomes important to those experiencing depression, such as the quality of interpersonal relationships or quality of life, could improve understanding of the impacts of depression and effectiveness of treatment. Drawing on data from in-depth interviews with adolescents, parents, and therapists, clinicians produced a novel coding framework that covers additional domains of interest that matter to adolescents, such as relationships, functioning, and well-being. In this paper, we examine whether large language model embeddings can be used to classify the outcomes of this framework from annotated  interviews. We compare the suitability of four language models across three different segmentations of interview transcripts, such as conversation turns or non-interviewer utterances. The level of performance achieved by our models makes them useful for a variety of applications, ranging from aiding human annotation of text transcripts to quantifying the presence of outcomes for downstream uses, such as estimating treatment effects or building prognostic models.

Main paper: 
 introduction: 
 Globally, in adolescents aged 10-19 years, the prevalence of major depressive disorder and dysthymia is estimated at 8% and 4%, respectively [1] . Advancing understanding of treatment outcomes is critical in addressing this public health problem. In clinical trials and routine specialist care, around 40% of youth leave treatment without showing meaningful improvement in depressive symptoms, which include low mood, anhedonia, sleep disruption, suicidality, or irritability, defined by the DSM and ICD-11. Less is known about the impact of treatment on other outcomes, such as relationships or quality of life. Between 2007 and 2017, a systematic review of clinical studies of depression found that 94% of studies measured depressive symptoms, 52% measured general functioning, and less than 10% measured any other outcome [2] .Previously, clinical researchers performed a post-hoc analysis of interview transcript data from the qualitative study IMPACT-My Experience (IMPACT-ME [3] ), a substudy nested within the Improving Mood with Psychoanalytic and Cognitive Therapies (IMPACT) study of the psychological treatment of adolescent depression [4, 5] . Using qualitative content analysis, they produced a systematic and comprehensive framework of adolescent depression treatment outcomes, identifying seven broad outcome domains, and twenty-nine specific outcomes of interest [6] . Analysis of these outcomes in qualitative data could complement traditional quantitative measurement of symptom change, providing a more holistic impression of how treatment affects depression.However, manual qualitative analyses of large volumes of qualitative data is time-intensive and may not always be feasible. Recent developments in natural language processing (NLP), particularly improvements in large language models (LLMs), can help address this challenge by automating the analysis of large volumes of data. A recent survey demonstrated that NLP enables automated screening for symptoms of several mental disorders from text data [7] , though most of these studies only address a single binary classification (e.g., depressed and non-depressed) or regression (e.g., severity). Additional limitations are the use of social media data, which complicates clinical integration, and lack of work focused on adolescents.We aim to train models capable of reliably detecting mentions of fine-grained depression outcomes in the IMPACT-ME study data. We compared several LLMs to investigate whether the use of different embeddings improved detection performance. We use opensource LLMs deployable within our own servers, which reduces concerns with protected health information or personally identifiable information. In this paper, we demonstrate the feasibility of using LLM embeddings as part of models for detecting outcomes across a range of high-level domains. We compare the performance of embeddings from various LLMs, including the recently released Llama 3. Overall, we find that LLM embeddings allow for effective classification of these outcomes, and could be useful for future work on understanding the holistic experience of depression and its treatment. 
related work: 
 Various natural language processing techniques can detect mental health disorders and symptoms by automating the analysis of large volumes of data, as described in a recent survey of nearly 400 articles [7] . Social media posts are the predominant data source (81%) [8] , followed by interviews (7%), EHRs (6%), screening surveys (4%), and narrative writing (2%) [7] . NLP transforms text into numerical representations, which may include specific linguistic features, language representation features, and others. NLP uses both traditional machine learning (ML) and deep learning-based methods for tasks related to depression, such as risk assessment, symptom detection, and more.Traditional ML methods usually extract handcrafted features in model training for classification or prediction tasks. Features include linguistic, statistical, and domain-specific features. For example, linguistic features identified by the Linguistic Inquiry and Word Count (LIWC) [9, 10] tool have proven effective in detecting depressive moods and other mental health indicators from language [11] [12] [13] [14] [15] . Part-of-speech (POS) tagging [16] [17] [18] or extraction of sentiments, emotions, topics, word usage, grammar, and readability have also been used [14, [19] [20] [21] . Statistical features also include bagof-words (BoW) [16, 22] , n-grams [23] [24] [25] , term frequency-inverse document frequency (TF-IDF) [26] , and sentence or passage length [27, 28] . Domain-specific features might involve ontologies and dictionaries, such as UMLS [29] or other specialized vocabularies [16, 30, 31] . Many traditional ML algorithms, such as support vector machines (SVMs) [12, 13, 32, 33] , decision trees [34] , random forests [34] , adaptive boosting [35] , k-nearest neighbors (KNN) [36, 37] , and logistic regression [13, [38] [39] [40] , have been applied for depression-related tasks.Deep learning-based methods garner significant attention due to their superior performance compared to traditional ML methods [7, 41] . In particular, LLMs have become foundational tools for transforming text inputs into quantitative vector representations, or embeddings. In contrast to traditional ML, embeddings are learned from data using various algorithms, such as neural networks, rather than being defined by human experts. These embeddings can then be used as inputs for classification models to predict annotations, such as the presence of specific depression markers. Various embedding techniques, including GloVe [42] , word2vec [43] , and transformer-based models like BERT [44] and RoBERTa [45] , effectively identify depression markers in text [25, 46] . Deep learning methods are generally categorized into convolutional neural network (CNN)-based, recurrent neural network (RNN)-based, and transformer-based approaches [7, 47] . CNN architectures incorporate convolutional, pooling, and fully connected layers [48, 49] . RNN architectures, such as long short-term memory (LSTM) and gated recurrent unit (GRU), often incorporate attention mechanisms and hierarchical attention networks for multi-level semantic information extraction, making them well-suited for sequential data like text [13, [50] [51] [52] [53] . Transformer-based methods, including BERT, RoBERTa, Llama [54] [55] [56] , Mistral [57] , and the GPT* series [58] , incorporate an attention mechanism that manages long-range dependencies, which are crucial in NLP applications. Transformers can be finetuned for various prediction and classification tasks, and large-scale pre-training improves performance, as demonstrated in specialized domains such as depression detection [59] [60] [61] [62] [63] [64] .Most previous studies tackle broad binary classification problems (i.e., depression and control group). Additionally, the lack of interpretability in many models prevents clinicians from relying on the outcomes of automated screening techniques. Therefore, the scientific community has initiated several efforts to improve the clinical applicability of machine learning studies, including the Early Risk Prediction on the Internet (eRisk) workshop, which has been part of the Conference Labs of the Evaluation Forum (CLEF) since 2017. eRisk provides a collaborative environment for developing methods and practical approaches for early detection of health risks on the Internet, including depression. In 2023, eRisk featured a depressionrelated task (Task 1) [65] that involved ranking sentences based on their relevance to each of the 21 symptoms of depression derived from the Beck Depression Inventory-II (BDI-II) [66] . Symptoms included pessimism, thoughts about suicide, or sleep problems, rated on a severity scale from 0 to 3. Outside of eRisk, other studies aggregate symptoms from different questionnaires, such as the BDI-II [67, 68] and PHQ-9 [69] , and transformer-based models, such as BERT, can screen for depression in patients [70] .However, these initiatives mainly rely on social media data, which limits clinical integration due to issues with standardization and reliability. Additionally, limited research focuses on adolescent participants, highlighting the need for studies that address the unique factors affecting depression detection in this age group. Our work differs in that it focuses only on the detection of symptoms particularly relevant to adolescents and uses a psychiatric clinical dataset rather than social media. In addition, given the sensitivity of the dataset, our approach was developed using the latest open-source large language models, such as Llama, rather than commercial ones such as GPT or Claude. In IMPACT-ME, interviews were conducted with adolescent patients, parents, and therapists at treatment start, end, and one-year follow-up, exploring therapy experiences and observed changes [6] . 
data: 
 3.1.2 Qualitative analysis and annotation. Krause et al. conducted a secondary qualitative analysis on these interviews to explore the range of treatment outcomes relevant to patients. Interviews from the end of treatment were transcribed verbatim and included pauses, filler words, interruptions, and typos. Participants were excluded if any of the three interviews were missing, if treatment ended within the first three sessions, or if they were referred to inpatient care. Of the remaining 34 cases (9 BPI, 9 CBT, and 16 STPP participants; 102 interviews), the average age was 16.2 years (𝑠 = 1.5, range = 12-19), and 21 (61%) were female. To categorize outcomes, Krause et al. first designed an a priori coding framework based on existing taxonomies of treatment outcomes. During annotation, outcomerelevant passages were extracted, and the coding framework was further modified to incorporate new themes. The final framework contained 29 specific outcome categories within seven high-level domains [6] , listed in Table 1 and described further in Table 4 in Appendix A. All annotations were performed by one researcher.We split the dataset of 34 subject cases into a training set of 26 subjects and a test (holdout) set of 8 subjects. Transcripts were grouped by subject (i.e., by triplets of interviews relating to an adolescent participant) to prevent training and teseting models on data from the same person. Test cases were determined by manually balancing positive and negative examples for all specific outcomes. The test set was not used in this paper and is reserved for future evaluation.Empty lines and header information, such as subject ID and interviewer ID, were removed from transcription files before analysis. Transcripts were split into speaker blocks, which were marked by the start of a new paragraph in the transcript. The original IMPACT-ME annotations created by Krause et al. were produced by highlighting excerpts of the transcript relevant to a specific outcome. These excerpts could start or end at any position in a speaker block, and a block could contain multiple annotations. For our models, an entire text block was Table 1 : Names of the high-level domains and the specific outcomes found during qualitative analysis [6] .Specific outcomes (A) Symptom change (1) Mood and affect, (2) anger and aggression, (3) appetite, (4) sleeping and energy, (5) self-harm, (6) suicidality, (7) anxiety, (8) other comorbidities (B) Coping and selfmanagement 4 ). The number of positive samples for each label can be found in in Appendix A Table 5 .The Original segmentation of text generated from annotations, containing 32,520 blocks, included various uninformative text segments. Outcome-relevant dialogue would often be interspersed with interjections, acknowledgments, or requests for elaboration, e.g., an interviewer saying "okay" or "yes" to encourage a patient would be included within the excerpt and labeled as positive in our dataset. To address these uninformative text blocks, we created two additional segmentations of the transcript, Monologue and Turns, described below. An example contrasting these segmentations with the Original segmentation can be found in Appendix A Table 3 .Monologue: We discarded all interviewer speech and blocks with twelve or fewer characters. We manually determined the cutoff by examination of the labeled text in the training set. By only retaining non-trivial interviewee text, we aimed to produce "monologues" about the study experience, although some interviews, such as those conducted jointly with both parents of a patient, retained multiple interviewees interacting in dialogue. Of the original 32,520 blocks, 12,941 were retained in this filtration.Turns: We partitioned blocks at each interview utterance, grouping together sequential pairs of utterances by interviewer and interviewee into "turns" of the conversation. By concatenating blocks, the Turns segmentation kept informative interviewer questions together with short interviewee responses that were otherwise uninformative (e.g., "I: How has your mood been?" "P: Fine... "). For interviews with multiple interviewees, all utterances between interviewer utterances were concatenated into the same turn. This process produced 16,139 blocks of text. 3.1 Large language model embeddings. Embeddings were produced with various transformer-based LLMs. As a baseline, we used the base variant of BERT [71] , a common choice for various NLP tasks, such as sentiment analysis or summary generation. Additionally, we included MentalBERT, a BERT model pretrained on additional data collected from various Reddit communities related to mental health discussion [72] . Because of the long passages present in all segmentations, we also included MentalLongformer, a derivation of Longformer [73] pretrained on the same mental health data as MentalBERT [74] . Furthermore, we included Llama 2-7B, and Llama 3-8B, state-of-the-art open source models [54, 56] .LLMs produced embeddings of size 𝑏 ×𝑙 ×𝑑, where 𝑏 is the batch size, 𝑙 is the sequence length, and 𝑑 is the hidden dimension of the model. For each model, we used 𝑏 = 1, i,.e., passing individual blocks to the LLM. Sequence length (number of tokens), varied based on a passage's length and the model-specific tokenizer. Passages exceededing a model's maximum sequence length were truncated before embedding. Embeddings were averaged across all tokens in the sequence to produce a 𝑑-dimensional vector of predictors for each text block. An overview of model details can be found in Table 2 .To classify labels, we trained L2-penalized logistic regression models on the 𝑑-dimensional averaged embedding vector for each passage. Models were trained and evaluated with a 4-fold cross-validation (CV) loop. For each of the 4 test folds, the 𝐶 hyperparameter of logistic regression was tuned with inner 3-fold CV, using the same fold partitions as the outer 4-fold CV. Data were grouped by subject ID and stratified by label. Models for labels A8, E4, and E5 could not be trained because fewer than four subjects were present in the development data. To adjust the loss function for the imbalance between positive and negative examples in every label, errors in positive examples were multiplied by the ratio of positive to negative examples in that label. 
classification performance for each label: 
 Our first goal was to investigate classification performance of each of the embedding models for our 39 binary labels (31 specific outcomes 1 , 7 high-level domains, and presence of any outcome). For each model, we computed the area under the ROC curve (ROC AUC) for each test fold and reported the average ROC AUC across folds [75] . Classification performance fell within 0.6-0.9 for the Original segmentation and 0.7-1.0 for the Monologue and Turns segmentations, seen in Figure 1 . (details in Table 6 in Appendix B).In the Original segmentation, D1 performs the best across all models. In Monologue, the best performer was one of D3 or F2. In Turns, A3 and D3 performed well for all models, but the top performer for MentalLongformer was F2. For any combination of model and segmentation, the lowest performer tended to be D2 or G1, with A2 performing poorly in Original. Many labels were inconsistent across models and segmentations. For example, A5 was in the top four for all models in the Original segmentation, but underperformed in Monologue and Turns in non-Llama models. The worst classified labels tended to have high variance in performance across embedding models, though the relative rankings are consistent. For every embedding, the averaged ROC AUC for models of "Any" outcome were between 0.75-0.85. Further details on relative classification performances can be found in Table 7 in Appendix B.Our second goal was to determine whether embeddings from a particular large language model had consistently better performance than others. For our 28 specific outcomes, we tested the null hypothesis of no difference in model performance with the Friedman test [76] . We excluded aggregate labels, i.e., the seven domain labels and the "Any outcome" label, to avoid double-counting. Friedman test results were 𝑄 3 = 8.571, 𝑝 = 0.0356 for Original; 𝑄 3 = 13.16, 𝑝 = 0.00431 for Monologue; and 𝑄 3 = 12.56, 𝑝 = 0.00570 for Turns. At significance level 𝛼 = 0.05, the Friedman test results supported rejection of the null hypothesis of model equivalence, and we proceeded with the post hoc Bayesian comparison tests [77] .The Bayesian post hoc test indicated that both Llama models had probability ≥ 0.94 of outperforming any other non-Llama model (Figure 2 ). BERT had a < 0.04 probability of outperforming any model except for MentalLongformer, where the probability of BERT being better was 0.14, 0.15 and 0.08 for Original, Monologue, and Turns, respectively. Llama 2-7B and Llama 3-8B had a 0.84, 0.71 and 0.92 probability of practical equivalence for Original, Monologue, and Turns. MentalLongformer and MentalBERT as well as BERT and MentalBERT had practical equivalence probabilities of 0.19 to 0.38 in Original and Turns. All other pairwise model comparisons returned ≤ 0.06 probability of practical equivalence. 
discussion and conclusion: 
 Models generally performed well, even for labels with very few positive examples. Across the 36 labels considered, performance was for each label are grouped vertically by domain (labeled by color) and horizontally by segmentation (labeled on right axis). Domains are "Symptom Change" (A), "Coping and selfmanagement" (B), "Functioning" (C), "Personal growth" (D), "Relationships" (E), "Peace of mind" (F), and "Parental support and wellbeing" (G). Numbers indicate the performance of each specific outcome within a domain, black letters the domains, and the red X represents "Any" of the outcomes. never below an average ROC AUC of 0.60 for every model within a segmentation. Given these results, we believe that classifiers using LLM embeddings as inputs could prove useful for detecting fine-grained outcomes. On the other hand, it is unclear why specific labels were easier or harder to classify. Even within the same domain, specific outcomes can run a wide range, e.g., D3 being best overall while D2 is worst overall. The relative performances of aggregated labels, such as labels for high-level domains A through G, tend to be consistent across models within a segmentation, suggesting that some variability may be due to the small number of positive examples.The Bayesian comparison test between models suggests that, of the models investigated, Llama models produce more informative embeddings for classification. The test also suggests that Llama 2-7B and Llama 3-8B have a high probability of practical equivalence, which is unsurprising considering their architectural similarities. The non-negligible probability of practical equivalence for Men-talBERT and MentalLongformer is also unsurprising, considering the models are pre-trained with the same set of mental health data. MentalBERT, though fine-tuned on domain-specific data, only has a high (0.98) probability of outperforming BERT on Monologue, and has a moderate probability of practical equivalence (0.32, 0.38) on Original and Turns. Additionally, although the Bayesian comparison test suggests that Llama models have high probabilities of outperforming the other models tested, the advantage is not very large. Other concerns, such as resource usage, may be a deciding factor in choosing an embedding model for different tasks. Llama 2-7B and Llama 3-8B, for example, require a GPU to perform inference, while BERT, MentalBERT, and MentalLongformer can produce embeddings using the CPU available in a standard laptop.Based on our results, our methodology may be useful for similar datasets or labels at a comparable level of granularity. However, it is unclear how reported performance could reflect idiosyncrasies of this dataset. The small sample size, both in total positive labels and number of subjects, may prove an obstacle when applying these particular models to other datasets. For example, the adherence to transcribing the exact utterances of the interviews is not common in written text or in machine transcription, which often removes pauses, filler words, and accent indicators.The 𝑘-fold cross-validation results should provide reasonable estimates of model performance in new participants, given that we have reported on all the experiments that we have carried out. Nevertheless, the final analysis of generalization, and model variance, should be performed on the test set, which we are currently withholding to allow for further model development on the training set. When testing on the holdout, we will produce performance estimates for labels for which models could not be trained and tuned during the 𝑘-fold cross-validation step due to sparsity.Future model development will work on two fronts. The first is to improve text representation, either through more advanced models or by fine-tuning LLMs for these prediction tasks. The second will be to improve the generalization of prediction models through, for example, supplementing the training data with additional synthetic examples, paraphrased by an LLM from existing ones. 
 
 Appendix: Table 3 illustrates how segmentations might be created from a passage, though the text example lacks many of the interview transcripts' idiosyncrasies, such as inclusion of hesitations and filler words. Additionally, the example Monologue segmentation is more coherent than the true dataset, as realistic answers are often difficult to understand without the context of the interview question.Table 4 provides more description of the specific outcomes of interest in the IMPACT-ME interviews. The addendum to Krause et al. also contains examples of specific outcomes to illustrate how they may appears in the transcript [6] . ROC AUCs reported in Table 6 are averaged across each of the 𝑘 outer folds, with 𝑘 = 4. By sorting the labels by average overall rankings (Table 7 ), we can observe that around half of the specific outcomes show wide variance in the relative performance for each model and segmentation.Table 4 : Names of the labels and a brief description. Unless otherwise indicated, assume descriptions refer to changes with the adolescent patient.Name Description Parental support Parents are better able to understand their child's difficulties and more aware of how their parenting practices may contribute to these difficulties; parents learn to support and parent their child more effectively G2 Parental wellbeing Parents feel less guilty, isolated, stressed, and worried; parents feel reassured, supported, and able to express their own frustrations and issues Any 𝐴 ∪ • • • ∪ 𝐺 A Symptom change 𝐴 1 ∪ • • • ∪ 𝐴 8 A1Results are reported for28/31 specific outcome labels (i.e., 36/39 binary labels), as three labels did not contain enough subjects for 4-fold CV. 



Reviews: review: title: title: Review of "Using Large Language Models to Detect Outcomes in Qualitative Studies on Adolescent Depression"
main_review: main_review: This manuscript attempts to leverage Large Language Models (LLMs) for extracting features related to depression from clinical data, presenting a novel approach. The use of detailed and extensive tables to present results provides clarity and enriches the content. However, there are several areas where the manuscript could be significantly improved.

- Justification for Data Choice: The manuscript states that the use of clinical data over social media data, but I'm wondering why this choice is superior for depression identification.
- Outdated Literature Review: The related work section is mostly about traditional machine learning and deep learning methods that are dated. This section needs substantial updates to include more recent advancements in NLP and specifically LLMs for depression detection.
- Lack of Comparative Analysis: The experimental results are presented only as absolute AUC values without any comparison to baseline methods, which undermines the persuasiveness of the results. Mentioning and comparing to at least some baseline methods in the related work would enhance the credibility.
- Redundancy in Classification Discussion: The section on using LR for classification seems redundant. The main focus of the manuscript is on using LLMs to extract certain features of depression, having already validated the AUC for each indicator and high-level domain AUC. If the intention is to further demonstrate that LLM-extracted indicators are beneficial for depression classification, a comparison should be introduced. Specifically, a baseline should be introduced to show that features extracted by other methods are less accurate in classification when using LR compared to features extracted by LLMs.
- Figure Clarity: Figure 1 is described as too vague and overloaded with information, making it difficult to grasp the main points. A suggestion is made to replace this figure with tables from the appendix that focus on AUC values for a clearer and more detailed presentation.
- Small Dataset: The dataset size is relatively small, with only 34 cases, which may not provide enough variability to generalize the findings effectively.
title: title: LLM for mental health on adolescent depression, but only in outcomes of detection
main_review: main_review: Summary:
In this paper, authors present a novel approach to use LLM to detect outcomes for mental health, especially on adolescent depression. Thus, they use the embeddings to classify the high-level domains from their qualitative analysis.

Strengths:
This article provides an interesting perspective to use the LLM to process the text on adolescent depression, and it is demonstrated to be successful on a range of classification tasks.  It is a solid work with complete experiments.

Weaknesses:
In general, the technical part of this article is relatively weak in innovation. It only uses language models to do some basic text classification tasks, just like traditional NLP tasks. 

In addition, the article only uses the most advanced LLM embedding for classification tasks, which may limit its performance to some extent. Because LLM, such as Llama 3, their training method and architecture (decoder-only) are not suitable for some text understanding tasks, even though they do well in text understanding. The author should try other classification tasks to test the performance of LLM as much as possible. For example, zero-shot classification, few-shot classification tasks, etc. These are the most suitable scenarios for LLM.

As mentioned above, I would mainly ask you to add some new tasks for LLM. I think It is easy to do that.