Venue: NAACL 2024

# Depression Detection in Clinical Interviews with

LLM-Empowered Structural Element Graph

 Zhuang Chen\({}^{1}\)   Jiawen Deng\({}^{2}\)   Jinfeng Zhou\({}^{1}\)    Jincenzi Wu\({}^{1}\)

**Tieyun Qian\({}^{3}\)  Minlie Huang\({}^{1}\)***

\({}^{1}\)CoAI Group, DCST, TUPM, IAI, BNRIST, Tsinghua University

\({}^{2}\)University of Electronic Science and Technology of China  \({}^{3}\)Wuhan University zhchen-nlp@mail.tsinghua.edu.cn aihuang@tsinghua.edu.cn

Corresponding author.

###### Abstract

Depression is a widespread mental health disorder affecting millions globally. Clinical interviews are the gold standard for assessing depression, but they heavily rely on scarce professional clinicians, highlighting the need for automated detection systems. However, existing methods only capture part of the relevant elements in clinical interviews, unable to incorporate all depressive cues. Moreover, the scarcity of participant data, due to privacy concerns and collection challenges, intrinsically constrains interview modeling. To address these limitations, in this paper, we propose a structural element graph (SEGA), which transforms the clinical interview into an expertise-inspired directed acyclic graph for comprehensive modeling. Additionally, we further empower SEGA by devising novel principle-guided data augmentation with large language models (LLMs) to supplement high-quality synthetic data and enable graph contrastive learning. Extensive evaluations on two real-world clinical datasets, in both English and Chinese, show that SEGA significantly outperforms baseline methods and powerful LLMs like GPT-3.5 and GPT-4.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8181–8194 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Depression is a pervasive mental health disorder that affects a significant portion of the global population. According to the World Health Organization (WHO)1, over 300 million people suffer from depression, casting a profound shadow on individuals, families, and society. Furthermore, in many communities, limited awareness and stigmatization surrounding mental health can lead to under-diagnosis and under-treatment, which underscores the imperative for effective depression screening.

Footnote 1: [https://www.who.int/news-room/fact-sheets/detail/depression](https://www.who.int/news-room/fact-sheets/detail/depression)

To date, clinical interviews remain the standard method for assessing depression severity He et al. (2022). In Figure 1, we present an example interview from DAIC-WOZ Gratch et al. (2014), a widely-used corpus for clinical depression detection. As can be seen, clinical interviews typically take the form of semi-structured multi-round question-answering dialogues. The interviewer follows an outline while flexibly posing questions relevant to personal experience and mental states, and the participant answers are recorded to assess the state of depression. Currently, conducting clinical interviews has become relatively feasible, by either an "Wizard-of-Oz" virtual interviewer Kelley (1983) or a fully autonomous scripted agent DeVault et al. (2014). However, analyzing clinical interviews for depression assessment remains a domain reserved for highly specialized clinicians.

As an alternative, automatic assessment of clinical interviews has emerged as a promising assistance for depression detection Valstar et al. (2016). Clinical interviews mainly encapsulate four types of elements: the _interview questions2_ and the _answer transcripts/audios/videos_, enabling super

Figure 1: A clinical interview for assessing depression.



vised learning for automatic depression assessment. However, existing studies still face two major challenges. **1)** As can be seen from Table 1, existing methods only capture part of the four elements in clinical interviews, unable to incorporate all relevant cues for depression detection. The main reasons are twofold. One is that the background noise in audio and video modalities poses difficulties for interaction among different elements, which may even degrade performance Baltrusaitis et al. (2018); Hazarika et al. (2022). Moreover, the interview questions are usually overlooked since they contain no participant information. The other is the potential complexity of interviews, which can include up to 50 rounds, leading to a compromise in using only partial elements for efficient aggregation Al Hanai et al. (2018). **2)** The scarcity of participant data, arising from privacy concerns and collection costs, directly constrains the quality of interview modeling. Commonly available datasets like DAICWOZ contain only \({\sim}\)100 training samples with \({\leq}30\) depressed participants. Hereby, training a sophisticated depression detection model becomes an intrinsical demanding task.

To address the above limitations, we propose an LLM-empowered **S**ructural **E**lement **G**r**Aph (SEGA) for comprehensive interview modeling under limited resources. **1)** To make full use of all four types of elements, we transform the clinical interview into SEGA, a directed acyclic graph. We first gather expertise about processing interview elements in depression assessment, then align SEGA's structure with this human experience to facilitate representation learning. Specifically, within each round, we predefine the information flow between elements with noise suppression, allowing for element interaction while avoiding interference. Across multiple rounds, we specify simple yet efficient information propagation and aggregation approaches, so as to preserve the complete element features for predicting depressive status. **2)** To alleviate the negative impact of data scarcity on interview modeling, we introduce a novel LLM-based data augmentation approach guided by principles of _integrity_, _authenticity_, _respectfulness_, _consistency_, and _informality_. Along with the graph contrastive learning with real and synthetic data, we further advance the performance of SEGA under limited resources. We conduct experiments on two real-world clinical interview datasets in English and Chinese, respectively. The results demonstrate that SEGA significantly surpasses existing baseline methods and powerful LLMs like GPT-4 for depression detection in clinical interviews.

## 2 Related Work

### Depression Detection

Depression is a debilitating mental condition that affects a person's thoughts, feelings, and behavior. The automatic analysis of clinical interviews has been recognized as a promising assistant for psychologists and psychiatrists to improve the time efficiency and diagnostic consistency of depression detection Valstar et al. (2016); Zou et al. (2022).

Most studies leverage the participant answers for depressive assessment, which can be categorized into three types: 1) Uni-modal methods typically model either transcripts or audios. For transcripts, existing work focuses on aggregating word representations for prediction Mallol-Ragolda et al. (2019); Burdisso et al. (2023), or further incorporates affective and mental health lexicons Xezonaki et al. (2020); Villatoro-Tello et al. (2021). For audios, existing studies have investigated the utility of shallow acoustic features like MFCCs Taguchi et al. (2018); Huang et al. (2022) and deep audio representations from neural networks like CNN Ma et al. (2016); Zhang et al. (2021); Sardari et al. (2022); Sun et al. (2022) . 2) Bi-modal methods typically combine transcripts and audios as the two most informative modalities via feature fusion Al Hanai et al. (2018); Wu et al. (2022); Guo et al. (2022); Seneviratne and Espy-Wilson (2022) or prediction ensembling Niu et al. (2021); Shen et al. (2022). Yoon et al. (2022) model both audio and video modalities with a cross-attention transformer, but the performance is relatively limited without transcripts. 3) Tri-modal methods consider the answer transcripts, audios, and videos via multimodal fusion conducted at word-level Rohanian et al. (2019) or utterance-level Guohou et al. (2020); Zhang et al. (2020); Fang et al. (2022); Prabhu et al. (2022).

\begin{table}
\begin{tabular}{l c c c c} \hline \multicolumn{1}{c}{Method} & \multicolumn{3}{c}{Elements in Clinical Interviews} \\  & \(\mathbb{T}\) & \(\mathbb{A}\) & \(\mathbb{V}\) & Q \\ \hline \multirow{2}{*}{\begin{tabular}{c} \vspace{1mm} \textbackslash{} \\ \textbackslash{} \\ \textbackslash{} \\ \textbackslash{} \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} \vspace{1mm} \textbackslash{} \\ 

In addition to the three modalities of the answers, a few studies Niu et al. (2021); Flores et al. (2022) have incorporated the interview questions as supplementary context to better extract salient cues from participant answers. Different from the above methods, we resort to human experience in depression assessment to construct the structural element graph, so as to effectively model all four types of interview elements under limited resources.

### LLMs Application in Healthcare

Recent years have witnessed the rapid development and powerful capability of large language models (LLMs), such as OpenAI's ChatGPT Ouyang et al. (2022), Anthropic's Claude Bai et al. (2022), Google's LaMDA Thoppilan et al. (2022), and Meta's LLaMA Touvron et al. (2023). LLMs have shown promising potential in powering the healthcare domain. Wang et al. (2023) and Chen et al. (2023) use LLMs to play doctor and patient roles, respectively, to generate medical dialogue data. Yang et al. (2023), Bisercic et al. (2023), and Jo et al. (2023) leverage LLMs to assist diagnosis via medical report generation, tabular data extraction, and emotional support deployment, respectively. Sarker et al. (2023) and Liyanage et al. (2023) conduct data augmentation via LLMs to enhance medication event identification and wellness dimension classification.

Despite these advances, LLMs remain underexplored in the realm of depression detection in clinical interviews. In this work, we meticulously craft principle-based prompts to direct LLMs in generating high-quality data for depression detection, complemented by responsible quality verification and ethical consideration.

## 3 Methodology

### Task Definition

In a clinical interview \(I\) from the corpus \(\mathcal{C}\), the interviewer asks the participant a series of \(m\) questions (recorded as text) \(Q=\{q_{1},...,q_{m}\}\), typically probing into their feelings, experiences and mental states. The participant's responses are recorded in three modalities - transcripts \(T=\{t_{1},...,t_{m}\}\) capturing the textual content, audios \(A=\{a_{1},...,a_{m}\}\) capturing tone and prosody, and videos \(V=\{v_{1},...,v_{m}\}\) capturing facial expressions and body movements. Each question (\(q_{i}\)) and transcript (\(t_{i}\)) comprises a word sequence, while each audio (\(a_{i}\)) and video (\(v_{i}\)) comprises a frame sequence sampled with a certain frequency. The goal is to analyze these elements and predict a label \(y\in\{1,0\}\) denoting whether the participant is depressed or not.

### Structural Element Graph

For depression detection in clinical interviews, we propose the structural element graph (SEGA) based on human experience for comprehensive interview modeling. Figure 2 presents the architecture of SEGA, which comprises three main components: element feature extraction, graph construction with expertise, and graph learning and prediction.

#### 3.2.1 Element Feature Extraction

In clinical interviews, each element serves as a unique indicator of the depressive state. Interview questions may probe for negative emotions, shedding light on underlying feelings of hopelessness or worthlessness. Answer transcripts may reveal a higher frequency of negative words, reflecting a depressed mindset. Answer audios can hint at depression through a flatter affect, slower speech rates, or prolonged pauses. Answer videos capture visual cues like reduced facial expressivity or subdued reactions. Together, these elements form a rich tapestry of information that helps in assessing an individual's depressive state.

Below we show how to extract element features. Within a question-answer round, the question (\(q_{i}\))/ transcript (\(t_{i}\)) is an utterance involving the text modality. We first map every word in the utterance using a pre-trained word embedder, then compute the mean of word vectors to obtain the utterance representation \(\mathbf{q_{i}}\) / \(\mathbf{t_{i}}\). For the audio (\(a_{i}\)) and video (\(v_{i}\)) modalities of the answer utterance, they are already processed into frames at a certain frequency, where the raw features are specified by the used dataset. We then process those frames according to the timestamps in the corresponding transcript. Specifically, we look at the start and end timestamps of the utterance in the transcript, then average the frames spanning the duration of this utterance to derive audio and video representations \(\mathbf{a_{i}}\) / \(\mathbf{v_{i}}\) for the answer utterance. By this means, we ensure that all element features are strictly aligned at the granularity of a question-answer round.

#### 3.2.2 Graph Construction with Expertise

Based on the element features, we follow specific expertise for depression assessment to derive the graph structure and facilitate interview modeling 



**Expertise within Each Round**: 1) Element Importance. Studies on multi-modal learning have demonstrated that texts contain the richest semantics Rohanian et al. (2019); Hazarika et al. (2022). Therefore, for each round, we anchor the transcript \(t_{i}\) as the central node, and treat the audio \(a_{i}\) and video \(v_{i}\) as auxiliary nodes. The question \(q_{i}\) is also recognized as a supplementary node since it completes the contextual information of the answer. 2) Information Flow. To reflect the status of nodes, we build a directed acyclic graph Shen et al. (2021) where information is passed unidirectionally from the auxiliary nodes to the central node. 3) Noise Suppression. To avoid direct interference from the background noise in audio and video elements Baltrusaitis et al. (2018), we additionally insert a proxy node \(p_{i}\) within each round. It aims to first distill evidence from the audio and video, transform it into the text vector space, and then let the transcript node selectively absorb useful information. The representation of \(p_{i}\) is initialized with the representation of \(t_{i}\). Besides, each element node has a self-loop edge to preserve its own information.

**Expertise across Multiple Rounds**: 1) Information Propagation. Since an interview topic (e.g., querying family relationships) may span multiple rounds, we link adjacent central nodes (\(t_{i}\)) for capturing temporal dependencies. 2) Information Aggregation. To represent the semantics of the entire clinical interview, we introduce an additional virtual "summary" node (\(s\)). We then pass information unidirectionally from all central nodes (\(t_{i}\)) to the summary node \(s\). The representation of \(s\) is initialized by averaging all \(t_{i}\) representations.

The structural element graph is a simple yet effective architecture. On the one hand, it fully utilizes the evidence in four types of elements embedded in clinical interviews. On the other hand, it benefits from prior human experience and sets a good starting point for interview modeling.

#### 3.2.3 Graph Learning and Prediction

Based on the constructed structural element graph, we employ Graph Attention Network (GAT) Velickovic et al. (2018) to capture the interactions among elements and accordingly update representations. For node \(i\) and its neighborhood nodes \(\mathcal{N}(i)\), we iteratively updates the node representation \(\mathbf{h_{i}}\in\mathbb{R}^{d_{h}}\) by aggregating neighborhood node representations using multi-head attention:

\[r_{ij}^{(k)}=\text{LeakyReLU}(\mathbf{w}^{\top}[\mathbf{W}^{(k)}\mathbf{ h_{i}}||\mathbf{W}^{(k)}\mathbf{h_{j}}]), \tag{1}\] \[\alpha_{ij}^{(k)}=\text{softmax}_{j}(r_{ij}^{(k)})=\frac{\exp( r_{ij}^{(k)})}{\sum_{m\in N_{i}}\exp(r_{im}^{(k)})},\] (2) \[\mathbf{h}_{i}^{(k)}=\text{ELU}\left(\sum_{j\in\mathcal{N}_{i}} \alpha_{ij}^{(k)}\mathbf{W}^{(k)}\mathbf{h}_{j}\right),\] (3) \[\mathbf{h}_{i}=||_{k=1}^{K}\mathbf{h}_{i}^{(k)}, \tag{4}\]

where \(k\) denotes an attention head, \(\mathbf{W}^{(k)}\) is the weight matrix, \(\mathbf{w}\) is the weight vector, \(\alpha_{ij}^{(k)}\) is the attention weight, \(\mathbf{h}_{i}^{(k)}\) is the output feature vector for head \(k\), \(||\) denotes vector concatenation. After the computation of \(L\) GAT layers, we collect the representation \(\mathbf{s}\) of the summary node after propagation as the final feature of the interview. We then feed it into a feed-forward layer to predict the depressive status \(\hat{y}\) and compute the cross-entropy loss \(\mathcal{L}_{ce}\) for detection:

\[\hat{y}=\text{softmax}(\mathbf{W}_{y}(\mathbf{s})), \tag{5}\] \[\mathcal{L}_{ce}=-\left(y\log(\hat{y})+(1-y)\log(1-\hat{y})\right). \tag{6}\]

### A Helping Hand from LLMs

While SEGA intrinsically aligns with the expertise of processing interviews, its learning capability is still constrained by the limited clinical resources. Due to collection difficulties and privacy concerns, the training set size of commonly-used public clinical depression detection corpora is only about 100. Nevertheless, large language models (LLMs) possess an extremely strong human-like

Figure 2: Architecture of SEGA. We first extract element features according to timestamps and frame index, then construct and learn a structural element graph for depression detection in clinical interviews.



text generation capability, offering immense potential for generating synthetic data to alleviate data scarcity. Hereby, we propose to design a principle-based prompt to carefully control the LLM for data augmentation. The goal is to simulate diverse participant responses by rephrasing transcripts with the same meanings but different vocabularies.

#### 3.3.1 Principle-based Augmentation

While LLMs are capable of generating fluent and coherent text, their outputs can sometimes deviate from the original meaning or contain inappropriate content if not properly constrained. This presents challenges when utilizing them for data augmentation in sensitive domains like mental healthcare. Therefore, to ensure LLMs rephrase transcripts faithfully without altering semantics or tones, we guide the synthesizing process with explicit principles. The five principles serve as "guard rails" to steer the model toward making edits that introduce diversity while preserving _integrity_, _authenticity_, _respectfulness_, _consistency_, and _informality_.

We list the principles as follows. 1) Integrity of Content: Retain the original meaning and sentiment. Do not change the substance. 2) Conversational Authenticity: Use natural, casual language. Avoid overly formal styles. 3) Respectful Communication: Maintain a respectful tone. Do not make inappropriate/offensive alterations. 4) Consistency in Length: Keep similar length to the original. Avoid excessive shortening/lengthening. 5) Tolerance for Informality: Tolerate some irregularities (omissions, repetitions, filler words) given the conversational context.

We rephrase transcripts in a sliding window with a size of three, applying LLMs with context awareness. In other words, when rephrasing each answer transcript, we use the last three rounds of question-answers as input. Besides, we also provide \(N\) hand-crafted demonstrations to facilitate task understanding. As shown in Figure 3, the final input prompt is composed of task instruction, rephrasing principles, and manual demonstrations. After prompting, we collect the outputs of LLMs as the augmented data. For the accompanying audios and videos, due to the untouchability of raw recordings and the lack of LLM-level synthesis tools, we adopt a simple random frame-swapping method. We sample \(a_{j}\neq a_{i}\), \(v_{j}\neq v_{i}\) in the same interview as augmentation features, capturing feasible variance. We do not augment interview questions as their content is predefined.

After synthesizing augmentation data via rephrasing, we manually inspect samples for quality verification. We verify that: 1) The rephrased sentences express the same semantic content as the originals, without introducing factual inconsistencies or changes in meaning. 2) The tone and sentiment remain consistent between the original and augmented versions. 3) No offensive, unethical, or otherwise inappropriate language is generated. 4) Personal details and identifiers are not altered or exposed. Ultimately, steered by the guiding principles, only very few (less than 1%) of the augmented data is filtered out and regenerated.

#### 3.3.2 Empowerment of \(\mathsf{SEGA}\)

After obtaining the synthetic data \(\hat{\mathcal{C}}\), we mix it with the original corpus \(\mathcal{C}\) to obtain the augmented training data. Furthermore, owing to the improved size and diversity with synthetic data, we introduce the self-supervised contrastive learning with InfoNCE [9] loss to encourage distinct representations for depressed and control participants:

\[\small\mathcal{L}_{cl}=-\sum_{c}^{c||\mathcal{C}}\log\frac{\exp(\mathbf{s}_{c}^{ \top}\mathbf{s}_{c}^{\top}/\tau)}{\exp(\mathbf{s}_{c}^{\top}\mathbf{s}_{c}^{+}/\tau)+\sum_ {j=1}^{J}\exp(\mathbf{s}_{c}^{\top}\mathbf{s}_{c,j}^{-}/\tau)} \tag{7}\]

where \(\tau\) denotes a temperature parameter, \(\mathbf{s}_{c}^{+}\) is a certain positive sample with the same label , \(\mathbf{s}_{c}^{-}\) denotes several negative samples with the opposite

Figure 3: Guided by principles, LLMs synthesize augmentation data by rephrasing the answer utterances.

 

label. By update the final training loss as \(\mathcal{L}=\mathcal{L}_{ce}+\mathcal{L}_{cl}\), we obtain the empowered model SEGA++.

## 4 Experiment

### Experimental Setup

DatasetsWe conduct experiments on two available corpora for clinical depression detection: DAIC-WOZ Gratch et al. (2014) and EATD Shen et al. (2022). DAIC-WOZ is a widely-used English dataset that collects interviews from 189 participants, recorded in transcripts, audios, and videos. Each participant is accompanied by a PHQ-8 score Kroenke et al. (2009), and those with scores \(>=\) 10 are labeled as depressed ([D]), while the others are labeled as control ([C]). DAIC-WOZ has a fixed data split, and we follow previous studies by using the development set for evaluation since the labels of the test set are unavailable. EATD is a newly released Chinese dataset that consists of interviews conducted with 162 student volunteers, recorded in transcripts and audios. Each participant is accompanied by an SDS score Zung (1965), and those with scores \(>=\) 53 are labeled as depressed. EATD is split into training and development sets via 3-fold cross-validation with train:dev = 2:1. The detailed statistics of datasets are shown in Table 2. The details of interview recordings and pre-processing methods can be found in Appendix A.

SettingsWe first discuss feature pre-processing. For DAIC-WOZ, we use the GloVe.84OB.300d embeddings Pennington et al. (2014) to vectorize interview questions and answer transcripts. The videos and audios have already been processed by COVAREP Degottex et al. (2014) and OpenFace Baltrusaitis et al. (2016), respectively. For EATD, we use the pre-trained Chinese BERT Cui et al. (2020) as the word embedder. EATD only provides unprocessed raw audios, so we extract the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) Eyben et al. (2016) via OpenSmilLE Eyben et al. (2010) as acoustic features.

For the implementation of SEGA, the dimension of all hidden states is \(d_{h}\)=256. The layer number of GAT is \(L\)=2. The number of attention head \(K\)=8. We use gpt-3.5-turbo-0613 for principle-guided data augmentation. In the prompt, we include \(N\)=2 manually written samples as the demonstration. We collect three rephrased results for each transcript, but only use the first one since no significant performance difference is observed, resulting in 2x training samples. We perform contrastive learning within a single batch with \(\tau\)=0.05. For DAIC-WOZ/EATD, we train the models for 300/100 epochs using Adam optimizer Kingma and Ba (2015) with a learning rate 1e-4/2e-5 and batch size 8. We run all experiments on a single Tesla V100 32G GPU in Ubuntu 16.04. SEGA contains 2.6M trainable parameters and requires 0.33 GPU hours for training. The hyperparameter ranges can be found in Appendix B.

Following previous studies Burdisso et al. (2023), we use depressed, control (i.e., non-depressed), and macro F1-scores as the metrics. We select the checkpoint for evaluation based on macro F1-scores. The final results for comparison are the average scores of 3 runs with random seeds (for DAIC-WOZ) or folds (for EATD)3.

Footnote 3: Code is available at [https://github.com/zhchen18/SEGA](https://github.com/zhchen18/SEGA).

### Compared Methods

After reviewing the literature on depression detection, we find that the pre-processing of clinical interviews in different methods varies considerably, making it inaccurate to compare their reported results directly. For instance, Niu et al. (2021) employ manual cleaning of transcripts, while Shen et al. (2022) group every ten participant answers as one sample. Additionally, there are hardly any reported results on the recently released EATD dataset. Therefore, we select the four latest top-performing methods involving participant transcripts, audios, videos, and interview questions used in SEGA. Since most models are not open-sourced, we carefully replicate them following the original papers. Our implementation may serve as a basis for fair comparison in future research.

\(\omega\)**-GCN**Burdisso et al. (2023) is a uni-modal model that leverages answer transcripts. **ETDP Fusion**Shen et al. (2022) is a bi-modal model that leverages answer transcripts and audios. **MFM-Att**Fang et al. (2022) is a tri-modal model that lever

\begin{table}
\begin{tabular}{l l c c c c} \hline \multicolumn{7}{c}{**Dataset Size Category**} & **Round** & **Token** & **Duration** \\ \hline \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & train & 107 & [D] 30 & 6,069 & 149,149 & 26h53m \\  & & [C] 77 & (\(\bar{x}\)=57) & (\(\bar{x}\)=1,394) & (\(\bar{x}\)=15m04s) \\ \cline{1-1}  & dev & 35 & [D] 12 & 1,909 & 53,588 & 10h01m \\  & & [C] 23 & (\(\bar{x}\)=55) & (\(\bar{x}\)=1,531) & (\(\bar{x}\)=17m09s) \\ \hline \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & train & 108 & [D] 237/20/24 & 324 & 19,994 & 1h22m \\  & & [C] 85/88/84 & (\(\bar{x}\)=3) & (\(\bar{x}\)=181) & (\(\bar{x}\)=47s) \\ \hline \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } & train & 107 & 71/10/6 & 162 & 9,968 & 1h1m \\  & & [C] 47/44/48 & (\(\bar{x}\)=3) & (\(\bar{x}\)=177) & (\(\bar{x}\)=49s) \\ \hline \end{tabular}
\end{table}
Table 2: Detailed statistics of DAIC-WOZ and EATD.

 

ages answer transcripts, audios, and videos. MFM-Att for EATD only uses transcripts and audios since there is no video modality in EATD. **HCAG**Niu et al. (2021) is a model that captures the interaction between interview questions and participant answers. Considering that we use LLMs for data augmentation, we further compare the performance of **GPT-3.5** and **GPT-4** by prompting them to determine whether the participant is depressed according to the clinical interview. We consider both zero-shot and few-shot settings, where the latter includes a control and a depressed training sample as two demonstrations. The implementation details of baseline methods can be found in Appendix C.

### Main Results

We present the performance on DAIC-WOZ and EATD datasets in Table 3. Obviously, SEGA achieves state-of-the-art performance, outperforming the best baseline by 1.12% and 2.36% in terms of macro F1-scores on two datasets, respectively. After LLM-empowered data augmentation, SEGA\({}^{++}\) obtains further gains of 2.83% and 1.81%, surpassing baselines by 3.95% and 4.17%. Moreover, all methods achieve markedly lower F1-scores on the depressed class compared to the control class, stemming from the severe class imbalance prevailing in the corpora. For the primary depressed class, SEGA and SEGA\({}^{++}\) demonstrate notably higher performance than all baselines, reaching (81.48%, 84.62%) and (73.18%, 75.31%) on DAIC-WOZ and EATD, respectively.

When examining the upper four baselines, we observe that simply introducing more elements does not necessarily lead to better performance. For instance, \(\omega\)-GCN, by ingeniously weighting the relations between words and interviews, achieves the best baseline results on DAIC-WOZ. Compared to baseline methods, SEGA employs a directed acyclic graph with prior expertise for interview modeling, which effectively captures relevant depressive cues in four types of elements.

We then inspect the performance of the GPT family. Despite the strong general capabilities, GPTs' performance still lags behind existing baselines. We hereby have the following observations: 1) GPT-4 generally outperforms GPT-3.5, benefiting from more parameters and pretraining data. 2) Few-shot models significantly exceed zero-shot models, where the additional depressed and control samples serve as anchors to aid judgment. 3) Performance on English DAIC-WOZ is markedly higher than Chinese EATD, indicating the language preference inside LLMs. 4) LLMs are more accurate at identifying the control class, showing the data encountered during LLM pretraining is also imbalanced, limiting discrimination of the depressed class. Overall, the primary reason for the inferior performance is that LLMs cannot explicitly model the specific pragmatic structure of clinical interviews, and they are inherently less adept at integrating multi-modal evidence such as useful MFCCs and eye movements.

Our key findings can be summarized into two aspects. One is that effective structure modeling, rather than simply introducing more elements, is more important. SEGA's structural element graph has shown strength in capturing clinical interviews. The other is that, directly applying LLMs to specialized domains like mental health still yields unsatisfactory performance. We effectively absorb applicable knowledge from LLMs with princiepatied augmentation, which empowers SEGA\({}^{++}\) to make further performance gains.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Element**} & \multicolumn{3}{c}{**DAIC-WOZ**} & \multicolumn{3}{c}{**EATD**} \\  & T & A & V & Q & Depressed & Control & Macro & Depressed & Control & Macro \\ \hline \(\omega\)-GCN & ○ & X & X & X & 78.26 & 89.36 & 83.81 & 66.63 & 78.40 & 72.52 \\ EATD-Fusion & ○ & ○ & X & X & 69.57 & 85.11 & 77.34 & 70.24 & 82.63 & 76.44 \\ MFM-Att & ○ & O & ○ & X & 78.57 & 85.71 & 82.14 & 70.06 & 82.10 & 76.08 \\ HCAG & ○ & ○ & X & O & 76.92 & 86.36 & 81.64 & 71.88 & 76.88 & 74.38 \\ \hline GPT-3.5 (Zero Shot) & ○ & X & X & ○ & 35.29 & 79.25 & 57.27 & 6.05 & 73.12 & 39.58 \\ GPT-3.5 (FewShot) & ○ & X & X & ○ & 52.63 & 82.35 & 67.49 & 31.38 & 74.29 & 52.84 \\ GPT-4 (Zero Shot) & ○ & X & X & ○ & 75.86 & 82.93 & 79.39 & 4.26 & 73.36 & 38.81 \\ GPT-4 (Few Shot) & ○ & X & X & ○ & 78.57 & 85.71 & 82.14 & 13.10 & 72.07 & 42.58 \\ \hline SEGA & ○ & ○ & ○ & 81.48\({}^{\dagger}\) & 88.37 & 84.93\({}^{\dagger}\) & 73.18\({}^{\dagger}\) & 84.42\({}^{\dagger}\) & 78.80\({}^{\dagger}\) \\ SEGA\({}^{++}\) & ○ & ○ & ○ & **84.62\({}^{\dagger}\)** & **90.91\({}^{\dagger}\)** & **87.76\({}^{\ddagger}\)** & **75.31\({}^{\dagger}\)** & **85.91\({}^{\dagger}\)** & **80.61\({}^{\dagger}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Depressed, Control, and Macro F1-scores. The best scores are in bold, and the best baseline scores are underlined. Results with \(\dagger\) are significantly better than baselines (\(p<0.05\)) based on a one-tailed unpaired t-test.

 

Analysis

### Ablation Study

We conduct extensive ablation studies to investigate the contribution of different components and present the results on DAIC-WOZ.

In Figure 4, we present models with variations on the graph structure. We first include variants "Transcript", "Video", and "Audio" representing variants using only the respective single modality, where the results emphasize the need for integrating different elements for interview modeling. We then examine the graph construction. "FC Graph" uses fully connected subgraphs for each round's four elements; "Bi Edge" alters unidirectional edges to bidirectional without modifying the structure; "w/o Proxy" involves removing proxy nodes, allowing auxiliary nodes to directly connect with central nodes. The evident performance drop in these structural variants underscores the effectiveness of SEGA based on human experience.

In Figure 5, we present models with different modalities. For bi-modal models, combining audios and videos as "A+V" leads to performance gains compared to these two single modalities. However, adding audios or videos to transcripts, forming "T+A" or "T+V", results in similar or even decreased performance. This suggests that audios and videos contain relatively limited information, thus potentially complementing each other. In contrast, transcripts contain abundant information, making it challenging for audio or video alone to effectively supplement valuable information. Nevertheless, when all these modalities are combined as "T+V+A", a slight improvement over the single transcript modality can be achieved. Moreover, modeling interview questions (i.e., SEGA) can further enhance the performance.

In Figure 6, we compare variants for empowerment. The absence of contrastive learning ("w/o CL") results in a performance decline. Removing principle-based augmentation ("w/o DA") leads to a performance drop to the level of the original SEGA, highlighting the effectiveness of synthetic data. "Free Gen" leverages LLMs to role-play depressed/control participants and answer questions for augmentation, resulting in a substantial performance decrease, indicating its impracticality. "EDA" Wei and Zou [2019] involves easy data augmentation like deletion, insertion, and swapping, which brings insignificant improvement.

### Case Study

To have a close look, we further select a correctly predicted depressed sample from the development set of DAIC-WOZ, and highlight a question-answer round heavily weighted by the summary node per SEGA's average graph attention scores. The answer transcript and audio's Mel spectrograms are presented in Figure 7. The video is omitted due to the unavailability of original recordings.

Figure 4: Ablation variants on graph structures.

Figure 5: Ablation variants on modalities.

Figure 6: Ablation variants on empowerment.

 

In the transcript, we mark negative emotion words in _blue_, expressing the participant's distress and regret in semantics. Moreover, from the _green_ highlights and Mel spectrograms, we observe slower speech and more pauses, which existing research identifies as typical depressive symptoms Sobin and Sackeim (1997). This demonstrates \(\mathbb{SECA}^{*}\)'s strength in integrating indicative cues across modalities via the structural element graph to make accurate detection.

### Implications on Mental Healthcare

Mental healthcare currently faces challenges such as insufficient specialized professionals, uneven distribution of resources, and stigma surrounding treatment seeking. By proposing SEGA, we absolutely do not intend to replace mental health experts like psychologists and psychiatrists. Rather, our goal is for SEGA to act as a qualified assistant.

For practical implementation, clinical depression assessment can be formulated as a two-step pipeline. The first step is conducting automated interviews, for which existing methods like Wizard-of-Oz or fully automated agents are viable options. The second step is the automatic detection by analyzing the interview recordings to aid in judging the participant's depressive state, like SFGA. By combining these two steps, we can alleviate the burden on professionals, compensate for imbalanced medical resources, expedite widespread depression screening, and ultimately promote public health and well-being for people worldwide.

## 6 Conclusion

In this paper, we propose a novel structural element graph SEGA for depression detection in clinical interviews. SEGA comprehensively captures all key elements - interview questions, answer transcripts, audios, and videos - within a simple yet effective directed acyclic graph. To further address data scarcity, we design the empowered SEGA++ via principle-guided augmentation with LLMs and graph contrastive learning. Experiments on two real-world datasets demonstrate the state-of-the-art performance of our method, surpassing the latest baseline methods and zero/few-shot GPT-3.5/4.

## Acknowledgments

This work was supported by the National Science Foundation for Distinguished Young Scholars (No. 62125604) and the NSFC Key Project (No. 61936010). This work was also supported by Tsinghua Precision Medicine Foundation and Tsinghua University - Beijing Tsingshang Architectural Decoration Engineering Co., Ltd. Joint Institute for Smart Scene Innovation Design.

## Limitations

We discuss the limitations of our work as follows.

**1) Task Setting.** In this work, we focus on depression detection in clinical interviews, a serious "laboratory setting" typically conducted by mental health experts like psychologists and psychiatrists, representing a standard approach in depression screening. Contrasting this is the "wild setting", which refers to informal early screenings for daily contents, such as analyzing social media posts (e.g., on Reddit). Due to significant differences between these settings in data structures, modalities, and judgment criteria, intuitively, our method designed for the "laboratory setting" might not effectively transfer to the "wild setting".

**2) Modality Specificity.** Due to our reliance on publicly available datasets, our methodology is limited to three modalities: text (interview questions and participant responses), audio (MFCC, formants), and video (facial actions, eye gazes, etc.). We do not include other medically relevant information such as electroencephalograms (EEGs), near-infrared signals, or magnetic resonance imaging (MRI) signals.

**3) Corpus Size.** Considering the challenges in data collection and privacy considerations, the datasets available for depression detection in clinical interviews usually have limited samples. In experiments, we report the average results across multiple runs and conduct significance testing to ensure that any performance improvements are statistically valid.

**4) Data Release.** Due to licensing constraints of the DAIC-WoZ and EATD datasets, we are unable to directly release raw interview recordings. Instead, under appropriate licensing or registration, we can provide pre-processed vector features to those who have already applied to obtain the raw data. Access to the original materials is possible for anyone who completes an application form and receives approval from the dataset authors.

## Ethical Considerations

We here elaborate on the potential ethical issues 



**1) Data Privacy and Consent.** We apply and use publicly available datasets, DAIC-WoZ and EATD, for depression detection in clinical interviews. According to the original dataset papers, both datasets have received approval from Institutional Review Boards. All participants are informed that their interviews are for academic research. All personal details like names, ages, and professions are either removed or anonymized, eliminating any risk of personal information exposure. The DAIC-WoZ dataset, involving a video modality, does not provide original videos but de-identified vector features of facial actions and eye gaze, making it impossible to reconstruct the participants' appearances. Therefore, the information of participants is comprehensively and rigorously protected, with no privacy breaches.

**2) Participant Demographics.** The DAIC-WoZ dataset, collected by researchers from the Institute for Creative Technologies at the University of Southern California, encompasses participants including the U.S. armed forces veterans and the general public from the Greater Los Angeles metropolitan area. The EATD dataset, collected by researchers from the School of Software Engineering at Tongji University, primarily includes teachers and students from universities in Shanghai. The focus on specific groups in both datasets is a result of objective conditions and research goals, rather than any intention of bias or unfairness towards race, nationality, age, or gender.

**3) Role of AI in Diagnosis.** Our method aims to serve as an intelligent assistant for mental health experts such as psychologists and psychiatrists, not to replace them. Directly using our method to calculate depression coefficients may cause or perpetuate algorithmic bias, potentially leading to inaccurate diagnoses. Therefore, the model predictions should only be used as a reference, while the final diagnosis must be cautiously determined by professionals. Our method is based solely on completed interview recordings for preliminary screening and does not offer medical advice or intervention.

**4) Dataset Access and Use.** For the DAIC-WOZ dataset, we meticulously review the DAIC-WOZ End-User License Agreement and submit the necessary official application forms, securing consent from the original authors. For the EATD dataset, we download the data from the author's official GitHub repository. We strictly use the two datasets exclusively for the research purposes of this work. No unauthorized dissemination or sharing of the data is conducted.

**5) LLM Data Augmentation**. In the use of large language models for data augmentation, we apply principle-guided rephrasing instead of free generation to control the model behavior. This approach ensures that the model performs semantic-preserving rewrites and does not introduce imaginative, additional information. We rigorously verify all synthetic data to ensure no introduction of any unsafe or harmful content.

## References

* Hanai et al. (2018) Tuka Al Hanai, Mohammad Ghassemi, and James Glass. 2018. Detecting Depression with Audio/Text Sequence Modeling of Interviews. In _INTERSPEECH_, pages 1716-1720.
* Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Baltrusaitis et al. (2018) Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. _IEEE transactions on pattern analysis and machine intelligence_, 41(2):423-443.
* Baltrusaitis et al. (2016) Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In _2016 IEEE winter conference on applications of computer vision (WACV)_, pages 1-10. IEEE.
* Bisercic et al. (2023) Aleksa Bisercic, Mladen Nikolic, Mihaela van der Schaar, Boris Delibasic, Pietro Lio, and Andrija Petrovic. 2023. Interpretable medical diagnostics with structured data extraction by large language models. _arXiv preprint arXiv:2306.05052_.
* Burdisso et al. (2023) Sergio Burdisso, Esau Villatoro-Tello, Srikanth Madikeri, and Petr Motlicek. 2023. Node-weighted graph convolutional network for depression detection in transcribed clinical interviews. _arXiv preprint arXiv:2307.00920_.
* Chen et al. (2023) Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023. Llm-empowered chatbots for psychiatric and patient simulation: Application and evaluation. _arXiv preprint arXiv:2305.13614_.
* Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. In _Findings of EMNLP_, pages 657-668.



collaborative voice analysis repository for speech technologies. In _ICASSP_, pages 960-964.
* DeVault et al. (2014) David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. Simsense kiosk: A virtual human interviewer for healthcare decision support. In _Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_, pages 1061-1068.
* Eyben et al. (2016) Florian Eyben, Klaus R. Scherer, Bjorn W. Schuller, Johan Sundberg, Elisabeth Andre, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukaa, Shrikanth S. Narayanan, and Khiet P. Truong. 2016. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. _IEEE Trans. Affect. Comput._, pages 190-202.
* Eyben et al. (2010) Florian Eyben, Martin Wollmer, and Bjorn W. Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In _ACM on Multimedia_, pages 1459-1462.
* Fan et al. (2022) Cunhang Fan, Zhao Lv, Shengbing Pei, and Mingyue Niu. 2022. Csenet: Complex squeeze-and-excitation network for speech depression level prediction. In _ICASSP_, pages 546-550.
* Fang et al. (2022) Ming Fang, Siyu Peng, Yujia Liang, Chih-Cheng Hung, and Shuhua Liu. 2022. A Multimodal Fusion Model with Multi-Level Attention Mechanism for Depression Detection.
* Flores et al. (2022) Ricardo Flores, ML Tlachac, Ermal Toto, and Elke Rundensteiner. 2022. Transfer learning for depression screening from follow-up clinical interview questions. In _Deep Learning Applications, Volume 4_, pages 53-78. Springer.
* Gratch et al. (2014) Jonathan Gratch, Ron Artstein, Gale M. Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, David R. Traum, Skip Rizzo, and Louis-Philippe Morency. 2014. The distress analysis interview corpus of human and computer interviews. In _LREC_, pages 3123-3128.
* Guo et al. (2022) Yanrong Guo, Chenyang Zhu, Shijie Hao, and Richang Hong. 2022. A topic-attentive transformer-based model for multimodal depression detection.
* Guohou et al. (2020) Shan Guohou, Zhou Lina, and Zhang Dongsong. 2020. What reveals about depression level? The role of multimodal features at the level of interview questions. _Information & Management_, page 103349.
* Hazarika et al. (2022) Devamanyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, and Soujanya Poria. 2022. Analyzing modality robustness in multimodal sentiment analysis. _arXiv preprint arXiv:2205.15465_.
* He et al. (2022) Lang He, Mingyue Niu, Pravag Tiwari, Pekka Marttianen, Rui Su, Jiewei Jiang, Chenguang Guo, Hongyu Wang, Songtao Ding, Zhongmin Wang, Xiaoying Pan, and Wei Dang. 2022. Deep learning for depression recognition with audiovisual cues: A review. _Information Fusion_, pages 56-86.
* Huang et al. (2022) Zhaocheng Huang, Julien Epps, and Dale Joachim. 2022. Investigation of speech landmark patterns for depression detection. _IEEE Trans. Affect. Comput._, pages 666-679.
* Jo et al. (2023) Eunkyoung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023. Understanding the benefits and challenges of deploying conversational ai leveraging large language models for public health intervention. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, pages 1-16.
* Kelley (1983) John F Kelley. 1983. An empirical methodology for writing user-friendly natural language computer applications. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_, pages 193-196.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR_.
* Kroencke et al. (2009) Kurt Kroencke, Tara W Strine, Robert L Spitzer, Janet BW Williams, Joyce T Berry, and Ali H Mokdad. 2009. The phq-8 as a measure of current depression in the general population. _Journal of affective disorders_, 114(1-3):163-173.
* Liyanage et al. (2023) Chandreen Liyanage, Muskan Garg, Vijay Mago, and Sunghwan Sohn. 2023. Augmenting reddit posts to determine wellness dimensions impacting mental health. _arXiv preprint arXiv:2306.04059_.
* Ma et al. (2016) Xingchen Ma, Hongyu Yang, Qiang Chen, Di Huang, and Yunhong Wang. 2016. DepAudioNet: An efficient deep model for audio based depression classification. In _AVEC_, pages 35-42.
* Mallol-Ragolda et al. (2019) Adria Mallol-Ragolda, Zipping Zhao, Lukas Stappen, Nicholas Cummins, and Bjorn W. Schuller. 2019. A hierarchical attention network-based approach for depression detection from transcribed clinical interviews. In _INTERSPEECH_, pages 221-225.
* Niu et al. (2021) Meng Niu, Kai Chen, Qingcai Chen, and Lufeng Yang. 2021. HCAG: A Hierarchical Context-Aware Graph Attention Model for Depression Detection. In _ICASSP_, pages 4235-4239.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. _CoRR_ 

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543. Cited by: SS6.
* S. Prabhu, H. Mittal, R. Varagani, W. Jha, and B. Singh (2022)Harnessing emotions for depression detection. Pattern Analysis and Applications, pp. 537-547. External Links: Document, ISSN 1948-0209, Link Cited by: SS6.
* M. Rohanian, J. Hough, M. Purver, et al. (2019)Detecting depression with word-level multimodal fusion. In Interspeech, pp. 1443-1447. Cited by: SS6.
* S. Sardari, B. Nakisa, M. N. Bastgoo, and P. Eklund (2022)Audio based depression detection using convolutional autoencoder. Expert Systems with Applications, pp. 116076. Cited by: SS6.
* S. Sarker, L. Qian, and X. Dong (2023)Medical data augmentation via chatgpt: a case study on medication identification and medication event classification. arXiv preprint arXiv:2306.07297. Cited by: SS6.
* N. Seneviratne and C. Y. Espy-Wilson (2022)Multimodal depression classification using articulatory coordination features and hierarchical attention based text embeddings. In ICASSP, pp. 6252-6256. Cited by: SS6.
* W. Shen, S. Wu, Y. Yang, and Xiaojun Quan (2021)Directed acyclic graph network for conversational emotion recognition. In ACL-IJCNLP, pp. 1551-1560. Cited by: SS6.
* Y. Shen, H. Yang, and L. Lin (2022)Automatic depression detection: an emotional audio-textual corpus and a gru/bilstm-based model. In ICASSP, pp. 6247-6251. Cited by: SS6.
* C. Sobin and H. A. Sackeim (1997)Psychomotor symptoms of depression. American Journal of Psychiatry154 (1), pp. 4-17. Cited by: SS6.
* G. Sun, S. Zhao, B. Zou, and Y. An (2022)Speech-based Depression Detection Using Unsupervised Autoencoder. In ICSIP, pp. 35-38. Cited by: SS6.
* T. Taguchi, H. Tachikawa, K. Nemoto, M. Suzuki, T. Nagano, R. Tachibana, M. Nishimura, and T. Arai (2018)Major depressive disorder discrimination using vocal acoustic features. Journal of affective disorders, pp. 214-220. Cited by: SS6.
* R. Thoppilan, D. De Freitas, J. Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. (2022)Lamda: language models for dialog applications. arXiv preprint arXiv:2201.08239. Cited by: SS6.
* H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, Timothete Lacroix, Baptiste Rozieere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. (2023)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: SS6.
* M. F. Valstar, J. Gratch, B. W. Schuller, F. Ringeval, D. Lalanne, M. Torres, S. Scherer, G. Stratout, R. Cowie, and M. Pantic (2016)AVEC 2016: depression, mood, and emotion recognition workshop and challenge. In AVEC@MM, pp. 3-10. Cited by: SS6.
* P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lo, and Y. Bengio (2018)Graph attention networks. In ICLR, Cited by: SS6.
* Esait Villatoro-Tello, Gabriela Ramirez-de-la-Rosa, Daniel Gatica-Perez, Mathew Magimaal-Doss, and Hector Jimenez-Salazar (2021)Approximating the mental lexicon from clinical interviews as a support tool for depression detection. In ICAI, pp. 557-566. Cited by: SS6.
* J. Wang, Z. Yao, A. Mitra, S. Osebe, Z. Yang, and H. Yu (2023)Umass_bionlp at media-chat 2023: can llms generate high-quality synthetic note-oriented doctor-patient conversations?. arXiv preprint arXiv:2306.16931. Cited by: SS6.
* J. Wei and K. Zou (2019)EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6382-6388. Cited by: SS6.
* W. Wu, M. Wu, and K. Yu (2022)Climate and weather: inspecting depression detection via emotion recognition. In ICASSP, pp. 6262-6266. Cited by: SS6.
* D. Xezonaki, G. Paraskevopoulos, A. Potquianos, and Shrikanth Narayanan (2020)Affective conditioning on hierarchical attention networks applied to depression detection from transcribed clinical interviews.. In INTERSPEECH, pp. 4556-4560. Cited by: SS6.
* B. Yang, A. Raza, Y. Zou, and T. Zhang (2023)Customizing general-purpose foundation models for medical report generation. arXiv preprint arXiv:2306.05642. Cited by: SS6.
* J. Yoon, C. Kang, S. Kim, and J. Han (2022)D-vlog: multimodal vlog dataset for depression detection. AAAI, pp. 12226-12234. Cited by: SS6.
* P. Zhang, M. Wu, H. Dinkel, and K. Yu (2021)DEPA: self-supervised audio embedding for depression detection. In ACM Multimedia Conference, pp. 135-143. Cited by: SS6.

 

Wenbo Zheng, Lan Yan, Chao Gou, and Fei-Yue Wang. 2020. Graph Attention Model Embedded With Multi-Modal Knowledge For Depression Detection. In _ICME_, pages 1-6.
* Zou et al. (2022) Bochao Zou, Jiali Han, Yingxue Wang, Rui Liu, Shenghui Zhao, Lei Feng, Xiangwen Lyu, and Huimin Ma. 2022. Semi-structural interview-based chinese multimodal depression corpus towards automatic preliminary screening of depressive disorders. _IEEE Transactions on Affective Computing_.
* Zung (1965) William WK Zung. 1965. A self-rating depression scale. _Archives of general psychiatry_, 12(1):63-70.

## Appendix A Details of Dataset Pre-Processssing

We here discuss the raw information provided by the dataset and the detailed preprocessing methods.

### Daic-woz

The Distress Analysis Interview Corpus - Wizard Of Oz (DAIC-WOZ) provides multi-modal data, including transcripts, audios, and videos of clinical interviews.

* Transcript: DAIC-WOZ provides complete textual transcripts of each interview containing interview questions and participant answers. We process transcripts with GloVe.840B.300d word embeddings.
* Audio: DAIC-WOZ provides the raw audio of each interview along with acoustic features extracted using COVAREP, including F0, VUV, NAQ, QOO, H1H2, PSP, MDQ, peakSlope, Rd, Rd\({}_{\text{conf}}\), MCEP\({}_{\text{0}}\)-24, HMPDM\({}_{\text{0}}\)-24, HMPDD\({}_{\text{0}}\)-12, and the first 5 formants.
* Video: DAIC-WOZ provides the visual features extracted from each interview using OpenFace, instead of the original video recordings. These include 2D/3D facial landmark points, facial action units, eye gazes, Felzenswalb's HoG features, and head poses.

### Eatd

The Emotional Audio-Textual Depression Corpus (EATD) provides multi-modal data, including transcripts and audio, without video.

* Transcript: EATD provides only the transcribed text of participant answers. We supplemented these with 26 additional interview questions based on the content of the answers. We use pre-trained Chinese BERT to obtain word embeddings for the text.
* Audio: EATD provides audio recordings of each participant's answers. Since no specific audio processing tool is specified, we extract the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) acoustic features using OpenSmile, which contains features related to frequency-related parameters like formant, energy-related parameters like loudness, and spectral parameters like MFCCs.

## Appendix B Details of Hyperparameter Ranges

We present the hyperparameter ranges in Table 4. We select all hyperparameters via manual tuning.

## Appendix C Details of Baseline Implementation

We here illustrate the implementation of baseline methods.

* **\(\omega\)-GCN** is a uni-modal model that leverages answer transcripts for depression detection. It constructs a graph between words and interview documents, and uses TF-IDF, PMI, and PageRank to obtain the edge weights for graph convolution.
* **EATD-Fusion** is a bi-modal model that leverages answer transcripts and audios for depression detection. It uses a gating mechanism for integrating predictions from two modalities.
* **MFM-Att** is a tri-modal model that leverages answer transcripts, audios, and videos for depression detection. It first learns features of single modalities, then uses an attention fusion network to obtain fused multi-modal representation. MFM-Att for EATD only uses transcripts and audios since there is no video modality in EATD.
* **HCAG** is a model that captures the interaction between interview questions and participant answers for depression detection. It concatenates the question and answer in each round as a vertex, and constructs edges between adjacent QA rounds in a

\begin{table}
\begin{tabular}{|l|c|c|} \hline Hyperparameter & Range & Selected \\ \hline GAT \& Bi+LSTM hidden state \(d_{k}\) & (64, 128, 256, 512) & 256 \\ \hline number of GAT layers \(L\) & (1,2,3,4) & 2 \\ \hline number of attention heads \(K\) & (1, 2, 4, 8, 16) & 8 \\ \hline number of demonstrations \(N\) & (1, 2, 3) & 2 \\ \hline temperature in contrastive loss \(\tau\) & (0.01, 0.05, 0.1, 0.5, 1.0) & 0.05 \\ \hline batch size & (4, 8, 16, 32) & 8 \\ \hline learning rate & (1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-5) & 1e-4/ 2e-5 \\ \hline \end{tabular}
\end{table}
Table 4: Ranges of hyperparameters.

 

QA graph. GAT is used to update vertex features and make predictions.

\({}^{\bullet}\)**GPT-3.5** and **GPT-4** are included as the competitors in experiments. Given the length of the clinical interview, for GPT-3.5, we employ the gpt-3.5-turbo-16k-0613 API (the \(4k\) context window of the default gpt-3.5-turbo-0613 is insufficient in length). As for GPT-4, we utilize the standard gpt-4-0613 API with an \(8k\) context window. The zero-shot and few-shot prompts used for evaluating the GPT family in depression detection are shown in Table 5.

## Appendix D Use of AI Assistants

We use ChatGPT to polish some of the content.

\begin{table}
\begin{tabular}{l} \hline
**Zero-Shot Prompt for GPT-3.5/GPT-4** \\ \hline Below is a transcript of an interview between an interviewer and a participant. Based on the content of the interview, determine whether the participant is depressed or not. Only answer ’Yes’ or ‘No’ without any explanations. \\ \end{tabular}
\end{table}
Table 5: The zero-shot and few-shot prompts for evaluating GPT-3.5 and GPT-4 for depression detection in clinical interviews.