Introduction
Depression is a major cause of disability globally 2 2 2 WHO (2023, March 31). Depressive disorder. . Its personal and societal impact makes optimising mental health care practices crucial. Existing diagnostic systems of depression are heavily dependent on clinicians’ proficiency in attending to patients’ verbal and non-verbal cues, but achieving this expertise requires extensive training Atzil-Slonim et al. ( 2024 ) . The growing demand for mental health care services, coupled with a shortage of qualified providers, means that many individuals with depression go undiagnosed and untreated Kazdin ( 2021 ) .
Detection of depression severity is crucial, as it can prevent deterioration and enable adequate and effective treatment. Current diagnostic systems have faced criticism for failing to capture the significant heterogeneity and variability of symptoms between individuals Bickman ( 2020 ) . Understanding how different symptoms vary between individuals could lead to more personalised and effective interventions.

A common way to assess depression or track depression throughout a treatment program is based on self-reported questionnaires like PHQ-8 (Kroenke et al., 2009 ) or BDI-II (Beck et al., 1996 ) . These questionnaires contain questions regarding depression symptoms, and patients need to score each question based on how frequently they encounter these symptoms. The sum of the scores from each question gives the patient’s depression severity score. However, such methods burden patients, especially when they are required to complete the questionnaires repeatedly as part of ongoing treatment monitoring Kazdin ( 2008 ) . Thus, to improve the speed and convenience of diagnosing and monitoring depression, efforts are devoted to building depression severity prediction methods through machine learning. Initial works on automatic depression detection focused on using social media data (De Choudhury et al., 2013 ) for binary depression classification. However, due to the lack of predicting depression severity, such a method is unable to prioritise people with higher levels of depression. So, the task was reformulated as classification among four depression levels (Naseem et al., 2022 ) . However, models trained on social media data are unsuitable for clinical settings (Wongkoblap et al., 2017 ) .

Also, social media data lacks multimodal cues, which are often used by therapists to infer the depression severity of a patient. Depression has identifiable verbal and nonverbal characteristics, such as facial expression (Slonim et al., 2024 ) , prosodic information (Cummins et al., 2015 ; Cohn et al., 2009 ; Scherer et al., 2014 ; Paz et al., 2024 ) and semantic features (Chim et al., 2024 ) . To utilise these cues for depression assessment, the AVEC challenges (Ringeval et al., 2017 , 2019 ) released semi-clinical datasets, DAIC-WOZ (Gratch et al., 2014 ) and E-DAIC (DeVault et al., 2014 ) , containing recorded interviews and self-reported PHQ-8 questionnaires. These questionnaires help detect symptoms and give more fine-grained depression severity levels.

Various works have tried to utilise the text, audio and video modality from the AVEC datasets and improve the fusion between them (Rodrigues Makiuchi et al., 2019 ; Sun et al., 2021 , 2022 ; Zhao and Wang, 2022 ; Ray et al., 2019 ; Yuan et al., 2024 ) to predict the depression severity score obtained from PHQ-8 questionnaires. However, these approaches only use one fusion module to fuse text, audio, and video information to predict the depression severity score (the sum of the scores for each question in the questionnaire). This design choice results in a failure to model the variable contributions of each modality depending on the questions in the questionnaire, leading to sub-optimal fusion.
For example, a question on being fidgety may require more attention to audio-visual modalities. On the other hand, text transcripts may contribute more significantly to a question regarding a person’s appetite. While Van Steijn et al. ( 2022 ) also perform question-wise modeling, they mainly use text features concatenated with a few hand-crafted audio features from Automatic Speech Recognition (ASR), like sentiment, speech rate, and repetition rate. So, they do not utilise the audio and video modalities effectively.
Another issue is that the current multimodal methods frame the depression severity score prediction as a regression task, resulting in sub-optimal training. Humans score each question in the questionnaire as 0 , 1 , 2 , or 3 , depending on the frequency of the symptoms experienced. Thus, the depression severity prediction task should be framed as a question-wise ordinal classification task.

Contributions: We propose a novel Question-wise Modality Fusion ( QuestMF ) framework for depression severity prediction. This framework contains question-wise fusion modules to ensure different contributions from modalities based on the question. In addition, we propose a novel Imbalanced Ordinal Log-Loss ( ImbOLL ) function to train our models for ordinal classification. We find that our method matches the performance of the current state-of-the-art methods on the E-DAIC dataset and enhances interpretability for clinicians by identifying an individual’s specific symptoms. We also analyse the importance of each modality for each question and find that a fusion of text and video modalities performs best in most questions.