RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Single Modality Methods
Earlier works in depression severity prediction focused on the text modality like the use of linguistic feature extraction based on LIWC (De Choudhury et al., 2013 ) , Bag-of-word models (Nadeem, 2016 ) , word2vec embeddings (Husseini Orabi et al., 2018 ) or using emotion features (Aragón et al., 2019 ) . With pre-trained language models like BERT (Devlin et al., 2019 ) improving performances on text-based tasks, depression severity prediction works also utilised them (Rodrigues Makiuchi et al., 2019 ; Fan et al., 2019 ; Sun et al., 2022 ) . Van Steijn et al. ( 2022 ) introduces a framework to predict scores of each question of a PHQ-8 questionnaire to add interpretability, which is missing in the prior works.
These methods also ignore the multi-turn dialogue present in therapy sessions. Thus, Milintsevich et al. ( 2023 ) introduces a turn-based method that encodes each dialogue turn using a sentence transformer (Reimers and Gurevych, 2019 ) . We use a similar turn-based model to encode the multi-turn dialogue data in each modality. We, however, use multihead self-attention instead of additive attention to improve the model.

With the advent of LLMs in recent times, Sadeghi et al. ( 2023 ) uses GPT-3.5-Turbo 3 3 3 https://platform.openai.com/docs/models/gpt-3-5-turbo with encoder models for depression severity prediction. However, its performance falls short of the state-of-the-art models. Moreover, data privacy requirements do not allow data to be sent to proprietary LLMs. These issues motivate us to only explore encoder models.

The AVEC challenges introduced the potential to use audio features for depression detection. This resulted in works utilising low-level audio features (Eyben et al., 2016 ) extracted by OpenSmile (Eyben et al., 2010 ) . Fan et al. ( 2019 ) uses CNNs over the low-level features, while Yin et al. ( 2019 ) and Sun et al. ( 2022 ) use LSTMs to capture the temporal relation among them.
However, LSTMs are sub-optimal at processing long sequences. Thus, Sun et al. ( 2021 ) uses transformers to process long sequences of audio features. There are also other methods that do not use OpenSmile features but rather use spectrograms (Rodrigues Makiuchi et al., 2019 ) or audio recordings directly (Han et al., 2024 ; Chen et al., 2023 ) . However, they are computationally expensive, making them difficult to use for multimodal fusion. Here, we use LSTM over low-level features. We break the session into turns and aggregate features at the turn level to make shorter sequences that can be processed using LSTMs.
Multimodal Fusion Methods
Multimodal methods focus on improving the fusion of the modalities for depression severity prediction. Van Steijn et al. ( 2022 ) follows an early fusion method where they concatenate extracted text features with hand-crafted audio features. Other methods generally follow a late fusion strategy. Initial late fusion works used simple concatenation (Rodrigues Makiuchi et al., 2019 ) or weighted concatenation (Sun et al., 2021 ) for the fusion of text, audio, and video encodings. Ray et al. ( 2019 ) uses attention modules to improve fusion. Some works also use hierarchical fusion at frame level (Yin et al., 2019 ) , word level (Rohanian et al., 2019 ) , or topic level (Guo et al., 2022 ) to capture the interaction between the modalities at fine-grained levels.
Fusion at the frame level, however, can cut words, and fusion at the word level lacks sufficient context, thus being too fine-grained. Meanwhile, fusion at the topic level is too course-grained. Thus, we use turn-level fusion in our framework.
For further improvement in fusion, MMFF (Yuan et al., 2024 ) exploits the high-order interaction between different modalities. However, it is computationally expensive. CubeMLP (Sun et al., 2022 ) uses MLPs to mix information among modalities to enhance the computational efficiency of fusion. However, it results in lower performance. Zhao and Wang ( 2022 ) uses Self-Attention GAN to augment training data to reduce the issue of data shortage. They use a cross-attention based fusion strategy (Tsai et al., 2019 ) . We use the same cross-attention-based fusion. However, these late fusion works use a single fusion module, thus ignoring the variable contribution of each modality according to the question. We use question-wise fusion modules to mitigate this issue.
Ordinal Classification Methods
Ordinal classification has been explored in tasks like sentiment analysis in Twitter (Nakov et al., 2016 ; Rosenthal et al., 2017 ) . While depression severity score prediction is also an ordinal classification task, prior multimodal methods (Rodrigues Makiuchi et al., 2019 ; Milintsevich et al., 2023 ; Ray et al., 2019 ; Yuan et al., 2024 ; Zhao and Wang, 2022 ) treat it as a regression task. As a result, ordinal classification has been rarely explored in depression severity score prediction (Van Steijn et al., 2022 ) . Ordinal classification methods include ordinal binary classification methods (Frank and Hall, 2001 ; Allwein et al., 2000 ) , threshold methods (Lin and Li, 2006 ; Verwaeren et al., 2012 ; Cao et al., 2020 ) and loss-sensitive classification methods (Rennie and Srebro, 2005 ; Diaz and Marathe, 2019 ; Bertinetto et al., 2020 ; Cao et al., 2020 ; Castagnos et al., 2022 ) . However, these methods are not suitable for imbalanced datasets. Since very few patients feel a specific symptom very frequently, the distribution of question-wise scores (labels) in depression severity prediction datasets is imbalanced. While Van Steijn et al. ( 2022 ) explores a weighted ordinal classification setup, their method can only be used with Kernel Extreme Learning Machine (KELM). Also, they obtain their best results with a regression framework and the results with weighted ordinal classification are far below state-of-the-art performance. Thus, we propose ImbOLL , a modified version of the OLL function (Castagnos et al., 2022 ) to handle the data imbalance, which can be used with any classification method.

In summary, our work is the first to perform question-wise modality fusion and present a loss function for imbalanced ordinal classification of depression severity prediction task that can be used with any classification method. Moreover, we are the first to analyse the contribution of each modality towards the score of each question, thus improving interpretability. We are also the first to show the robustness of our method through multiple seed runs.
Dataset
Multimodal clinical data collection for depression detection is difficult due to privacy issues, thus resulting in small datasets. In this work, we use the E-DAIC dataset from the AVEC 2019 DDS (Ringeval et al., 2019 ) challenge. While the E-DAIC dataset is also small, to the best of our knowledge, it is the only dataset with more than 200 data points available for research on depression severity prediction. We do not use the DAIC-WoZ dataset since it is a subset of the E-DAIC dataset. Other available datasets are either smaller (Zou et al., 2023 ) or are not clinically grounded with self-reported questionnaires (Yoon et al., 2022 ) .
The E-DAIC dataset collected recorded interview sessions with a virtual agent and filled out self-reported PHQ-8 questionnaires for each participant. All the interviews were conducted in English. The dataset provides text transcripts of participant dialogues, recorded audio clips, and visual features like ResNet, VGG, and OpenFace for each interview session. The recorded videos have not been released due to privacy concerns. The dataset contains 275 sessions. The training set includes 163 sessions, and the validation and test sets each contain 56 sessions. However, one session from the validation and one from the test set have incomplete video feature files. Thus, we do not use them in the evaluation. The dataset also provides the PHQ-8 scores of all participants. The PHQ-8 score ranges from 0 to 24 . While the training and validation sets contain scores for each of the eight PHQ-8 questions ( 0 to 3 ), the test split only contains the total PHQ-8 scores. More details on the PHQ-8 questionnaire are provided in Appendix A .
QuestMF Framework
In this section, we present our novel Question-wise Modality Fusion ( QuestMF ) framework. In this framework, we use n different single modality encoders for each modality and n different modality fusion models corresponding to n questions (thus question-wise modality fusion) in a questionnaire. Each of the n fused models outputs the score for its corresponding question ( 0 , 1 , 2 , or 3 ), ensuring different contributions from each modality depending on the question, which was lacking in previous works. These question-wise scores are added to get the total questionnaire score ( 0 to 3n ). Figure 1 shows the proposed framework. Moreover, the QuestMF framework improves interpretability by predicting the score of each question. This allows clinicians to understand the symptoms affecting a patient and create interventions accordingly. This framework will also enable clinicians to track the progression of each symptom through the question-wise scores throughout the multiple therapy sessions during the treatment.

Moreover, current multimodal methods (Rodrigues Makiuchi et al., 2019 ; Milintsevich et al., 2023 ; Ray et al., 2019 ; Yuan et al., 2024 ; Zhao and Wang, 2022 ) train the multimodal methods to predict the total depression severity score as a regression task treating it as a continuous variable. However, the question-wise scores belong to 4 classes: 0,1,2,3 depending on the frequency of the symptoms experienced, thus making it an ordinal classification task. While Van Steijn et al. ( 2022 ) tries this, they predominantly use text features and concatenate them with a few hand-crafted audio features. Also, they obtain their best result with regression setup, and their ordinal setup results are significantly worse than the state-of-the-art models. Treating question-wise scores as continuous variables also reduces interpretability as fractional scores like 1.5 can mean experiencing a symptom at the frequency of either class 1 or class 2 . Thus, framing the question-wise score prediction as an ordinal classification task ensures improved interpretability as we get the predicted probabilities of the 4 classes: 0,1,2,3 and choose the class with the highest probability. This is also more similar to how humans fill out the questionnaires.

Next, we discuss the single modality encoders used in the framework in Section 4.1 and the fusion methods used to combine the single modality encodings in Section 4.2 . Finally, we introduce the novel ImbOLL function used to train the models for ordinal classification in Section 4.3 .
Single Modality Encoder Models
All the single modality encoder models follow a turn-based method similar to Milintsevich et al. ( 2023 ) to better encode the interviews containing multi-turn dialogues. We start by encoding the dialogue turns and then use these turn encodings to generate an encoding for the whole session. The overall structure is uniform across all modalities, as shown in Figure 2 . Now, we describe each single modality encoder in detail.
Text Encoder Model
For text, we use the textual transcripts from interview sessions. We break the transcripts into dialogue turns and only consider the dialogue turns from participants. We encode the turns using a pre-trained sentence transformer (Reimers and Gurevych, 2019 ) . Now, we get turn encoding, X_{i}\in\mathbb{R}^{tokens_{i}\times D_{model}} , where i\in(1,2,..,m) for m turns, tokens_{i} is the number of tokens in turn i and D_{model} is the model output dimension. Next, we use mean pooling over the tokens and normalise them to get X_{mean,i}\in\mathbb{R}^{D_{model}} . Then, we pass these turn encodings through a Bidirectional LSTM layer. The Bidirectional LSTM layer ensures that the turns can interact among themselves. This gives us X_{lstm,i}\in\mathbb{R}^{2\cdot D_{lstm}} . Next, we use a multihead attention layer to determine the importance of each turn and get an updated encoding X_{att,i}\in\mathbb{R}^{2\cdot D_{lstm}} . Following this, we concatenate and flatten the turn encodings obtained after the multihead attention layer to get a session-level encoding representation of X_{session}\in\mathbb{R}^{(2m\cdot D_{lstm})} . Finally, we pass this session-level encoding through a multilayer perceptron (MLP) to get the score probabilities.
Audio Encoder Model
For audio, we use the low-level features extracted by OpenSmile. Details on the low-level features extracted using OpenSmile are provided in Appendix C . These features are extracted at every 0.01 seconds. Like the text model, we process the information at the turn level. For each dialogue turn i , the dataset contains a starting time t_{start,i} and an ending time t_{end,i} . We get the features extracted from time t_{start,i} to t_{end,i} and apply mean pooling to get the aggregated features in a turn, X_{mean,i}\in\mathbb{R}^{D_{Features}} . Here, D_{Features} is the number of the features extracted by OpenSmile. After this, we pass them through a Bi-LSTM layer and two attention layers to get updated turn encodings. These are concatenated, flattened, and passed through an MLP to get the score probabilities.
Video Encoder Model
For the video encoder model, we use ResNet (He et al., 2016 ) features. Similar to text and audio, we aggregate information at turn level. For this, we get the ResNet features for the frames in a dialogue turn, i.e., frames in t_{start,i}*sr:t_{end,i}*sr , where t_{start,i} is the starting time and t_{end,i} is the ending time of the dialogue turn and sr is the frames rate at which the video is recorded. We pass these ResNet features in a dialogue turn through a mean pooling layer and normalise them to get turn-level encoding X_{mean,i}\in\mathbb{R}^{D_{ResNet}} where D_{ResNet} is the output dimension of the ResNet model. After this, we follow the same architecture as the audio encoder to get the score probabilities.
Modality Fused Models
For modality fusion, we use cross-attention based methods introduced by Tsai et al. ( 2019 ) . The cross-attention layers are sometimes accompanied by X \rightarrow Y. This denotes that the encoding of the Y modality is used as the query, and the encoding of the X modality is used as the key and value in the cross-attention layer. Next, we describe the modality fused models in detail.
Two-Modality Fused Models
Figure 3 shows our two-modality fused models. We use the output from the multihead attention layers of the trained single modality encoders as the modality encoding. We use multihead cross-attention layers over these encodings to exchange information among the modalities. Considering modality encodings M1 and M2 , we use two cross-attention layers M1\rightarrow M2 and M2\rightarrow M1 for interaction among the modalities. This is followed by a multihead self-attention layer for each cross-attention layer. Finally, we concatenate the encodings from the self-attention layers to get a fused encoding. We flatten this fused encoding and pass it through an MLP to get the score probabilities.
Three-Modality Fused Models
Figure 4 shows our three-modality fused model. We use the output from the multihead attention layers of the trained single modality encoders as the modality encoding. Then, we use multihead cross-attention layers to pass information among the modalities. In this case, we have six combinations of query and (key, value) pairs.
Now, we accumulate the encoding for each modality with information from the other two modalities. We perform this by concatenating the outputs from two cross-attention layers using the same modality as the query. For example, the audio encoding with information from text and video modalities will concatenate encodings obtained from cross-attention layers T \rightarrow A and V \rightarrow A.
We pass these encodings for each modality with the information from other modalities through a multihead self-attention layer. Next, we concatenate them to get a combined encoding of all three modalities. Finally, we flatten the combined encoding and pass it through an MLP to get the score probabilities.
ImbOLL Function
Now, we introduce the novel ImbOLL function we use to train our models. The ImbOLL function is a modified version of the OLL (Castagnos et al., 2022 ) function. The OLL function is used to train models for ordinal classification. The OLL function for N classes is defined as:

\mathcal{L}_{OLL-\alpha}(P,y)=-\sum_{i=1}^{N}log(1-p_{i})d(y,i)^{\alpha} (1)

where y is the actual class, p_{i} is the predicted probability of class i , \alpha is a hyperparameter and d(y,i) is the distance between the classes y and i which is defined as:

d(y,i)=|y-i| (2)

The OLL function is based on the principle of penalising a model for bad decisions instead of rewarding good decisions. However, the OLL function is not suitable for imbalanced datasets. The questions of the PHQ-8 questionnaire consist of 4 possible classes according to the frequency of symptoms: 0 , 1 , 2 , and 3 . However, very few participants give a score of 2 or 3 to a question as very few patients feel a particular symptom so often. This results in an imbalanced score distribution. To consider this, we introduce weights w(y) , which gives a harsher punishment to a model when it makes a wrong decision for a rarer ground truth score. The weights are defined as:

w(y)=\frac{n_{T}}{n_{y}} (3)

Where n_{T} is the total number of data points in the training set, and n_{y} is the number of data points in the training set belonging to class y . Our novel loss function, ImbOLL , is defined as follows:

\mathcal{L}_{ImbOLL-\alpha,\beta}=-\sum_{i=1}^{N}log(1-p_{i})d(y,i)^{\alpha}w(y)^{\beta} (4)

where \alpha and \beta are hyperparameters.
Experiments
In this section, we describe the experiments with QuestMF and its ablation frameworks. We use the following ablation frameworks:

Total : We train the models with the MSE loss function to predict the total questionnaire score. This framework consists of a single modality encoder for each modality and a single fused model.

QuestMF (MSE): We train the QuestMF framework with the MSE loss function.

QuestMF (OLL): We train the QuestMF framework with OLL function.

QuestMF ( ImbOLL ): We train the QuestMF framework with ImbOLL function. This is our proposed framework .

To evaluate the performance of the methods in the depression severity prediction task, we use the standard metrics used in prior works: Concordance Correlation Coefficient (CCC), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). CCC is defined as:

\rho_{c}=\frac{2\rho\sigma_{x}\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}+(\mu_{x}-\mu_{y})^{2}} (5)

Where \rho is the Pearson correlation between variables x and y . \sigma_{x} and \sigma_{y} are the standard deviations of variables x and y . \mu_{x} and \mu_{y} are the means of variables x and y . We use CCC as the primary metric because it is unbiased by changes in scale and location (Lin, 1989 ) . Psychologists use CCC to assess the agreement between test scores from different raters. It was also used as the evaluation metric for the AVEC 2019 challenge. We also report RMSE and MAE. A higher CCC score is desirable to show that the predicted and actual outputs correlate well. For RMSE and MAE, a lower value is desired, as it shows a smaller difference between the predicted and the actual output. More details on the metrics are provided in Appendix B . For the ImbOLL function, we empirically find the parameters \alpha=1 and \beta=0.5 to be the best. For the OLL function, we find \alpha=1 to give the best results. The detailed results of the experiments with hyperparameters of ImbOLL and OLL are presented in Appendix D . To show the robustness of our model, we run our experiments on three different seeds: 42 , 100 , and 1234 . We are the first in this domain to run experiments on multiple seeds. The training strategy, checkpoint selection, and hyperparameter details of the models are provided in Appendix E .
Conclusions & Future Work
In this work, we show that our Question-wise Modality Fusion ( QuestMF ) framework trained with Imbalanced Ordinal Log-Loss ( ImbOLL ) function improves the interpretability in depression severity score prediction by predicting scores of specific questions. This can help clinicians identify particular symptoms or symptom combinations, enabling them to tailor their interventions to the individual’s specific needs. The QuestMF ( ImbOLL ) framework also shows performance comparable to current state-of-the-art models on the E-DAIC dataset. We also show its robustness over different seeds. Our framework can assist clinicians in diagnosing and monitoring depression and reduce the burden placed on patients in filling out self-reported questionnaires.
Additionally, we perform an extensive analysis to understand the importance of each modality for each question in the questionnaire. By releasing the code, we hope to enable future research of this framework on other questionnaires for mental health assessment and on real world longitudinal therapy data.
Limitations
While the question-wise modality fusion framework trained with ImbOLL function offers a solution to considering the variable contribution from modalities based on questions and framing the problem as an ordinal classification task, the data used for training and evaluation are not ideal. While the E-DAIC dataset was released to improve multimodal research in depression severity prediction, the training split only contains 163 sessions. As a result, the trained models are prone to overfitting and high bias and are unlikely to perform well in out-of-distribution data. The validation and test splits also contain only 56 sessions each. As a result, they are far from representing the general population. Moreover, bigger and more diverse datasets are unavailable due to privacy issues. Thus, the QuestMF ( ImbOLL ) framework is only tested on the E-DAIC dataset, which further constrains testing the generalisability of the model. Thus, the results and analysis obtained in this work need to be verified with bigger and more diverse datasets in the future. Also, the E-DAIC dataset only contains first time interviews of participants with a virtual agent similar to enrolling interviews for therapy and does not contain real therapy session interviews. Real world depression tracking also requires longitudinal data, i.e., multiple therapy sessions with the same participant and tracking changes in depression severity throughout their treatment. Since the E-DAIC dataset does not contain such data, we cannot test the effectiveness of our model in such real world situations.

Another limitation is the language and culture coverage. In this work, we only cover the English language, and the dataset is collected in the US. However, people use different languages to express themselves, and people from different cultures express themselves differently, thus affecting therapy. However, QuestMF ( ImbOLL ) could not be developed and tested for such generalisation due to the lack of suitable datasets.

Our focus in this work is to present a more intuitive methodology that considers the variable contribution from modalities according to the question in a questionnaire, frames the task in its true nature of ordinal classification task, gives question-wise scores that can help clinicians design more personalised interventions and analyse the results to understand the contribution of each modality towards the score of each question.
Ethical Considerations
While this work is focused on presenting a methodology and analysis for automatic depression detection, the methods need to be trained on larger datasets to ensure the method’s generalisation capabilities. The method should also be assessed for generalisability through clinical trials. Deploying these methods without proper training and assessment through clinical trials could lead to introducing harmful biases in real world situation. Therefore, the framework trained with the E-DAIC dataset may not be used in clinical practice. It requires a broader evaluation and clinical validation before being used in real-world clinical settings.