Results & Analysis
The results comparing our proposed QuestMF ( ImbOLL ) framework with its ablations and current state-of-the-art methods are presented in Table 1 . Since the prior works only show their best results on a single run, we also pick our best results on CCC for a fair comparison. As we can see, our proposed QuestMF ( ImbOLL ) framework matches the performance of state-of-the-art models in CCC, the primary metric used to evaluate depression severity prediction tasks. In addition to comparable performance, QuestMF framework provides question-wise scores, improving interpretability over current methods, thus allowing clinicians to design personalised interventions. We also show the robustness of our frameworks over multiple runs, which is not done by the previous works. The mean and standard deviation of the performance over 3 runs are presented in Table 2 . We observe that Total and QuestMF (MSE) perform similarly. However, training with ordinal classification objective improves the performance, as we can see from the results of QuestMF (OLL). This shows the effectiveness of combining QuestMF with ordinal classification training. Training with our novel ImbOLL function further improves the results on CCC. It is also robust, with a standard deviation of 0.022 .

Next, we present an ablation study to observe the performance of single modality models, two-modality fused models, and the three-modality fused model with the different frameworks. We present the results in Table 3 . For all single modality models and two-modality fused models, we observe that QuestMF (OLL) and QuestMF ( ImbOLL ) frameworks show better performance than the other frameworks. All the frameworks except QuestMF ( ImbOLL ) and QuestMF (OLL) show the best performance with only text and video fusion, while the performance drops when all three modalities are fused. This shows that training with an ordinal classification task gives a better optimisation objective. Moreover, we also see that QuestMF ( ImbOLL ) and QuestMF (OLL) show the best performance gains when adding more modalities. Comparing the results of Text + Audio + Video with the Text models, we observe that QuestMF ( ImbOLL ) achieves an improvement of 0.047 (Text + Audio + Video ( 0.662 ) - Text ( 0.615 )) on CCC and QuestMF (OLL) achieves an improvement of 0.048 (Text + Audio + Video ( 0.640 ) - Text ( 0.592 )). Among the regression methods, Total achieves the best improvement when comparing the results of Text + Audio + Video with the Text models. However, it only achieves an improvement of 0.011 (Text + Audio + Video ( 0.602 ) - Text( 0.591 )). This further shows that the QuestMF ( ImbOLL ) framework improves fusion. Appendix F provides more detailed results for this ablation.

Finally, we analyse the importance of each modality toward predicting the score for each question in the framework.
This analysis can be used to build a framework where each question only uses the modalities important for its scoring. However, we do not build such a framework as it requires more data to generalise the analysis.
Since we lack fine-grained labels in the test split, we use the validation set CCC for each question in this analysis. A higher CCC shows greater importance. The CCC for each question with different modality combinations is shown in Figure 5 . From them, we observe:

Q1: Feeling no interest. Text modality gives the best results. This may be because the content of the conversation in the interview is the best indicator to determine loss of interest in hobbies.

Q2: Feeling depressed. Text + Audio gives the best results with very close results from Text + Video and Text + Audio + Video. Audio cues like flat speech and visual cues like sadness or blunted facial expressions might help predict this score. However, a fusion of all three struggles to train due to a small number of training data.

Q3: Irregular sleep. Text + Video gives the best results. Sleep disturbances are often directly reported by individuals, which explains the contribution of the text modality. The video modality contributes since sleep issues can often be observed from posture and general demeanour.

Q4: Feeling tired. Text + Audio + Video gives the best results as tiredness is often visible in a person’s face, body language, and way of speaking.

Q5: Irregular appetite. Text + Video gives the best results. A person’s appetite can be determined by directly asking them, so the text has the highest importance here. The video also contributes as physical appearance may influence the prediction.

Q6: Feeling like a failure. Text + Audio + Video gives the best results. Acoustic cues like a disappointed voice and visual cues like a saddened face help in the prediction.

Q7: Trouble concentrating. Text + video gives the best results. Visual cues like gaze can be an important factor. Looking away and not making eye contact may indicate concentration problems.

Q8: Irregularities in moving and speaking. In this case, Text + Video gives the best results, with Text + Audio + Video closely following. Movement can be captured through the video, and irregularities in a speech can be captured from transcripts and audio recordings.