Multimodal Depression Detection with Contextual Position Encoding and Latent Space Regularization

Authors: Enshi Zhang, Christian Poellabauer
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: depression detection, speech and language processing, contextual position encoding, latent space regularization

Abstract:
Clinical interviews are the gold standard for detecting depression, and previous work using multimodal features from participants' audio, transcribed text, and video have shown promising results. Recent approaches further improve performance by incorporating an additional textual modality—the interviewer’s prompts—during training. However, these approaches risk introducing biases, as models may over-rely on specific prompt-response pairs, which may not always be present in real-world settings. This leads to models exploiting these cues as shortcuts for detecting depression rather than learning the language and behaviors that genuinely indicate the subject's mental health, ultimately undermining consistency and objectivity. To address this, we propose a novel approach that combines Contextual Position Encoding (**CoPE**) and Latent Space Regularization (**LSR**), leveraging both subjects' responses (audio) and the interviewer's prompts (text). CoPE captures the evolving context of the interview, ensuring that the model utilizes insights from the entire conversation, preventing over-reliance on isolated or late-stage cues. This helps the model understand interactions holistically and more accurately reflect mental health indicators. LSR introduces constraints to enforce consistency in the model’s learned representations, reducing overfitting to superficial cues and guiding the model toward more generalizable patterns. By smoothing the latent space, LSR helps the model focus on meaningful, high-level representations of both audio and text.
Our approach yields competitive results on the **DAIC-WOZ** benchmark and surpasses the state-of-the-art on the **EATD** benchmark. The code is released.

Decision: reject
Reviewer scores: 6, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730127204587

thread_type:
official_review

<thread_object>
creator: reviewer_eyjc

<content>
summary:
This paper provides a method for detecting depression in clinical interviews by combining Contextual Position Encoding and Latent Space Regularization. This involves tracking the context as it evolves over the conversation, and subsequently smoothing latent representations. This is meant to better capture the ‘flow’ of the conversation and avoid overfitting. Although this provides good results on DAIC-WOZ and EATD, additional modalities (like video) could have been pursued.

strengths:
- the more contextual approach of CoPE is a good addition to the (varied and numerous) approaches to this problem. This might be a better way to obtain pragmatic cues or more speaker-specific baselines than focusing on cross-sectional aspects across speakers. Using LSR to smooth the latent space is also a correct step to take, although there are alternatives. This is, on the whole, positive but fairly evolutionary.
- SotA performance is, of course, reported on this important task. Ablations are run on the appropriate parts of the model in Table 3, and various alternatives are mentioned in Table 2 for comparison.

weaknesses:
- While it is positive that DAIC-WOZ and EATD were used, this is still a somewhat reduced and well-trodden set. They are also fairly curated and may not reflect real-world variability or noise, which limits claims of generalization or robustness.
- Despite the (almost necessary) ablations, the evaluations are fairly superficial, focusing on precision, recall, and F1 but not going deeper into the results to vary the parameters further, or to explain them, or to identify patterns in performance on aspects of the dataset. The result is a fairly ‘paint-by-numbers’ set of experiments with unclear gain in knowledge.

questions:
- Could you check your references for, e.g., completeness (e.g., how the WHO 2017 paper was published) or formatting?
- What real-world aspects may be relevant to consider for additional analysis? E.g., how might environmental considerations or more unstructured data collection relate to your results? What are the limitations of this work that you are able to identify?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644981780

thread_type:
official_review

<thread_object>
creator: reviewer_adxy

<content>
summary:
The paper presents an architecture for automatic depression detection from clinical interviews. The input information is obtained as a combination of the patient's audio recordings and the interviewer's textual prompts. The key components of the architecture are the Contextual Positional Encoding, which is a combination of absolute and relative positional encodings, and the application of L2 regularization to the fused modality features. The proposed approach achieves competitive results on the DAIC-WOZ benchmark and state-of-the-art results on the EATD benchmark.

strengths:
* The paper does a good job on presenting the current progress on the field of automatic depression detection.
* Ablations are performed for each architectural component.

weaknesses:
* The novelty of the proposed approach comes into question, as it appears a combination of pre-existing methods, applied to well-studied datasets
* There are some issues with the presentation of the method. Specifically, in L257 the equation describing the LSR loss includes a dropout term, while in L340 no such term exists in the loss. My guess is that L257 is inaccurate and simply aims to demonstrate that dropout is applied during training, but this may confuse future readers.
* The presentation of Table 1 could be improved by including some extra columns (Text encoder, Audio encoder etc.)
* No statistical significance testing is presented about the results. This is especially relevant because the explored datasets are fairly small, and the improvements are rather marginal.
* Regarding formatting there are two issues worth of mention. 1) all equations are unnumbered. Including equation numbering can help referencing equations during reviewing or analyzing the paper. 2) The gap after Fig. 1 could be utilized to present more relevant information in the main part of the paper.

questions:
* The Equations about the LSR loss need further clarifications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744484237

thread_type:
official_review

<thread_object>
creator: reviewer_1gmd

<content>
summary:
This paper introduces a novel approach to detecting depression from clinical interviews by combining Contextual Position Encoding (CoPE) and Latent Space Regularization (LSR) to overcome limitations of existing methods that rely too heavily on specific question-answer pairs. The system processes both audio responses and interviewer prompts through a multimodal framework, achieving competitive results on the DAIC-WOZ dataset and state-of-the-art performance on EATD.

strengths:
Main contribution/ideas: 1) Proposed to use Contextual Position Encoding (CoPE) to capture the evolving nature of clinical interviews, 2) Uses Latent Space Regularization (LSR) to prevent overfitting and encourage more generalizable patterns, 3) Combines both audio features and interviewer prompts. 1-is the main contribution and should have been the focus of the paper.

The performance is strong compared to other audio and audio+interviewer baselines. Good ablation analysis verifies the importance of CoPE for this dataset

Well-written paper, well-presented ideas, good technical detail.

weaknesses:
I would have personally preferred to focus just on the CoPE embeddings and similar ideas to explore the nature of clinical interviews and how transformers can better model this structures. I think this is the main idea/contribution of the paper and it gets diluted as the authors are chasing SoTA. 

Questions about generalizability: I am missing a more detailed deep dive into the databases to understand the impact of each proposed methods to a differently structured interaction, e.g., would CoPE be relevant for a podcast interview, what is the power of representation it is adding to the model etc. Also can you better explain the effect of I, why is it important, is the question order reshufle that you do important?

All methods proposed are not new (cope, lsr, cross-modal attention, feature fusion) so the originality of the paper is limited. 

Missing Comparisons with the SoTA multimodal approaches.

questions:
Why don't you present any results even for simple fusion for text and video, since SoTA is what you are after?

See also question above about generalizability and deeper data analysis,
</content>
</thread_object>

</thread>
<thread>
creation date:
1731124847830

thread_type:
official_review

<thread_object>
creator: reviewer_skaj

<content>
summary:
The document presents a novel approach to detecting depression by leveraging multimodal data from clinical interviews. This method combines **Contextual Position Encoding (CoPE)** and **Latent Space Regularization (LSR)** to enhance model accuracy and generalizability. By analyzing both participants' audio responses and interviewers' text prompts, CoPE enables the model to capture the full conversational context rather than relying on isolated cues. Meanwhile, LSR helps prevent overfitting by encouraging the model to focus on meaningful patterns in the data. The applicability of the proposed approach has been evaluated using two state-of-the-art datasets, including **DAIC-WOZ** and **EATD**.

strengths:
* Multimodal approach relying on the inclusion of contextual positional embeddings and latent space regularization for depression detection.
* Inclusion of participant's audio responses and the interviewer's text prompts for improving the accuracy and generalizability of depression detection models.
* State-of-the-art results on the DAIC-WOZ and EATD datasets for multimodal depression detection.

weaknesses:
* The interview is classified as a distinct modality; however, it would be more accurately rephrased as a text modality. 
* In **Table 1**, it is unclear whether CoPE has been applied to the text modality within the first set of audio-specific models. 
* Why was the visual modality not considered in the modeling phase?
* In **Table 2**, the table should be divided into sections based on respective datasets.
* LSR implements regularization at the level of latent embeddings. How does this compare to weight-based approaches, such as L2 and L1 regularization techniques? 
* Instead of utilizing attention-based weighting, what was the rationale behind not considering cross-attention-based fusion for the text and audio modalities?

questions:
* Have the authors thought about utilizing RoPE embeddings for encoding relative positions regarding contextual position encoding?
* Do the existing contextual CoPE embeddings apply effectively to interview sessions of varying lengths, from short to long?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732191606278

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author response & revision

comment:
Dear reviewers, ACs, and PCs,

We appreciate the time you have taken to review our work and for giving us the opportunity to address the concerns raised and improve the quality of our paper 'Multimodal Depression Detection with Contextual Position Encoding and Latent Space Regularization.' We have summarized our responses to each reviewer’s concerns for easier tracking and review, including the changes made in the **updated pdf file** and **additional experiments** conducted.

We have just uploaded the revised version with **highlighted changes** for your review and will upload a **clean version** before the discussion period ends. During the discussion period, we welcome any further feedback from the reviewers. Thank you again!

Following Reviewer SKAj’s suggestions:

1.	Clarifications in Abstract.

2.	Updated the caption of Table 1.

3.	Experiments in Section A.10 and discussed via response.

4.	Restructured Table 2 and Table 1.

5.	Updates in Section 3.1 and experiments in Section A.6.

6.	Discussed via response.

7.	Discussed via response.

8.	Added Figures 3 and 4, experiments in Section A.8, and discussed via response.



Following Reviewer 1GMd’s suggestions:

1.	Discussed via response

2.	Discussed via response

3.	Experiments in Section A.8, Section A.9

4.	Discussed via response

5.	Discussed via response; Section A.10

6.	Experiments in Section A.6, Section A.8


Following Reviewer adxY’s suggestions:

1.	Discussed via response

2.	Section 3.1

3.	Table 1

4.	Experiments in Section A.9

5.	All equations (numbered); extra space eliminated

6.	Section 3.1


Following Reviewer eyjC’s suggestions:

1.	Discussed via response; Section A.10

2.	Discussed via response; experiments in Section A.7, Section A.8

3.	Reference fixed

4.	Discussed via response; Section A.10
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a multimodal model for automatic depression detection from clinical interviews which combines the patient's audio recordings with the interviewer's textual prompts. The authors propose a Contextual Positional Encoding combining both absolute and relative positional encodings, and the application of a smoothing regularization term to the latent space to enforce consistency in the model’s learned representations. The proposed approach achieves competitive results on the DAIC-WOZ benchmark and state-of-the-art results on the EATD benchmark.

3 reviewers were negative about the paper, and one gave marginal above acceptance. All reviewers found the problem setting interesting, that the performance was strong, and the ablation studies and analysis were solid.

However, all reviewers were also concerned that the proposed methods and combination of modules (contextual positional encodings, latent space regularization, cross-modal attention, feature fusion) already exist in the literature, so they found limited originality in the paper. I am aligned with the majority of the reviewers' judgement that the application of these methods to this new problem of multimodal depression detection warrants research, but that the application-centric and practical tricks proposed here are too specific to be broadly significant for the ML community, and therefore the paper falls below the threshold for acceptance.