A Foundation Model for Patient Behavior Monitoring and Suicide Detection

Authors: Rodrigo Oliver, Josué Pérez Sabater, Diego Herrero-Quevedo, Antonio Artés, Alejandro Lancho, Pablo M. Olmos
Primary area: applications to neuroscience & cognitive science
Keywords: Foundation models for patient monitoring, Suicide detection via deep learning

Abstract:
Foundation models have achieved remarkable success across various domains, yet their adoption in healthcare remains limited, particularly in areas requiring the analysis of smaller and more complex datasets. While foundation models have made significant advances in medical imaging, genetic biomarkers, and time series from electronic health records, the potential for patient behavior monitoring through wearable devices remains underexplored. Wearable device datasets are inherently heterogeneous and multisource and often exhibit high rates of missing data, presenting unique challenges. Notably, missing patterns in these datasets are frequently not-at-random, and when adequately modeled, these patterns can reveal crucial insights into patient behavior. 
This paper introduces a novel foundation model based on a modified vector quantized variational autoencoder (VQ-VAE), specifically designed to process real-world data from wearable devices. Our model excels at reconstructing heterogeneous multisource time-series data and effectively models missing data patterns. We demonstrate that our pretrained model, trained on a broad cohort of psychiatric patients with diverse mental health issues, can perform downstream tasks without fine-tuning on a held-out cohort of suicidal patients. This is illustrated through the use of a change-point detection algorithm that identifies suicide attempts with high accuracy, matching or surpassing patient-specific methods, thereby highlighting the potential of VQ-VAE as a versatile tool for behavioral analysis in healthcare.

Decision: reject
Reviewer scores: 3, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729686716645

thread_type:
official_review

<thread_object>
creator: reviewer_4ffh

<content>
summary:
The paper presents a foundational model for imputing PDP data with missing MNAR values using VQ-VAE. This system aims to encode the time series into discretized embeddings that represent latent profiles, which help capture patient behavior and improve performance in downstream tasks.

The results appear comparable to existing alternatives in the literature. However, the strength of this model lies in its flexibility, as it can be applied to tasks for which it was not explicitly trained. This demonstrates that the generated embeddings capture meaningful representations of the input data, allowing their use across a broader range of objectives without requiring fine-tuning.

strengths:
- **Model Flexibility**: One of the standout features of the proposed model is its flexibility. The embeddings learned are generalizable across different tasks, even those for which the model was not explicitly trained. This demonstrates its robustness and potential for transfer learning.

- **Potential for Interpretability**: The use of latent profile embeddings offers promise for improving interpretability in clinical and behavioral data, an essential feature for real-world healthcare applications. This adds an additional layer of value to the model beyond just predictive accuracy.

- **Description of the method**: The article presents a detailed explanation of the method.

weaknesses:
- **Limited Evaluation on Downstream Tasks**: Although the paper emphasizes the model’s flexibility and its applicability to a variety of downstream tasks, the evaluation is limited to only two tasks. Expanding the analysis to include a broader range of scenarios would have provided stronger evidence of the model's generalizability.

- **Lack of Exploration on Interpretability**: The paper asserts that the learned latent profiles enhance interpretability, yet no concrete evidence or case studies are provided to substantiate this claim. A more thorough investigation, possibly through qualitative analysis or correlations with clinically relevant features, would significantly strengthen the paper's contributions.

- **Omission of Spatio-Temporal Architectures**: While the authors mention that exploring spatio-temporal architectures is left for future work, the choice not to use models such as GNNs, RNNs, or Transformers, which are well-suited for these types of tasks, is not sufficiently justified. This omission limits the model's applicability to more complex time series problems.

- **Clarity in Results Presentation:** The presentation of results could be made clearer. The use of multi-line graphs can be confusing, and this would be better supplemented with numerical information in the main text. Additionally, incorporating a wider range of classification metrics beyond AUC, such as AUPRC, would provide a more comprehensive evaluation of the model's performance.

questions:
- When training the model, synthetic missing values are introduced. Why are these not also used during the calculation of the loss function to optimize their imputation, potentially improving the model’s capacity to handle missing values in the time series?

- What is the class imbalance in the Suicide Detection task? If there is significant imbalance, the AUC score might be misleading. In clinical settings, the AUPRC is often used as an alternative metric for imbalanced data [1]. Perhaps it could be considered for this problem as well.

- The imputation errors in Table 6 seem quite high, although this may be due to the nature of the data. Could the authors provide results from comparison techniques? It would be useful to include at least one method of a common benchmark [2], and possibly show results from methods like linear interpolation and mean interpolation.

- The paper mentions that these profile embeddings could enhance the interpretability of individual behaviors. Have the authors explored this? Were they able to find any correlation between the presence of specific embeddings and labels? If not, could they outline how they would approach this in future work, if it falls outside the scope of this paper?

- Why were more suitable architectures for capturing spatio-temporal representations not used? While it is mentioned that this is left for future work, it’s unclear why models such as GNNs, RNNs, Transformers, or even newer blocks like Mamba were not considered [2] [3].

- Figure 4 is difficult to interpret in terms of the quality of imputations and reconstructions. It would be helpful to complement this with a table, similar to Table 6.


[1] Moor, M., Rieck, B., Horn, M., Jutzeler, C. R., & Borgwardt, K. (2021). Early prediction of sepsis in the ICU using machine learning: a systematic review. Frontiers in medicine, 8, 607952.

[2] Cini, A., Marisca, I., & Alippi, C. (2021). Filling the g_ap_s: Multivariate time series imputation by graph neural networks. arXiv preprint arXiv:2108.00298.

[3] Liu, M., Huang, H., Feng, H., Sun, L., Du, B., & Fu, Y. (2023, April). Pristi: A conditional diffusion framework for spatiotemporal imputation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE) (pp. 1927-1939). IEEE.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729784158579

thread_type:
official_review

<thread_object>
creator: reviewer_pvxg

<content>
summary:
This paper explores patient behavior monitoring and suicide detection by adapting the vector quantized variational autoencoder (VQ-VAE) to analyze data from real-world wearable devices. By reconstructing heterogeneous multi-source time-series data and accounting for patterns of missing data, the proposed model is validated to be effective in detecting change-time events within a cohort of suicidal patients.

strengths:
S1. This research provides significant medical potential by modeling heterogeneous, multi-source time-series data from wearable devices, aiming to develop advanced approaches for identifying potential behavioral shifts that signal serious mental health risks.

S2. The proposed model, which leverages VQ-VAE as the foundation model, captures the missing data patterns, and incorporates a change-point detection algorithm, is overall reasonable from a technical standpoint for this specific medical application.

S3. The experimental evaluation includes a demonstration of imputation performance, an assessment of downstream tasks, results from statistical tests, and several representative case studies.

weaknesses:
W1. My primary concern regarding this paper is its technical novelty. Although the proposal adeptly uses the VQ-VAE model as the foundational framework, modifies its design to capture missing data patterns in time-series wearable device data, and connects to change-point detection as a downstream task, it relies heavily on existing techniques. This reliance diminishes the perceived novelty of the proposed model. Moreover, the paper does not sufficiently articulate the challenges and complexities associated with integrating these techniques, which is crucial for demonstrating its innovation.

W2. The evaluation of imputation performance lacks comparative baselines, which undermines the robustness of the findings. Additionally, the downstream task evaluation includes only one baseline, HetMM, which is insufficient. Incorporating a broader range of advanced and recent baselines would help demonstrate the superiority of the proposal in this research area.

W3. The evaluation relies solely on a single proprietary dataset, limiting the generalizability of the findings. Expanding the evaluation to include multiple datasets would enhance the credibility and applicability of the proposed model across different contexts.

W4. The performance improvements offered by the proposed model are marginal when compared with HetMM. Given that the proposal claims advantages in scalability and efficiency, it is essential to substantiate these claims with experiments on larger datasets or real-time analysis scenarios.

W5. As the proposal targets a specific healthcare application—suicide detection—it would be advantageous to include medical validation of the findings or to offer insights into how the proposed model could inform real-world clinical decision-making. Such contributions would significantly elevate the practical relevance of the proposal.

W6. The writing of this paper could be improved by addressing the following aspects:

(i) The paper should clearly identify which model among Model A0, Model A1, and Model A2 performs best and should be recognized as the final proposed model.

(ii) The inclusion of a detailed related work section is necessary. This section would help situate the paper within the existing body of knowledge, providing a comprehensive backdrop against which the current research can be assessed and appreciated.

questions:
Please refer to points W1 through W6 for detailed concerns and recommendations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650939433

thread_type:
official_review

<thread_object>
creator: reviewer_w2p2

<content>
summary:
The paper introduces a foundation model for patient behavior monitoring and suicide detection based on a modified Vector Quantized Variational Autoencoder (VQ-VAE). This model effectively reconstructs heterogeneous time-series data from wearable devices and captures missing data patterns. The paper highlights its application in predicting suicide attempts using a probabilistic change-point detection (CPD) algorithm on data from a cohort of psychiatric patients. The model shows promise for scalable, efficient patient monitoring by outperforming patient-specific methods without requiring fine-tuning.

strengths:
The approach is novel in integrating VQ-VAE with CPD for behavioral analysis and suicide prediction, particularly in using discrete latent representations for temporal health data. The paper is methodologically strong, with well-designed experiments and performance evaluations. The work has substantial potential in clinical monitoring and predictive modeling, addressing a critical healthcare application with an approach that could be extended to other domains. The paper is well-written, with clear articulation of concepts and objectives, making complex methodologies accessible. Figures and tables are effectively used to illustrate the architecture and experimental results.

weaknesses:
The evaluation is conducted on a single private dataset. 
The use of a proprietary dataset limits reproducibility and prevents validation or benchmarking with public datasets, reducing transparency. Expanding to public datasets could provide a more comprehensive assessment of the model's generalization and facilitate future comparisons.

The study lacks a comparison with other generative models for reconstruction tasks, which could demonstrate the superiority or limitations of the proposed VQ-VAE model, comparisons with more generative models in the reconstruction phase could bolster the paper’s robustness.

Although three VQ-VAE variants (A0, A1, and A2) are tested, there is limited discussion on the specific performance differences across these variants and their impact on handling missing data.

Certain technical details, such as the choice of hyperparameters, could be clarified to enhance reproducibility.

questions:
1. Could you elaborate on the performance differences among the three VQ-VAE variants (A0, A1, and A2) in terms of handling specific missing data patterns?

2. Have you considered evaluating the model on additional, publicly available behavioral datasets to better assess generalizability and facilitate reproducibility? Do the authors plan to publish or share the dataset to reproduce and enable further research validation?

3. Have you explored using alternative generative models, such as GANs or VAEs, for comparison on data reconstruction? How do you envision these models performing relative to VQ-VAE in terms of capturing behavioral patterns?

4. Can the authors clarify the rationale for choosing specific hyperparameters for CPD (e.g., the hazard function parameter λ)?

A minor typo: Line 188 - The term "D" is used without prior definition or context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681488959

thread_type:
official_review

<thread_object>
creator: reviewer_sgoo

<content>
summary:
The authors introduce a model based on VQ-VAE, trained in a self-supervised way with different (informative) ways of modeling missingness. The authors look at reconstruction and imputation accuracy of the model on a behavioral data set of wearable readings. Using the trained model the authors implement a Bayesian online learning procedure to infer the MAP of the presence of a change point in the data with proposed application to suicide prevention, which is benchmarked against HetMM on their behavioral data set in terms of ROC curves.

strengths:
- Time series modeling and change point detection are important problems in clinical applications

weaknesses:
- A related work section for other time series models is missing, but important to place the work within the context of the broader literature
- For imputation, the model should be compared to baselines.
- Other baselines for change-point detection than HetMM would be beneficial, e.g. point process frameworks, time series foundation models
- The novelty of the work (as opposed to applying a known framework to a novel interesting problem) is not entirely clear to me.

questions:
- What is the calibration of the pseudo-probabilities as compared to HetMM?
- How does the model deal with censoring due to loss of follow-up?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732317889836

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments on baseline comparison, new downstream task, interpretability and reproducibility

comment:
# GENERAL
 To address the primary concerns raised by the reviewers, we have made the following updates:

1.  **Comparison with Baselines**: We compared our approach with the HMM-Transformer model proposed by Arbaizar et al. ([DOI: 10.2196/preprints.63962](https://preprints.jmir.org/preprint/63962)). Although the HMM-Transformer model showed good performance in downstream tasks like sentiment analysis, it struggled with suicide detection due to its high-dimensional continuous latent space. In contrast, our VQ-VAE model's discrete latent space proved more interpretable and manageable, achieving competitive results across multiple tasks.
    
2.  **New Downstream Task**: We introduced a new task—sentiment analysis—mirroring the work by Arbaizar et al. Our VQ-VAE achieved a notable AUC of 0.81, compared to the HMM-Transformer’s 0.98, despite the VQ-VAE not being explicitly tuned for this task. This highlights its versatility in handling missing data and adaptability to different contexts without additional training steps.
    
3.  **Interpretability**: To address concerns about interpretability, we applied Latent Dirichlet Allocation (LDA) to the discrete latent representations learned by VQ-VAE. This analysis revealed meaningful groupings of patients based on behavioral patterns, offering insights into different types of patient behavior and variability. Details of this analysis, including figures, can be found at [https://figshare.com/s/3b97194103b7b5b6254c](https://figshare.com/s/3b97194103b7b5b6254c).
    
4.  **Reproducibility**: We acknowledge the challenges posed by working with private datasets. If the paper is accepted, we will release a partial dataset to enhance reproducibility.
    

We believe these revisions address the key concerns and elevate our work to meet ICLR's standards.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a new foundation model for wearable data. The authors showcase the effectiveness of the method in modeling missing data patterns and reconstructing heterogeneous multi source time-series data. The model is used for downstream tasks on a test cohort of suicidal patients (identifying suicide attempts via a change-point algorithm).

The importance and difficulty of the clinical problem was appreciated by all the reviewers. 

Concerns were rained concerning the positioning of this paper in the broader literature, and missing comparisons to other foundation models, as well as baselines that perform imputation and change point detection. 

Concerning the novelty, the authors have identified it as: “no previous studies have used this type of discrete latent-space model for reconstructing heterogeneous missing data in time series from wearable and mobile devices. This unique approach is specifically motivated by the characteristics of the dataset and the downstream tasks addressed in this work.” This might be sufficient, but more arguments need to be presented concerning why existing methods do not succeed at this.

The authors have also committed to including a related work section on time series reconstruction and imputation. However, despite of the references and explanations they provided, it’s difficult to say whether this will be cogent enough to meet ICLR standards. 

The sentiment analysis results are very far from the state of the art model (0.81 vs 0.98) - with the caveat that the proposed foundation model was not fine tuned for sentiment analysis. I don’t see this as evidence in favor of the proposed model being generalizable, but it’s also not evidence against it, as it’s a completely different domain. While foundation models should by definition be more generalizable than models train for a single dataset/task, there are limits to this. Maybe applying the model to some datasets and task that is closer to what it was trained on would be more useful in showing versatility (for instance: using wearables for stress detection or clinical outcome forecasting). 

The authors have argued this is an application-driven paper, which, in my opinion, could be in scope for ICLR. Here is what I would recommend that the authors include for such a paper:

1. A clear exposition of the importance of the applications, its settings, and the characteristics that make it challenging; this appears to have been done well in the paper.

2. An overview of existing techniques that in the same area as the task described. Some of these techniques might not be applicable to the problem without extensive modifications, in which case clear arguments should be presented with respect to why this can’t be done. This was not done in the paper, though the authors did make a good start of it in the rebuttal.

3. For the existing methods that are applicable to (or easily modifiable for) the task, experiments showing that they fall short of solving this particular problem, motivating the methodological gap. This part was missing from the original paper. The addition of the HMM-Transformer experiment is a good step in the direction, though more methods need to be considered, including as one reviewer suggested, methods from the Gaussian Process literature).

4. Description of the new method and experiments showing that it overcomes the pitfalls of the other techniques. In this respect, the new method is well described, but without extensive comparisons, it’s difficult to judge its merits.


Overall, this paper deals with an incredibly important application, and would have potential, however the comparisons to the related work need to be improved considerably to meet the standards of ICLR. However, I encourage the authors to revise and submit to ICML.

Should the paper not be accepted at methods-focused conferences, the authors might consider sending it to venues such as MLHC, CHIL, or the ML4H Symposium at NeurIPS.