# Use of immersive data by medical practitioners for mental health prevention

Noah Richert, Antoine Widmer, Yannick Prié

► To cite this version:

Noah Richert, Antoine Widmer, Yannick Prié. Use of immersive data by medical practitioners for mental health prevention. 2025 - II International Congress: Metaverse for the Good, Part I - II EMRN Conference: “Exploring the Intersection of AI and the Metaverse, Jun 2025, Alicante, Spain. hal-05332785

HAL Id: hal-05332785

https://hal.science/hal-05332785v1

Submitted on 27 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Use of immersive data by medical practitioners for mental health prevention

Noah Richert 1,3, Antoine Widmer 1 and Yannick Prié 2

1 HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland, 3960 Sierre, Switzerland

2 Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France

3 Lemanic Neuroscience Doctoral School (LNDS), University of Lausanne, University of Geneva, and EPFL, Switzerland

Correspondence*: Corresponding Author
noah.richert@hevs.ch

## 2 ABSTRACT

With the growing demand for accessible and scalable mental health care, immersive technologies offer different solutions for prevention and early intervention. This paper presents the preliminary work of a Ph.D. project within the framework of the EU funded THCS project titled Immersive Prevention Centers for Mental Health. This research explores the integration of eXtended Reality (XR) platforms within Immersive Prevention Centers (IPCs) to enhance decision-making with data support in mental health management. It presents a framework that will collect, interpret and visualize behavioral and biometric data to support clinicians in real-time and retrospective reviews of patient activity. By incorporating Machine Learning (ML) and clinically approved scenario design, this project aims to improve early identification, personalize interventions and monitor treatment outcomes in immersive environments. The interdisciplinary approach brings XR, Artificial Intelligence (AI) and clinical expertise together to build a foundation for scalable and interactive mental health support systems.

Keywords: immersive technology, mental health prevention, extended reality (XR), biometric data, artificial intelligence, co-design, virtual reality, digital health

## 1 INTRODUCTION

Mental health is a state of mental well-being that enables people to cope with the stresses of life (WHO, 2022). However, mental health disorders affect millions worldwide and access to care remains a challenge due to geographical, financial, and social barriers (Samira Abdul et al., 2024). According to the World Health Organization "Mental Disorders" fact sheet (WHO, 2022), these disorders are characterized by a disturbance in an individual's cognition, emotion regulation, or behavior, often leading to significant distress or impairment in functioning. With rising global awareness of the mental health crisis, there is an urgent need for innovative approaches that enhance prevention, early detection, and patient engagement in mental health interventions.

Among emerging solutions, digital health innovations, particularly immersive technologies, have shown promise in transforming how mental health services are delivered. Among these, immersive technologies
27 such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) collectively referred to as eXtended Reality(XR) (Rauschnabel et al., 2022) have opened up new avenues for experiential learning, 28 patient monitoring, and remote intervention (Freeman et al., 2017). 29

30 Our literature review synthesizes insights from three main research areas: the use of XR in healthcare 31 settings (Worlikar et al., 2023), the integration of mental health management strategies in healthcare 32 organizations (Samira Abdul et al., 2024), and the application of Machine Learning (ML) and Artificial 33 Intelligence (AI) to the interpretation of biometric data (Greener et al., 2022). 34

35 Within XR in health, one of the dimensions most actively explored is its integration into telehealth 36 services (Worlikar et al., 2023). Other Major areas of application include telerehabilitation (Caminero 37 et al., 2025; Isernia et al., 2020), telepsychiatry (Dan Shao, 2020), telediagnosis (Filippeschi et al., 2019), 38 and teleconsultation (Hu et al., 2019) using VR, telementoring, telesurgery (Guo et al., 2016; Kutzin et al., 39 2017) and telemonitoring (Thøgersen et al., 2020) using MR and AR. 40

41 Simultaneously, healthcare organizations are beginning to adopt structured mental health management 42 strategies that prioritize early intervention, reduce stigma and encourage scalable, tech-driven approaches to 43 care (Samira Abdul et al., 2024). The main classes of strategies by healthcare organizations are preventive 44 (Colizzi et al., 2020), diagnostic, therapeutic, and supportive measures (Schlak et al., 2022). 45

46 Furthermore, advances in artificial intelligence (AI) and machine learning (ML) provide new opportunities 47 to interpret biometric and behavioral data collected during digital interventions. These technologies can 48 support real-time assessment of emotional and physiological states, deliver adaptive and personalized 49 interventions, and enable predictive modeling for early crisis detection (Greener et al., 2022). When 50 integrated with XR platforms, ML algorithms can enhance interactivity by adjusting therapeutic content 51 based on user responses, further improving the precision and effectiveness of mental health interventions 52 (Altozano et al., 2025; Lee, 2024). 53

54 The research presented in this paper is part of a broader project investigating how Immersive Prevention 55 Centers for Mental Health (IPC4MH) within the Metaverse can address the following tasks. IPCs will be 56 providing structured, interactive, and evidence-based prevention programs for mental healthcare. The IPC 57 project focuses on prevention of mental health in two layers. The first is to act on early identification and 58 intervention of mental distress to avoid developing into diagnosable conditions. The second is to avoid the 59 worsening of a diagnosed patient's condition with better supervision of the therapy. 60

60 This work aims to help medical professionals deliver data-driven mental health interventions in immersive 61 environments, exploring how to best capture, interpret, and apply XR-derived data to support clinicians 62 with real-time insights and structured review data for ongoing patient monitoring and personalized care 63 planning. 64

## 2 PROJECT OBJECTIVES

65 Building on the previously outlined context, this project defines a practical and technical roadmap for 66 developing the IPC framework. The overarching goal is to support clinicians in the early identification, 67 intervention, and management of patients' mental health conditions within Immersive Prevention Centers 68 (IPCs). To achieve this, the project aims to create a comprehensive framework for the collection, 69 interpretation, and restitution of data generated from patient activities within XR environments. This 70 framework will ultimately enable the development of a clinician-facing dashboard, providing structured 71 and actionable insights into patients' engagement and progress. 72
<figure><img src="image_2.png" /><figcaption>**Figure 1.** IPC data flow structure</figcaption></figure>

67

The project addresses four main usage scenarios, illustrated in Fig. 1:

68

* **Clinician-present activity:** When the clinician is actively supervising a session and assigns an activity to the patient, real-time data is made available to assist the clinician in monitoring and immediate assessment.

69

70

71

* **Clinician-absent activity:** When the clinician assigns an activity (either individual or group-based) but is not present during its execution, patient data is collected and securely stored for later review.

72

73

* **Individual review sessions:** During subsequent consultations, clinicians can access and review the data collected over time to evaluate the patient's progress and to adjust therapeutic interventions accordingly.

74

75

76

* **Additional annotations:** Clinicians may also add notes or complete questionnaires through the dashboard interface. While this feature enhances patient monitoring, it involves handling highly sensitive personal information and thus requires strict privacy safeguards.

77

78

79

To translate this vision into practice, the project is structured around three key objectives that drive both clinical and technical development:

81

**Objective 1: extract behavioral and biometric data from IPC**

82

83

84

Data will be collected from immersive activities in IPCs using various modules. Patients will use Head-Mounted Displays (HMD) with possible add-ons for eye tracking or EEG. They can also use tablets or web platforms. Data collection must operate across multiple scenarios, in group or solo, and diverse tasks,
85

86

87

88

89

90

91

**Objective 2: determine clinically relevant indicators for mental health professionals**

92

93

94

95

96

97

98

99

100

101

102

mental or physical, to ensure robustness. The goal is to capture detailed behavioral and biometric indicators including posture (Caminero et al., 2025), movement dynamics (Duvivier et al., 2021), micro-gestures (Steptoe and Steed, 2012), attention shifts and physiological responses. Automation of collection and pre-processing pipelines need to be emphasized to support future clinical scalability and integration into time-constrained environments. While collecting this data is important, ensuring its clinical relevance is equally important.

103

**Objective 3: develop a clinician-oriented framework for the visualization and review of patient data**

104

105

106

107

108

109

110

The final objective is to support the clinician's decision-making and intervention process with a structured and interpretable data interface. This includes building a dashboard tool that offers real-time visualizations during live sessions and retrospective views of patient behavior for asynchronous review. The system will incorporate tools to annotate sessions, view trends, and optionally integrate sensitive data such as questionnaires or notes. A User-Centered Approach (UCA) (Kutzin et al., 2017; Filippeschi et al., 2019) will guide the design of HMD and tablet compatible modules to ensure usability, interpretability, and seamless integration into clinical workflows.

## 3 METHODS

111

112

113

114

115

This project adopts a mixed-methods approach, beginning with an interdisciplinary literature review and iterative stakeholder engagement. These foundational steps will inform the design of structured scenarios and guide the development of a data-oriented framework for IPCs. This framework will enable the systematic collection, interpretation, and restitution of behavioral and biometric data generated during XR-based mental health interventions.

116

### 3.1 Literature Review

117

118

119

120

121

The literature review will focus on three intersecting domains: (1) the use of XR technologies in healthcare settings (Worlikar et al., 2023), (2) strategies for managing mental health in clinical practice (Samira Abdul et al., 2024), and (3) the application of Artificial Intelligence (AI) and Machine Learning (ML) for analyzing biometric data (Greener et al., 2022; Altozano et al., 2025). This review will provide both theoretical grounding and empirical insights for the project.

122

123

Primary academic databases such as PubMed and Google Scholar will be used to ensure comprehensive coverage of relevant literature. The review will also incorporate existing research on immersive telehealth
platforms, including the work by Prié et al., *A Teleclinic for Neuro-psychological Assessment in the Metaverse* (Prié et al., 2025), which provides a foundational model for XR-based clinical environments.

The outcome of the review will be a consolidated list of relevant behavioral and biometric indicators that can be integrated into IPC scenarios. These indicators will serve as a basis for establishing a structured and continuous dialogue with mental health professionals, ensuring clinical relevance and facilitating co-design of XR interventions.

## 3.2 Dialogue with mental healthcare professionals

To ensure clinical relevance and usability healthcare professionals will be engaged throughout the project via focus groups and semi-structured interviews. Focus groups will be conducted with professionals from different fields within mental health such as child psychiatrists, psychomotricians and nurses. During these we will be able to discuss their experiences with the mental health system, current workflow, digital mental health tools, current diagnostic gaps and the perceived potential of IPCs. In parallel, individual semi-structured interviews will allow deeper analysis and case by case study of the data necessity of the clinician's field. Insights gathered from this ongoing engagement will directly inform the development of the data framework.

## 3.3 Framework development for data collection

Based on the two previous subsections, a framework for data collection will be co-developed in iteration with the clinicians to specify the types of information recorded, the methods of collection and their relevance to mental health monitoring. Data will be collected via a standard HMD with hand tracking and an integrated eye tracking system. These data are based on a project that we did previously using the Unity game development framework and either the XRIntegration Toolkit or the Oculus Integration Toolkit packages. With the technical infrastructure in place, the project will advance to design and validate scenario-based tasks tailored to elicit clinically meaningful responses.

## 3.4 Scenario development with practitioners

As shown in Table 1, Scenario-based design will ensure the ecological and diagnostic validity of the data collected through tasks within the IPC. These scenarios are tailored to elicit data relevant to early screening, symptom monitoring, and stratification of mental health conditions.

<table><caption>**Table 1.** Examples of clinician-designed scenarios for IPC environments</caption><thead><tr><td>Scenario</td><td>Description</td><td>Data Captured</td><td>Clinical Use</td></tr></thead><tbody><tr><td>Social Stress Simulation</td><td>Patient navigates a virtual group conversation</td><td>Gaze aversion, voice tremor, body motion</td><td>Detection of social anxiety and avoidance behavior</td></tr><tr><td>Cognitive Flexibility Task</td><td>Color-word Stroop test in immersive form</td><td>Response time, error rate, hesitations</td><td>Executive dysfunction, attention deficits</td></tr><tr><td>Virtual Interview</td><td>Semi-structured avatar-led conversation</td><td>Facial affect, verbal fluency, eye contact</td><td>Indicators of depression or psychomotor slowing</td></tr></tbody></table>

To further anchor these scenarios in validated clinical practice, psychometric tools such as the PHQ-9 (Patient Health Questionnaire-9) (Ford et al., 2020), GAD-7 (Generalized Anxiety Disorder-7) (Spitzer et al., 2006), BDI (Beck Depression Inventory) (Osman et al., 2004), and STAI (State-Trait
154 **Anxiety Inventory) (Julian, 2011)** will be integrated into the scenarios as either pre-session screening
155 or post-session reflective assessments. These tools first serve to get a baseline on self-reported symptoms
156 against behavioral and biometric data. On a second note, they are here to evaluate whether immersive tasks
157 provoke consistent responses in individuals with known profiles.

158 Altogether, these steps are setting the groundwork for a structured, clinician-informed framework
159 that supports immersive mental health prevention through XR-based data collection, interpretation, and
160 restitution.

CONCLUSION

161 This research is part of a broader exploration of how IPCs can play a transformative role in mental healthcare
162 by combining XR environments with clinically relevant data workflows. The aim is to enhance early
163 intervention and therapeutic support and to bridge the gap between clinicians and immersive technologies
164 with interpretable insights derived from real-time or ongoing behavioral and biometric inputs. The process
165 will engage iteratively with mental health professionals, from indicator selection to scenario and dashboard
166 interface design, this work will link clinicians and their workflows while building toward scalable and
167 adaptive digital care tools. Through the integration of XR technologies and data-driven dashboards, the
168 project seeks to create a structured framework that supports individualized care and broader mental health
169 prevention strategies.

REFERENCES

170 Altozano, A., Minissi, M. E., Alcañiz, M., and Marín-Morales, J. (2025). Introducing 3DCNN ResNets for
171 ASD full-body kinematic assessment: A comparison with hand-crafted features. *Expert Systems with
172 Applications* 270, 126295. doi:10.1016/j.eswa.2024.126295

173 Alvari, G., Vallefuoco, E., Cristofolini, M., Salvadori, E., Dianti, M., Moltani, A., et al. (2024). Exploring
174 Physiological Responses in virtual reality-based interventions for Autism Spectrum Disorder: A Data-
175 Driven Investigation

176 Bell, I. H., Nicholas, J., Alvarez-Jimenez, M., Thompson, A., and Valmaggia, L. (2020). Virtual reality as
177 a clinical tool in mental health research and practice. *Dialogues in Clinical Neuroscience* 22, 169–177.
doi:10.31887/dcns.2020.22.2/lvalmaggia

179 Caminero, R. V., Méndez-Herrero, L. A., Díaz-Pernas, F. J., Campo, J. C. d., Rodríguez, M. A., and
180 Martínez-Zarzuela, M. (2025). Telekin - Tele-rehabilitation System for Musculoskeletal and Cognitive
181 Disorders using Natural Movement Interface Devices. 198–203

182 Colizzi, M., Lasalvia, A., and Ruggeri, M. (2020). enPrevention and early intervention in youth mental
183 health: is it time for a multidisciplinary and trans-diagnostic model for care? *International Journal of
184 Mental Health Systems* 14, 23. doi:10.1186/s13033-020-00356-9

185 Dan Shao, I.-J. L. (2020). Acceptance and Influencing Factors of Social Virtual Reality in the Urban
186 Elderly

187 Duvivier, V., Perreira da Silva, M., Prié, Y., and Nantes Université, C. L. U. . . N. F., Ecole Centrale Nantes
188 (2021). *A conceptual framework for immersive behavioral data exploration assisted by a deep learned
189 representation of data*. Tech. rep.

190 Felsberg, D. T., McGuirt, J. T., Ross, S. E., Raisbeck, L. D., Howard, C. K., and Rhea, C. K. (2025).
191 Clinician perspectives on virtual reality use in physical therapy practice in the United States. *PLoS ONE
192* 20, e0320215. doi:10.1371/journal.pone.0320215
193 Filippeschi, A., B., Filippo, R., Emanuele, J. V., Juan Manuel, L., Lorenzo, et al. (2019). Evaluation of diagnostician user interface aspects in a virtual reality-based tele-ultrasonography simulation. *Advanced Robotics* 33, 840–852. doi:10.1080/01691864.2019.1635909. Publisher: Taylor & Francis

197 Ford, J., Thomas, F., Byng, R., and McCabe, R. (2020). Use of the Patient Health Questionnaire (PHQ-9) in Practice: Interactions between patients and physicians. *Qualitative Health Research* 30, 2146–2159. doi:10.1177/1049732320924625

200 Freeman, D., Reeve, S., Robinson, A., Ehlers, A., Clark, D., Spanlang, B., et al. (2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. *Psychological Medicine* 47, 2393–2400. doi:10.1017/s003329171700040x

203 Greener, J. G., Kandathil, S. M., Moffat, L., and Jones, D. T. (2022). enA guide to machine learning for biologists. *Nature Reviews Molecular Cell Biology* 23, 40–55. doi:10.1038/s41580-021-00407-0. Publisher: Nature Publishing Group

206 Guo, J., Guo, S., Tamiya, T., Hirata, H., and Ishihara, H. (2016). enA virtual reality-based method of decreasing transmission time of visual feedback for a tele-operative robotic catheter operating system. *The International Journal of Medical Robotics and Computer Assisted Surgery* 12, 32–45. doi:10.1002/rcs.1642. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/rcs.1642

210 Hu, H.-z., Feng, X.-b., Shao, Z.-w., Xie, M., Xu, S., Wu, X.-h., et al. (2019). enApplication and Prospect of Mixed Reality Technology in Medical Field. *Current Medical Science* 39, 1–6. doi:10.1007/s11596-019-1992-8

213 Isernia, S., Di Tella, S., Pagliari, C., Jonsdottir, J., Castiglioni, C., Gindri, P., et al. (2020). EnglishEffects of an Innovative Telerehabilitation Intervention for People With Parkinson’s Disease on Quality of Life, Motor, and Non-motor Abilities. *Frontiers in Neurology* 11. doi:10.3389/fneur.2020.00846. Publisher: Frontiers

217 Julian, L. J. (2011). Measures of anxiety: State-trait anxiety inventory (STAI), Beck anxiety inventory (BAI), and hospital anxiety and depression scale-anxiety (HADS-a) 63, S467–S472. doi:10.1002/acr.20561

219 Kutzin, J. M., Milligan, Z., and Chawla, S. (2017). EnglishUsing Simulation to Conduct a Usability Study of Wearable Technology. *Clinical Simulation In Nursing* 13, 64–70. doi:10.1016/j.ecns.2016.12.003. Publisher: Elsevier

222 Lee, A. G. (2024). enAI- and XR-Powered Digital Therapeutics (DTx) Innovations. In *Digital Frontiers - Healthcare, Education, and Society in the Metaverse Era* (IntechOpen). doi:10.5772/intechopen.1006619

225 Osman, A., Kopper, B. A., Barrios, F., Gutierrez, P. M., and Bagge, C. L. (2004). Reliability and Validity of the Beck Depression Inventory–II With Adolescent Psychiatric Inpatients. *Psychological Assessment* 16, 120–132. doi:10.1037/1040-3590.16.2.120

228 Prié, Y., Vigier, T., Acier, D., Bonneville, H., Georges, M., Thevenet, S., et al. (2025). A teleclinic for Neuro-psychological Assessment in the Metaverse. In *Abstracts and Workshop Proceedings, IEEE 2025* (Saint-Malo, France)

231 Rauschnabel, P. A., Felix, R., Hinsch, C., Shahab, H., and Alt, F. (2022). What is XR? Towards a Framework for Augmented and Virtual Reality. *Computers in Human Behavior* 133, 107289. doi:10.1016/j.chb.2022.107289

234 Samira Abdul, Ehizogie Paul Adeghe, Bisola Oluwafadekemi Adegoke, Adebukola Adejumoke Adegoke, and Emem Henry Udedeh (2024). Mental health management in healthcare organizations: Challenges and strategies - a review. *International Medical Science Research Journal* 4, 585–605. doi:10.51594/imsrj.v4i5.1157
238 Schlak, A. E., Rosa, W. E., Rushton, C. H., Poghosyan, L., Root, M. C., and McHugh, M. D. (2022). en-239 USAn expanded institutional- and national-level blueprint to address nurse burnout and moral suffering 240 amid the evolving pandemic. *Nursing Management* 53, 16. doi:10.1097/01 NUMA.0000805032.15402. 241 b3

242 Spitzer, R. L., Kroenke, K., Williams, J. B. W., and Löwe, B. (2006). A brief measure for assessing 243 generalized anxiety disorder: The GAD-7. *Archives of Internal Medicine* 166, 1092–1097. doi:10.1001/ 244 archinte.166.10.1092

245 Steptoe, W. and Steed, A. (2012). Multimodal Data Capture and Analysis of Interaction in Immersive 246 Collaborative Virtual Environments. *Presence: Teleoperators and Virtual Environments* 21, 388–405. 247 doi:10.1162/PRES_a_00123

248 Thøgersen, M., Andoh, J., Milde, C., Graven-Nielsen, T., Flor, H., and Petrini, L. (2020). 249 EnglishIndividualized Augmented Reality Training Reduces Phantom Pain and Cortical Reorganization 250 in Amputees: A Proof of Concept Study. *The Journal of Pain* 21, 1257–1269. doi:10.1016/j.jpain.2020. 251 06.002. Publisher: Elsevier

252 WHO (2022). *World mental health report: Transforming mental health for all* (World Health Organization)

253 Worlikar, H., Coleman, S., Kelly, J., O'Connor, S., Murray, A., McVeigh, T., et al. (2023). Mixed 254 Reality Platforms in Telehealth Delivery: Scoping Review. *JMIR Biomedical Engineering* 8, e42709. 255 doi:10.2196/42709