# A teleclinic for Neuro-psychological Assessment in the Metaverse

Yannick Prié, Toinon Vigier, Didier Acier, Hélène Bonneville, Manon Georges, Solène Thevenet, Samuel Bulteau

► To cite this version:

Yannick Prié, Toinon Vigier, Didier Acier, Hélène Bonneville, Manon Georges, et al. A teleclinic for Neuro-psychological Assessment in the Metaverse. IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), Mar 2025, Saint-Malo, France. hal-04907281

HAL Id: hal-04907281

https://hal.science/hal-04907281v1

Submitted on 22 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A teleclinic for Neuro-psychological Assessment in the Metaverse

Yannick Prié*

Nantes Université

Toinon Vigier*

Nantes Université

Didier Acier†

Nantes Université

Hélène Bonneville*

Nantes Université

Manon Georges*

Nantes Université

Solène Thevenet‡

CHU Nantes

Samuel Bulteau‡

CHU Nantes

## ABSTRACT

Metaverse technologies allow new tele-medicine services and applications. Our interdisciplinary team conducted a user-centered design process, and implemented an immersive teleclinic for clinicians to carry out neuro-psychological testing sessions with distant patients. A session is a mix of discussions and desk-based or independent test activities, accompanied and monitored by the clinician with various dashboards. We carried out a preliminary evaluation focusing on usability, presence and acceptability for healthy participants playing the role of patients and practitioners. Results shows all the participants could project into using the system, but practitioners were more reluctant, due so some usability issues, and insufficient quality of current technology to observe patients.

**Index Terms:** Cognitive testing, Immersive Clinic, Metaverse, Neuropsychology, Teleconsultation, Telemedicine, Virtual Reality.

## 1 INTRODUCTION

The Metaverse in medicine is a hot topic, yet many of its foreseen applications appear as rebranding of existing technologies, and rarely take into account its fundamental social aspects. The “mental Metaverse” where various mental-health activities would be carried out by patients with both therapist and peers [1] is still at the agenda. In the TECNIS project, we investigated how to leverage social VR technologies to explore patient-practitioner encounter, focusing on making use of existing working VR applications — neuro-psychological tests for cognitive evaluation— so as to include them into social VR patient-practitioner scenarios. We have designed a first working example of what could be a clinic in the metaverse, with a focus on the journey of a patient that has an appointment for neuro-psychological testing.

## 2 RELATED WORK

Neuropsychological testing is crucial in many areas of medicine and the expertise of practitioners for conducting it is invaluable. VR testing is a whole new way to carry out cognitive testing in an embodied and ecological ways. Many VR tests have been proposed, showing their value [2], and commercial solutions already exist. Yet for the moment being VR testing is limited to using stand-alone applications within health facilities, patients carrying their tasks alone, either with a automated battery of several tests, or the accompaniment of a practitioner using a laptop or tablet.

VR can be considered as an effective “immersive therapeutic delivery device” [3]. VR telerehabilitation allows patients to self-administer VR activities at home [4], with good adherence and satisfaction, related to the easiness of staying at home vs going to a care center. Synchronous monitoring, if any, is only provided using audio, and no avatar-based / social VR interaction. VR teleconsultation has been shown feasible, but studies are limited for mental health testing or diagnosis, though it can provide practitioners with “a sense of “being at” the place and seems critical and beneficial for patient monitoring” [5]. Moreover, impersonating and interacting with avatars in simulation of social activities was “crucial to increasing the effectiveness of treatment” [6].

## 3 SYSTEM

Our vision is that we should integrate activities within immersive remote teleconsultations between medical practitioners and patients (Fig. 1). These should take place in *immersive teleclinics*, run by socio-economic players (hospitals, telemedicine operators). Teleclinics are mainly a set of *spaces*, e.g., welcoming spaces, social rooms, waiting rooms, consultation rooms, collective or individual activity rooms, *etc*. Patients accessing a teleclinic can either possess their own HMD, or necessitate that an assistant be present, who would bring the HMD, set up the access, and accompany. Activities can be proposed as external modules by third-party vendors.

Our prototype (Fig. 2) implements several workflows (Fig. 1). Both users begin in their home space, where they can consult their agenda, specify their avatar, *etc*. Then patients teleport in the waiting room, where several waiting activities are proposed, while practitioners go in the consultation room to prepare the upcoming session. When ready, practitioners go to the waiting room to welcome patients and invite them to join the consultation room. Once there, practitioners can lead the consultation, propose desk test activities, and discuss the results. At some point, they can invite patients to stand up and head towards the independent activity space, where standing testing can take place, before going back at the desk for other activities or data-based debriefing. We used Unity and Photon to develop the VR app for Meta Quest Pro. We collected events in a server-based MySQL database, and integrated practitioners' dashboards in VR as webpages generated with Node.js.

## 4 PRELIMINARY EVALUATION

A first, non-clinical, evaluation was carried out with healthy subjects to test the system from the patients' point of view, and with practitioners to get feedback from professionals. 10 pseudo-patients and 4 practitioners had two 30min sessions: 1/ initial consultation between clinician and patient, and tutorials for two cognitive tests; 2/ consultation focusing on neuropsychological assessment and debriefing. These sessions were conducted separately, with one co-experimenter playing the second role. The final test was carried out under more realistic conditions, bringing together 1 healthy subject at home, and 1 practitioner in the lab.

**Pseudo-patients** did not experience cybersickness in either session and usability was very good for both. Presence scores show very good levels of *Physical* and *Social Presence* for both sessions (mean > 4.3), and a good level of *Self Presence* (mean ≈

<footnote>*e-mail: [yannick.prie, toinon.vigier, helene.bonneville, manon.georges]@univ-nantes.fr, Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France</footnote>

<footnote>†e-mail: didier.acier@univ-nantes.fr, Nantes Université, Centre de recherche en Éducation de Nantes, CREN, UR2661, F-44000 Nantes, France</footnote>

<footnote>‡e-mail: [solene.thevenet, samuel.bulteau]@chu-nantes.fr, Nantes Université, CHU Nantes, INSERM, MethodS in Patients-centered outcomes and HEalth Research, SPHERE, F-44000 Nantes, France</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Simplified workflow for a consultation. Activity modules plugged into the clinic in gray.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: Left: The waiting room and the garden feature several chairs where patients can carry out various waiting activities, such as gardening, respiration, video watching, therapeutical information. When ready, the practitioner teleports into the room to meet the patient (right of the image), before they enter together the consultation room. Middle: The consultation room features a desk where desk-based testing activities can take place (here *Tower of London*). The practitioner can monitor the test from the control panel, and a shared screen allow to comment results with patients. The green teleportation mat behind the patient is where he needs to go to teleport to independent testing environments. Right: Independent environment for testing executive functions where the patient has to take books from a pile on the table and put them at the right place in the library. The practitioner is also present in the environment, observing the patient directly and by attending at behavioral indicators on her panel. Video: https://youtu.be/fLZfp1H8gas</figcaption></figure>

4). Acceptability was assessed only after the second session on -3 to +3 scales for the dimensions: Performance Expectancy, Effort Expectancy, Social Influence, Facilitating Conditions, Hedonic Motivation, Behavioral Intention. Scores for patients’ point of view are very good: mean > +2 for all dimensions except Performance Expectancy (mean = +1.85) and Social Influence (mean = +1.30).

**Practitioners** had no or minimal symptoms of cybersickness, and usability was acceptable or good for all in the 1st session, and in the same range for the 2nd one, except for one who rated it as poor. **Physical and Social Presence** were rated slightly positively during the first session but **Self Presence** got a mean score below 3. All dimensions of the presence were rated lower during the 2nd session (mean scores < 3). Acceptability score show important dispersion between practitioners. **Performance Expectancy, Social Influence and Behavioral Intention** got negative or neutral scores for all. **Habit** got neutral or negative scores for all except one with a slightly positive score. **Effort Expectancy** got positive scores for all and **Hedonic Motivation** got positive scores for all except one with a slightly negative score. **Facilitating conditions** got 3 medium positive, one slightly negative and one strongly negative scores.

# 5 CONCLUSION

TECNIS is the first of its kind application for neurocognitive consultation in the social VR. All participants were able to project how they would use the system in the future. Patients had no problem, and are willing to use it for neuropsychological appointment. Practitioner were more reluctant, explanations being related to remaining usability problems, the lack of knowledge on the value of VR

tests, as well as a quality of the avatars deemed insufficient for psychological interviews. This work is a first step towards a medical Metaverse based on social VR, and numerous challenges remain.

# REFERENCES

[1] S. Navas-Medrano, J. L. Soler-Dominguez, and P. Pons, "Mixed Reality for a collective and adaptive mental health metaverse," *Frontiers in psychiatry*, vol. 14, p. 1272783, Jan. 2023. 1

[2] N. Ribeiro, T. Vigier, J. Han, G. H. Kwon, H. Choi, S. Bulteau, and Y. Prié, "Three virtual reality environments for the assessment of executive functioning using performance scores and kinematics: An embodied and ecological approach to cognition," *Cyberpsychology, Behavior, and Social Networking*, vol. 27, no. 2, pp. 127–134, Feb. 2024. 1

[3] T. Maddox, C. Y. Sparks, L. Oldstone, M. Chibbaro, J. Sackman, E. Judge, R. Maddox, R. Bonakdar, and B. D. Darnall, "Perspective: The Promise of Virtual Reality as an Immersive Therapeutic," *Journal of Medical Extended Reality*, vol. 1, no. 1, pp. 13–20, Mar. 2024. 1

[4] H. Worlikar, S. Coleman, J. Kelly, S. O’Connor, A. Murray, T. McVeigh, J. Doran, I. McCabe, and D. O’Keeffe, "Mixed Reality Platforms in Telehealth Delivery: Scoping Review," *JMIR Biomedical Engineering*, vol. 8, no. 1, p. e42709, Mar. 2023. 1

[5] R. Chengoden, N. Victor, T. Huynh-The, G. Yenduri, R. H. Jhaveri, M. Alazab, S. Bhattacharya, P. Hegde, P. K. R. Maddikunta, and T. R. Gadekallu, "Metaverse for Healthcare: A Survey on Potential Applications, Challenges and Future Directions," *IEEE Access*, vol. 11, pp. 12765–12795, 2023. 1

[6] A. Cerasa, A. Gaggioli, F. Marino, G. Riva, and G. Pioggia, "The promise of the metaverse in mental health: the new era of MEDverse," *Heliyon*, vol. 8, no. 11, Nov. 2022. 1