# An Intelligent Cloud-Based Platform for Effective Monitoring of Patients with Psychotic Disorders

Ilias Maglogiannis, Athanasia Zlatintsi, Andreas Menychtas, Dennis Papadimatos, Panayiotis P. Filntisis, Niki Efthymiou, George Retsinas, Panayiotis Tsanakas, Petros Maragos

► To cite this version:

Ilias Maglogiannis, Athanasia Zlatintsi, Andreas Menychtas, Dennis Papadimatos, Panayiotis P. Filntisis, et al.. An Intelligent Cloud-Based Platform for Effective Monitoring of Patients with Psychotic Disorders. 16th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2020, Neos Marmaras, Greece. pp.293-307, 10.1007/978-3-030-49186-4_25. hal-04060682

HAL Id: hal-04060682

https://inria.hal.science/hal-04060682

Submitted on 6 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

This document is the original author manuscript of a paper submitted to an IFIP conference proceedings or other IFIP publication by Springer Nature. As such, there may be some differences in the official published version of the paper. Such differences, if any, are usually due to reformatting during preparation for publication or minor corrections made by the author(s) during final proofreading of the publication manuscript.

Correction to: Chapter “An Intelligent Cloud-Based Platform for Effective Monitoring of Patients with Psychotic Disorders” in: I. Maglogiannis et al. (Eds.): *Artificial Intelligence Applications and Innovations*, IFIP AICT 584, https://doi.org/10.1007/978-3-030-49186-4_25

The original version of this chapter was revised. The first name of one of the authors inadvertently contained a typo. The author's first name has been corrected to “Niki.”
# An intelligent cloud-based platform for effective monitoring of patients with psychotic disorders*

I. Maglogiannis<fnref n="1" />, A. Zlatintsi<fnref n="2" />, A. Menychtas<fnref n="1" />, D. Papadimatos<fnref n="1" />, P.P. Filntisis<fnref n="2" />, N. Efthymiou<fnref n="2" />, G. Retsinas<fnref n="2" />, P. Tsanakas<fnref n="2" />, and P. Maragos<fnref n="2" />

**Abstract.** The therapy of patients with psychotic disorders (i.e., bipolar disorder and schizophrenia) could benefit from the constant monitoring of their physiological and motor parameters. In this paper, we present an innovative and advanced cloud based platform that facilitates the effective monitoring of such patients. A commodity smartwatch is used for biosignal and motion data collection at a 24/7 basis. The paper describes the technical details of the implemented application both on the smartwatch and the cloud server side. Technical challenges regarding the upload, the storage and the battery constraints of the smartwatch are also discussed, along with the initial results regarding data visualization and processing.

**Keywords:** Healthcare platforms · Biosignal collection · Patient monitoring · Psychotic disorders · Motion analysis

## 1 Introduction

A key issue in the therapy of patients with psychotic disorders (i.e., bipolar disorder and schizophrenia) is the ability to constant monitor their physiological and mental status, even 24/7 if that is possible. In this research work, we present an innovative and advanced computer-based platform that facilitates the effective monitoring and can support the relapse prevention in such patients. This research work is done within the e-Prevention research and development project that is coordinated by the National Technical University of Athens (https://eprevention.gr/) and it is co-funded by the European Commission and Greek National funds.

The innovative key offerings of the envisioned system, are as follows: 1) long-term continuous recordings of biometric signals and motion data through simple

<footnote n="1">¹ Department of Digital Systems, University of Piraeus, Greece</footnote>

<footnote n="2">² School of ECE, National Technical University of Athens, 15773 Athens, Greece
{imaglo,amenychtas}@unipi.gr, {nzlat,panag,maragos}@cs.ntua.gr
dpapadimatos@gmail.com, {filby,nefthymiou,gretsinas}@central.ntua.gr</footnote>

<footnote>* This research has been co-financed by the European Regional Development Fund of the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH – CREATE – INNOVATE (project code: T1EDK-02890, acronym: e-Prevention).</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 1: The overall architecture of the proposed system.</figcaption></figure>

wearable sensors (i.e., smartwatch), 2) a portable device (tablet) installed in the patient's residence, which records short-term audio-visual videos of the patient while communicating with the doctor; and by using affective computing methodologies is able to understand the emotional status of the patient, and 3) automatic and systematic storing and management of the captured data in a Cloud Infrastructure. The stored data are then processed through machine learning and signal processing techniques in order to detect changes and patterns, to facilitate the prediction of clinical symptoms and side effects of the patient's medication.

The final user of the platform, who is the attending physician, is able to continuously monitor and optionally also annotate the data and receive information about the patient's daily mental and physiological status by observing the data from the wearable sensors and the emotion analysis of the short videos. Additionally, the computational intelligence algorithms incorporated in the platform enable big data processing and statistical analysis and prediction of salient events regarding the patient's daily routine along with the corresponding visualizations. The overall architecture of the proposed e-Prevention system is illustrated in Fig. 1.

## 2 Related Work

The evolution and rapid dissemination of mobile and wearable consumer technology, such as smartwatches and fitness trackers, has created unique opportunities for personalized data collection in an unobtrusive and even affordable way. Furthermore, the enormous technological advances have enabled the reliable recording and quantification of a large number of behavioral and biometric indexes through their sensors [23, 25]. Such sensors usually include accelerometers, gyroscopes and heart rate monitors among others for measuring and detecting the user's motion/kinetic activity, sleep patterns and autonomic function [9, 26], opening this way the possibility for non-intrusive acquisition of activity data.

Over the last years, wearable technology due to the inclusion of these sensors has in fact become more proximal to human activity, having as an eminent result
to become increasingly acceptable even in healthcare. As early as 2006, the Insti-
tute of Medicine recognized the potential to transform mental health services by
providing more continuous and precise information on patient-specific behavior,
symptoms, and medication side effects. In order to do that, novel methods are
demanded to be developed, to transform raw data into knowledge and then use
this in turn to support personalized interventions [6] so as to transform hospital-
centered healthcare practice to proactive, individualized care. Behavioral and
biometric indexes have been already used in general medicine and sports and
nowadays the evidence shows that they could be introduced into clinical psy-
chiatry [6], as well. Specifically, psychosis is a spectrum of disorders that under-
lie different etiopathogenic mechanisms acting on the Central Nervous System
(CNS), leading to common symptoms [31]. Despite extensive research over the
last 60 years in neurobiology and neurophysiology of psychotic disorders, their
cause remains unclear and reliable biometric indexes for the diagnosis and pre-
diction of the course of the psychotic symptomatology have not yet been found.
Based on the fact that the process of psychosis is continuous and relapse is a
"biological" process that evolves over time [13, 21, 32], it would be expected to
observe changes in such indexes that are related and likely precede the onset
and/or worsening of psychotic conditions. To mention a few, some of the typical
early warning signs of mental illnesses include parkinsonian type symptoms, i.e.,
rigidity, tremor, jerking arm movements, or involuntary movements of the limps,
awkward gait, unusual gestures or postures, decreased physical activity and so-
cial interactions, abnormal sleep patterns [1, 5, 20]. The continuous monitoring
assessed through such wearables, opens the way for more precise and personalized
digital interventions, and complement by enabling detection of early signs of ill-
ness relapse, medication adherence or even treatment efficacy and thus may help
increase the number of positive clinical outcomes in mental healthcare [10, 28].

Major research areas in psychiatry [7, 8, 17, 22] have suggested employing
technological advancement for accurate and continuous monitoring of patients
to reduce the impact of mental illness on a patient’s daily activities, for early
diagnosis and prevention of psychotic relapses so as to increase the effectiveness
of treatments. For instance, sensor technology, has been used in order to un-
derstand patterns of daily behavior that may be indicative of trends or changes
in factors related to mental health, including sleep, mood, and stress [15, 24],
or to characterize a diverse array of psychopathology, from schizophrenia and
depression to general mental health [11]. In [30] a combination of accelerometer
and location sensors could detect relapse in depressive symptoms before patients
reported such changes. It has also been found that such sensor technology could
predict changes in affect and behavior (for a review, please see [15]).

## 3 System Architecture

The availability of commodity consumer devices (i.e., smartwatches) has eased
motion data collection but in the context of the e-Prevention project the require-
ment to collect and transmit constantly large amounts of raw biosignal data did
not fit the normal use of available consumer devices. We evaluated a number of
available devices and selected the Samsung Gear S3 Frontier as the user data col-
lection device. Compared to other smartwatches that were tested, the Samsung
Gear has the ability to store and send data from acceleration, angular velocity
and pulse sensors, providing also the heartbeat interval period value (R-R inter-
val), while it has a large storage capacity (4 GB) capable of storing data for a few
days and a WiFi interface removing the need for a smartphone for connectivity.

In addition, the smartwatch has the ability to run specific applications to
collect data and upload them to a server. An application was developed in the
Tizen Studio environment using technologies to implement Tizen applications
(Tizen Hybrid Application) [2]. Hybrid Application applications consist of sec-
tions implemented either with native C code to achieve efficient operation or
with javascript code to implement interfaces with available Tizen Advanced UI
(TAU) libraries. The application implements smartwatch data collection func-
tions and transmission to a cloud infrastructure. Due to the large volume of data
produced, various compression techniques without loss of information were con-
sidered taking into account the limited computational and energy resources of
the smartwatch. Analytics were also incorporated to gather critical smartwatch
operating parameters such as power levels, free storage and network availability.

**3.1 Architecture of the smartwatch application**

The architecture of the smartwatch application is illustrated in Fig. 2 and con-
sists of the following modules:

1. The Sensor Service module that interacts with the smartwatch sensors to collect data and store them temporarily on the device. Due to performance requirements it is implemented as a Tizen native service.

2. The Data Transmission Module (Network Service) that compresses and trans-
fers the data to the server; also implemented as a Tizen native service.

3. The Control Service that coordinates the data collection and transmits units taking into account the state of the internet connection and the smartwatch energy status. It also aggregates the smartwatch operating parameters ( analytics) for sending to the server.

4. The Tizen Web App interface that provides the user with basic information on the functionality of the application. The implementation utilizes advanced web design technologies based on TAU wearable interface libraries.

**3.2 Biosignal data collection and energy consumption of smartwatch**

The sensor data are provided by the Tizen operating system in various ways
dependent on the characteristics of the measured parameters. Extensive testing
during the pilot development of the system leaded to the optimum data sampling
rate in relation to the optimum time between successive battery charges.

Table 1 shows the sampling frequencies that were selected according to the
requirements of the project's medical team to ensure continuous recordings be-
tween charges of approximately 24 hours.
<figure><img src="image_4.png" /><figcaption>Fig. 2: Smartwatch application architecture.</figcaption></figure>

<table><caption>Table 1: Biosignal collection parameters.</caption><thead><tr><td>Data Parameter/hr</td><td>Sampling frequency</td><td>Data volume</td></tr></thead><tbody><tr><td>Linear Acceleration</td><td>20 Hz</td><td>ca. 7 MB/hr</td></tr><tr><td>Angular Velocity</td><td>20 Hz</td><td>ca. 7 MB/hr</td></tr><tr><td>Pulse data</td><td>5 Hz</td><td>ca. 1.5 MB/hr</td></tr></tbody></table>

Step data and sleep duration are also stored as calculated by the smartwatch, while GPS tracking was not used as it significantly increased power consumption and reduced the time required between successive loads. The actual output data amounts to about 300 MB/user/day, where the day of use includes approximately 22 hours of continuous recording. The data is compressed at about 100-120 MB/user/day. About 20 users require about 2.5 GB of storage per day. In total, 24 months of compressed logging require approximately 1.7 TB of storage.

## 3.3 Data collection module

The requirements for collecting large volumes of data from smartwatch sensors require an appropriate software architecture based on efficient event-driven sensor management. Tizen uses an asynchronous programming interface (API) for sensors based on function callbacks that allow data transfer when available. The data collection module initializes the sensors required and attaches appropriate management routines to transfer the data and store it along with a timestamp in the smartwatch storage.

Data are organized into separate folders that contain files depending on the sensor type as well as timing information. Each data structure corresponds to a smartwatch usage period between two charging events. The above organization allows smartwatch data to be cached if there is no network connection or data upload problems on the server. Also independent storage in separate structures reduces the risk of data loss.
<figure><img src="image_5.png" /><figcaption>Fig. 3: Flowchart of data collection and transmission.</figcaption></figure>

## 3.4 Data transmission module

The watch has advanced connectivity to local wireless WiFi networks but the volume of data requires efficient management of the data upload. The upload process is demanding on watch resources especially concerning energy, so it only takes place when the smartwatch is in the charging state.

Due to the high volume, the data collection module ensures efficient storage with minimal processing (Fig. 3). The data are stored in the smartwatch's storage in chronological order and are organized into groups corresponding to the intervals between device charging events. The available smartwatch storage capacity (approx. 4 GB) allows continuous recording and storage of measurements corresponding to 8 days. If the smartwatch is connected to a network, then as described below the data are transferred to the cloud and this time is renewed.

Transmission is organized into packets and compressed with the DEFLATE algorithm as implemented by the zlib library [3]. This library is a reliable solution with efficient use of the limited smartwatch computing resources and it is supported by a middleware on the node.js server platform. Running the decompression processes on the server concurrently allows data to be received from multiple smartwatches simultaneously.

## 3.5 Architecture of the biosignal collection server

The server uses the ‘~ okeanos’ national public infrastructure-as-a-service (IaaS), see Fig. 4 for the server architecture. The application is implemented with node.js technologies and uses the NoSQL MongoDB database. The application runs in an isolated Docker container and is structured as follows:

1. Authentication Controller that implements system resources access policies according to user roles.
<figure><img src="image_6.png" /><figcaption>Fig. 4: Biosignal Collection server architecture.</figcaption></figure>

2. Device Controller that implements the process of registering new devices in the system and gathering watch information on the functionality of the connected devices.

3. Biosignals Controller that manages the efficient transfer of biosignal data.

4. Admin Controller that implements the system management and monitoring.

5. The WEB UI module that constitutes the interface of the administrators with the application described in Sec. 4.1.

6. Web REST interface (REST interface) that routes REST calls from the devices and Web UI application to the appropriate module.

# 4 The system in practice

## 4.1 Web UI for system administrators

The Administrator Web UI interface (Fig. 5) allows devices to monitor and access downloaded data. It consists of a client subsystem implemented with front-end responsive frameworks such as Bootstrap and Vue.js and a server subsystem implemented with node.js. The portal features are: i) access to user log data, ii) new device registration, iii) e-mail notification system to administrators when a device has not uploaded data for more than 1 day, and iv) watch analytics.

The alert system uses information from the Device Controller module concerning device connectivity to generate alerts to administrators. The administrator interface allows access to functionality concerning activity alerts, user data and smartwatch analytics. It also provides access to data collected from the accompanying Android application used to collect activity data, which are manually annotated by the users.

## 4.2 Data management

As mentioned above, for efficient use of limited smartwatch computing resources, data is stored as provided by the available sensors. Motion and pulse tracking
<figure><img src="image_7.png" /><figcaption>Fig. 5: Administrator Web UI.</figcaption></figure>

sensors produce data at regular intervals depending on the sampling rate, while step and sleep data are provided by the smartwatch operating system comprising a very small amount of data compared to the original sensor data.

Various methodologies for storing the original sensor data were tested, but prior to the development of the system, priority was given to recording all available information in a way that allowed for thorough processing to eliminate errors caused by sensor operating problems and smartwatch software. In addition, in the pilot phase, the storage of the .csv format was chosen to save the processing time so as not to limit the processing capabilities.

The sheer volume of data produced and the requirement for flexible ways of securely accessing data create high storage requirements; thus was mitigated by the use of proven data compression techniques. Structuring the application of the server as a set of “micro services” for data transfer, without loss of information, entails the transfer of biosignals from the efficient processing and storage process. As part of the pilot process, the data is stored either uncompressed in the form of .csv files for immediate local processing or as compressed packets for a specific data collection time period ready for off-server transfer.

The need to store a large volume of data led to the creation of a new virtual machine in the ~okeanos infrastructure for the ultimate storage of the compressed data, and it is available for further processing through an appropriate sftp interface.

# 5 Initial data processing and results

## 5.1 Qualitative Analysis

We performed an exploratory data analysis (i.e., kinetic and heart rate data), employing both traditional signal processing techniques, such as short-time analysis, as well as more sophisticated non-linear methods; i.e., multi-scale fractal and non-linear dynamics analysis, in order to extract descriptors that efficiently convey behavioral and biometric information. Such non-linear descriptors have been shown to be of importance for the modeling of a series of other 1D signals, see as for instance [16, 19, 27, 33]). Our first preliminary results validate the efficiency of the features, thus they could be employed for building tools that
<figure><img src="image_8.png" /><figcaption>Fig. 6: Visualization of user movement and heart rate data, i.e., acceleration data (top row), gyroscope data (middle row) for all three axis (x, y, z-axis) and data collected by the heart rate monitor (i.e., heart rate and RR-interval data).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7: Short Time Energy of the acc and gyr signals (x-axis) for two different states (walking and other, e.g., resting, standing), using a 10 sec sliding window.</figcaption></figure>

can be used for more complex pattern extraction, i.e., behavioral changes, that
can be correlated with mental health issues or relapses. Figure 6 shows examples
of the raw data from the accelerometer (acc), gyroscope (gyr) and heart rate
monitor (hrm) collected from a control user. All data is shown for a 24 Hour
interval (where the horizontal blue line indicates when the user was asleep), so
obvious differences can be observed.

**Short-time energy analysis:** We extracted the short-time energy, to explore differences when the user is walking, sleeping, or doing other activities. Figure 7 presents the calculated short-time energy, on the acc and gyr signals (x-axis), showing two 10-min. intervals corresponding to the different states of walking vs. other activity. The short-time energy is computed using a 400-sample window applied on the signal and provides insights for the signal content. We can observe that the short-time energy presents a multitude of peaks in the acc and gyr data during walking, and fewer peaks with higher values of energy
<figure><img src="image_10.png" /><figcaption>Fig. 8: Raw signals and mean and std (error bars) of the multiscale fractal dimension distribution of the acc (x-axis) and the hrm data for walking, other and sleep (the MFDs are shown for 60-sec analysis windows, updated every 30 sec).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(a) Walking $D_e = 4$, $T_d = 5$, $D2 = 1.94$, $SE = 0.94$</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Sleeping $D_e = 8$, $T_d = 3$, $D2 \sim 7$, $SE = \infty$</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) Other $D_e = 8$, $T_d = 44$, $D2 = 0.44$, $SE = 0.01$</figcaption></figure>

<caption>Fig. 9: Reconstructed attractors for the three states. Each additional dimension above 3 and up to 6 is shown using different colors. $D2$ is the correlation dimension, and $SE$ the sample entropy.</caption>

when doing other activities. On the contrary, during sleep, the energy was very low, mainly including the energy of the noise produced by the sensors, thus we omitted showing the respective figures. Concluding, the results are intuitive, and show good discriminative abilities for the different states.
**Multi-scale fractals:** Regarding non-linear analysis, we explored an efficient algorithm [18] that measures the short-time fractal dimension, based on the Minkowski-Bouligand dimension [12]. Fractal dimension *D* is between 1 and 2 for 1D signals; and the larger the *D* is, the larger the amount of geometrical fragmentation of the signal. We conducted our analysis at multiple time scales, since real-world signals do not have the same structure over all scales [18], and we measured the multiscale fractal dimension (MFD) profile as a descriptor for 10-min. intervals of continuous data. Figure 8 shows MFD profiles measured for the three different states, i.e., walking, other and sleeping for the acc and the hrm data. The various profiles show quite distinctive patterns for every state and each type of data. Specifically, we note high fractal dimension up to *D* = 1.2 and *D* = 1.4 for the acc data during walking and other, while it gets up to *D* = 1.6 for the acc-sleep data. Regarding the hrm data, the higher fractal dimension is noticed for sleeping (up to *D* = 1.4), while walking is around *D* = 1 getting higher and up to *D* = 1.3 for larger scales *s*.

**Nonlinear dynamics and attractor reconstruction:** Next, we considered that the human movements, as well as the human heart rate system, constitute *separate* nonlinear dynamics systems. The two variables that must be calculated in order to unfold the attractor is the time delay $T_d$ and the dimension $D_e$. The most common methods for calculating the lag $T_d$ constitute of either selecting the first local minimum of the mutual information between the original signal $s(n)$ and its time-delayed version $s(n + T_d)$ or the first zero of the autocorrelation function. The dimension $D_e$ is increased until the false neighbours vanish, in which case we consider the attractor has unfolded [4].

Figure 9 shows the reconstructed attractors for short 1-min. intervals. During walking we observe that the reconstructed phase space exhibits a quite clear topology, due to the fact the user performs specific movements. During sleeping, due to the low amount of movement, the sensor noise is prevalent, with few random movements, while during other activities the phase space is dominated by seemingly random movements without a pattern. We generally notice that a nonlinear dynamical systems approach could be proven to be useful for deducing activities and other behavior throughout everyday life. The reconstruction of the attractor is a necessary step towards computing the nonlinear dynamics of the system. Such values are for example the correlation dimension of the reconstructed attractor [14], and the sample entropy [29]. In Figure 9 we also show those values; $D2$ is the correlation dimension and SE the sample entropy.

## 5.2 Quantitative Analysis

**Database Description:** In order to perform the quantitative analysis, we created a small dataset by randomly sampling 10 volunteers. We then selected from each volunteer 120 time intervals of 10-min. length, i.e., 40 intervals during which the subject was sleeping, 40 intervals during walking/running, and 40 intervals during other activities (rest, standing, working). The selected intervals were random, and we only made sure that the data collected during these intervals were valid, since in some cases, sensor malfunctions resulted in data loss, or the heart rate monitor was unable to accurately detect the heart rate. The analysis that
<table><caption>Table 2: Results on the collected data for various linear and non-linear features.</caption><thead><tr><th>Features</th><th>Acc(%)</th><th>Features</th><th>Acc(%)</th></tr></thead><tbody><tr><td>STE</td><td>72.17</td><td>STE+CH</td><td>74.58</td></tr><tr><td>MFD-G</td><td>72.08</td><td>STE+MFD-G</td><td>76.92</td></tr><tr><td>MFD-A</td><td>74.16</td><td>STE+MFD-GA</td><td><strong>77.66</strong></td></tr><tr><td>MFD-H</td><td>60.50</td><td>STE + MFD-GAH</td><td>77.33</td></tr><tr><td>CH</td><td>65.30</td><td>STE + MFD-GAH+CH</td><td>77.58</td></tr></tbody></table>

follows use this dataset and attempts to identify useful cues/features, that can be used for accurately detecting the state of a person (sleeping, walking, running), so that later on can generalize to other physiological states. It should be finally mentioned that according to the the act for data protection a written consent was collected for all participating users providing the motion data that are described and analyzed in Sec. 5.

**Experimental Evaluation:** In order to validate the performance of the features explored in Sec. 5.1, we employed a Support Vector Machine (SVM) with an RBF kernel, using 5-fold cross validation. Table 2 presents the results for different subset combinations.

With STE we denote the statistics (mean and std) of short-time energy analysis performed on the: 3 accelerometer (x,y,z-axis), 3 gyroscope (x,y,z-axis), and 1 heart rate signals. By taking the mean and std of the short-time energy calculated over each interval we have a total of 14 features for each interval. CH denotes the correlation dimension and the sample entropy calculated by performing delay embedding with $D_e = 4$ for the kinetic data and $D_e = 2$ for the heart rate data. MFD-G, MFD-A, and MFD-H denote features from the multi-scale fractal analysis of the gyroscope (x-axis), the accelerometer (x-axis), and the heart rate, respectively. In order to reduce the number of features extracted by the MFD analysis (900 dimensions for kinetic data and 300 for heart rate), we perform linear sampling, picking a total of 30 points for the heart rate and 32 points for the accelerometer and gyroscope MFD analysis.

As we can see from the accuracy (%) results in Table 2, both the traditional speech processing methods, such as the STE analysis, as well as the multi-scale fractal analysis can be used in order to discriminate between the different states. The same can be observed using the CH features which combined with STE, increase the accuracy of the system. Our results show that the extracted descriptors examined in this paper can provide useful insights and can be used in order to build a tool able to assess the state of the user.

# 6 Conclusions

Concluding, the novelty of this work is the implementation of a cloud based platform for the extraction of key markers and indicators that can possibly assist the medical personnel in monitoring patients with mental disorders and predict psychotic symptom's relapses and adverse medicine side effects. This is done
initially by long-term recordings of biosignal and motion related indexes and further processing. As this project evolves the collected data will be combined with the short-time videos and their affective analysis as well as with responses of patients to standard questionnaires. Nevertheless, the results so far prove the feasibility of our research effort in the continuous collection of biometric and motion data along with their prognostic value.

## References

1. Schizophrenia.com, http://schizophrenia.com/, http://schizophrenia.com/

2. https://developer.tizen.org/ko/development/guides/native-application/location-and-sensors/device-sensors (2019)

3. http://zlib.net (2019)

4. Abarbanel, H., Frison, T.W., Tsimring, L.S.: Obtaining order in a world of chaos [signal processing]. IEEE Signal Processing Magazine 15(3), 49–65 (1998)

5. Association, A.P., et al.: Desk reference to the diagnostic criteria from DSM-5®. American Psychiatric Pub (2014)

6. Aung, M.H., Matthews, M., Choudhury, T.: Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies. Depression and anxiety 34(7), 603–609 (2017)

7. Bauer, M., Wilson, T., Neuhaus, K., Sasse, J., Pfennig, A., Lewitzka, U., Grof, P., Glenn, T., Rasgon, N., Bschor, T., et al.: Self-reporting software for bipolar disorder: validation of chronorecord by patients with mania. Psychiatry research 159(3), 359–366 (2008)

8. Blum, J., Magill, E.: M-psychiatry: Sensor networks for psychiatric health monitoring. In: Proc. of The 9th Annual Postgraduate Symposium The Convergence of Telecommunications, Networking and Broadcasting, Liverpool John Moores University. pp. 33–37. Citeseer (2008)

9. Boletsis, C., McCallum, S., Landmark, B.F.: The use of smartwatches for health monitoring in home-based dementia care. In: Int’l Conf. on Human Aspects of IT for the Aged Population (2015)

10. Collins, P.Y., Patel, V., Joestl, S.S., March, D., Insel, T.R., Daar, A.S., Bordin, I.A., Costello, E.J., Durkin, M., Fairburn, C., et al.: Grand challenges in global mental health. Nature 475(7354), 27–30 (2011)

11. Cornet, V.P., Holden, R.J.: Systematic review of smartphone-based passive sensing for health and wellbeing. Journal of biomedical informatics 77, 120–132 (2018)

12. Falconer, K.: Fractal geometry: mathematical foundations and applications. John Wiley & Sons (2004)

13. Gaebel, W., Frick, U., Köpcke, W., Linden, M., Müller, P., Müller-Spahn, F., Pietzcker, A., Tegeler, J.: Early neuroleptic intervention in schizophrenia: areprodromal symptoms valid predictors of relapse? The British Journal of Psychiatry 163(S21), 8–12 (1993)

14. Grassberger, P., Procaccia, I.: Measuring the strangeness of strange attractors. Physica D: Nonlinear Phenomena 9(1-2), 189–208 (1983)

15. Javelot, H., Spadazzi, A., Weiner, L., Garcia, S., Gentili, C., Kosel, M., Bertschy, G.: Telemonitoring with respect to mood disorders and information and communication technologies: overview and presentation of the psyche project. BioMed research international 2014 (2014)
16. Kokkinos, I., Maragos, P.: Nonlinear speech analysis using models for chaotic systems. IEEE Transactions on Speech and Audio Processing **13**(6), 1098–1109 (2005)

17. Koutsouleris, N., Davatzikos, C., Bottlender, R., Patschurek-Kliche, K., Scheuerecker, J., Decker, P., Gaser, C., Möller, H.J., Meisenzahl, E.M.: Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification. Schizophrenia bulletin **38**(6), 1200–1215 (2011)

18. Maragos, P.: Fractal signal analysis using mathematical morphology. In: Advances in electronics and electron physics, vol. 88, pp. 199–246. Elsevier (1994)

19. Maragos, P., Potamianos, A.: Fractal dimensions of speech sounds: Computation and application to automatic speech recognition. JASA **105**(3), 1925–1932 (1999)

20. Maxhuni, A., Muñoz-Meléndez, A., Osmani, V., Perez, H., Mayora, O., Morales, E.F.: Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients. Pervasive and Mobile Computing **31**, 50–66 (2016)

21. McCandless-Glimcher, L., McKnight, S., Hamera, E., Smith, B.L., Peterson, K.A., Plumlee, A.A.: Use of symptoms by schizophrenics to monitor and regulate their illness. Psychiatric Services **37**(9), 929–933 (1986)

22. McGorry, P., Keshavan, M., Goldstone, S., Amminger, P., Allott, K., Berk, M., Lavoie, S., Pantelis, C., Yung, A., Wood, S., et al.: Biomarkers and clinical staging in psychiatry. World Psychiatry **13**(3), 211–223 (2014)

23. Menychtas, A., Tsanakas, P., Maglogiannis, I.: Automated integration of wireless biosignal collection devices for patient-centred decision-making in point-of-care systems. Healthcare technology letters **3**(1), 34–40 (2016)

24. Mohr, D.C., Zhang, M., Schueller, S.M.: Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual review of clinical psychology **13**, 23–47 (2017)

25. Panagopoulos, C., Malli, F., Menychtas, A., Smyrli, E.P., Georgountzou, A., Daniil, Z., Gourgoulianis, K.I., Tsanakas, P., Maglogiannis, I.: Utilizing a homecare platform for remote monitoring of patients with idiopathic pulmonary fibrosis. In: GeNeDis 2016, pp. 177–187. Springer (2017)

26. Patel, S., Park, H., Bonato, P., Chan, L., Rodgers, M.: A review of wearable sensors and systems with application in rehabilitation. Journal of neuroengineering and rehabilitation **9**(1), 21 (2012)

27. Pitsikalis, V., Maragos, P.: Analysis and classification of speech signals by generalized fractal dimension features. Speech Communication **51**(12), 1206–1223 (2009)

28. Reinertsen, E., Clifford, G.D.: A review of physiological and behavioral monitoring with digital sensors for neuropsychiatric illnesses. Physiological measurement **39**(5), 05TR01 (2018)

29. Richman, J.S., Moorman, J.R.: Physiological time-series analysis using approximate entropy and sample entropy. American Journal of Physiology-Heart and Circulatory Physiology **278**(6), 2039–2049 (2000)

30. Saeb, S., Zhang, M., Karr, C., Schueller, S., Corden, M., Kording, K., Mohr, D.: Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study. Jour. of Medical Internet research **17**(7) (2015)

31. Van Os, J., Kapur, S.: Schizophrenia. seminar. Lancet **374** (2009)

32. Wiersma, D., Nienhuis, F.J., Slooff, C.J., Giel, R.: Natural course of schizophrenic disorders: a 15-year followup of a dutch incidence cohort. Schizophrenia bulletin **24**(1), 75–85 (1998)

33. Zlatintsi, A., Maragos, P.: Multiscale fractal analysis of musical instrument signals with application to recognition. IEEE Transactions on Audio, Speech, and Language Processing **21**(4), 737–748 (2012)