# Brain Inspired Health Monitoring Supported by the Cloud

Fernando Luis-Ferreira, Sudeep Ghimire, Ricardo Jardim-Gonçalves

► To cite this version:

Fernando Luis-Ferreira, Sudeep Ghimire, Ricardo Jardim-Gonçalves. Brain Inspired Health Monitoring Supported by the Cloud. 6th Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Apr 2015, Costa de Caparica, Portugal. pp.273-281, 10.1007/978-3-319-16766-4_30. hal-01343494

HAL Id: hal-01343494

https://inria.hal.science/hal-01343494

Submitted on 8 Jul 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Brain Inspired Health Monitoring Supported by the Cloud

Fernando Luis-Ferreira¹,², Sudeep Ghimire¹,², Ricardo Jardim-Gonçalves¹,²

¹ Departamento de Engenharia Electrotécnica, Faculdade de Ciências e Tecnologia, FCT, Universidade Nova de Lisboa, 2829-516 Caparica, Portugal

² Centre of Technology and Systems, CTS, UNINOVA, 2829-516 Caparica, Portugal
{flf, sud, rg}@uninova.pt

**Abstract.** The health status of a person can be assessed by specialized professionals with the help of medical devices. The assessment, in real-time, would be useful as most of the times a person is not in presence of health professionals. But own assessment lacks precision and reaction time that could be risky in critical conditions. The pervasiveness of Internet connections and ubiquity of smart devices makes it possible to overcome that limitation. By using sensing capabilities of portable/wearable devices in conjunction with medical knowledge and clinical history it is possible to monitor a person's health status in real-time with reaction mechanisms. The objective is to keep track of collected evidence, and relate with existing knowledge for better assessment. This paper proposes a framework, inspired by the brain and physiology, capturing real-time health related information, at a person's location, submitting that to the cloud for reasoning and decision and triggers advice or request medical assistance.

**Keywords:** Cloud, Health Monitoring, Internet of Things, Brain Models

## 1 Introduction

A person living in modern societies is surrounded by an ever-growing number of technological devices [1]. The pervasiveness of computing and sensing devices offers different types of services that, ultimately, represent a source of data that can be used for the most different objectives and applications. As a person makes daily family or work activities, he/she is permanently exposed to an environment with which is necessary to properly interact and get valuable output for his/her life.

With the emerging developments of mobile devices and the Internet of Things becoming a reality, the offering of information provided by sensors or integrated services is growing fast. It would be desirable that a citizen could benefit from the best performance of all those systems, being supported by accurate context awareness thus improving quality of life, and security. In that direction, the information that devices and systems either portable or installed in surrounding environment (e.g. car, house, city, etc.) [2].

Most of animals have surveillance mechanisms, in which mammals stand out by the complexity of their sensorial organs and brain functions. Humans have, a sensorial capacity and processing at brain that detect and identify dangerous (e.g. fire in the surroundings) or risky situations (e.g. if you step there you may fall). In fact much of these mechanisms rely on how the brain makes assessment of the environment and
what are the present dangers and the risk of following actions. In the current environment, of applications in portable devices, (e.g. smartphones) such equipment should be empowered to identify the situations surrounding us and, eventually, warning us about identified risks in the proximities. That strategy is pursued by the emerging research area on context awareness [3]. Our current research, alike the brain, tries to identify our current physiological situation (e.g. temperature and heartbeat) and that of the surrounding environment (e.g. temperature, humidity) in order to evaluate our current health status and potential risks.

In the next section, the perspective of personal health monitoring powered by cloud with real-time data collection is explored. Section 3 provides research questions and hypothesis in close relation with the ongoing doctoral studies. Section 4 contains a discussion on methodology and concepts proposed by this research work and finally, section 5 presents conclusions and future work.

## 2 Personal Health Information Management using the Cloud

Sensors and sensor networks are in expansion with the need to obtain data from persons or from the environment. Devices with the ability to capture data can have different capabilities; they can be standalone (e.g. RFID tags) or be part of a larger setup (e.g. data from meteorological stations). That range of devices includes devices that need to be small and cheaper so that they can be installed in many places (e.g. clothes) but also devices that must operate alone for long periods of time (e.g. car identifiers). That means the need to make those devices as simple as possible with low consumption with consequent constraints in their capabilities. The need for additional computational resources is an identified problem for small sensors and similar devices as they have small or inexistent computational power, they also may be passive, and must have low energy consumption.

The usage of small (and cheaper) low consumption devices, with limited or inexistent computational power, can be empowered by the usage of external resources. For that, the cloud can be a differentiating factor to overcome the limitation of small devices, either sensors or actuators, by providing storage capacity and computational power. The cloud can be used to ensure that sensors can generate data that is stored remotely at the cloud and processed remotely where computational power is available. In what refers to personal health information management, the usage of services at the cloud can provide reasoning over knowledge bases to evaluate a person's health and to generate evaluation reports in real-time [4]. For a person doing normal life, wearable or local sensors can permanently monitor generic activities or a specific exercise (e.g. jogging, playing sports). Data being permanently sent to the cloud can be remotely analyzed and, if dangerous events are identified, the adequate warnings will be issued and, eventually, request medical assistance.

## 3 Followed Research Method

The observation of the current fitness and eHealth oriented devices and how they are operating makes us to question about its potential usefulness, for users, and how it can
be improved. The problems of real-time evaluation of a person and her circumstances
lead to the following question.

**Research Question:** How to adopt lessons learned from the brain and neurophysiology to computing devices in order to monitor, inform and assist users for self-help when necessary?

**Hypothesis:** We propose, by hypothesis, that if we monitor bodily functions along with environmental condition, it will be possible to establish an overview of a person's health status and warn about potential risks by utilizing computation in cloud.

## Background Observation

The brain and the nervous system permanently monitor a person's body. Information is acquired by our sensorial organs, is perceived and interpreted by the brain using existing knowledge, gained by learning and experience. The reasoning is based on a combination of different inputs, acquired by sensorial organs [5], which generate better assessment (e.g. I felt a spike on my feet, the sand is red so i have a cut and I am bleeding). Thus the brain uses multiple inputs to assess if there is danger for the person or if we only need to take care, avoiding multiple attendances to a doctor.

People use many devices that have computational power and embedded sensorial capacities. Smartphones have processing capacities, usually are equipped with a GPS, have a camera and can capture sound. They also have schedulers and planners where people can shape their foreseen activities for the day. Fitness devices provide information about physiological measurements (e.g. heartbeat) and are also capable of determining at what pace you are walking or running and in general the amount of calories being spent [6][7]. Based on all those capabilities, it is possible these days to determine if you are jogging (e.g. reading your electronic scheduler) and measuring your pace, or determine that you are in a work meeting. In those cases, an elevated heartbeat has different interpretations; in one case it is normal that you have higher heart frequency as you are running as in the other case you can be facing some disturbance, as a work meeting is not going as you would like. Using all this information it is possible to make an assessment on the health status of a person and, eventually, warn her to slow exercise or to control their reaction to a meeting.

In summary, with the existing devices and services, it is possible to monitor health status according to the context, and trigger events to inform or warn actions to be executed.

## 4 Methods and Associated Concepts

In this section we will first address the data processing flow to be followed in the proposed system, which starts with data acquisition from the real world until the actions to be performed in the real world.
<figure><img src="image_2.png" /><figcaption>Fig. 1 Data processing flow</figcaption></figure>

Fig. 1 shows the data processing flow. In the first step data is captured from the real world by various types of sensors or from other service providers. Thus collected data act as measurement of the parameter under consideration like temperature, humidity etc. In the next step, data infusion is performed leading towards more meaningful state like the temperature and humidity of an object at a particular instant, which leads to the generation of virtual sensors (VS). VSs will have data infusion rules such as if the temperature > x and humidity < y then activate VS. These are useful for generating simple instances of events, with multiple parameters, thus allowing more understanding of the real world and may lead to reflex-like reactions as appends in the human body. The next step termed “Pre-Processing” is used for pattern extraction and will remove the unwanted data from the collected samples of instances thus providing filter over useful data. The patterns detected in this phase are passed to for post-processing, where domain knowledge and rules, are applied over the collected samples to generate facts. Interpretation of facts leads to generation of actions to be taken, over which actors can decide the follow-up procedures.

In the processing flow diagram, we also see that in each stage actions are generated, which we term as “reflex”- immediate response to the state of the real world. In order to act over the reflexes users should use their own knowledge and understanding of the overall scenario. The final actions are suggested based on wider processing of the scenario thus allowing users to respond with lower level of understanding. It is important to define some important concepts for further understanding of the methodology.

**Sensor:** Sensors are devices that are capable of reading physical and physiological parameters. A sensor $S$ can be defined by a tuple $<D, C>$ where $D$ represents data-stream and $C$ the context of the sensor. $D$ will have a data-type and an ordered set of readings obtained at various time which can be represented as $D=<DataType, \{R_1, ..., R_n\}>$. $C$ is used for understanding the contextual use of the sensor and can have an URI to an external Sensor Networks Ontology<fnref n="1" /> or can also be linked with other instances of metadata resources.

**Virtual Sensor (VS):** VS are termed virtual because they don't have physical occurrence but are formed by logical infusion of various sensors. So, a VS can be defined by a relation $\odot$ s.t. $VS= S_x \odot S_y$, where $S_x$ and $S_y$ are sensors as defined previously. The relation $\odot$ can have different syntax and semantics as needed based on the type of data infusion that should be established between sensors. For instance if VS is used for taking into consideration humidity and temperature of a body part (e.g. arm, armpit) at the same time then a one to one union over the reading of the sensors for humidity and temperature will give an ordered set of reading for this new VS. And if the operator is to find the difference of the reading of two sensors (e.g. environment and body temperature) then the ordered difference between each element of the

<footnote n="1">¹ http://www.w3.org/2005/Incubator/ssn/ssnx/ssn</footnote>
reading of two sensors will give the dataset for the VS. Note that the context of the VS can always be obtained by the union of the individual contexts of the sensors forming the VS.

**Complex-Event Processing (CEP):** CEP is a widely accepted term which is used to combine data from multiple sources (e.g. sensors or virtual) to infer events by observing specific patterns over observed data. This is important to identify meaningful events or threats by observing a series of patterns and an application of possible rules. For instance if it is observed that a person is having heartbeat above a given threshold and the difference between the outside and body temperature crossed a defined limit, then by comparing to similar patterns recorded as dangerous for that person, a warning is issue signalling that some immediate action has to be taken. Also note that based on the concept of adding context to the sensors and propagated to VS will be determinant for CEP to take into consideration the context of the environment. Same patterns can raise different scenarios in different contexts thus adding more value to the CEP.

For further understanding of the concepts of sensors and VSs, let us consider a simplified scenario in which body temperature, heartbeat and Scheduler is regularly monitored to detect critical situations. Table 1 shows a snapshot of some of the situations.

<table><caption>**Table 1** – Data generated by the sensors and analyzed by the framework</caption><tr><td>Time [H]</td><td>9h00</td><td>9h30</td><td>10h00</td><td>10h30</td><td>11h00</td></tr><tr><td>T_body [°]</td><td>36</td><td>36.5</td><td>37</td><td>36.2</td><td>36.5</td></tr><tr><td>Heartbeat [BPM]</td><td>70</td><td>110</td><td>80</td><td>70</td><td>90</td></tr><tr><td>Scheduler</td><td>Jogging</td><td>Jogging</td><td>Showering</td><td>Work</td><td>Work</td></tr><tr><td>Result</td><td>0OK</td><td>1OK</td><td>2OK</td><td>3OK</td><td>4Warning</td></tr></table>

The data are read by individual sensor, which is infused to produce warnings by considering situations that can arise at a particular instance. This can be modelled with the definitions of sensors and VSs given before. Note that context of sensors (which can be position, data type, personal identity etc.) is not considered for simple understanding of the modelling process.

Sensors are:

$S_{T_body} = \{36,36.5,37,36.2,36.5\}$,

$S_{HeartBeat} = \{70,110,80,70,90\}$,

$S_{Time} = \{9, 9.30, 10, 10.30, 11\}$,

$S_{Scheduler} = \{(9, Jogging), (9.30, Jogging), (10, Showering), (10.30, Work), (11, Work)\}$.

Now VS can be modelled as follows:

$VS_{TBodyAndHeartBeat} = S_{T_body} \odot S_{HeartBeat}, \odot$ is s.t. for $i=1...n$
$VS_{TBodyAndHeartBeat} = \{(x_i, y_i)\}$ where $x_i \in S_{T_body}$ and $y_i \in S_{HeartBeat}$

In our example:

$VS_{TBodyAndHeartBeat} = \{(36,70),(36.5,110),(37,80),(36.2,70),(36.5,90)\}$

$VS_{TimeAndScheduler} = S_{Time} \odot S_{Scheduler}, \odot$ is s.t. for $i=1...n$
$VS_{TimeAndScheduler} = \{(z_i)\}$
where $x_i \in S_{Time}$, $(y_i, z_i) \in S_{Scheduler}$ and $x_i=y_i$

So we have: $VS_{TimeAndScheduler} = \{Jogging, Jogging, Showering, Work, Work\}$

We define: $VS_{Warning} = VS_{TBodyAndHeartBeat} \odot VS_{TimeAndScheduler}, \odot$ is s.t. for $i=1...n$
$VS_{Warning} = \{(x_i, y_i, z_i)\}$ where $(x_i, y_i) \in VS_{TBodyAndHeartBeat}$ and $z_i \in VS_{TimeAndScheduler}$ and
$x_i>36, y_i>80$ and $z_i=\text{work}$ or $y_i>150$
Which gives us $VS_{warning} = \{(36.5, 90, \text{work})\}$

This model shows the fusion of data from sensors and VS to detect warning. The $VS_{warning}$ thus detects the situation that the person is at work, has body temperature above 36 and heartbeat above 80 or if the heartbeat is above 150. In the scenario as shown in Table 1, this VS issues a warning for the last column, as expected.

## 4.1 Conceptual Architecture and Technology adoption

This section will present the conceptual architecture that is being followed for the realization of the system. The system should have layered architecture with well-defined interfaces and clear distinction between the independent components to be implemented at each stage of data processing flow. Fig. 2 shows the conceptual architecture with distinct layers divided, with components designated for specific functionality, and a clear demarcation on the cloud and private deployments.

<figure><img src="image_3.png" /><figcaption>Fig. 2 Conceptual Architecture</figcaption></figure>

**Data Collection Layer:** This layer is responsible for the collection of data via different sensors deployed in real world. Sensors interact with this layer through the adaptors. Adaptors are sensor specific implementations that act as sink for sensors, the output of which is data streams. The context of the sensor can be registered through admin configuration. This configuration will also define the adaptor for the particular type of sensor. This methodology allows easy integration of the sensor into the system with minimal changes for the new type of sensor. This layer also consists of VSs, which collect data from the outputs of the adaptors. The operation to be performed on the data for generation of outputs for virtual sensors is defined in the configuration details assigned to the virtual sensor. VS configs are the configuration files (XMLs), which will generate a new set of outputs by selecting parameters from other sensors or VSs. Since VS can also be specified with constraints, some simple events (or reflexes) will be generated and notified to the user through local application within the reach of user even in the offline mode.

**Pre-Processing Layers:** This layer implements CEP functionality and provides a module for publication and subscription to data. It also provides interface to query over data, patterns and events that have been generated by this or lower layers. Events Generation Module is an implementation of CEP and is used for sampling and filtering of data, the rules for which are defined in the configuration file (configs). Publish/Subscribe module acts as a subscriber of events from the data collection layer
and publisher for Post-Processing Layer. This module pulls data from the lower layer, passes it to modules for processing and pushes the pre-processed data towards post-processing layer for further treatment. The query interface is utilized by the application for providing notifications and statistics to the user.

**Post-Processing layer:** This is a heavy-duty layer of the overall system, where critical processing of the data is performed such as reasoning and learning. This layer makes use of domain knowledge created by experts and reasoning algorithms based on logical reasoning, and/or probabilistic reasoning, to extract new facts for understanding of the overall situation. This layer also implements a learning mechanism to keep the knowledge base updated with new facts generated by automated reasoning. This layer thus acts like a decision support system (DSS) to help the responsible person to have better insight of the real-world scenario.

**User-Interface:** This layer is used for interaction with the users supported by local applications and applications deployed as SaaS. Local applications are for immediate response and notification of reflexes upon detected events. Other applications are provided as SaaS, in the cloud, and deliver detailed analysis of the scenario, historical data and suggested actions to be taken. The 'admin' functionality allows configuration changes of the sensors and virtual sensors. The applications can be configured to trigger alarms to the user, case an emergency situation is detected.

## 4.2 Technology Adoption

For the purpose of system implementation this research work adopts services and enablers, being developed by projects, under the European FI-PP program like FI-WARE<fnref n="2" /> and FITMAN<fnref n="3" />, which provide enablers for data acquisition from the physical world, complex event processing, context aware data handling, data analysis etc. Shop-floor Data Collection<fnref n="4" /> (SFDC) is the implementation for data collection from various sensor networks (protocol agnostic methodology), and also for integration of tagged objects into the system. SFDC implements creation and configuration of the VSs and provides adaptors for some standard sensors with easy integration of new types of sensors. VSs are created by writing XML with defined schema. Complex Event Processing (CEP)<fnref n="5" /> and Publish/Subscribe Context Broker<fnref n="6" /> are configured and used in the data pre-processing layer. Both of these enablers are the important components, which are adapted for the realization of events generation and publish/subscribe modules respectively. In the implementation of post-processing layer it is very important to consider knowledge extraction over real time data, which we have adopted from some current research work as described in [8] and [9]. This module also makes use of ANN with semantics for implementing probabilistic learning algorithm, which have been inspired by some works as described in [10] and [11].

<footnote n="2">² http://www.fi-ware.org/</footnote>

<footnote n="3">³ http://www.fitman-fi.eu/</footnote>

<footnote n="4">⁴ http://catalogue.fitman.atosresearch.eu/enablers/shopfloor-data-collection</footnote>

<footnote n="5">⁵ http://catalogue.fi-ware.org/enablers/complex-event-processing-cep-ibm-proactive-technology-online</footnote>

<footnote n="6">⁶ http://catalogue.fi-ware.org/enablers/publishsubscribe-context-broker-orion-context-broker</footnote>
# 5 Concluding Remarks and Future Work

Research presented in this paper has potential impact in many societal and business aspects. New business models can be deployed to support real-time monitoring as proposed and, from the human side, a citizen can be monitored and assisted while performing a normal life. As by research presented in this paper, those objectives can be achieved by building computational methodology to integrate sensorial data and their interpretation, generating advice or asking for support as needed. This research work provides a framework for raw data collection from real world and multi-stage processing of data leading towards automated understanding of the surrounding. Proposed methodology will leverage cloud computing to utilize open linked data in the domain of health and big data processing for more reliable self-help health monitoring services. On-going validation and iterative testing of the implemented system in real scenarios is the most important task to be completed in a near future.

**Acknowledgments.** The research leading to these results has received funding from the EC 7th Framework Programme under grant agreement FITMAN nr 604674 (http://www.fitman-fi.eu)

# References

1. H. Sundmaeker, P. Guillemin, P. Fries, and S. Woelfflé, Vision and Challenges for realising the Internet of Things. European Commission, 2010.

2. F. Luis-Ferreira and R. Jardim-Goncalves, "A behavioral framework for capturing emotional information in an internet of things environment," in 11TH INTERNATIONAL CONFERENCE OF NUMERICAL ANALYSIS AND APPLIED MATHEMATICS 2013: ICNAAM 2013, 2013, vol. 1558, no. 1, pp. 1368-1371.

3. H.-W. Gellersen, Ed., "Towards a Better Understanding of Context and Context-Awareness," in Handheld and Ubiquitous Computing, vol. 1707, Berlin, Heidelberg: Springer Berlin Heidelberg, 1999, pp. 304-307.

4. F. Luis-Ferreira, "A Pervasive Computing Framework Supported by the Cloud for Personal Health Management," in Proceedings of CE2014, 2013.

5. K. S. Oie, T. Kiemel, and J. J. Jeka, "Human multisensory fusion of vision and touch: detecting non-linearity with small changes in the sensory environment," Neurosci. Lett., vol. 315, no. 3, pp. 113-116, Nov. 2001.

6. "Fitbit® Flex, One & Zip Wireless Activity & Sleep Trackers." [Online]. Available: http://www.fitbit.com/. [Accessed: 30-Jun-2013].

7. "Nike+ FuelBand. Tracks your all-day activity and helps you do more.. Nike.com." [Online]. http://www.nike.com/us/en_us/c/nikeplus-fuelband. [Accessed: 29-Jun-2013].

8. P. Angelov, "Nature-Inspired Methods for Knowledge Generation from Data in Real-Time," URL http://www.nisis.risk-technologies...., 2006.

9. J. Gama and P. Rodrigues, Knowledge discovery from data streams. 2010.

10. T. Mikolov and G. Zweig, "Context dependent recurrent neural network language model.," SLT, 2012.

11. S. Bowman, C. Potts, and C. Manning, "Recursive Neural Networks for Learning Logical Semantics," arXiv Prepr. arXiv1406.1827, pp. 1-10, 2014.