Patent ID: 12226196

DETAILED DESCRIPTION OF THE INVENTION

The present invention aims at providing a new method for predicting the HR of a user during a physical activity. For this, the present invention proposes the Personalized Exponential Model (PEM), an approach that learns to estimate the HR in real-time during physical exercises using 3-axis of an accelerometer signal, the user's demographics, and an HR starting point. In addition, it uses a continual learning approach that allows personalization in real-time without the need of storing past predictions on memory. In this sense, our method is able to better map the user's heart rate kinetics, which results in a more reliable HR estimation.

As previously discussed, estimating the heart rate (HR) from photoplethysmography (PPG) sensors embedded in wearable devices during physical exercises is a challenging task due to different issues. In addition, Motion Artifacts (MA) is the type of noise responsible for causing most of the wrong PPG-based HR estimations in wrist-type devices (e.g. smartwatches). Nonetheless, there are other problems such as sweating and loosely placed devices that also contribute to poor estimations and are quite hard to be detected by the current methods that deal with this issue. In order to illustrate how the noise affect the PPG signal, and consequently, the HR estimation,FIGS.1Aand1B show two examples of spectrograms obtained from two different PPG signals collected during a running workout session.

In the first spectrogram (FIG.1A), it is possible to distinguish the HR spectrum101from the MA102. In the second one (FIG.1B), it is possible to see strong disruptions103and104that are caused by MA and other types of noise. For this case, it is quite difficult to extract the spectrum for the HR curve.

InFIG.2, it is illustrated an example of poor HR estimation caused by noise in the PPG signal also during a running workout session. The plot shows the HR estimation throughout the time where the Y-axis204and X-axis205represent the HR in beats per minute (bpm) and the time in seconds (sec), respectively. The signal202indicates the reference HR obtained from an electrocardiogram (ECG) heart rate sensor attached to the user's chest, while signal203indicates a PPG-based HR estimation obtained from a smartwatch using a common method for reducing the impact of the MA in the PPG signal. Lastly, the gray area201represents the period of time in which the PPG signal is strongly affected by noise and any estimation achieved using this part of the signal is indicates as unreliable.

In order to tackle this problem, the present invention proposes a method for HR estimation during strong movements without using the PPG signal, which would be unreliable. In this way, the proposed method is applied to support the PPG-based HR estimation by replacing the poor estimation achieved within the gray area201by one closer to the reference signal202.

More specifically, the present invention proposes a method for predicting the heart rate (HR) of a user during a physical activity, the method comprising: monitoring the user during the physical activity with an wearable electronic device comprising a photoplethysmography (PPG) sensor, an accelerometer, a memory and a processor; wherein the memory is configured to store user demographic data, PPG data and accelerometer data detected by the PPG sensor and accelerometer, respectively; computing a PPG heart rate estimation and a PPG quality flag using the PPG data and accelerometer data, wherein the PPG quality flag indicates whether the PPG data is reliable or not; estimating an exponential heart rate estimation by applying the PPG heart rate estimation, the PPG quality flag and the user's demographic data on an exponential approximation model; if the PPG quality flag is reliable, outputting the PPG heart rate estimation; otherwise, outputting the exponential heart rate estimation.

Therefore, the present invention proposes a lightweight HR estimator method that, in real-time and in a recurrent way, estimates the next HR using as inputs 3-axis of an accelerometer signal, the user's demographics (age, gender, height, and weight), and the previous HR. The proposed estimator does not depend on the PPG signal to estimate the HR. However, it does not aim to replace PPG-based HR estimators, but to support them when the PPG signal is strongly affected by MA or any other type of noise, in particular, during physical exercises, which is when the majority of the wrong HR estimations occur.

Beyond the aforementioned input data, the proposed invention also receives, from the PPG-based approach, the HR estimated and the PPG signal quality information (hereinafter referred to as quality flag) that indicates if the signal is reliable or not. As such, we may see the proposed invention as another block that is attached to the PPG-based method output. According to the quality flag, it determines the value of the HR that is returned. In summary, if the quality flag is reliable, the PPG-based estimation will always be outputted. On the other hand, if it is unreliable, the HR estimation results from a composition obtained from both methods, preferably, with a smooth transition between them. In this sense, the present invention is preferably composed of a complete pipeline that includes:a preprocessing method to generate an activity level measurement;an Exponential Approximation (EA) model that estimates the HR;a continual learning approach to personalize the EA model in real-time without storing past predictions on the device's memory;a manager block that uses the quality flag to ensure a smooth transition between the exponential model and the PPG-based approach whenever the quality flag is unreliable.

This entire pipeline is named Personalized Exponential Model (PEM) and is designed to run in real-time in devices with strong memory restrictions, which is mandatory to allow its deployment in wearable devices.

More specifically, the first embodiment of the present method will be described with reference toFIG.3, which shows an illustrative pipeline that represents the present method according to a preferred embodiment of the present invention.

In order carry out the proposed invention, it is necessary to monitor the user during the physical activity via a wearable electronic device comprising a photoplethysmography (PPG) sensor and an accelerometer. Furthermore, the wearable device also comprises a memory for storing the user's demographics303, the PPG data301and accelerometer data302detected by the PPG sensor and accelerometer, respectively. Preferably, considering the memory limitations of wearable devices, the memory may store such data temporarily, being used for the calculation and being discarded or overwritten during future iterations of the method. However, the memory may alternatively be used for storing such data for longer periods, depending on the availability of memory in the electronic device.

In addition, the wearable electronic device may also comprise a processor for processing and executing the steps of the proposed method.

InFIG.3,312represents step of computing a PPG heart rate estimation304and a PPG quality flag305using the PPG data301and accelerometer data302. The PPG quality flag305indicates whether the PPG data301is reliable or not. The PPG-based method refers to the prediction of the heart rate and the PPG quality flag305with basis on both the signals provided by the PPG sensor and the accelerometer. However, the motion artifacts affect the results. Accordingly, the present invention proposes the use of a secondary heart rate estimation method.

The proposed method is composed of three main blocks: (1) a accelerometer preprocess block415, (2) a prediction block416, and (3) a manager block315. Particularly, the Personalized Exponential Model (PEM) is illustrated in pipeline316that shows all three blocks, how they communicate with each other, and how these blocks are integrated with a PPG based approach. The inputs are the PPG signal301, 3-axis of an accelerometer signal302, and the user's demographics303(age, weight, height, and gender). The outputs are the PPG-based HR estimation304and the quality flag305.

The Accelerometer preprocessing block415receives the accelerometer signal302and returns a value that represents the level of movement412and another value that indicates the intensity of the physical activity413. These values are sent to the prediction block416that also takes the user's demographics303, PPG-based HR estimation304, and quality flag305as input to produce the HR estimation based on the present model308and the personalization timer309, which indicates for how long the predictor model was personalized using the user's data. The personalization algorithm, which is carried out in block416, uses the user's data to update the model in order to improve its predictions. Finally, the manager block315gets as inputs the PPG-based HR estimation304, quality flag305, HR estimation based on the present model308, and the personalization timer309to generate the final HR estimation310and the final quality flag311. As previously mentioned, the exponential approximation block316is attached to the PPG-based approach and the manager block315ensures a smooth transition between the methods if the PPG signal is unreliable.

In view of the features above, the proposed invention presents the following main advantages.

It estimates the HR without using the PPG signal, which is the standard approach to obtain such measurement in wearable devices. As stated, this feature is important because it allows the method to cover poor estimations when the quality flag is unreliable. In other words, the accelerometer is not used to clean the PPG signal. Instead, the accelerometer is used as a main resource to predict the HR during a period of time in when the PPG is disturbed by noise.

It presents a strategy to personalize the prediction model in real-time using the own user's data. To do so, the present invention involves a learning approach to optimize the model whenever the PPG quality flag is reliable. Therefore, the model according to the present invention is capable of continuously learning, which is important to improve its predictions. It is important to note that the personalization algorithm does not hinder the model's performance in terms of computational time and memory burden, since it does not need to save past workout data.

The present method is agnostic to a PPG-based approach. In other words, the pipeline presented inFIG.3may be applied to improve any method in which the PPG signal is disrupted by noise.

Moreover, it demands less than 1 KB of memory for both HR estimation and personalization, which allows it to be easily embedded in wearable devices that have strong constraints in terms of memory.

Therefore, the present method can run completely on-device, which means that no cloud or local server is necessary for training the model. The model is constantly retrained on-device in real-time during physical activities, such, as a workout session. In addition, the on-device solution also has the advantage of keeping user's data privacy since no data needs to be shared or uploaded to a server neither for inference or training. All the steps of the method can be carried out on-device.

In the remaining of this section, the three main blocks depicted inFIG.3ill will be thoroughly described.

Accelerometer Preprocess and HR Prediction

InFIG.4, the interactions between the accelerometer preprocess block415and the HR prediction block416are presented in detail. Each part of this figure is described in the following subsections.

Accelerometer Preprocess Block (415)

The accelerometer preprocess block415executes the step of determining the level of movement412and the intensity of the physical activity of the user413using the accelerometer data302. Preferably, the accelerometer preprocess block415is designed to take 3-axis (x, y, z) of the accelerometer signal302in order to produce two values that represent the activity level (AL)412and intensity state of the workout413. To do so, this block is composed of three different modules: filtering and aggregating405, activity level computation406, and intensity handler407, which are thoroughly described as follows.

Filtering and Aggregating (405)

This module consists of two steps. First, the 3-axis of the accelerometer302are filtered using signal processing filters including, but not limited, a band-pass Finite Impulse Response (FIR) filter. This step aims to clean the accelerometer signal and reduce low frequency noise. Next, we aggregate the 3-axis of the accelerometer, which were filtered in the previous step, using a norm function such as:
αgg(αx,αy,αz)=αx2+√{square root over (αy2+α22)}  (1)
where αx, αyand αz, represent the 3-axis coordinates of the accelerometer.

The aggregated accelerometer value410is used as input to the next module406.

Activity Level Computation (406)

In this module, the aggregated signal410is integrated using a sliding window of time to obtain a raw value of the activity level ALraw, which may be approximated, for example, by the trapezoidal rule:

A⁢Lraw=∫t1t2agg⁡(a)≈Δ⁢a⁢∑k=1N-1agg⁡(ak)+agg⁡(an)+agg⁡(a0)2(2)
where a=[ax, ay, az] and N is the number of data points within the window, which depends on the accelerometer signal frequency. The ALraw411is sent to the intensity handle module407. The value indicated in412is the normalized version of the raw activity level. To achieve this value, we apply the clipping operation to ensure that 411 lies within [0,1]. Next, we integrate it and normalize the result by dividing it by the size of the time window (i.e. t2−t1), which is the maximum value that the integral may assume.
Intensity Handler (407)

This module takes as input the raw activity level411in order to compute the intensity of the physical activity413. This intensity may assume two states: low or high. To select this state, we define a threshold φithat is obtained from data. If φi<ALrawfor a predefined period of time, the state is low, otherwise, high. The selected state is used as input by the prediction block.

Prediction Block (416)

The prediction block416executes the step of estimating an exponential heart rate estimation308by applying the PPG heart rate estimation304, the PPG quality flag305and the user's demographic data303on an exponential approximation model408. It takes as inputs the user's demographics303, which consists of age, weight, height, and gender, the HR estimation at time t403, the activity level412, and the intensity state413to estimate the HR at time t+1. The block is composed of two modules: the exponential approximation (EA) model408and the continual learning409, which we detail in the following.

Exponential Approximation Model (408)

The exponential approximation (EA) model comes from the observation that the cardiovascular kinetics during physical exercised may be modeled by an exponential function over time [2, 3]. As such, we present a new dynamic and recursive model that is able to predict future HRs according to the following equation:

H⁢R⁡(t+1)=-[-HR⁡(t)]×e1τ(3)
wherein HR(t+1) is the predicted HR404and HR is defined as:
=DS×WT+{blow,bhigh}  (4)

In this equation, Dsis an array defined as [AL, BMI, age, male, female], where the AL is the activity level412and the remaining data are obtained from the demographics303, i.e., BMI is the body mass index (weight/height2) and the gender is mapped to male and female that may assume 1 or 0. In addition, the parameter b is determined according to the intensity of the physical activity413.

As we may see, the exponential approximation model has only eight trainable parameters to predict the HR: W, τ, blow, and bhigh. In summary, the predicted HR is obtained according to the previous HR and the, which is a measurement that indicates how much the predicted HR must increase or decrease from its previous value HR(t). This change is controlled by the exponential

e-1τ
that determines how fast or slow the HR(t+1) should increase or decrease. As previously stated, this is a recurrent model, i.e., we must know HR(t) in order to estimate HR(t+1) and so on. Nonetheless, if the very first HR value is unknown, our invention is able to estimate it from.
Continual Learning (409)

The continual learning block409implements a continual learning approach that uses the own user's data to improve the exponential approximation model by optimizing the eight trainable parameters previously described. The core idea is: if the HR obtained from the PPG-based approach is available and the quality flag is reliable, it is possible to fine-tune the exponential approximation model parameters in real-time. Essentially, this feature works as a model personalization for a given user.

Let us consider y and ŷ to be the HR estimated from the PPG-based approach and from exponential approximation model, respectively. In order to achieve a continual learning, we perform the gradient descent algorithm using the derivatives of the mean absolute error (MAE) throughout the time. The MAE is defined by the following equation:

MAE⁡(y,yˆ)=1N⁢∑i=0N❘"\[LeftBracketingBar]"yi-❘"\[RightBracketingBar]"(5)

Since the MAE is not continuous, which is problematic to use the gradient descent strategy, we use the module definition to compute the derivatives for yi−>0 with respect to Θ={W, τ, blow, bhigh}:

∂MAE⁡(y,yˆ)∂Θ=∂∂Θ[1N⁢∑i=0N(yi-)]=1N⁢∑i=0N∂(yi-)∂Θ=1N⁢∑i=0N∂yi∂Θ-∂∂Θ=1N⁢∑i=0N-∂∂Θ(6)

When yi−>0, the signal of the derivatives just needs to be inverted.

Now, the derivatives of the HR (t+1) are computed:

∂H⁢R⁡(t+1)∂Θ=∂∂Θ-∂∂Θ[(-HR⁡(t))⁢e-1τ]=∂∂Θ-[∂(-HR⁡(t))∂Θ⁢e-1τ+∂e-1τ∂Θ⁢(-HR⁡(t))]=∂∂Θ-[(∂∂Θ-∂H⁢R⁡(t)∂Θ)⁢e-1τ+∂e-1τ∂Θ⁢(-HR⁡(t))]=∂∂Θ-[∂∂Θ⁢e-1τ+∂e-1τ∂Θ⁢(-HR⁢(t))](7)

Therefore, for each parameter in Θ, there are the following derivatives:

∂HR⁡(t+1)∂W=Ds-Ds⁢e-1τ(8)∂HR⁡(t+1)∂τ=-e-1ττ2[Ds×WT-HR⁡(t)](9)∂HR⁡(t+1)∂{blow,bhigh}=1-e-1τ(10)
which result in the following parameters update rules:

Wt+1=Wt-α⁡(Ds⁢e-1τt-Ds)(11){bt+1low,bt+1high}={btlow,bthigh}-α⁡(e-1τt-1)(12)τt+1=τt-α⁢-e-1τtτt2[Ds×Wt⊤-HR⁡(t)](13)
where, α is the learning rate and the parameters blowand bhighare selected according to the intensity of the workout413.

The update rule equations are applied throughout the time. In other words, whenever the optimization is allowed, it performs one step of the gradient descent algorithm. In this sense, the personalization pipeline does not save any past workout data within the memory device. Instead, it performs the optimization in real-time.

Therefore, with basis on the exponential heart rate estimation308obtained from the exponential approximation model408, the method is capable of outputting a secondary estimation to the user whenever the PPG signal is unreliable. Hence, according to the present method, if the PPG quality flag305is reliable, the PPG heart rate estimation305is shown to the user. Otherwise, it shows the exponential heart rate estimation308.

Manager Block

Returning toFIG.3, the manager block315takes as input the HR estimated using the PPG-based method304, the quality flag305, the HR predicted by the exponential approximation model308, and the personalization timer309. The main goal of the manager block315is to ensure a smooth transition between the HR estimated using the PPG-based method304and the HR predicted by the exponential approximation model308when the quality flag changes from reliable to unreliable. Essentially, the pipeline illustrated inFIG.4with the accelerometer preprocess block415and the prediction block416is enough to predict the HR even without using the PPG signal. Nonetheless, since it is preferably intended to support the estimation provided by the PPG-based method, it turns out that transitioning between the methods when the quality flag changes may lead to strong discontinuity in the HR curve when both estimations disagree with each other. Therefore, the manager block315implements an algorithm in order to make the transition smoother and more natural.

There are two transition possibilities: (1) when the quality flag changes from reliable to unreliable and it is necessary to switch the estimation from the PPG-based approach to the exponential approximation model; (2) when the flag quality changes from unreliable to reliable and the PPG-based approach takes back the control over the HR estimation based on the exponential approximation model. In both cases, a stabilization equation is applied with basis on the exponential approximation that was described in the previous sections:

(t)=HRTarget-(HRTarget-(t-1))×e-Δ⁢tσ(14)
where:(t) is the transition HR;(t−1) is the HR obtained right before the transition starts;HRTargetis the target HR (from PPG-based approach or exponential approximation model) in which(t) must stabilize;Δt is one step of time that depends on the PPG signal frequency; andσ controls how fast or slow the transition should happens.

The idea of the transition equation is straightforward: it works as an interpolation function between the transition HR and the target HR. In addition to this equation, the algorithm waits tsseconds to stabilize the(t), if within this window of time the absolute error between(t) and HRTargetis greater than Es bpm, the algorithm forces it to assume the value of HRTarget.

In order to illustrate how the manager block315works, inFIG.5is illustrated an example of the HR estimation for part of a running session using and without using the manager. As it was previously described, the manager block315outputs the final HR310, which is basically the composition of the estimation obtained from the PPG-based approach and the exponential approximation model. As we may see,FIG.5presents the measured HR502, the HR obtained from the PPG signal506, and the exponential approximation model estimation501. When the time is 160s the quality flag305changes from reliable to unreliable, which is indicated by the grey area504. Therefore, the manager block315must switch from the PPG-based method to the exponential approximation model. However, as we may observe, the estimations506and501disagree too much with each other. In this case, the PPG-based method is far from the reference502. Thus, the HR curve indicated by503shows the result without applying the transition algorithm, while the HR curve indicated by505applies it. As we can note, curve503presents a significant discontinuity, since the transition is hard. On the other hand, curve505presents a smoother transition, which looks more natural and closer to a HR curve than503.

It is interesting to note that the transition enforced by the manager block315may lead to a higher MAE when compared to the same result without applying it. It is easy to note it by observing that the curve503is closer to curve501than curve506. Indeed, it is a trade-off between MAE and integrity of the HR curve. The magnitude of this error is controlled by the transition parameters σ, tsand Es. If the integrity is more important in this situation, the manager block315must always be applied as described in this section.

Beyond ensuring the smooth transition between the HR estimations, the manager block315also decides when to switch the estimations. The simplest approach would be switching them whenever the quality flag305changes from reliable to unreliable and vice-versa. However, since the model is personalized throughout the time, in order to increase performance, safeness, and integrity, preferably, a personalization threshold time (ρt) that the model must achieve before allowing it to take control of the final HR estimation may be defined. In other words, the threshold time (ρt) indicates the minimum required training time for the EA model before usage. The idea is straightforward, the manager block315takes as input the personalization timer309and if it is greater than ρtthe transition is allowed; otherwise, the manager always estimates the HR and quality flag305obtained in the PPG-based method according to block312.

A Step-by-Step Example of the Proposed Invention

In order to thoroughly illustrate the present method's workflow, a step-by-step example will be described to exemplify each block, input, and output presented inFIGS.3and4. The method outputs a new HR estimation every second. However, the length of the accelerometer and PPG input data depends on their frequency. The method works with any frequency, but for this example, we consider it as 25 Hz. Thereby, let us consider the workout illustrated inFIG.5. Our goal is to compute all input data for t=160 to predict HR(t=161), where the PPG quality flag is unreliable.

Step 0: Determining the Inputs

First of all, it is necessary to define the inputs for the method. InFIG.3is shown that the method takes as input the PPG signal301, 3-axis of an accelerometer signal302, and the user's demographics303. In addition, the proposed invention is agnostic to the PPG-based method, i.e., it works to any method that takes the PPG signal and returns a PPG heart rate estimation304and the PPG quality flag305. For this example, the PPG signal has eight channels. Therefore, we define each input as follows:

PPG signal301: since the signal is in 25 Hz and it has four channels, every second it produces a matrix P with shape equal to 25×8. For t=160 s:

P[t=160]=[20897482086539211501321121562156491215331320914742088684208959620864302114686211185921555022152341208843120856062087929208467521130622110150215344721501992084467208160320829102079733211138021084882151776214859320810512078252207415920711192109006210611421498362146545207553820728272066701206367321030272100164214471221414192069958206733520665052063429209976920969312142078213878820685152065962206812320649392098844209602721415682138215206795120653862068589209555220982772095490214142421381342069159206677020695632066545209791220951972141659213832120709102068655207015820672012097820209510921415932138315207115520689822071418206854020978112095165214148821382722071331206921020724022099428209763320949302141389213814420707152068447206522820622462095263209254221400362136794206848420663302065804206265620922292089390213700921337272061535205889720668852063607209091820880082136085213278620614432058689206861920652912089735208682221345572131244206094020581082071091206779720886702085726213232121290092061019205812120729002069550208783520848592130807212750820611392058224207663220733032086575208359221283212125071206007720572092079849207651020862062083238212704421237452060131205716520802732076872208638620833942127030212375720607412057784208064620771232086478208343721268852123581206110620581232079845207651620861652083179212682321235322061210205825420762952072957208643720834632127406212412220601152057261]

Accelerometer signal302: since the signal is in 25 Hz and it has 3 axis (x, y, z), every second it produces a matrix A with shape equal to 25×3:

A[t=160]=[-2.138785-6.275047-6.557664-2.639352-2.483673-4.92663-3.791374-9.630521-4.905075-0.342493-7.187563-11.4818993.683596-11.077135-17.0240596.689392-16.65522-17.1725528.013858-12.293823-11.3142455.848727-7.795908-5.3529517.45102-4.418879-3.6620417.764772-1.872934-0.847857.316896-0.052691-0.5843947.8893150.320938-0.3927897.87255-3.441696-2.9986112.054317-17.613243-15.8217413.932682-17.287516-11.2399994.737421-18.726944-9.9897790.383209-14.851744-4.579348-1.350812-9.570644-1.250219-2.375896-6.8043550.067062-3.695572-1.2574042.35434-3.240511-1.6980950.337703-3.472831-5.987641.281355-4.138657-10.928641-2.986635-0.589184-15.335545-6.4666520.42153-23.186539-7.407909]

User's demographics303: in this example, the physical activity is performed by a 23 year old woman, weighting 61.6 kg, and 1.63 m tall. Therefore, the demographics is represented by an array d=[BMI, age, male, female], where BMI=(weight/height2). Thus:
d=[23.2,23,0,1]

The PPG data P301and the accelerometer data A302are the inputs for the block312, which represents the PPG-based method. As previously mentioned, the model proposed by the present invention is agnostic to this method. Thus, what is important is that we need to get the PPG-based HR estimation304and the PPG quality flag305. For this example, they assume the following values HRppg(t=160)=149 and FlagPPG=unreliable, respectively, the FlagPPGbeing the quality indicator.

Step 1: Determining the Activity Levels

In this step, the activity levels are obtained, which represents the user's motion information. To do so, the Accelerometer preprocess block415is carried out, which is composed of the Filtering and aggregation405, Activity level computation406, and intensity handler407subblocks.

Following the workflow presented inFIG.4, firstly, a band-pass Finite Impulse Response (FIR) filter is applied to the accelerometer data A302with order and cut-off frequencies equal to 2 and (8 Hz, 12 Hz), respectively. It results in the Afmatrix illustrates as follows:

Af[t=160]=[0.130740.1116911.81941-0.8071740.466119-0.3020581.1406-0.698129-0.206817-0.947085-0.08175890.7603861.174410.312469-2.10798-0.4158080.3359041.74074-0.259664-1.97762-0.06985310.1082373.81550.837646-0.656075-2.61985-0.7170311.043441.02823-0.24824-0.708639-0.5518460.3142020.08383740.356188-0.2991650.203105-0.600293-0.126399-0.0884237-0.130125-0.07363230.423789-1.46469-1.5893-1.735652.95453.455061.67703-1.16453-1.64657-0.5064160.0344491-0.0128342-0.6702341.260760.909221.5553-1.85453-1.58975-1.616261.794291.502761.52287-1.94451-1.65235-1.167060.4488691.419030.716502-0.177439-1.434130.0710520.3872870.702242]

Next, each row of Afmatrix is aggregated using Equation (1) to obtain the aggregated accelerometer Agg array:

Agg[t=160]=[1.827520.9798141.353191.217312.43321.820961.995823.907862.794311.485820.9515390.472650.6462050.1737032.202454.866112.622930.5077491.692752.895782.84432.971621.891341.612940.805099]

The Agg array is the output410of the Filtering and aggregating module405, which is the input of the Activity level and computation module406.

The activity level raw ALraw411is computed through Equation (2), which results in the following array:

ALraw[t=160]=[2.074782.067572.059062.06282.060392.059742.078222.085382.079392.068112.052152.038092.019452.021282.049732.055462.039982.036512.04512.05312.062282.060572.05612.043592.03353]

The activity level AL412is also computed through Equation (2), but using the normalized version of the Aggarray, which results in the following:

AL[t=160]=[0.9108410.9117330.9126150.9134890.9143540.9152110.9160590.9168980.9177290.9180670.9136130.9109390.9035660.9045310.9054860.9064310.9024440.9034190.9043850.9053410.9062880.9072250.9081530.9071220.908051]

This array represents the activity level AL412, which is the output of the Accelerometer preprocess block415. Finally, the intensity of the workout IW413is determined within the Intensity handler module407. For this example, we set the threshold φi=1.25 and tϕ=5, which results the state high. It is worth to note that the algorithms to compute such measurements are not essential to this invention. Although they are important to the next block and module, they refer to preferential features of the invention.

Step 2: Predicting the HR and Updating the Trainable Parameters

As previously stated, the Prediction block416is a key component of our invention. It takes as input the AL=[t=160] array412, IW[t=160]413, demographics, d,303, and for this example, the previous HR(t)403is represented by HRPPG=(t=160) (since the goal is to predict HR(t=161), the previously HR available is the one obtained by the PPG. If we want to predict HR(t=162), for example, the previous HR would be HR(t=161) since the PPG signal will be unreliable).

The predicted HR is computed according to Equations (3) and (4). To do so, we need to define the parameters W, τ, blowand bhigh. For this example, they are defined as follows:
W={80,0.82,−0.9,−6.44,−0.09}
blow=80,bhigh=120, and τ=75

Next, from Equation (4), we obtain=139.16. Next, from Equation (3), we obtain the HR estimated from the EA model408for HREA—(t=161)=140 which represents the output308.

After estimating the HR, the next step is to perform the Continual Learning (personalization) module409. This module takes the EA model parameters (W, τ, blowand bhigh), the HREA, HRPPG, and FlagPPGto perform the personalization algorithm whenever the FlagPPG=reliable. Thus, for this example, the personalization is not performed since the FlagPPG=unreliable. However, let us suppose that in t=180 the FlagPPGbecomes reliable. In this case, to update (W, τ, blowand bhigh) it is necessary to perform Equations (11), (12), and (13) with a=5.0, which fine-tunes the EA Model according to the error between the HREA—(t=180) and HRPPG=(t=180). In addition, the personalization module returns the personalization timer309, which represents the amount of time in seconds that the EA Model was fine-tuned to the current user.

In this step, there are two features that further differentiate this invention from the prior art methods. First, we propose a new dynamic and recursive exponential function (Equation (4)) that is able to adapt to different user's demographics and workout intensity. Second, the personalization runs in real-time, without saving or using any user data in the device's memory. Due to memory limitations in wearable devices, this represents a great advantage when compared to the prior art.

Step 3: Delivering the Final HR Prediction

As previously mentioned, the proposed invention ensures that the EA Model and the PPG-based method work in a collaborative way. To do so, the last step is to perform the transition algorithm within Manager block315, which is important to ensure the smoothness and integrity of the HR final curve (seeFIG.5). Whenever the FlagPPGchanges from unreliable to reliable (or vice-versa), Equation (14) is applied to achieve the final HR prediction310. If there is no change, the transition is not applied. Therefore, for this example, we perform Equation (14) using the following parameters:tϕ=5, ρt=120, Δt=1/25, σ=25, ts=10, and Es=3. This results in(t=161)=139, which is the output310. The final quality flag311depends on the personalization threshold time (ρt) and the FlagPPG. If FlagPPG=reliable, the final quality flag is always reliable. On the other hand, if FlagPPG=unreliable and the personalization timer309is greater than ρt, the final quality flag is reliable. Otherwise, unreliable. For this example, we set ρt=120, which means that the EA Model must be personalized for the user for 120 seconds to be able to collaborate with the PPG-based method. To predict(t=162), all steps must be performed again.

This step is another difference from the prior art methods. In brief, the transition algorithm (Equation (14)) is important to avoid strong discontinuities in the HR final curve (seeFIG.5), which makes it unnatural and unreliable. In addition, it ensures a better prediction of the regression model since we force the model's personalization for ρtseconds.

Alternative Embodiments

As described in the previous section, the best mode of the present invention is to support the HR estimation of a PPG-based method. Nonetheless, beyond this application, this invention may be adapted to work for other problems that we describe in the following.

Predicting the HR Curve Using Few Data

It is possible to use the EA model to predict the HR curve for a given workout session using only a few data points sampled from the original activity level (AL) measurements. This scenario is illustrated inFIG.6where the measured HR curve601is sampled into multiple time points indicated by603. For each time point, we save the AL that is computed during the real-time estimation. From these values, we approximate the HR curve602using the EA model. This application may be interesting to predict the whole HR curve of a past workout by saving just a few data points related to it.

Saving Battery Life

Although PPG-based methods provide reliable HR estimation, the PPG sensor consumes much more energy than the accelerometer. The PPG may take up to 5000 times more energy than the accelerometer. Obviously, it directly affects the device's battery lifetime and hinder its autonomy. In this sense, the present invention may be adapted to assist the host device to save battery by intercalating the HR estimations from the PPG and the accelerometer signals according to a given rule. InFIG.7is illustrated a use case scenario that exemplify how the EA model may be adapted to save the device's battery. In this figure is presented the HR estimated from the PPG702and the accelerometer701signals. As we may see, they are both close to the reference HR represented702. The grey areas704show when the estimation from the accelerometer should be replaced by the PPG one. In this case, we define the following rule: in every ns seconds, we compare the performance of the HR obtained from the PPG-based and the EA model. If the difference between them is lower than E bpm, the device would stop the PPG measurements for the next ns seconds and the result HR estimation would come from the EA model. On the other hand, if the different is greater than E bpm, the estimation would keep coming from the PPG-based method until the difference be measured again (within ns seconds) and start the cycle again. In this example, the accelerometer is applied to estimate the HR in 68.5% of the time. Since it spends less battery than the PPG, this approach would save the devices battery.

Hardware Implementation

The example embodiments described herein may be implemented using hardware, software or any combination thereof and may be implemented in one or more computer systems or other processing systems. Additionally, one or more of the steps described in the example embodiments herein may be implemented, at least in part, by machines. Examples of machines that may be useful for performing the operations of the example embodiments herein include wearable electronic devices, such as smartwatches.

For instance, one illustrative example system for performing the operations of the embodiments herein may include one or more components, such as one or more microprocessors, for performing the arithmetic and/or logical operations required for program execution, and storage media, such as one or more disk drives or memory cards (e.g., flash memory) for program and data storage, and a random access memory, for temporary data and program instruction storage.

Therefore, the present invention is also related to a wearable electronic system for predicting the heart rate (HR) of a user during a physical activity comprising a processor, and a memory comprising computer readable instructions that, when performed by the processor, causes the processor to perform the method steps previously described in this disclosure.

The system may also include software resident on a storage media (e.g., a disk drive or memory card), which, when executed, directs the microprocessor(s) in performing transmission and reception functions. The software may run on an operating system stored on the storage media, such as, for example, UNIX or Windows, Linux, Android, and the like, and can adhere to various protocols such as the Ethernet, ATM, TCP/IP protocols and/or other connection or connectionless protocols.

As is well known in the art, microprocessors can run different operating systems, and can contain different types of software, each type being devoted to a different function, such as handling and managing data/information from a particular source, or transforming data/information from one format into another format. The embodiments described herein are not to be construed as being limited for use with any particular type of server computer, and that any other suitable type of device for facilitating the exchange and storage of information may be employed instead.

Software embodiments of the illustrative example embodiments presented herein may be provided as a computer program product, or software, that may include an article of manufacture on a machine-accessible or non-transitory computer-readable medium (also referred to as “machine-readable medium”) having instructions. The instructions on the machine accessible or machine readable medium may be used to program a computer system or other electronic device. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks or other type of media/machine readable medium suitable for storing or transmitting electronic instructions.

Therefore, the present invention also relates to a non-transitory computer readable storage medium for predicting the heart rate (HR) of a user during a physical activity, comprising computer readable instructions that, when performed by the processor, causes the processor to perform the method steps previously described in this disclosure.

The techniques described herein are not limited to any particular software configuration. They may be applicable in any computing or processing environment. The terms “machine-accessible medium”, “machine-readable medium” and “computer-readable medium” used herein shall include any non-transitory medium that is capable of storing, encoding, or transmitting a sequence of instructions for execution by the machine (e.g., a CPU or other type of processing device) and that cause the machine to perform any one of the methods described herein. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, and so on) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that the execution of the software by a processing system causes the processor to perform an action to produce a result.

Effect

In order to evaluate the effect of the method according to the present invention, experiments were carried out using three different exercise datasets performed on treadmills that were collected:Walking: a dataset containing 103 different subjects performing a three different walking sessions from 10 to 15 minutes totaling 4,625 minutes of data.Running: a dataset containing 136 different subjects performing a three different running sessions from 12 to 18 minutes totaling 5,464 minutes of data.Walk-and-run: a dataset containing 14 different subjects that intercalates in the same session one minute walking and two minutes running until reach 10 minutes. There are six different sessions totaling 881 minutes of data.

In each dataset, all subjects wear a Polar H10 chest strap to measure the reference HR using ECG and a Galaxy Watch 3 device to collect the PPG and accelerometer signals.

To play the role of the PPG-based approach, the latest HR estimation algorithm from Samsung Galaxy Watch 3 (GW3-HR), which uses multiple PPG channels and the 3-axis of the accelerometer to output the HR, was used. In addition, this algorithm returns a flag that classifies the PPG signal as reliable or unreliable (most of the time, the flag becomes unreliable because of MA). As a baseline method for comparison, a Multilayer Perceptron (MLP) was applied with 2 hidden layers containing 6 neurons each. This may be seen as an adaptation of the method proposed by McCoville et. al [3] that allows the use of an online personalization similar to the one described in the first embodiment. For all experiments, the methods' performances are evaluated according to the following metrics:The mean average error (MAE) in beats per minute (bpm). The lower the better.The workout coverage (COV) that is the % of time in which the quality flag is reliable. In other words, it shows the % of time the user sees a HR prediction on the device's display. The higher the better.The pass rate (PR) that is the % of time in which the MAE is lower than 10 bpm. The higher the better.

First of all, the accuracy of the HR prediction the EA model was assessed by comparison with the other ones. In Table 1 is presented the performance of each HR estimation method for each dataset considering the entire workout. In other words, the results consider both the reliable and unreliable quality flags and the PPG-based approach GW3-HR and the EA model are working individually, i.e., the manager block is not applied. As we may see, the GW3-HR estimation is generally better than the other methods, which is expected since it uses the PPG signal as primary source. Nonetheless, the EA model outputs a fair HR estimation, in particular for the walking dataset. In addition, its performance is better than the MLP for all dataset and metrics.

TABLE 1the performance of each method for eachdataset considering the entire workoutsGW3-HREA modelMLPMAEPRMAEPRMAEPRDataset(bpm)(%)(bpm)(%)(bpm)(%)Walking5.58 ±70.33 ±6.06 ±78.9 ±10.98 ±51.82 ±6.1333.955.6727.756.6234.67Running6.37 ±69.36 ±8.16 ±66.13 ±12.41 ±51.56 ±6.1927.434.8926.477.6127.59Walk-and-3.66 ±87.87 ±5.32 ±85.11 ±12.28 ±53.73 ±run2.7114.432.7716.293.0712.64

In Table 2 is shown the performance of the GW3-HR method stratified by quality flag. As we may note, the method's performance when the quality flag is unreliable is much worse than considering only the reliable one. As previously described, most of the time the unreliable flag occurs due to MA. In this way, our goal is to improve the GW3-HR performance presented in Table 1 by replacing the unreliable HR estimations by the ones obtained using our invention. For example, considering the walking dataset, our method would run around 13% of the time to cover the unreliable flag. The GW3-HR covers the rest of the time. Therefore, both methods work in a collaborative way, which is ensured by the manager block.

TABLE 1the performance of the GW3-HR stratified by quality flag. The coverageconsidering both quality flags together does not sum to 100% becausethe algorithm may take a few seconds to start the estimation.AllReliableUnreliableMAECOVPRMAECOVPRMAECOVPRDataset(bpm)(%)(%)(bpm)(%)(%)(bpm)(%)(%)Walking5.58 ±86.74 ±70.33 ±4.56 ±75.77 ±85.42 ±15.04 ±10.98 ±32.47 ±6.1328.6233.954.5434.8221.519.6323.5536.71Running6.37 ±90.1 ±69.36 ±4.37 ±66.6 ±85.88 ±14.04 ±23.5 ±39.55 ±6.1921.8527.433.035.0817.669.1231.9534.48Walk-and-3.66 ±96.28 ±87.87 ±3.02 ±84.81 ±94.72 ±9.2 ±11.47 ±62.82 ±run2.716.714.431.5122.37.376.5621.2335.25

In Table 3 is presented the performance of both EA model+Manager (PEM) and MLP+Manager replacing the HR estimation when the quality flag of the GW3-HR is unreliable. For both methods, the manager block315performs the HR transition when the quality flag changes in the same way as described in the first embodiment. First of all, comparing the results of the Manager-like methods with the GW3-HR in Table 2, we note that the performance, in terms of MAE and PR, improved for the stratification all and is close to the reliable one. As expected, replacing the unreliable method by the manager results in a better performance if compared to using only the GW3-HR alone. It is interesting to note that the GW3-HR also applies a traditional algorithm to deal with MA; however, it is not enough to produce a proper estimation during the whole workout. Lastly, the coverage for both stratifications is slightly lower because of the integrity rule defined by the personalization threshold time (pr), which is set to 120 seconds.

TABLE 2the performance of the PEM and MLP working in collaborationwith the GW3-HR, which is run by the manager blockPEMMLP + ManagerMAECOVPRMAECOVPRDataset(bpm)(%)(%)(bpm)(%)(%)Walking4.2 ±79.91 ±86.7 ±5.01 ±79.8 ±81.63 ±3.0133.2318.694.0431.7824.86Running4.63 ±71.25 ±83.95 ±5.76 ±76.12 ±80.1 ±3.1134.2819.235.6333.2523.66Walk-and-3.26 ±95.12 ±93.61 ±4.0 ±95.7 ±90.28 ±run1.698.659.482.528.0912.86

Comparing the performance of the Manager using the EA model (PEM) and the MLP, we see that the average performance for our invention is better than the MLP for all datasets. In addition to better performance, there are other two advantages of using the PEM rather the MLP:

It demands much less memory to run the estimation and personalization than the MLP. Since it must work in real-time in devices with strong memory restriction, this characteristic is mandatory.

Since the PEM has only eight trainable parameter it is easier to personalize for a given user, to explain the contribution of each parameter (consequently, to debug), and its performance is steady regardless the workout session.

InFIG.8is depicted an example of the performance of each method individually for a given workout session of the walk-and-run dataset. In this workout, the performance of the GW3-HR and the PEM in terms of MAE is quite close, 1.64 and 2.78 bpm, respectively. On the other hand, the MLP performance is 7.84, which far from the other methods. In addition,801indicates a region in which the MLP presents a quite wrong estimation. This behavior is common for this method and it is hard to identify the reason since we do not know the role of each parameter in this model. The HR curve obtained by the PEM looks more natural and results in better performance than the MLP one.

Finally, inFIG.9is illustrated an example of the collaboration between the EA model and the GW3-HR provided by the manager block, which is the PEM. As we can see,901indicates the start of the transition between the methods, i.e., when the quality flag switches from reliable to unreliable. In this workout, the unreliable flag remains as unreliable for the entire grey area902. It is easy to see the improvement that the manager introduces into the algorithm. Quantitatively, in terms of MAE, it decreases from 22.2 to 3.31 bpm, which demonstrates the benefits of the present invention.

Technical Advantages of the Proposed Solution

In view of the foregoing, the present invention provides a solution based on the collaboration of the comparison between the HR estimation from motion information and a PPG based method to improve the final estimation. In addition, the present invention proposes the use of only one regression model designed from an exponential function, which allows to output the HR estimation without using multiple models. Further, the present invention relies only in the current user data without using previous data or data from a plurality of users to achieve personalization. Finally, the present invention proposes alternative embodiments wherein a transition equation is applied to improve the smoothness and integrity of the final HR estimated from the collaboration between the proposed exponential model and the PPG-based approach.

The present invention proposes an exponential approximation model developed to estimate the HR of a given person. Using an exponential function over time to model cardiovascular kinetics was previously presented by Hill & Lupton [1] and Zakynthinaki [2]. Although the assumption that the HR may be approximated by an exponential function is correct, proposing such a function and parametrize it is challenging because the HR profile of a well-trained and a sedentary person is quite different and the previously models do not consider it. In addition, they do not present an approach to dynamically consider both the increase and decrease of the HR curve. In this way, the non-obviousness characteristics that differs the exponential regression model of the present invention from the others are described as follow:it dynamically updates the HR estimated in real-time based on a motion measurement. In other words, the model does not rely in a constant measurement and for each time t it outputs a new estimation.its equation incorporates a mechanism to automatically identify when the HR should increase or decrease according to the exponential function.it also incorporates the user's demographics (age, height, weight, and gender) into its equation in order to adapt the model to different HR profiles.

In addition to the characteristics related to exponential model, the present invention also presents the following innovations:It has a methodology to personalize the exponential model in real-time without saving any data from the previous workout in the device's memory. In this way, the model is continuously learning the user's HR profile during workout execution in order to improve its estimations.It presents a transition algorithm to improve the smoothness and integrity of the final HR estimated from the collaboration between the exponential model and the PPG-based approach.

While various example embodiments have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein.