Human motion feature extraction in personal emergency response systems and methods

A non-wearable Personal Emergency Response System (PERS) architecture is provided, having a synthetic aperture antenna based RF interferometer followed by two-stage human state classifier and abnormal states pattern recognition. Systems and methods transmit ultra-wide band radio frequency signals at, and receive echo signals from, the environment, process the received echo signals to yield a range-bin-based slow signal that is spatio-temporally characterized over multiple spatial range bins and multiple temporal sub-frames, respectively, and derive from the slow signal a Doppler signature and a range-time energy signature as motion characteristics of human(s) in the environment and optionally also derive location data as movement characteristics thereof. The decision process is carried out based on the instantaneous human state (local decision) followed by abnormal states patterns recognition (global decision).

FIELD OF THE INVENTION

The present invention relates to the field of elderly monitoring, and more particularly, to a system architecture for personal emergency response system (PERS).

BACKGROUND OF THE INVENTION

Elderly people have a high risk of falling, for example, in residential environments. As most of elder people will need immediate help after such a fall, it is crucial that these falls are monitored and addressed in real time. Specifically, one fifth of falling elders are admitted to hospital after staying on the floor for over one hour following a fall. The late admission increases the risk of dehydration, pressure ulcers, hypothermia and pneumonia. Acute falls lead to high psychological effects of fear and negatively impact the quality of daily life.

Most of the existing personal emergency response systems (PERS), which take the form of fall detectors and alarm buttons, are wearable devices. These wearable devices have several disadvantages. First, they cannot recognize the human body positioning and posture.

Second, they suffer from limited acceptance and use due to: elders' perception and image issues, high rate of false alarms and miss-detects, elders neglect re-wearing when getting out of bed or bath, and the fact that long term usage of wearable devices might lead to user skin irritations. Third, the wearable PERS are used mainly after experiencing a fall (very limited addressable market).

Therefore, there is a need for a paradigm shift toward automated and remote monitoring systems.

SUMMARY OF THE INVENTION

Some embodiments of the present invention provide a unique sensing system and a breakthrough for the supervision of the elderly during their stay in the house, in general, and detect falls, in particular. The system may include: a UWB-RF Interferometer, Vector Quantization based Human states classifier, Cognitive situation analysis, communication unit and processing unit.

One aspect of the present invention provides a method comprising transmitting ultra-wide band (UWB) radio frequency (RF) signals at, and receiving echo signals from, an environment including at least one human, processing the received echo signals to yield a range-bin-based slow signal that is spatio-temporally characterized over a plurality of spatial range bins and a plurality of temporal sub-frames, respectively, and deriving from the slow signal a Doppler signature and a range-time energy signature as motion characteristics of the at least one human.

According to some embodiments of the present invention, the system may be installed in the house's ceiling, and covers a typical elder's apartment with a single sensor, using Ultra-Wideband RF technology. It is a machine learning based solution that learns the elder's unique characteristics (e.g., stature, gait and the like) and home primary locations (e.g. bedroom, restroom, bathroom, kitchen, entry, etc.), as well as the home external walls boundaries.

According to some embodiments of the present invention, the system may automatically detect and alert emergency situation that might be encountered by elders while being at home and identify the emergency situations.

According to some embodiments of the present invention, the system may detect falls of elderly people, but may also identify other emergencies situations, such as labored breathing, sleep apnea, as well as other abnormal cases, e.g., sedentary situation, repetitive non-acute falls that are not reported by the person. It is considered as a key element for the elderly connected smart home, and, by connecting the system to the network and cloud, it can also make use of data analytics to identify new patterns of emergencies and abnormal situations.

DETAILED DESCRIPTION OF THE INVENTION

Prior to the detailed description being set forth, it may be helpful to set forth definitions of certain terms that will be used hereinafter.

The term “slow signal” as used in this application refers to the signal derived from received echo (fast) signals and is spatio-temporally characterized over multiple range bins (as spatial units) and multiple sub-frames (as temporal units).

The term “motion” as used in this application refers to the motion of the body and/or of body parts without displacement of the whole body as a bulk, such as gestures, limb motions, posture changes such as sitting down or standing up, gait (separated from the displacement), motion suddenness (e.g., possible fall or collapse) etc.

The term “movement” as used in this application refers to the displacement of a person's body as a whole, irrespective of the motion of body parts such as the limbs. In certain embodiments, the term “movement” may be used to refer only to radial displacements and radial components of displacement with respect to the antenna, whereas tangential displacement may be discarded. In certain embodiments, tangential components of the displacement may be taken into account as movements as well.

FIG. 1is a block diagram illustrating a non-limiting exemplary architecture of a system100in accordance with some embodiments of the present invention. System100may include a radio frequency (RF) interferometer120configured to transmit signals via Tx antenna101and receive echo signals via array110-1to110-N. It should be noted that transmit antennas and receive antennas may take different forms, and, according to a preferred embodiment, in each antenna array they may be a single transmit antenna and several receive antennas. An environmental clutter cancellation module may or may not be used to filter out static non-human related echo signals. System100may include a human state feature extractor130configured to extract from the filtered echo signals, a quantified representation of position postures, movements, motions and breathing of at least one human located within the specified area. A human state classifier may be configured to identify a most probable fit of human current state that represents an actual human instantaneous status. System100may include an abnormality situation pattern recognition module140configured to apply a pattern recognition based decision function to the identified states patterns and to determine whether an abnormal physical event has occurred to the at least one human in the specified area. A communication system150for communicating with a remote server and end-user equipment for alerting (not shown here). Communication system150may further include two-way communication system between the caregiver and the monitored person for real-time assistance.

FIG. 2Ais another block diagram illustrating the architecture of a system in further details in accordance with some embodiments of the present invention as follows:

UWB-RF interferometer220—this unit transmits an Ultra wideband signal (e.g., pulse) into the monitored environment and receives back the echo signals from multiple antenna arrays to provide a better spatial resolution by using the Synthetic Antenna Aperture approach. In order to increase the received signal-to-noise (SNR), the transmitter sends multiple UWB pulse and receiver receives and integrates multiple echo signals (processing gain). The multiple received signals (one signal per each Rx Antenna) are sampled and digitally stored for further signal processing.

Environmental Clutter Cancellation230(possibly as part of a processing unit225). The echo signals are pre-processed to reduce the environmental clutter (the unwanted reflected echo components that are arrived from the home walls, furniture, etc.). The output signal mostly contains only the echo components that reflected back from the monitored human body. Environmental Clutter Cancellation230is fed with the trained environmental parameters232. In addition, the clutter cancellation includes a stationary environment detection (i.e., no human body at zone) to retrain the reference environmental clutter for doors or furniture movement cases.

Feature extractor240that processes the “cleaned” echo signals to extract the set of features that will be used to classify the instantaneous state of the monitored human person (e.g. posture, location, motion, movement, breathing, see more details below). The set of the extracted features constructs the feature vector that is the input for the classifier.

Human state classifier250—The features vector is entered to a Vector Quantization based classifier that classifies the instantaneous features vector by statistically finding the closest pre-trained state out of a set of N possible states, i.e., finding the closest code vector (centroid) out of all code vectors in a codebook234. The classifier output is the most probable states with its relative probability (local decision).

Cognitive Situation Analysis (CSA) module260—This unit recognizes whether the monitored person is in an emergency or abnormal situation. This unit is based on a pattern recognition engine (e.g., Hidden Markov Model—HMM, based). The instantaneous states with their probabilities are streamed in and the CSA search for states patterns that are tagged as emergency or abnormal patterns, such as a fall. These predefined patterns are stored in a patterns codebook234. In case that CSA recognizes such a pattern, it will send an alarm notification to the healthcare center or family care giver through the communication unit (e.g., Wi-Fi or cellular). Two-way voice/video communication unit150—this unit may be activated by the remote caregiver to communicate with the monitored person when necessary. The UWB-RF interferometer unit220may include the following blocks: (i) Two-Dimensional UWB antenna array110-1-110-N to generate the synthetic aperture through all directions, followed by antenna selector. (ii) UWB pulse generator and Tx RF chain to transmit the pulse to the monitored environment UWB Rx chain to receive the echo signals from the antenna array followed by analog to digital converter (ADC). The sampled signals (from each antenna) are stored in the memory, such as SRAM or DRAM.

In order to increase the received SNR, the RF interferometer repeats the pulse transmission and echo signal reception per each antenna (of the antenna array) and coherently integrates the digital signal to improve the SNR.

Environmental Clutter Cancellation

The environmental clutter cancellation is required to remove unwanted echo components that are reflected from the apartment's static items, such as walls, doors, furniture, etc.

The clutter cancellation is done by subtracting the unwanted environmental clutter from the received echo signals. The residual clutter represents the reflected echo signals from the monitored human body.

According to some embodiments of the present invention, the clutter cancellation also includes stationary environment detection to detect if no person is at the environment, such as when the person is not at home, or is not at the estimated zone. Therefore, a periodic stationary clutter check is carried out, and new reference clutter fingerprint is captured when the environment is identified as stationary.

The system according to some embodiments of the present invention re-estimates the environmental clutter to overcome the clutter changes due to doors or furniture movements.

Multiple Features Extraction

The “cleaned” echo signal vectors are used as the raw data for the features extraction unit. This unit extracts the features that mostly describe the instantaneous state of the monitored person. The following are examples for the set of the extracted features and the method it's extracted: Position—the position is extracted as the position (in case of 2D—angle/range, in case of 3D—x,y,z coordinates) metrics output of each array baseline. The actual person position at home will be determined as a “finger print” method, i.e., the most proximity to the pre-trained home position matrices (centroids) codebook. Posture—the person posture (sitting, standing, and laying) will be extracted by creating the person “image” by using, e.g., a back-projection algorithm. Both position and posture are extracted, for example, by operating, e.g., the Back-projection algorithm on received echo signals—as acquired from the multiple antennas array in SAR operational mode.

The following is the used procedure to find the human position and posture: Dividing the surveillance space into voxels (small cubic) in cross range, down range and height; Estimating the reflected EM signal from a specific voxel by the back projection algorithm; Estimating the human position by averaging the coordinates of the human reflecting voxels for each baseline (Synthetic Aperture Antenna Array); Triangulating all baselines' position to generate the human position in the environment; Estimating the human posture by mapping the human related high-power voxels into the form-factor vector; and Tracking the human movements in the environment (bedroom, restroom, etc.)

Human motion—The monitored human body may create vibrations and other motions (as gestures and gait). Therefore, it introduces frequency modulation on the returned echo signal. The modulation due to these motions is referred to as micro-Doppler (m-D) phenomena. The human body's motion feature may be extracted by estimating the micro-Doppler frequency shift vector at the target distance from the system (down range). The following description andFIGS. 2B-2Oelaborate on the aspect of human motion features extraction.

It is noted that the term “motion” refers to the motion of the body and/or of body parts without displacement of the whole body as a bulk, such as gestures, limb motions, posture changes such as sitting down or standing up, gait (separated from the displacement), motion suddenness (e.g., possible fall or collapse) etc. The term “movement” refers to the displacement of a person's body as a whole, irrespective of the motion of body parts such as the limbs (in case of movement detection by backpropagation algorithms, the movement may comprise only the radial components of displacement).

Non-wearable monitoring system100may comprise ultra-wide band (UWB) radio frequency (RF) interferometer120configured to transmit UWB RF signals at, and to receive echo signals from, an environment including at least one human, processing unit225configured to processing derive, e.g., at a slow signal derivation module226, a range-bin-based slow signal from the received echo signals, the slow signal being spatio-temporally characterized over a plurality of spatial range bins and a plurality of temporal sub-frames, respectively, and feature extractor240configured to derive from the slow signal a Doppler signature and a range-time energy signature as motion characteristics of the at least one human.

The Doppler signature may be derived by comparing spectral signatures of sub-frames in the slow signals, which are related to identify human-related range bins and sub-frames. The energy signature may derived by evaluating powers of the slow signal at identified human-related range bins and sub-frames. The Doppler signature and/or the energy signature may be derived with respect to different body parts of the at least one human.

Feature extractor240may be further configured to derive location data to yield movement characteristics of the at least one human. The location data may be derived by detecting displacements of the at least one human using back-projection and/or by identifying human-related range bins and sub-frames in the slow signal. The derivation of the location data may be carried out using a spatio-temporal histogram of the range-time energy signature, by identifying on the histogram range changes of at least body parts of the at least one human.

System100may further comprise human state classifier250configured to classify the motion and movement characteristics of the at least one human to indicate a state of the at least one human, and abnormality situation pattern recognition module262, e.g., as part of cognitive situation analysis module260configured to generate an alert once the indicated state is related to at least one specified emergency. The classification may carried out by identification of a most probable fit of one of a plurality of predefined states to the motion and movement characteristics and wherein the alert generation is based on pattern recognition with respect to previously indicated states.

FIG. 2Bis a high-level schematic flowchart illustration of exemplary human motion features extraction241in feature extractor240, according to some embodiments. The Human Motion Features Extractor system receives a UWB echo signal401and processes it according to the following blocks. Detailed descriptions of modules inFIG. 2Bare presented in consecutive figures.

Echo (fast) signal preprocessing unit405receives the echo signals from antennas110-1to110-N. Each pulse transmission is represented by a vector that is referred to in the following as the ‘fast time signal’. The transmission-reception cycle is performed repeatedly for a frame of, e.g., Tframe=2 to 5 seconds at a rate of, e.g., Fslow=100 Hz to 300 Hz as non-limiting values. The output of unit405is a matrix of the received echo signals, where each row is a fast time signal of a different transmission.

Range bin based slow signal constructor (Fast2Slow)410rearranges the downrange echo (fast) signals vectors (the matrix rows) to represent the cross-range (slow) signals411(the matrix columns), as illustrated inFIG. 2Cbelow. The slow signal vector represents a single downrange distance (bin) with a sampling rate, e.g., Fslow=100 Hz to 300 Hz as a non-limiting value. These vectors are referred as the ‘slow time signals’.

Human body (target) detection is carried out by detecting its representation by a range bins window of e.g., RWtarget=50 to 200 range bins (assuming, in a non-limiting manner, that each range bin is approximately 1 cm), in a non-limiting example. The target location may be determined by the range bins window with the highest motion power among all of the RWtargetbins windows. The slow signal may be preprocessed for each range bin separately and may include DC removal, which is done by the subtraction of the estimated average DC signal from the original signal as well as other optional signal adjustments for example gain and phase mismatch correction between all the range bins slow signals and out-of-band noise reduction filtering.

Feature extraction241may be separated into two components—motion Doppler characteristics derivation420A (motion Doppler features) and motion change over range bins and time characteristics derivation420B (motion energy features). Motion features extraction241yields a motion features vector440which is then used for further processing and classification in classifiers130and/or250. The following demonstrates in a non-limiting manner possible embodiments of derivations420A,420B.

Motion characteristics detection420may comprise deriving from the slow signal a Doppler signature, e.g., by block420A, and a range-time energy signature, e.g., by block420B, as motion characteristics of the at least one human.

Motion characteristics detection420may comprise, concerning derivation of Doppler signature420A, Doppler preprocessing and segmentation422in which the slow signal frame is divided into Msubframessub-frames using Equation 6 (see below). The spectrogram may be generated by fast Fourier transform (FFT) for each slow time signal sub-frame within the human target range. A maximal Doppler frequency extractor424may use the maximum Doppler frequency to identify the instantaneous moment and range that a rapid motion (such as falling) has occurred. This feature is extracted by scanning all the slow time signal sub-frames per each range bin and accumulating the related power spectrum with the highest motion (Doppler) frequency that is selected out of each range bin. The maximal Doppler feature is extracted from the accumulated range bins power spectrums. A Motion Energy Extractor426may estimate the motion energy features in the frequency domain. There are a few features that are extracted to better represent the overall motion energy.

Motion characteristics detection420may comprise, concerning derivation of energy signature420B, Range over Time preprocessing and segmentation432in which the signal is preprocessed and segmentation of the data into histograms is performed. For example, at a first stage, a Dynamic Time Wrapping (DTW) process may be implemented to estimate the human motion path along the range bins window and at a second stage, e.g., three histograms, which contain information about the distribution of the motion activity and energy signature over range, are generated to represent: (i) Cumulated energy of all the range bins selected; (ii) The numbers of appearances of each range bin in the top 5 range bins; and (iii) The number of average energy for each range bin that appeared in the top 5 ranges bins list. For each histogram, a set of features may be extracted to represent the histogram form factor, for example: (i) Motion energy distribution analysis434which comprises the extraction of features that represent the distribution of the energy over the range bins, carried out e.g. by using the energy distribution histogram analysis over range bins; (ii) Motion over range distribution analysis436to represent the distribution of the active range bins during the motion period and helps determine if the motion is stationary in space or distributed among several range bins; and (iii) Motion route energy estimator438which extracts the motion route energy by accumulating the power over the motion path (the selected range bins power as a result of the DTW at the pre-processing unit).

FIG. 2Cis a high-level schematic illustration of fast and slow signal mapping410,411, according to some embodiments. The received preprocessed fast signals are mapped in a two dimensional matrix X (Equation 1). Each echo sample is an element on the matrix x[n][k]; n=1 . . . NRanges; k=1 . . . Ksampels, where n is the downrange bin indicator of spatial range bin, and k is the cross-range (slow) time indicator of temporal bins. The number of total range bins is determined by the scanning window, while each range bin represents C/Ffastmeters (Ffastis the echo signal sampling rate). The matrix is separated into its rows. Each row xnis the echo signal from the same range from the interferometer (radar), sampled in Fslow=250 Hz. Those vectors are referred as the slow time signals.

FIG. 2Dis a high-level schematic flowchart illustration of exemplary human body target detection452, according to some embodiments. Human Body Target Detection unit452narrows the focus of the analysis to the relevant range bins with human presence. Unit452may operate with various inputs, according to the required features to be extracted. The process of the target detection given the slow time signals of all the N range bins is performed by the following blocks, as an exemplary embodiment. A range bin power calculator452A calculates the power of each slow time vector by Equation 2, where k and n are the time and range bin indicators respectively, to yield N power values.
p[n]=Σk=1Kxn2[k]forn=1 . . .NRanges(Equation 2)

Following, the power sequence over a sliding window of RWtargetrange bins is calculated along the (NRanges−RWtarget+1) windows (Eq. 2.2) and accumulated by accumulator452B.
s[n]=Σj=0M-1p[j+n]forn=1 . . . (NRanges−RWtarget+1)  (Equation 3)
Finally, the human target location region is detected452C and indicated at the most powerful windowed power as expressed in Equation 4.
Windicator=argmaxn(s[n])  (Equation 4)

FIG. 2Eis a high-level schematic flowchart illustration of an exemplary slow signal preprocessing unit454, according to some embodiments. The slow time signal preprocessing may be carried out in a generic unit having its input determined by the extracted features (e.g., of features vector440) and optionally operating on each slow time signal separately. Preprocessing unit454may perform the following blocks: (i) Adaptive DC removal454A by continuously calculating the estimated DC signal (time varying DC) for each time bin by Equation 5, using the current slow signal vector x[k],
s[k]=(1−a)s[k−1]+ax[k],k=1 . . .Ksamples(Equation 5)
where a, is the learning coefficient. The estimated DC signal is subtracted from the original signal, namely y[k]=x[k]−s[k]. Gain mismatch correction454B may optionally be performed to the selected range bins' slow signals to compensate the path losses differences among the selected range bins. The additional path loss of Riversus Rminmay be calculated as

Δ⁢⁢P.L.[d⁢⁢B]=20⁢⁢log⁡(RiRmin),
where Riis the range bin i distance out of the selected set of range bins and Rminis the first (closest) range bin. A slow signal phase mismatch correction454C among the selected range bins may be carried out to compensate for the motion offset over the time/range bin. That is, the same motion profile may be preserved between neighbor range bins with a delayed version. The slow signal phase mismatch correction may estimate the phase error between SlowSigRiand SlowSigRref, where SlowSigRiis the slow signal of range bin Ri, and SlowSigRrefis the slow signal that is considered the reference range bin out of the selected range bins. Optionally, an out of band (O.O.B.) noise reduction filter454D may be enabled to filter out the irrelevant slow signal components or interferences that might influence the performance of the various energy based features extraction.

FIG. 2Fis a high-level schematic flowchart illustration of exemplary Doppler preprocessing and segmentation422, according to some embodiments. A spectrogram422A for each range bin may be generated and used for extraction of signal's spectral features, for every short time period, termed herein sub-frame (e.g., a plurality of specific fast signal times, i.e., a range of k values). The sub-frame period should be short enough to consider the motion as stationary). In order for a spectrogram to be created from a slow time signal vector of a specific range bin in the target region, slow time signal411of each range bin is first preprocessed by a preprocessing unit422B, and then a human motion target detection unit422C is being used to find the target range bin. Spectrogram422A of each range bin is generated by segmenting the original signal x[k] to Msubframessub-vectors. For a given window size and number of overlaps, a new vectors group is constructed according to Equation 6.
{vm[i]}={x[Lstepm+i]};i=1 . . .D;m=1 . . .Msubframe(Equation 6)

Each vector may have, as a non-limiting example, DWinSizemay be between 50 and 200 samples (equivalent a subframe length of between 0.15 and 2 seconds) with overlaps of (DWinSize−Lstep) samples from the previous vector in the sequence (Lstep=is the samples step size between subframes). Then, a power spectrum Vmmay be computed for each sub-frame by Equation 7, where h is a hamming window with length D.
{Vm}=FFT{vm·h};m=1 . . .Msubframes(Equation 7)
This process is repeated for every range bin within the target region. RWtargetspectrograms422D are gathered for further processing.

FIG. 2Gis a high-level schematic flowchart illustration of an exemplary maximal Doppler frequency extraction424A, according to some embodiments. Maximal Doppler frequency extractor424is configured to find the highest velocity of the motion, which is represented by the motion's Doppler frequency along the motion's down-range route. The timing of the human peak motion activity is not common for all range bins, due to the fact that the motions can cross range bins versus the time. Therefore, the maximal Doppler frequency feature is extracted by scanning all the slow time signal sub-frames per each range bin and accumulating the related power spectrum with the highest motion (Doppler) frequency that is selected out of each range bin. The max Doppler feature may be extracted from the accumulated range bins power spectrums. In order to extract the action power spectrum by extractor424B from each range bin spectrogram, the following process is performed: Noise level threshold estimation computation424C calculates the noise level threshold for the spectrogram energy by considering the spectrogram values below noise level are considered as not related to human motion. A threshold T1(measured in dB) may be determined by Equation 8, using the mean of the upper four frequency bins of the spectrogram, while Q, P are respectively the numbers of frequency and time bins of the spectrogram matrix S.

T1=14⁢∑q=(Q-4)q=Q⁢⁢1P⁢∑p=1p=P⁢⁢s⁡[p,q],s∈S(Equation⁢⁢8)
The maximal motion frequency bin is defined and estimated in424D, as the first frequency bin to its power below the motion threshold when scanning the spectrum from Fminto Fmaxwhich is the motion (active) region for the p power spectrum as defined by Equation 9.
fp=argminq(s[q,p]<(T1+1))forp=1 . . .P(Equation 9)
where fpis the maximal frequency at the p power spectrum that its power is <T1+1 dB. An example for that region from a full spectrogram can be seen in spectrogram424E ofFIG. 2H.FIG. 2His an exemplary illustration of a spectrogram424E of motion over a single range bin in the active area, according to some embodiments. Action power spectrum extractor424B further carries out a selection424F of the power spectrum with the highest frequency—the selected power spectrum at time bin p is the one that has the highest value of fp(referred as action power spectrum of range q). This power spectrum is extracted for farther analysis. The averaged action power spectrum Pavis created (424G) using action power spectrums424E from all range bins. Then, a new noise threshold T2is calculated from Equation 10, by using the average value of the upper four frequency bins of the averaged (accumulated) power spectrums, in a non-limiting example.

T2=14⁢∑q=(Q-4)q=Q⁢Pav⁡[q](Equation⁢⁢10)
The maximal frequency feature is calculated by Equation 11:
fmax=argminq(Pav[q]<(T2+1))  (Equation 11)

FIG. 2Iis a high-level schematic flowchart illustration of an exemplary motion energy features extractor426, according to some embodiments. The motion energy features may be estimated in the frequency domain. There are a few features that are extracted to better represent the overall motion energy. The motion energy might be affected by several conditions which are not related to the motion itself. For example, the relative distance from the interferometer as well as the motion duration. Unit426may create several spectrograms for all the target range bins to extract the various features that represent the energy signature. The motion energy features may be extracted by the following exemplary process. Two spectrogram versions may be created for each target range bin. The first spectrogram may be created after a range gain mismatch correction (to compensate the path loss variations over the range bins). The other spectrogram may be created without the gain mismatch correction (426A). The gain mismatch may be implemented at preprocessing unit422B. Therefore, two spectrogram sets are created for the complete range bins {S1n} and {S2n}. For each set of spectrograms, an average spectrogram426B S1avS2avmay be created by Equation 12.

Si,av⁡[q]⁡[m]=1RW⁢∑n=1RW⁢⁢Sn⁡[q]⁡[m];for⁢⁢i=1,2⁢⁢m=1⁢⁢…⁢⁢Msubframes,q=1⁢⁢…⁢⁢Qfreqbins(Equation⁢⁢12)
In order to emphasize the motion power in higher frequencies, each averaged spectrogram frequency bin {right arrow over (S)}avq, may be processed with a corresponding weight, into a new weight-averaged spectrogram426C by Equation 13.

SW→avq=S→avq*f⁡[q]fmax;for⁢⁢q⁢=1⁢⁢…⁢⁢Qfreqbins(Equation⁢⁢13)
where f[q] is the frequency value of the q frequency bin, and fmaxis the maximal frequency bin value. Two vectors of the power peaks may be created (426D) for each the two spectrograms, with and without power correction. A first vector {right arrow over (o)}1contains the maximal power of each sub-frame vector {right arrow over (s)}avm(Equation 14A), and the second vector {right arrow over (p)}2contains the maximal values of each frequency bin vector {right arrow over (s)}avq(Equation 14B).
p1[m]=max({right arrow over (s)}avm);form=1 . . .Msubframes(7.3)  (Equation 14A)
p2[m]=max({right arrow over (s)}avq);form=1 . . .Mfreqbins(7.4)  (Equation 14B)
Each of the four (2×2) vectors—with the different procedures for gain processing and for maximal power values extraction, are accumulated into four energy features.

FIG. 2Jis a high-level schematic flowchart illustration of an exemplary range-time preprocessing and segmentation flow432as part of derivation of energy signature420B, according to some embodiments. The motion features over range-time helps profiling the energy signature of each motion, not only by characterizing its power and velocity, but by also characterizing its distribution over space along the motion time. Module432may be configured to create three histograms that express the distribution of motion energy and activity over the range bins during motion period. The energy related histograms may be created by the following algorithms. After normalization of the slow time matrix X (defined in Equation 1) using the highest absolute value amplitude as

X=XX∞
(the notation X is maintained for simplicity), the target region is located by a target region detector unit432A. The two axis of the slow time matrix X correspond to the time of the sample (time bin) and the range of the sample (range bin). Each range bin vector, with KSamples=Tframe·Fslowlength as an example, may then segmented into 10 sub-frames as a non-limiting example and mapped in as new matrix Xndefined in Equation 15, with Ksamples=800 as a non-limiting example, with each row of the new matrix having Ksamples/10 samples with an overlap.

Xn=[xn⁡[1]…xn⁡[80]⋮⋱⋮xn⁡[720]…xn⁡[800]](Equation⁢⁢15)
With Enbeing the temporal energy vector of each range bin as calculated in Equation 16, j being the sub-frame number.

En⁡[j]=∑i=1i=K10⁢⁢Xn⁡[j,i]2⁢⁢for⁢⁢n=1⁢⁢…⁢⁢RWtarget(Equation⁢⁢16)
A Matrix E defined by Equation 17 is constructed by gathering all the temporal energy vectors from each range bin.

E=[E1⋮EN](Equation⁢⁢17)
The columns of E are the energies of all the ranges along the new wider time bins, and the rows are the energy of a specific bin along time. From each column with indicator k, the five highest elements values may be extracted into wk(r), r=1 . . . 5; together with their row indexes gk(r), as a non-limiting example. The three histograms432B are created from elements wk(r) as defined by Equations 18A-C.
An accumulated range histogram with elements calculated by Equation 18A:
hacc(n)=Σk=1k=Kwk(r)*l(gk(r)=n)forn=1 . . .RWtarget(Equation 18A)
The indicator function l(ωεΩ)is equal to 1 if the condition in the brackets is true.
An activity in range over time histogram, with elements calculated by Equation 18B:
happ(n)=Σk=1k=Kl(gk(r)=n)forn=1. . . RWtarget(Equation 18B)
A normalized energy histogram, with elements calculated by Equation 18C:

FIG. 2Kis a high-level schematic flowchart illustration of an exemplary range energy distribution analysis434as part of derivation of energy signature420B, according to some embodiments. Range energy distribution analysis module434extracts features from the accumulated and normalized energy histograms, which relate to the amount and distribution of motion energy over the range bins. Range energy distribution analysis434includes the extraction of the total and maximal (peak) energy over the range bins out of the histogram. In addition, the histogram form factor, defined as the percentage accumulated distribution points, is extracted (for example I20—identifies the range bin point that covers 20% of the accumulated motion energy, I40—identifies the range bin point that covers 40% of the accumulated motion energy, etc.).

FIG. 2Lis a high-level schematic flowchart illustration of an exemplary motion over range distribution analysis436, according to some embodiments. Motion over range distribution analysis unit436extracts features that relate to the distribution of active range bins over time, which is related to the motion's and varying over the down range. Unit436extracts the number of times that most of active region has been selected, the total number of active range bins and the mean number of repeated selection of the range bin as an active region.

FIG. 2Mis a high-level schematic flowchart illustration of an exemplary motion route energy estimation438, according to some embodiments. The motion route energy is defined as the accumulated power along the motion route in the range bins window during the motion period (time) relatively to the overall energy. This feature may be extracted in two major stages: (i) Estimating the motion route by using a dynamic Time Warping (DTW) approach438A, (ii) accumulating the estimated power along the selected range bin route, and normalizing by the overall power438B and calculating the motion route peak to average power ratio438C. The DTW may be performed by selecting the highest range bin power for every sub-frame, as illustrated inFIG. 2N, being a schematic matrix illustration of DTW-based motion route estimation438A, according to some embodiments. The relative Motion Route Energy (MRE) may be calculated as expressed in Equation 19:

The motion route Peak to Average Power Ratio (PAR), measured by the ratio between maximal and average power of the motion route, may be calculated as in Equation 20:

FIG. 2Ois a high level schematic flowchart of a method600according to some embodiments of the invention.

Method600may comprise transmitting signals (e.g., UWB RF signals) at a specified area (such as an environment including at least one human) and receiving echo signals via an antenna array (stage603), filtering out static non-human related echo signals (stage606), extracting from the filtered echo signals, a quantified representation of position postures, movements, motions and breathing of at least one human located within the specified area (stage610), identifying a most probable fit of human current state that represents an actual human instantaneous status (stage690) and applying a pattern recognition based decision function to the identified states patterns and determine whether an abnormal physical event has occurred to the at least one human in the specified area (stage693) (see additional details below).

Method600may further comprise finding the best match to a codebook which represents the state being a set of human instantaneous condition/situation which is based on vector quantized extracted features (stage691).

Method600may further comprise ensuring, by the filtering out, that no human body is at the environment, using static clutter estimation and static clutter subtraction (stage607)

Method600may further comprise quantizing the known states features vectors and generating the states code-vectors (stage692A), measuring the distance between unknown tested features vectors and pre-defined known code-vectors (stage692B) and finding the best fit between unknown tested features vector and pre-determined code-vectors set, using the most probable state and the relative statistical distance to the tested features vector (stage692C).

Method600may further comprise generating the set of abnormal states patterns as a reference codebook, a set of states transition probabilities, and a states-patterns matching function to find and alert on a match between a tested states pattern and the pre-defined abnormal pattern of the codebook (stage694). Method600may further comprise communicating an alert upon determining of an abnormal physical event (stage695).

Method600may further comprise estimating the reflected clutter from a specific voxel to extract the human position and posture features (stage612A), extracting the human motions and breathing features using Doppler signatures (stage612B) and creating a quantized vectors of the extracted features (stage612C).

Method600may further comprise processing the received echo signals to yield a range-bin-based slow signal that is spatio-temporally characterized over a plurality of spatial range bins and a plurality of temporal sub-frames, respectively (stage620) and deriving from the slow signal a Doppler signature and a range-time energy signature as motion characteristics of the at least one human (stage630). Method600may comprise deriving the Doppler signature by comparing spectral signatures of sub-frames in the slow signals, which are related to identified human-related range bins and sub-frames (stage622) and deriving the energy signature by evaluating powers of the slow signal at identified human-related range bins and sub-frames (stage624). Method600may comprise deriving the Doppler signature and/or the energy signature with respect to different body parts of the at least one human (stage626).

Deriving630may further comprise deriving location data as movement characteristics of the at least one human (stage632). Deriving of the location data632may comprise detecting displacements of the at least one human using back-projection (stage634) and/or identifying human-related range bins and sub-frames in the slow signals (stage636). Deriving of the location data632may be carried out using a spatio-temporal histogram of the range-time energy signature and by identifying on the histogram range changes of at least body parts (e.g., limbs) of the at least one human (stage638). The motion characteristics and/or movement characteristics may comprise gait parameters.

Method600may comprise classifying the position and/or posture and/or motion and/or movement and/or respiration characteristics of the at least one human to indicate a state of the at least one human (stage688). Classification688, e.g., by identifying the most probable fit690, may be carried out by identifying a most probable fit of one of a plurality of predefined states to the motion characteristics.

Communicating the alert695may be carried out by generating the alert once the indicated state is related at least one specified emergency. The alert generation may be based on pattern recognition with respect to previously indicated states.

Human breathing—During the breathing (respiration) the chest wall moves. The average respiratory rate of a healthy adult is usually 12-20 breaths/min at rest (˜0.3 Hz) and 35-45 breaths/min (˜0.75 Hz) during labored breathing. The breathing frequency feature is extracted by estimating the spectrum on the slow-time sampled received echo signal at the target distance (down range) from the system.

The features vector is prepared by quantizing the extracted features with a final number of bits per field and adding the time stamp for the prepared vector. This vector is used as the entry data for the human state classifier (for both training and classifying stages).

FIG. 2Pis a schematic illustration of the possibility to separate different types of motions based on the derived parameters, according to some embodiments of the invention.

The two illustrations inFIG. 2Pare of the same 3D graphics and are taken from different angles to illustrate the separation of the two types of motion in the 3D parameter space.FIG. 2Pclearly illustrates the ability of the analysis described above to separate motions that are categorized, in the non-limiting illustrated case, as fall motions and as regular motions. The results may be used independently to detect falls, or be provided to the classifier for verification and augmentation with additional data and analysis results. Classification of the human state, as described in detail below, may relate to the derived motion characteristics as well as optionally to posture characteristics, respiration characteristics and position characteristics that may be derived from the received echo signals by implementing the disclosed methods, approaches and/or additional analysis of the received echo signals.

Human State Classifier

The Human state classifier is a VQ (Vector Quantization) based classifier. This classifier consists of two main phases: (i) Training phase—it's done offline (supervised training) and online (unsupervised training), where a stream of features vectors reflecting various states are used as a preliminary database for vector quantization and finding the set of code-vectors (centroids) that sufficiently representing the instantaneous human states. The set of the calculated code-vectors are called codebook. Some embodiments of the training sessions are provided in more details hereinafter. (ii) Classifying phase—it's executed during the online operation while an unknown features vector is entered into the classifier and the classifier determines what the most probable state that it represents. The classifier output is the determined states and the set of the measured statistical distances (probabilities), i.e., the probability of State-i given the observation-O (the features vector). The aforementioned probability scheme may be formulated by: P (Si|O). The determined instantaneous state is called “Local Decision”. The VQ states are defined as the set of instantaneous states at various locations at the monitored home environment. Therefore, any state is a 2 dimension results which is mapped on the VQ state matrix. (iii) The State matrix consists of the state (row) and location (Column) followed by a time stamp. Typical elderly home environment consists of the specific locations (Primary zones) and others non-specified locations (Secondary zones). State is defined as the combination of posture/motion at a specific location (e.g. S21will indicate sleeping at Bedroom).

FIG. 3is a diagram illustrating conceptual 2D Synthetic Aperture Antennas arrays in accordance with some embodiments of the present invention. Antenna array system300may include several arrays of antennas320,330,340, and350. Each row may have at least one transmit antenna and a plurality of receive antennas. The aforementioned non-limiting exemplary configuration enables to validate a location of a real target310by eliminating the possible images310A and310B after checking reflections received at corresponding arrays of antennas330and320, respectively. It is well understood that the aforementioned configuration is a non-limiting example and other antennas configurations may be used effectively.

FIG. 4is a table134illustrating an exemplary states definition in accordance with some embodiments of the present invention.FIG. 5is a table135illustrating an exemplary states matrix in accordance with some embodiments of the present invention.

Cognitive Situation Analysis (CSA)

The CSA's objective is to recognize the abnormal human patterns according to a trained model that contains the possible abnormal cases (e.g., fall). The core of the CSA, in this embodiment, may, in a non-limiting example a Hidden Markov Model (HMM) based pattern recognition. The CSA engine searches for states patterns that are tagged as an emergencies or abnormal patterns. These predefined patterns are stored in a patterns codebook. The output of the CSA is the Global recognized human situation.

FIG. 6is a table136illustrating exemplary abnormal patterns in accordance with some embodiments of the present invention. It can be seen that in the first abnormal case (Critical fall), it appears that the person was sleeping in the leaving room (S25), then was standing (S45) and immediately fell down (S65). He stayed on floor (S15) and start being in stress due to high respiration rate (S75). The CSA may contain additional codebook (irrelevant codebook) to identify irrelevant patterns that might mislead the system decision.

Communication Unit

The communication unit creates the channel between the system and the remote caregiver (family member or operator center). It may be based on either wired (Ethernet) connectivity or wireless (e.g., cellular or WiFi communication or any other communication channel).

The communication unit provides the following functionalities:1. This unit transmits any required ongoing situation of the monitored person and emergency alerts.2. It enables the two way voice/video communication with the monitored person when necessary. Such a communication is activated either automatically whenever the system recognizes an emergency situation or remotely by the caregiver.3. It enables the remote system upgrades for both software and updated codebooks (as will be in further detail below)4. It enables the communication to the centralized system (cloud) to share common information and for further big data analytics based on multiple deployments of such innovated system

FIG. 7is a diagram illustrating cloud-based architecture700of the system in accordance with embodiments of the present invention. Raw data history (e.g., states stream) is passed from each local system100A-100E to the central unit located on a cloud system710and performs various data analysis to find correlation of states patterns among the multiple users' data to identify new abnormal patterns that may be reflected just before the recognized abnormal pattern. New patterns code vectors will be included to the CSA codebook and cloud remotely updates the multiple local systems with the new code-book. The data will be used to analyze daily operation of local system100A-100E.

FIG. 8is a diagram illustrating a floor plan800of an exemplary residential environment (e.g., an apartment) on which the process for the initial training is described herein. The home environment is mapped into the primary zones (the major home places that the monitored person attends most of the time as bedroom810, restroom820, living room830and the like) and secondary zones (the rest of the barely used environments). The VQ based human state classifier (described above) is trained to know the various primary places at the home. This is done during the system setup while the installer10A (being the elderly person or another person) stands or walks at each primary place such as bedroom810, restroom820, and living room830and let the system learns the “fingerprint” of the echo signals extracted features that mostly represents that place. These finger prints are stored in the VQ positions codebook. In addition, the system learns the home external walls boundaries. This is done during the system setup while the installer stands at various places along the external walls and lets the system tune its power and processing again (integration) towards each direction. For example, in bedroom810, installer10A may walk along walls in route840so that the borders of bedroom810are detected by tracking the changes in the RF signal reflections throughout the process of walking. A similar border identification process can be carried out in restroom820, and living room830. Finally, the system learns to identify the monitored person10B. This is done by capturing the fingerprint of the extracted features on several conditions, such as (1) while the person lays at the default bed812(where he or she is supposed to be during nighttime) to learn the overall body volume, (2) while the person is standing to learn the stature, and (3) while the person walks to learn the gait. All the captured cases are stored in the VQ unit and are used to weight the pre-trained codebooks and to generate the specific home/person codebooks. According to some embodiments, one or additional persons such as20can also be monitored simultaneously. The additional person can be another elderly person with specific fingerprint or it can be a care giver who needs not be monitored for abnormal postures.

FIG. 9is a diagram illustrating yet another aspect in accordance with some embodiments of the present invention. System900is similar to the system described above but it is further enhanced by the ability to interface with at least one wearable medical sensor910A or910B coupled to the body of human10configured to sense vital signs of human10, and a home safety sensor920configured to sense ambient conditions at said specified area, and wherein data from said at least one sensor are used by said decision function for improving the decision whether an abnormal physical event has occurred to the at least one human in said specified area. The vital signs sensor may sense ECG, heart rate, blood pressure, respiratory system parameters and the like. Home safety sensors may include temperature sensors, smoke detector, open door detectors and the like. Date from all or some of these additional sensors may be used in order to improve the decision making process described above.

It is to be understood that where the specification states that a component, feature, structure, or characteristic “may”, “might”, can or “could” be included, that particular component, feature, structure, or characteristic is not required to be included.