Patent Description:
Presence detection, intrusion detection, and other activity recognition, is typically performed by motion detectors, which come in many forms including optical and thermal/infrared cameras, passive/active infrared motion detectors, acoustic sensors, vibration sensors, window magnetic sensors and/or glass break sensors. The most common motion sensor used for intrusion detection is passive infrared sensors (PIRs), which rely on sensing the heat radiated by human bodies. The PIRs may be deployed at entrance or transition points in a building through which an intruder may enter.

More recently, research and advancements have developed motion and/or presence sensing techniques that exploit changes in the radio frequency electromagnetic fields (i.e., often called RF fields) generated by wireless devices. Some systems include multiple wireless nodes/transceivers, where each node can determine changes in the signal strength and/or link quality of a specific coded or a generic RF signal received from other nodes. Decision logic, then, determines motion /presence. Other systems are based on a single transmitter and receiver to determine motion and/or presence in an area, either using a single direction measurement, or bi-directional measurements. Unfortunately, these systems rely on the deployment of specific devices for generation and sampling of the RF field. Such deployment may contribute toward deployment costs. Moreover, improvements in the preprocessing of radio frequency data streams is desirable to increase detection confidence.

QINHUA GAO ET AL: "CSI-Based Device-Free Wireless Localization and Activity Recognition Using Radio Image Features",In:.

<CIT> discloses an intruder detection system based on the analysis of ambient RF signals. XIAO JIANG ET AL: "Pilot: Passive Device-Free Indoor Localization Using Channel State Information",In:.

A method of operating an activity recognition system in a building, wherein a plurality of wireless radio devices, each including a transmitting component configured to transmit a radio frequency, RF, and a receiving component configured to receive the RF to accomplish a primary task are located in the building; the method comprising: capturing ambient RF data from each of the plurality of wireless radio devices by an RF sniffer located in the building; receiving the ambient RF data by a processor; reducing noise content of the ambient RF data by the processor; subtracting background from the ambient RF data by the processor; converting the ambient RF data with reduced noise and subtracted background into an image by the processor; generating a successive image for each one of a plurality of time intervals by the processor; and apply an image processing algorithm storing in a storage medium and executed by the processor to each successive image to compare the plurality of successive images to the RF background data and to thereby determine activity recognition.

Furthermore, the noise content of the ambient RF data is reduced by removing a mean value of multiple Channel State information (CSI) subcarriers at the same time index to subtract common mode noise.

Optionally, the subtracting background includes converting the ambient RF data to a first order derivative of time.

Optionally, the conversion to an image includes the combination of RF data from a plurality of antenna channels.

Optionally, the plurality of time intervals is associated with characteristics of a building region containing the RF sniffer.

Optionally, the image processing algorithm applies a deep learning network.

Optionally, the deep learning network is a convolutional neural network (CNN).

Optionally, the ambient RF data is ambient WiFi data.

Optionally, the ambient WiFi data is Channel State Information (CSI) data.

The method involves operating the activity recognition system in a building system as discussed below.

A building system comprising: a wireless radio device including a transmitting component configured to transmit a radio frequency, RF, and a receiving component configured to receive the RF to accomplish a primary task; and an activity recognition system configured to perform an activity recognition task, the activity recognition system including an RF sniffer configured to sample and measure ambient RF signals over time, control circuitry including one or more processors and one or more storage mediums, RF background data stored in at least one of the one or more storage mediums and indicative of no activity, a computer instruction stored in at least one of the one or more storage mediums and executed by at least one of the one or more processors, wherein the computer instruction is configured to process the measured ambient RF signals, convert the process ambient RF signals to a plurality of successive images, and apply an image-based algorithm to compare the plurality of successive images to the RF background data, and thereby determine activity recognition; wherein the transmitting device, the receiving device, and the sniffer are located in a building, and wherein the wireless radio device is one of a plurality of wireless radio devices each transmitting respective RF signals sampled by the sniffer.

Optionally, the sniffer is one of a plurality of sniffers each located in a respective region of a plurality of regions of the building.

Optionally, the wireless radio device is a WiFi device.

Optionally, the activity recognition system is an intruder alert system.

The foregoing features and elements may be combined in various configurations without exclusivity, unless expressly indicated otherwise. However, it should be understood that the following description and drawings are intended to be exemplary in nature and non-limiting.

In the present disclosure, activity recognition detection is built on existing wireless sensors previously deployed in the building. Since radio frequency (RF) signals are increasingly available because of the penetration of wireless IoT devices, especially in indoor building automation, the present disclosure proposes to leverage the ambient RF field generated by devices that are previously deployed and not specifically for intrusion detection purposes. A decision system is presented that determines the devices that are suitable for the purposes of motion, intrusion, and/or activity recognition detection. The system is further configured to facilitate novel preprocessing of RF fields (e.g., WiFi) for improved detection confidence.

In addition, more traditional systems may entail wireless nodes deployed around an area of interest (e.g., a room or a house perimeter). However, these systems may not address the false alarm issues that arise from movements outside the area of interest. In the present disclosure, such issues are addressed by a methodology that explicitly determines the area of interest in any arbitrary deployment. Furthermore, the present disclosure incorporates a machine learning and/or neural network routine that learns the variations in the RF field corresponding to the movement within the area of interest. The machine learning and/or neural network routine, consequently, can reject false alarms caused by movements outside the area of interest.

Referring to <FIG>, an exemplary embodiment of a building system <NUM> (e.g., wireless communication system) includes one or more commodity wireless radio devices, or links, <NUM> (i.e., two illustrated in <FIG>) and one or more activity recognition systems <NUM> (i.e., two illustrated in <FIG>) that apply radio frequency sensing. The commodity wireless radio devices <NUM> and the activity recognition system <NUM> are generally located in, or proximate too, a building <NUM>. Each commodity wireless radio device <NUM> is generally stationary, and may include a transmitting component <NUM> configured to transmit a radio frequency (RF) signal (see arrow <NUM>) and a receiving component <NUM> configured to receive the RF signal <NUM>. Non-limiting examples of the wireless radio devices <NUM> and the associated RF signals <NUM> include WiFi devices, Zigbee devices, iBeacons, and others. Non-limiting applications of the commodity wireless radio device <NUM> may include a wireless phone, an entertainment system, a television system, and any other type of wireless, RF, system typically used in, or proximate too, the building <NUM>. One, non-limiting, example of an activity recognition system <NUM> may be a building activity recognition system.

Each respective commodity wireless radio device <NUM> is constructed to perform a respective primary task, and the respective RF signals <NUM> enable the accomplishment of such primary tasks. For example, a wireless television system may stream a movie from a transmitting component <NUM> (e.g., router) and to a receiving component <NUM> (e.g., a smart television). In another example, a telephone system may transmit verbal communications as the RF signal <NUM>, and from a transmitting component <NUM> (e.g., power charger base) and to a receiving component <NUM> (e.g., hand-held phone). All of the RF signals <NUM>, taken together in a given space, amount to an ambient RF signal <NUM> having various characteristics such as signal strength, channel state information (CSI), and others. CSI generally represents the combined effect of, for example, scattering, fading, and power decay with distance. In one embodiment, the plurality of commodity wireless radio devices <NUM> is a network configured to communicate in one of a mesh topology and a star topology.

The activity recognition system <NUM> is configured to leverage the ambient RF signal <NUM> by generally detecting variations in prescribed characteristics of the ambient RF signal indicative of, for example, a moving presence <NUM>. That is, the ambient RF signal <NUM> is generally leveraged to serve a dual purpose, the primary task when applied to one or more of the wireless radio devices <NUM> (as previously described with regard to signal <NUM>), and an activity recognition alert task when applied to the activity recognition n system <NUM>. In one, non-limiting, example, the presence <NUM> may be a human intruder and the activity recognition n system <NUM> may be an intrusion detection system.

Referring to <FIG> and <FIG>, the activity recognition system <NUM> includes one or more RF sniffers <NUM> (i.e., two illustrated), control circuitry <NUM>, RF data <NUM> (e.g., RF background data), and prescribed instructions <NUM> (i.e., software program). The RF sniffer <NUM> may be a RF device that measures physical characteristics of the received RF signal, such as signal strength, CSI, and others. Non-limiting examples of the RF sniffer <NUM> is a Wi-Fi network interface card, a CSI monitor, and others. Each RF sniffer <NUM> is located in a respective region <NUM> of the building <NUM>. For example, the regions <NUM> may be individual rooms, or, a first region may be an area proximate to a first entry door, and a second region may be an area proximate to a second entry door of the same building <NUM>. Each RF sniffer <NUM> is configured to sample and measure characteristics of the ambient RF signals <NUM> in the respective region <NUM>.

It is understood, that an RF signal strength of the same RF signal <NUM> may be different from one region <NUM> to the next region due to, for example, attenuation (i.e., traveling through objects like walls) and/or distance from the transmitting component <NUM>. The region <NUM> is defined and configured during the commissioning of the system. In one embodiment, the installer may traverse the corners of the region and let the RF sniffer <NUM> collect measurements of the characteristics of the ambient RF signal <NUM>. This could be stored in a site-specific database and a machine learning algorithm infers if the variations in the characteristics of ambient RF signal <NUM> is indicative of an activity and/or a moving presence <NUM> that is within the configured region <NUM>. The characteristics of the ambient RF signals <NUM> are further measured over time, because such measurements may differ over time depending upon, for example, the usage of the wireless radio devices <NUM>.

In one embodiment and as illustrated in <FIG>, the control circuitry <NUM> may be located in each one of the sniffers <NUM> as a single, self-contained, unit. In another embodiment, each sniffer <NUM> may communicate with a single control circuitry <NUM> that may be located in the building <NUM>, or remotely located. Upon the detection of the moving presence, the control circuitry <NUM> may output a notification signal (see arrow <NUM> in <FIG>) to a notification device <NUM> for notification to a user, government authority, and/or others.

The control circuitry <NUM> includes one or more processors <NUM> (e.g., microprocessor) and one or more storage mediums <NUM> (e.g., non-transitory storage medium) that may be computer writeable and readable. The RF data <NUM> and the instructions <NUM> are stored in the storage medium <NUM>. In operation, the RF data <NUM> is used by the processor <NUM> along with an input signal (see arrow <NUM> in <FIG>) indicative of the measured ambient RF signal <NUM> when executing the instructions <NUM> to determine the existence of the moving presence <NUM> within the region of interest <NUM>. The RF data <NUM> includes RF background data that is indicative of no moving presence. The RF background data may be learned by the processor <NUM> via, for example, machine learning algorithms as part of the instructions <NUM>. The RF data may include features that allow determining if the ambient RF signal <NUM> is generated by a transmitting component <NUM> that is stationary either by matching the MAC address or by looking at the temporal variations of the RF data <NUM> or both. The RF data <NUM> may further include extracted features associated with the signal characteristics that are attributed to motion of a presence <NUM>, which may be invariant to RF background changes (i.e., temporal variation in the ambient RF signals <NUM> attributed to motion).

Referring to <FIG>, and in operation, the activity recognition system <NUM> when initialized may be in an auto configuration at block <NUM>. At block <NUM>, the control circuitry <NUM> may determine if the system is armed. If no, the system <NUM> loops back to apply block <NUM> again. If the system is armed, and at block <NUM>, the control circuitry <NUM> applies an auto calibration that entails self-learning of the RF background data and extracted features previously described. At block <NUM>, the system <NUM> may continuously monitor for a moving presence <NUM> within the region of interest <NUM> by generally comparing the measured ambient RF signal <NUM> to the RF background data and/or extracted features stored as part of the RF data <NUM>. At block <NUM>, and based on this comparison, the control circuitry <NUM> determines if an activity (e.g., moving presence <NUM>) is detected. If not, the system loops back to block <NUM>. If a moving presence <NUM> is detected, and at block <NUM>, the control circuitry <NUM> may affect the triggering of an alarm via the notification device <NUM>.

Referring to <FIG>, a method of preprocessing the ambient RF signals, or data <NUM>, before applying machine learning or other data analysis algorithms, is illustrated. At block <NUM>, the ambient RF data <NUM> is captured by the RF sniffer <NUM> of the activity recognition system <NUM> and sent to the control circuitry <NUM> as an input signal <NUM> (see <FIG>). In one example, the captured ambient RF data may be WiFi CSI data, and the RF sniffer <NUM> may be a CSI monitor. At block <NUM>, noise as part of the ambient RF data <NUM> is reduced. This noise reduction of the data <NUM> is facilitated by removing a mean value of multiple CSI subcarriers at the same time index to subtract common mode noise. The operations of blocks <NUM> and <NUM> may be part of the prescribed instructions <NUM> stored in the storage medium <NUM> of the control circuitry <NUM>, and executed by the processor <NUM>.

At block <NUM>, background is subtracted. In one example, subtraction of the background is facilitated by converting the ambient RF data <NUM> with noise reduced, to the data's first order derivative of time. This step "flattens" the environmental background data to assist in the detection of signals attributable directly to activity recognition. At block <NUM>, the background subtracted, ambient, RF data <NUM> is converted to an image <NUM> that may be stored in the storage medium <NUM> (see <FIG>). In one example, conversion to the image <NUM> is facilitated by the combination of data from multiple antenna channels, that is then converted into an image format. As an image, the processed ambient RF data <NUM> may be analyzed by image processing algorithms <NUM> as part of the instructions <NUM>. In this way, the algorithms <NUM> may make use of the coherence between adjacent RF signal subcarriers that are typically not considered by other, more traditional, algorithms. The operations of blocks <NUM> and <NUM> may be part of the prescribed instructions <NUM> stored in the storage medium <NUM> of the control circuitry <NUM>, and executed by the processor <NUM>.

At block <NUM>, a successive image <NUM> is generated for each one of a plurality of successive images for each one of a plurality of time intervals <NUM> preprogrammed and stored in the storage medium <NUM>. The duration of each time interval <NUM> is established by, and associated with, characteristics of the specific building region <NUM> in which the ambient RF data <NUM> is detected. The plurality of successive images <NUM> facilitates use of a deep learning network for training of a shift invariant property. One example of a network is a Convolutional Neural Network (CNN). The operations of blocks <NUM> may be part of the prescribed instructions <NUM> stored in the storage medium <NUM> of the control circuitry <NUM>, and executed by the processor <NUM>. At block <NUM>, the image processing algorithm <NUM> is applied to the plurality of successive images <NUM>.

Advantages and benefits of the method for preprocessing the ambient RF signals <NUM> is a reduction of noise level of the ambient signals and only keeping activities information with background subtraction. Moreover, the method combines different channels into an image data format that enriches the information level in the training data to achieve optimal recognition results.

The various functions described above may be implemented or supported by a computer program that is formed from computer readable program codes and that is embodied in a computer readable medium. Computer readable program codes may include source codes, object codes, executable codes, and others. Computer readable mediums may be any type of media capable of being accessed by a computer, and may include Read Only Memory (ROM), Random Access Memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or other forms.

Terms used herein such as component, module, system, and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, or software execution. By way of example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. It is understood that an application running on a server and the server may be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.

Claim 1:
A method of operating an activity recognition system (<NUM>) in a building (<NUM>), wherein a plurality of wireless radio devices (<NUM>), each including a transmitting component configured to transmit respective radio frequency, RF, signals, and a receiving component configured to receive the RF to accomplish a primary task are located in the building; the method comprising:
using an RF sniffer in a region of the building, capturing ambient RF data by sampling and measuring characteristics of the ambient RF signals in the region of the building ;
receiving the ambient RF data by a processor (<NUM>);
reducing noise content of the ambient RF data by the processor;
subtracting background from the ambient RF data by the processor;
converting the ambient RF data with reduced noise and subtracted background into an image by the processor;
generating a successive image for each one of a plurality of time intervals by the processor; and
apply an image processing algorithm storing in a storage medium (<NUM>) and executed by the processor to each successive image to compare the plurality of successive images to the RF background data and to thereby determine activity recognition;
wherein the noise content of the ambient RF data is reduced by removing a mean value of multiple Channel State information, CSI, subcarriers at the same time index to subtract common mode noise.