Patent Description:
Infrared detectors used for intrusion and presence detection are typically limited to pixel counts of about four-by-four (4x4) elements to stay reasonable in terms of cost and performance. Even with advances in MEMS, the pixel counts remain less than approximately one hundred-by-one hundred (100x100). The manufacturing process for these low cost detectors does not scale well in terms of cost as pixel count increases. Additionally, the physical size of an infrared focal plane array is large compared to the same pixel count for, as one example, complementary metal oxide silicon (CMOS) visible sensors because of the longer wavelength. As such, one-by-one (1x1) to four-by-four (4x4) pyroelectric elements are commonplace as, for example, occupancy detectors, but even in sizes up to approximately one hundred-by-one hundred (100x100) they are not able to count with the fidelity needed to achieve more efficiently controlled heating, ventilation, and air conditioning (HVAC) systems and lighting. Yet further, energy consumption of infrared focal plane arrays becomes larger than desired for arrays having sufficient pixels to meet fidelity needs when supporting other systems such as HVAC and lighting.

Typical state-of-the-art presence detection may use a two-element passive infrared (PIR) sensor. These sensors typically include faceted lens designs and may include masks and variable detection thresholds to achieve useable performance. Unfortunately, such PIR sensors may have difficulty in distinguishing people from other heat sources (e.g., animals, HVAC operation, etc.), may not be able to localize or track the detected object, and may not be able to count the number of objects.

Typical state-of-the-art algorithms for people detection, classification, tracking and counting have been developed in the field of computer vision. For instance, there are state-of-the-art object detection algorithms for people detection and tracking including Support Vector Machines (SVM) on Histogram of Oriented Gradient (HOG) features, and discriminatively trained Deformable Part Models (DPM). Unfortunately, these algorithms are designed to work on visible spectrum, multi-color video with many hundreds or thousands of pixels on target. It is desirable to design detector systems and associated algorithms for infrared video and/or video with relatively few pixels on target (i.e., tens to a few hundreds of pixels). It is further desirable to develop cost effective detector systems and methods that perform occupancy detection and people counting with improved fidelity and reduced energy consumption to, for example, support HVAC and lighting systems.

According to an aspect of the invention there is provided a method of operating an object detection and tracking system as defined in claim <NUM>.

The Bayesian Estimator may be a Kalman Filter.

The Bayesian Estimator may be a Particle Filter.

However, it should be understood that the following description and drawings are intended to be exemplary in nature and non-limiting.

Referring to <FIG>, a building management system <NUM> of the present disclosure is illustrated. The building management system <NUM> may include at least one of an ambient air temperature control system <NUM>, a security system <NUM>, a lighting or illumination system <NUM>, a transportation system <NUM>, a safety system <NUM> and others. Each system <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may be associated with and/or contained within a building <NUM> having a plurality of predefined spaces <NUM> that may generally be detached or substantially isolated from one-another, may be accessible and/or interconnected via a door and/or through hallways (not shown) and other means.

The ambient air temperature control system <NUM> may be a forced air system such as a heating, ventilation, and air conditioning (HVAC) system, a radiant heat system, and others. The security system <NUM> may be configured to detect intruders and provide various forms of alerts and notifications. The lighting system <NUM> may control and/or monitor lighting in each one of the predefined spaces <NUM> based on any number of factors including natural background lighting, occupancy, and others. The transportation system <NUM> may include the control and/or monitoring of elevators, escalators, and other transportation devices associated with and/or within the building <NUM>. The safety system <NUM> may include the detection of conditions that may pose a risk or health hazard to occupants of the building <NUM>. All of these systems <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may require a variety of devices to perform any variety of functions including detection, monitoring, communication, data referencing and collection, user control, and others. Many devices may be shared between systems.

The building management system <NUM> may further include a computing device <NUM> that controls and/or supports each system <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. The computing device <NUM> may include a processor <NUM> (e.g., microprocessor) and a computer readable and writeable storage medium <NUM>. It is further contemplated and understood that the building management system <NUM> may include more than one computing device <NUM> with any one computing device being dedicated to any one of the systems <NUM>, <NUM>, <NUM>, <NUM>, <NUM>.

Referring to <FIG> and <FIG>, in one embodiment, the HVAC system <NUM> of the building management system <NUM> may include a common supply duct plenum <NUM> that flows heated or cooled air into a plurality of ducts <NUM> routed to respective spaces <NUM> of the building <NUM>. A temperature thermostat <NUM> may be located in each space <NUM> for controlling and monitoring ambient air temperature. In operation, the thermostat <NUM> may send a signal over pathway <NUM> to the computing device <NUM> when a demand for heating and/or cooling is needed. In-turn, the computing device <NUM> may initiate the HVAC system <NUM> that may include the control and/or energization of a <NUM> located in each duct <NUM>. The control and/or energization of the dampers <NUM> is conducted over pathways <NUM>, and enables the controlled flow of heated or cooled air (see arrows <NUM>) from the supply plenum <NUM> and into the respective spaces <NUM>.

Referring to <FIG> and <FIG>, the building management system <NUM> includes an object detection and tracking (ODT) system <NUM>. The ODT system <NUM> utilizes low cost and low resolution sensors and associated computer vision algorithms to accurately detect, classify, track and count objects (e.g., people) in a given space using minimal energy consumption; and in one embodiment, enabling functions of the building management system <NUM> (e.g., HVAC system <NUM>, lighting system <NUM>, security system <NUM> and others). For example, the computing device <NUM> may receive a signal from the ODT system <NUM> indicative of no persons in a given space <NUM>. Upon such a signal, the computing device <NUM> may output a command signal to the HVAC system <NUM> to, for example, lower a temperature set-point, and/or send a signal to the lighting system <NUM> to turn off the lights in the given space <NUM>. As another example, the computing device <NUM> may receive a signal from the ODT system <NUM> indicative of a person entering or in the space <NUM>. Upon such a signal, the computing device <NUM> may output a signal to the security system <NUM> indicative of an intrusion whereupon an alarm may be initiated.

The ODT system <NUM> may include a plurality of local units <NUM> with at least one local unit located in at least one space <NUM>. The local units <NUM> may be configured to communicate with the computing device <NUM> over wired or wireless pathways (see arrow <NUM>). Each local unit <NUM> is configured to monitor a pre-scribed scene <NUM> in its associated space <NUM> (e.g., the entire space or a particular restricted Field of View (FOV)) for detection, classification, tracking and counting of objects <NUM>. An example of an object <NUM> may be a human or may be anything that emits heat. The local unit <NUM> may include a sensor <NUM> that is low resolution, a memory module <NUM>, a sensor data compression block <NUM>, a processor <NUM>, a communication module <NUM>, a power management module <NUM>, and a power source <NUM>.

Referring to <FIG>, the low resolution sensor <NUM> is an infrared focal plane array (FPA) configured to sense and detect radiated heat emitted by the objects <NUM> (e.g., a human). The FPA <NUM> is `low resolution' because it includes less than one thousand pixels and preferably ten to three hundred pixels. The space <NUM> is a 'large' space relative to the low resolution FPA <NUM> (i.e., relatively low number of pixels per unit area). The FPA <NUM> may include a row decoder <NUM>, a column decoder <NUM> (which are part of the Read-Out Integrated Circuit (ROIC)), and the plurality of pixels <NUM> that may be infrared sensors arranged in a series of rows and columns (i.e., six rows and nine columns illustrated in <FIG>). The row and column decoders <NUM>, <NUM> are electrically coupled to the respective rows and columns of the pixels <NUM>, and are configured to receive intensity information (e.g., heat intensity) recorded over a time interval. As one example, the pixels <NUM> may be configured to sense radiated energy having an infrared, long wavelength that may be within a range of about three (<NUM>) to fifteen (<NUM>) micrometers. This range is a thermal imaging region, in which the pixels <NUM> may obtain a passive image of the objects <NUM> that are in whole or in part at only a slightly higher temperature than, for example, room temperature. This image may be based on thermal emissions only and may require no visible illumination.

The memory module <NUM> of the local unit <NUM> is generally a computer readable and writeable storage medium and is configured to communicate with the processor <NUM> and generally stores intensity data from the sensors <NUM> for later processing, stores executable programs (e.g., algorithms) and their associated permanent data as well as intermediate data from their computation. The memory module <NUM> may be a randomaccess memory (RAM) that may be a ferroelectric RAM (FRAM) having relatively low power consumption with relatively fast write performance, and a high number of writeerase cycles. It is further contemplated and understood that the ODT system <NUM> may be integrated in-part with the computing device <NUM> that may also perform, at least in-part, a portion of the data processing of data received from the FPA <NUM>.

The radiant energy intensity information/data received by the decoders <NUM>, <NUM> may be conditioned via a signal conditioning circuit (not shown) and then sent to the processor <NUM>. The signal conditioning circuit may be part of the ROIC. Signal conditioning may include analog-to-digital converters and other circuitry to compensate for noise that may be introduced by the sensors <NUM>. The processor <NUM> may be configured to provide focal plane scaling of the intensity value data received from the signal condition circuit and may further provide interpolation techniques generally known in the art. The processor <NUM> is generally computer-based and examples may include a postprocessor, a microprocessor and/or a digital signal processor.

The sensor data compression block <NUM> of the local unit <NUM> is known to one having skill in the art and is generally optional with regard to the present disclosure.

The communication module <NUM> of the local unit <NUM> is configured to send and receive information and commands relative to the operation of the remote unit <NUM>. The communication module <NUM> may include a network coding engine block <NUM>, an analog to digital converter (ADC) <NUM>, a receiver <NUM> (e.g. wireless), and a transmitter <NUM> (e.g., wireless). As is well-known in the art, the transmitter <NUM> and receiver <NUM> may be implemented as a transceiver or could be replaced by a well-known wired communication link (not shown). Equally as is well known in the art, transmitter <NUM> performs digital to analog conversion (DAC) when the communication channel is inherently analog (e.g., wireless). The network coding engine block <NUM> is configured to interface the input and output of the processor <NUM> to transmitter <NUM>, receiver <NUM> (through ADC <NUM>), provide encoding (e.g., for error detection and correction), security via encryption or authentication, and other features.

The ADC <NUM> of the local unit <NUM> is configured to convert received analog information to digital information for eventual use by the processor <NUM>. The network coding engine <NUM> provides any decoding necessary for error detection and correction, and/or security.

The receiver <NUM> and the transmitter <NUM> of the local unit <NUM> are configured to respectively receive and transmit communications to and from other systems or components such as the computing device <NUM> of the building management system <NUM> and/or the HVAC system <NUM>. Such communications may be conducted over pathways that may be wired or wireless.

The power management module <NUM> of the local unit <NUM> is configured to control the power acquisition and power consumption of the remote unit <NUM> by controlling both the power source <NUM> and power consuming components. Such power consuming components may include the processor <NUM>, the optional data compression block <NUM>, the memory <NUM>, the FPA <NUM> and the communication module <NUM> (e.g., transmitter <NUM>, receiver <NUM>, and ADC <NUM>). It is contemplated and understood that other energy consuming components of the remote unit <NUM> may be controlled. Such control may simultaneously maintain the remote unit <NUM> functionality while maximizing life (i.e., the length of time the remote unit <NUM> can remain functional). In one embodiment, this control is achieved by receding horizon control (optimization). In alternative embodiments other control strategies such as model predictive control may be used. In one embodiment, the power consumption of processor <NUM> and memory <NUM> may be controlled by the power management module <NUM> by reducing the clock rate.

The power source <NUM> of the local unit <NUM> provides power to the other components of the unit, and may include at least one of a super capacitor <NUM>, a battery <NUM> and a solar cell <NUM>. The power management module <NUM> is configured to draw power from any one of the power sources as dictated by the needs of the system. The power management module <NUM> may also facilitate a power scheduling function that controls the simultaneous use of the various on-chip component functions to minimize unwanted current spikes. It is contemplated and understood that other short-term energy storage devices may be used in place of the super capacitor <NUM>, other long-term energy storage devices may be used in place of the battery <NUM>, and other energy harvesting or recharging devices may be used in place of the solar cell <NUM> including power from a power grid.

Referring to <FIG>, the FPA <NUM> (including the ROIC), the memory module <NUM>, the processor <NUM>, the power management module <NUM> and the communication module <NUM> may generally be integrated together on a single substrate platform or chip <NUM> that may be silicon-based. More specifically, the components may generally share the focal plane of the FPA <NUM>. Together, the integrated components may be aimed toward minimal power consumption, small overall size/weight and low cost. Integration of these components may be further enhanced via a power scheduling function conducted by the power management module <NUM> as well as coordinated design of the individual functions of each component to work harmoniously. That is, the power scheduling function may, for example, minimize unwanted current spikes by controlling the simultaneous use of the various on-chip components functions.

By placing individual subsystem components on the same die or substrate platform <NUM>, signal integrity, resistive losses and security is generally improved through elimination of interconnects and sources of extraneous electrical and radiative noise typically present in systems with similar functionality but that use several individually packaged integrated circuits (IC's). Moreover, by placing all components on the same substrate platform <NUM>, economy of scale is achieved that enables chip-scale cost reduction. Yet further, power management and consumption may be optimized potentially achieving long life battery operation, and facilitating packaging of various circuitry components on a single substrate platform <NUM>. The detector unit <NUM> may be built upon a ferroelectric memory platform using either active or passive detection; and, may be built upon a thermal isolator rather than a MEMS bridge, thereby improving yield, reducing across device response variations, and may be compatible with wafer production having small feature sizes.

The infrared FPA <NUM> functions via absolute intensity (i.e., chopped). In an absolute intensity sensor, an additional device known as a chopper (also called a rotating shutter), provides a reference constant-temperature image source and the difference from this reference view is the image provided to the user. Conversely, in a relative intensity sensor there is no chopper and only intensity differences from the previous image are provided. The key characteristic of these relative intensity images is that an object disappears if it does not move, because there is no temperature difference at a given pixel location from the previous image.

Referring now to <FIG> and <FIG>, a chopper <NUM> is interposed between the low resolution sensor <NUM> and the scene <NUM> so as to interrupt electromagnetic radiation (see arrow <NUM>) emanating from the scene <NUM> from reaching the low resolution sensor <NUM>. Chopper <NUM> may be any of a variety of types such as, for example, a rotating wheel with alternating transmissive areas <NUM> and non-transmissive areas <NUM>. The rotation of the chopper <NUM> is coordinated with the acquisition of images by the low resolution sensor <NUM> such that an image is acquired only when one of the transmissive areas <NUM> or non-transmissive areas <NUM> is fully in front of low resolution sensor <NUM>. When non-transmissive area <NUM> blocks electromagnetic radiation <NUM>, the low resolution sensor <NUM> receives a background amount of radiation from the chopper <NUM>. When transmissive area <NUM> passes electromagnetic radiation <NUM>, the low resolution sensor <NUM> receives radiation from scene <NUM>. Since low resolution sensor <NUM> is sensitive only to differences in received radiation, the entire scene <NUM>, both stationary and moving objects, will create signals on the pixels <NUM>.

In the ODT system <NUM>, the system utilizes advanced machine vision techniques to exploit both spatial and temporal relationships of small objects <NUM> (i.e., up to a few hundred pixels <NUM> on an object <NUM>) in low-cost, absolute intensity, infrared video from a small FPA <NUM> (e.g., <NUM>×<NUM>, 64x64 pixels (which however does not form part of the invention), etc.). Two approaches in tracking objects that subtend a small number of pixels in low-resolution video are described. The first approach, not falling within the scope of the present invention, entails using a sensor-specific object model comprising fitting non-normalized Mixture of Gaussian (MOG) distributions and Kalman Filter tracking of the distributions' means at sub-pixel resolution. The omission of normalization conserves the magnitude of the infrared radiation. While matched filtering with more sophisticated object models is known, the low number of pixels on the object <NUM> and the variation of the object shape (e.g., two-dimensional projection of people movement) may not support the use of sophisticated object models. In the second approach, falling within the scope of the present invention, a sparse dictionary learning approach is used for detection, where the magnitude of the sparse coefficient and location of the object (according to labels associated with the dictionary atoms) are used in tracking filter state vector. In alternative embodiments, other Bayesian Filters may be used for tracking such as a Particle Filter. The magnitude of non-normalized MOG estimate may be included in the state vector to help disambiguate tracks of objects and allow estimation of the count of the number of objects <NUM> (e.g., people).

The Mixture of Gaussian algorithm may be a fast, incremental algorithm that has a complexity of O(m*p*f) where 'm' is the number of Gaussian distributions (i.e., number of people in the scene at any one time), 'p' is the number of pixels, and `f i is the update frequency. The Kalman filter has a known complexity of O(p<NUM>*f) where again 'p' is the number of pixels and `f i is the update frequency.

Referring to <FIG>, one example of a method of operating the ODT system <NUM> according to the invention includes, as block <NUM>, receiving the next frame of data from a sensor. The sensor is a two-dimensional infrared FPA. The frame data may be received at a periodic rate, on demand from the "Receive Next Frame" function, or at any scheduled, pseudo random, or random rate.

As block <NUM>, a static or a dynamic background is estimated. The background estimate may be performed a priori and may be non-adaptive (e.g., a codebook model). Alternatively, the background estimate may be updated based on the current frame using, for example, a Mixture of Gaussians approach. Other approaches as are well-known in the art may also be used.

As block <NUM>, foreground objects are estimated by comparing the received frame to the background estimate. This step may optionally include various morphological operations (not shown) to, for example, filter noise.

As block <NUM>, an object is detected. Unique to the detection of small objects in low-resolution chopped data is the use of a sensor-specific (chopped-data) object model, see block <NUM>. In an alternate embodiment not falling within the scope of the present invention, the object model may be a saturated Gaussian Mixture Model (GMM) where the highest values are truncated to a threshold. The object model of the alternate embodiment not falling within the scope of the present invention may be parameterized in part or in total by perspective data (see block <NUM>) (e.g., the variance of the distribution may vary by the geometry of the scene (i.e., nearer objects are larger). In an embodiment falling within the scope of the present invention, an empirical object model is employed where the object model is learned by discriminative dictionary learning from training data (see block <NUM>). The detection itself comprises computing the parameters of the object model and inferring from those parameters the number and location of the objects to sub-pixel accuracy. The mean of each object two-dimensional Gaussian distribution may be the location.

As block <NUM>, the objects, once detected, are tracked, preferably using a Kalman Filter. In one embodiment, a Kalman Filter as one type of Bayesian Estimator is employed. In another embodiment, a Particle Filter as another type of Bayesian Estimator, is employed.

As block <NUM>, the detected objects may be counted and classified by any well-known technique.

Benefits of the present invention include a low-cost, accurate, people presence detection and counting for building management systems including energy optimization and security.

Claim 1:
A method of operating an object detection and tracking system (<NUM>) comprising at least one low resolution sensor (<NUM>), wherein the low resolution sensor is an infrared focal plane array including less than one thousand pixels and configured to sense and detect radiated heat emitted by objects (<NUM>), the method comprising:
estimating (<NUM>) a current background of a current frame of sensor data generated by the low resolution sensor, based on a previous frame of sensor data by a computer-based processor;
estimating (<NUM>) a foreground of the current frame of sensor data by comparing the current frame of sensor data to the current background; and
detecting (<NUM>) an object (<NUM>) using a sensor-specific object model (<NUM>), wherein the sensor is an absolute intensity sensor utilizing a chopper (<NUM>) and wherein the sensor-specific object model is a chopped-data object model; characterised by
tracking (<NUM>) the object via a Bayesian Estimator;
wherein the chopped-data object model (<NUM>) is learned by discriminative dictionary learning (<NUM>).