Patent Description:
Traditional approaches for detecting fire include particle sampling (i.e., smoke detectors) and temperature sensors. While accurate, these methods include a number of drawbacks. For instance, traditional particle or smoke detectors require smoke to physically reach a sensor. In some applications, the location of the fire or the presence of heating, ventilation, and air conditioning (HVAC) systems prevents smoke from reaching the detector for an extended length of time, allowing the fire time to spread. A typical temperature sensor requires the sensor to be located physically close to the fire, because the temperature sensor will not sense a fire until a sufficient amount of the heat that the fire produces has spread to the location of the temperature sensor. In addition, neither of these systems provides as much data as might be desired regarding size, location, or intensity of the fire.

<CIT> and <CIT> disclose methods for detecting a flame using an image sensor.

A method of detecting a flame according to the invention is defined in claim <NUM>.

Optionally, the step of calculating the image flow vectors for each of the series of infrared images includes calculating an absolute magnitude of the image flow vectors.

Optionally, the step of calculating the image flow vectors for each of the series of infrared images includes calculating an amplitude of the image flow vectors at least one direction.

Optionally, the infrared metrics for the 3D image flow data set and the 3D pixel intensity data set include at least one of a minimum value, a maximum value, an average value, a median value, and a percent modulation.

Optionally, the infrared metrics for the 2D array of Fourier Transforms based on the at least one 3D image flow data set and the 2D array of Fourier Transforms based on the 3D pixel intensity data set include temporal frequency metrics of at least one of peak amplitude, frequency at peak amplitude, and percent modulation.

Optionally, the 2D threshold array includes an analysis of each pixel for each of the series of infrared images indicating a presence of the flame located at each of the pixels.

Optionally, a series of second images are acquired from a second imager.

Optionally, the method further includes the steps of calculating image flow vectors for each of the series of second images and storing the image flow vectors for each of the series of second images in a corresponding second 2D vector array. At least one second 3D image flow data set is generated from the corresponding second 2D vector arrays. A 2D array of 1D Fourier Transforms is calculated based on the at least one second 3D image flow data set. Second imager metrics are calculated based on the at least one second 3D image flow data set and the 2D array of Fourier Transforms based on the at least one second 3D image flow data set. The second metrics are compared to second imager threshold criteria to detect a flame. A corresponding second 2D threshold array is generated for each of the series of images from the second imager and a second 3D threshold mapping data set is generated from the second 2D threshold arrays. The flame status is validated based on the second 3D threshold mapping data set. An updated flame status is conveyed to the user device.

Optionally, the second imager includes a visible light imager and the image flow vectors for each of the series of second images are based on at least one of red light reception, green light reception, or blue light reception by the visible light imager.

Optionally, the second imager includes at least one of an ultraviolet imager or a near-infrared imager.

Optionally, the second imager includes at least one of a mid-wave infrared imager or long-wave infrared imager.

Optionally, the method further includes generating a second 2D pixel intensity array for each of the series of second images. A second 3D pixel intensity data set is generated based on the second 2D pixel intensity arrays for each of the series of second images. A 2D array of 1D Fourier Transforms is calculated based on the second 3D pixel intensity data set. Second infrared metrics are calculated for the second 3D pixel intensity data set and the 2D array of Fourier Transforms based on the second 3D pixel intensity data set.

Optionally, the flame status is indicated to a user device by highlighting a region on a display identifying a flame region.

Optionally, determination of the flame status includes image processing with at least one of spatial blob analysis or blob motion analysis.

An image recognition system according to the invention is defined in claim <NUM>.

Optionally, the controller is further configured to perform the step of acquiring a series of second images from a second imager.

Optionally, the controller is further configured to perform the following steps including calculating image flow vectors for each of the series of second images and storing the image flow vectors for each of the series of second images in a corresponding second 2D vector array. At least one second 3D image flow data set is generated from the corresponding second 2D vector array. A 2D array of 1D Fourier Transforms is calculated based on the at least one second 3D image flow data set. Second imager metrics are calculated based on the at least one second 3D image flow data set and the 2D array of Fourier Transforms based on the at least one second 3D image flow data set. The second metrics are compared to a second imager threshold criteria to detect a flame. A corresponding second 2D threshold array is generated for each of the series of images from the second imager and a second 3D threshold mapping data set is generated from the second 2D threshold arrays. The flame status is validated based on the second 3D threshold mapping data set. An updated flame status is conveyed to the user device.

Optionally, the second imager includes a visible light imager. The image flow vectors for each of the series of second images are based on at least one of red light reception, green light reception, and blue light reception by the visible light imager.

Optionally, the second imager includes at least one of a mid-wave infrared imager or a long-wave infrared imager.

Optionally, the controller is further configured to perform the following steps including generating a second 2D pixel intensity array for each of the series of second images. A second 3D pixel intensity data set is generated based on the second 2D pixel intensity arrays for each of the series of second images. A 2D array of 1D Fourier Transforms is calculated based on the second 3D pixel intensity data set. Second infrared metrics are calculated for the second 3D pixel intensity data set and the 2D array of Fourier Transforms based on the second 3D pixel intensity data set.

<FIG> illustrates an example image recognition system <NUM> for identifying the presence of a flame <NUM> in a scene <NUM>. The image recognition system <NUM> acquires a series of images with at least one imager <NUM>. In the illustrated example, the at least one imager includes an infrared imager 22A. The series of images are processed by a controller <NUM> to provide information regarding the presence of a flame <NUM> to a display <NUM> to be viewed by a user or to trigger an alarm <NUM> to notify the user of a potential flame <NUM> in the scene <NUM>. In order to perform the image processing, the controller <NUM> includes a microprocessor in communication with memory for storing programs to analyze the series of images collected by the at least one imager <NUM>. The system <NUM> can also include additional imagers 22B-F for validating the potential of the flame <NUM> in the scene <NUM> determined by the infrared imager 22A as will be discussed further below.

<FIG> illustrates a method <NUM> of operating the image recognition system <NUM>. The method <NUM> is stored on a computer readable medium, such as the memory, which is configured to cause the controller <NUM> to perform the method <NUM> outlined in <FIG> and described below. Initially, the system <NUM> acquires a series of infrared images <NUM> (<FIG>) from the infrared imager 22A of the scene <NUM> for determining the presence of the flame <NUM>. Step <NUM>. The infrared imager 22A is capable of capturing imagery that indicates the heat signature of the scene on a per pixel basis of the images <NUM>. Each image <NUM> in the series of images <NUM> is composed of an array of pixels defined in an X-Y or Cartesian coordinate system. The series of images <NUM> may be of a fixed number of recent sequential images which is maintained on an ongoing basis by adding new images and removing old images as new images are available.

With the series of infrared images <NUM> captured by the infrared imager 22A and stored in the memory in the controller <NUM>, the controller <NUM> calculates an image flow <NUM> (<FIG>) for each image <NUM> in the series of infrared images <NUM>. Step <NUM>. The calculated image flow <NUM> includes a plurality of vectors <NUM> that correspond to each pixel in a corresponding one of the images <NUM>. The image flow <NUM> is calculated using existing methods and algorithms, such as the Lucas Kanade method. Furthermore, the image flow <NUM> is calculated for a given infrared image <NUM> by using the infrared image <NUM> immediately preceding the given infrared image <NUM> in the series as a reference frame. The vectors <NUM> in the image flows <NUM> can be represented in a Cartesian coordinate system as having an absolute magnitude or length and an angle θ relative to the X-axis as shown in <FIG>.

The magnitude for each vector <NUM> in the image flow <NUM> is stored in a corresponding cell of a 2D magnitude array <NUM> as shown in <FIG>. The 2D magnitude arrays <NUM> are stored in the memory for each image flow <NUM> of the series of images <NUM> for a predetermined number of images <NUM> to generate a 3D magnitude data set <NUM> (2D for spatial position and 1D for image number) as shown in <FIG>.

Similarly, the angle θ for each vector <NUM> in each image flow <NUM> is stored in a corresponding cell of a 2D angle array <NUM> as shown in <FIG>. The 2D angle arrays <NUM> are stored in the memory for each image flow <NUM> in the series of images <NUM> for a predetermined number of images <NUM> to generate a 3D angle data set <NUM> (2D for spatial position and 1D for image number) as shown in <FIG>.

Additional information can be generated from the information stored in the 3D magnitude data set <NUM> and the 3D angle data set <NUM>. For example, the magnitude values from the 2D magnitude arrays <NUM> can be used in connection with a corresponding angle from the 2D angle arrays <NUM> to determine an amplitude of the vectors <NUM> in each cell in a number of directions. As shown in <FIG>, the amplitude of the vector <NUM> can be taken in the zero degree direction, <NUM> degree direction, <NUM> degree direction, and <NUM> degree direction.

The amplitude in the zero degree direction is provided in Equation <NUM> below by multiplying the absolute magnitude V from the 2D magnitude array <NUM> by the cosine of the angle θ from a corresponding cell in the 2D angle array <NUM>. The amplitude in the <NUM> degree direction is provided in Equation <NUM> by multiply the absolute magnitude V from the 2D magnitude array <NUM> with the sine of the angle θ from the corresponding cell in the 2D angle array <NUM>. The amplitude in the <NUM> degree direction is provided in Equation <NUM> by multiplying the absolute magnitude V from the 2D magnitude array <NUM> with the cosine of (θ + <NUM> degrees) from the corresponding cell in the 2D angle array <NUM>. The amplitude in the <NUM> degree direction is provided in Equation <NUM> by multiplying the absolute magnitude V from the 2D magnitude array <NUM> with the sine of (θ + <NUM> degrees) from the corresponding cell in the 2D angle array <NUM>. <MAT> <MAT> <MAT> <MAT>.

With reference to <FIG>, applying the above equations to the 2D magnitude arrays <NUM> and the 2D angle arrays <NUM> will result in a series of 2D directional arrays 46A-46D for each infrared image <NUM> that form a 3D directional data set 48A-48D over time in each of the chosen directions above. Step <NUM>. As shown in <FIG>, once the 3D directional data sets 48A-48D (<FIG>) are formed from 2D vector arrays <NUM>, <NUM>, the controller <NUM> calculates a magnitude curve over time <NUM> based on the 3D magnitude data set <NUM> and, for each of the 3D directional data sets 48A-48D, respectively, calculates amplitude curves representing the directional amplitude of the vector <NUM> over time 52A-52D. The representations of data shown in <FIG>, and <FIG> are for illustrative purposes only. The actual data will vary for each case as well as across the image.

A Fourier Transform of each of magnitude curve <NUM> and the amplitude curves 52A-52D is then performed by the controller <NUM> to produce a respective magnitude transform curve <NUM> and amplitude transform curves 56A-56D. Step <NUM>. The magnitude transform curve <NUM> and the amplitude transform curves 56A-56D represent a frequency (Hz) of values based on their respective curves. The controller <NUM> then determines metrics for the magnitude curve <NUM> and amplitude curves 52A-52D including at least one of a minimum value, a maximum value, an average value, a median value, and a percent modulation and metrics for the transform magnitude curve <NUM> and the amplitude transform curves 56A-56D including at least one of peak amplitude, frequency at peak amplitude, and percent modulation. Step <NUM>. These metrics are obtained from the spatial and temporal aspects of the images <NUM>, using efficient calculation methods such as frequency analysis and image flow calculations, and are relatable to the physical characteristics of flames <NUM> (ex. flicker) for the purpose of flame detection.

Because the images <NUM> are infrared images, the images convey a heat signature through a pixel intensity for each pixel in the series of images <NUM>. The pixel intensities for the infrared images <NUM> are stored in a series of corresponding 2D pixel intensity arrays <NUM> that forms a 3D pixel intensity data set <NUM> (<FIG>). Step <NUM>. The controller <NUM> calculates a magnitude curve <NUM> representing the magnitude of the pixel intensity for each corresponding cell in the 2D pixel intensity arrays <NUM> that form the 3D pixel intensity data set <NUM>. A Fourier Transform of the magnitude curve <NUM> is then performed by the controller <NUM> to produce a transform magnitude curve <NUM>. The transform magnitude curve <NUM> indicates a frequency of the magnitude curve <NUM>. Step <NUM>. The controller <NUM> then determines metrics for the magnitude curve <NUM> including at least one of a minimum value, a maximum value, an average value, a median value, and a percent modulation and metrics for the transform curve <NUM> include at least one of peak amplitude, frequency at peak amplitude, and percent modulation. Step <NUM>.

The controller <NUM> then compares to threshold values the metrics determined for the curves <NUM>, <NUM>, 52A-52D, 56A-56D, <NUM>, and <NUM>. Step <NUM>. The controller <NUM> evaluates the comparison between the metrics for the curves <NUM>, <NUM>, 52A-52D, 56A-56D, <NUM>, and <NUM> and the threshold values for each pixel and generates a 2D logic array <NUM> indicating whether or not a flame is detected in a given pixel (<FIG>). Step <NUM>. For example, a value of "<NUM>" in an array location would indicate a flame was detected in a corresponding pixel and a value of "<NUM>" would indicate that a flame was not detected in the corresponding pixel. The 2D logic array <NUM> is saved over time to create a 3D detection data set <NUM> (<FIG>. Step <NUM>).

The controller <NUM> then determines a 2D flame status based on the 3D detection data set <NUM>. In one example, the controller <NUM> applies at least one of blob motion or blob analysis and/or applies a predetermined criterion for clustering and persistence. Step <NUM>. The controller <NUM> then conveys the flame status to a user device of the system <NUM>. Step <NUM>. A user device may include a control panel interface, a mobile device such as a smart phone or other electronic user device with a display and an interface, a laptop, etc. The flame status can include highlight of a region of a portion of one of the images <NUM> on a display device, in order to indicate a region with a possible flame. Alternatively, the controller <NUM> could signal an alarm, for example, by sending a message indicating an alarm status to a control panel, or by sounding an audio device.

As discussed above, additional imagers <NUM> can be used in connection with the infrared imager 22A to further validate the results based on the series of infrared images and eliminate false positive flame detections. In one example, the additional imagers <NUM> could include a MidWave Infrared (MWIR) imager 22B and/or a LongWave Infrared (LWIR) imager 22C. These two imagers 22B and 22C cover wavelengths from approximately <NUM> to <NUM> microns with a <NUM> to <NUM> micron band being covered by the MWIR imager 22B and the <NUM> to <NUM> micron band being covered by the LWIR imager 22C. There is a lot of atmospheric absorption in about the <NUM> to <NUM> micron region, which makes it less useful in general.

Objects emit energy as a function of their temperature and as a function of wavelength. The imagers 22B, 22C may have a sensitivity over a wavelength band that is different than the wavelength band covered by the infrared imager 22A and can also provide approximate temperature or energy output of objects in the scene <NUM>. Because the imagers 22B and 22C are infrared, the controller <NUM> processes a series of images from the imagers 22B and 22C in a similar manner as the series of images collected by the infrared imager 22A. Although the same metrics will apply to the images collected by the imagers 22B, 22C as the infrared imager 22A, the physical meaning of the metrics are different such that the threshold values in step <NUM> are different to account for variations in reception by the imagers 22B, 22C.

A visible light imager 22D could be used in addition to the imagers 22B, 22C or in place of the imagers 22B, 22C. The same method <NUM> and metrics can be used as above for MWIR or LWIR imagers 22B, 22C, but the thresholds would be applied differently as the results from the visible light imager 22D have a different physical meaning. Moreover, unlike infrared based imagers, the intensity values from a visible light imager are not relatable to physical quantities of energy or temperature.

Furthermore, since visible light is often measured in three channels, such as RED, GREEN, and BLUE, the opportunity exists to maintain additional metrics resulting from the ratios or summations of the channels. For example, analogous to the pixel intensity maintained for infrared, a 2D array of RED + GREEN + BLUE may be determined to provide a total intensity that is stored over time to maintain a 3D light intensity data set. Other combinations could be stored in a 2D array over time to form a 3D data set such as just red or RED / (GREEN + BLUE). The same metrics can be derived as were derived for the infrared (related to frequency, image flow, etc.).

One additional feature of the visible light imager 22D, is that water and/or water vapor in the atmosphere are fairly transparent in the visible light region measured by the visible light imager 22D. Additionally, visible light imagers 22D are inexpensive compared to infrared based imagers.

One additional type of imager that could be used in connection with the infrared imager 22A is an ultraviolet imager 22E. For flame detection, the most useful UV band is the UV-C band of approximately <NUM> to <NUM>. The same metrics can be used as above for MWIR/LWIR imagers 22B, 22C, but the thresholds that would be applied would be different, as the results from the UV imager have a different physical meaning.

UV flame detectors are typically filtered to include just the 'UV-C' band because the sun's radiation at this band is absorbed by the earth's atmosphere. The result is that the ultraviolet imager 22E will not cause an alarm in response to radiation from the sun. Ultraviolet detectors and imagers 22E are sensitive to most fires with a sensitivity that differs from an infrared based imager. So a system that operates using both UV and IR has a validating feature over a system that operating solely in infrared. For example, a burning hydrogen flame radiates strongly in the UV-C band and a coal fire emits more weakly in the UV-C band but more strongly in infrared. However, UV imagers are obstructed by oil or grease in the optics and are sensitive to non-flame sources such as a plasma glow from an arc welding machine or lightening.

Yet another imager used in connection with the infrared imager 22A would be a Near-Infrared imager 22F. Near-Infrared imagers 22F covers wavelengths over approximately <NUM>-<NUM>. The same metrics can be used as above for MWIR and LWIR imagers 22B, 22C, but the thresholds that would be applied would be different as the results from the near IR imager have a different physical meaning. Similar to the visible light imager 22C, water and water vapor in the atmosphere are fairly transparent in Near-Infrared wavelength and Near-Infrared imagers can be relatively inexpensive compared to MWIR or LWIR imagers 22B, 22C.

Claim 1:
A method of detecting a flame (<NUM>) in an image comprising the steps of:
acquiring a series of infrared images (<NUM>) from an infrared imager (22A);
calculating image flow vectors (<NUM>), that correspond to each pixel in a corresponding one of the infrared images (<NUM>), for each of the series of infrared images and storing the image flow vectors in a corresponding 2D vector array (<NUM>, <NUM>);
generating at least one 3D image flow data set (48A-48D) from the corresponding 2D vector arrays;
generating a 2D pixel intensity array (<NUM>) for each of the series of infrared images;
generating a 3D pixel intensity data set (<NUM>) from the 2D pixel intensity arrays; the method characterised in that it further comprises the steps of:
calculating a Fourier Transform (<NUM>, 56A-56D, <NUM>) based on the at least one 3D image flow data set and the 3D pixel intensity data set;
calculating infrared metrics based on the at least one 3D image flow data set (48A-48D), the 3D pixel intensity data set (<NUM>), the Fourier Transform (<NUM>, 56A-56D) based on the 3D image flow data set, and the Fourier Transform (<NUM>) based on the 3D pixel intensity data set;
comparing the infrared metrics to an infrared threshold criteria to detect a flame;
generating a corresponding 2D threshold array for each of the series of infrared images based on the infrared metrics and generating a 3D threshold data set from the 2D threshold arrays;
determining a flame status based on the 3D threshold data set; and
conveying the flame status to a user device.