Patent Description:
This invention was made with government support under N66001-<NUM>-<NUM>-<NUM> awarded by the Defense Advanced Research Projects Agency. The government has certain rights in the invention.

The present disclosure generally relates to automated heat radiation analysis, and in particular, to a computer vision method of creating textures in infrared thermal images.

This section introduces aspects that may help facilitate a better understanding of the disclosure. Accordingly, these statements are to be read in this light and are not to be understood as admissions about what is or is not prior art.

Autonomous vehicle navigation technologies are becoming more commonplace in certain vehicles. Object detection and ranging (distance of object from the vehicle) is of vital importance for these technologies. However, object detection and ranging proves to be specially challenging given the wide-ranging obstacles (e.g., road signs, road objects, other vehicles, etc.), harsh environments (bright daylight as well as pitch blackness of dark nights), and unknown terrains encountered in scenarios such as off-road driving.

The technology of choice for detection and ranging is LIDAR (light detection and ranging). LIDAR is based on illuminating an object by a laser and measuring laser return times and wavelength. These parameters can be used to generate a three dimensional (3D) map of the object and its distance from the vehicle (i.e., source of the illuminating laser). However, LIDAR's effectiveness falls rapidly with distance. Additionally, with a growing number of autonomous vehicles, LIDAR detection becomes cumbersome as the same object may be illuminated with multiple LIDARs.

An alternative technology involves use of passive 3D vision, which use optical (visible) stereovision; where cost-effective red-green-blue (RGB) cameras are used for scene analysis. This approach however suffers from challenges associated with stereovision, where the errors accumulated in ranging (i.e. depth estimation) increase quadratically with distance. Furthermore, there exist no systematic procedures for target recognition or semantic segmentation for applications such as off-road navigation.

Thermal images provide completely passive approach to detection and ranging. However, the lack of texture in thermal images leads to lack of discernible object features to the eye or to a computer system. This is a serious issue in using infrared thermal images for autonomous navigation.

Related methods are disclosed in <CIT>, <CIT>, and <NPL>.

Therefore, there is an unmet need for improving and creating textures in an infrared thermal image to address object detection, classification, and ranging for autonomous vehicle navigation.

A method of generating object surface texture in thermal infrared images is disclosed. The method includes receiving heat radiation from a scene by a spectropolarimetric imaging system adapted to generate a plurality of spectral frames associated with the scene. The method also includes generating the plurality of spectral frames associated with the scene, each frame having a plurality of pixels. Furthermore, the method includes for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel-specific temperature representing an object's temperature, and thermal texture factor representing the object's texture. Additionally, the method includes for each of a plurality of materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture. Furthermore, the method includes matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation, and extracting spectral metadata from the matched reference spectral information for the associated material based on the match.

According to one embodiment of the method, the plurality of spectral frames from the spectropolarimetric imaging system are each generated by applying a plurality of associated bandpass filters to the spectropolarimetric imaging system and passing the heat radiation therethrough.

According to one embodiment of the method, the extracted spectral information associated with the scene from the spectropolarimetric imaging system for each pixel from the generated plurality of spectral frames is based on <MAT>.

According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library is obtained from: <MAT>.

According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves i) based on a plurality of temperatures and ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between <NUM> and <NUM>.

According to one embodiment of the method, the matching of the extracted spectral information for each pixel from the spectropolarimetric imaging system from the generated plurality of spectral frames to the generated reference spectral information is based on matching Sv to Svm.

According to one embodiment of the method, the statistical method includes sum of least mean squares between the Sv and Svm meeting a predetermined threshold.

According to one embodiment of the method, the statistical method includes a minimum least mean squares between the Sv and Svm.

According to one embodiment of the method, the spectropolarimetric imaging system is further adapted to generate a plurality of polarization frames associated within the scene.

According to one embodiment the method further includes generating the plurality of polarization frames associated with the scene, each frame having a plurality of pixels.

According to one embodiment of the method, the plurality of linear polarization frames from the spectropolarimetric imaging system includes liner polarization at <NUM>°, <NUM>°, <NUM>°, and -<NUM>°, thereby generating I<NUM>, I<NUM>, I<NUM>, and I-<NUM> frames.

According to one embodiment of the method, for each pixel from the generated plurality of polarization frames, further extracting spectral information associated with the scene based on the polarization angles (Svp).

According to one embodiment of the method, the generated reference spectral information from the spectropolarimetric imaging system as a function of temperature and material texture for each material in the library includes a family of spectral curves (Svmp) i) based on a plurality of temperatures, ii) for each temperature of the plurality of temperatures, based on variation of thermal texture factor (X), wherein the thermal texture factor is a variable between <NUM> and <NUM>, and for each thermal texture factor (X), based on variation of polarization angle including <NUM>°, <NUM>°, <NUM>°, and -<NUM>°.

According to one embodiment of the method, the statistical method includes sum of least mean squares between the Svp and Svmp meeting a predetermined threshold.

According to one embodiment of the method, the statistical method includes a minimum least mean squares between the Svp and Svmp.

It will nevertheless be understood that no limitation of the scope of this disclosure is thereby intended.

In the present disclosure, the term "about" can allow for a degree of variability in a value or range, for example, within <NUM>%, within <NUM>%, or within <NUM>% of a stated value or of a stated limit of a range.

In the present disclosure, the term "substantially" can allow for a degree of variability in a value or range, for example, within <NUM>%, within <NUM>%, or within <NUM>% of a stated value or of a stated limit of a range.

The present disclosure provides a novel approach for object ranging that can be used in a variety of applications including autonomous vehicle navigation. This novel approach is based on detection of heat signature of objects, near and far away. Towards this end, the present disclosure describes a heat assisted detection and ranging (HADAR) approach which is based on capturing heat radiation - the intrinsic heat signature of a body - and can provide the unique spectral fingerprint for tactical semantic segmentation of scenes. Additionally, as infrared heat radiation in the <NUM>-<NUM> micron range - long wavelength infrared, (LWIR) - is omnipresent and can be exploited at day or night.

In order to use heat signature as a primary source of information for autonomous vehicle navigation, several challenges must be addressed: <NUM>) Ghosting (i.e., since heat radiation is omnipresent, the signal is cluttered with environmental thermal signals that cause diminished features or textures in thermal images, thereby necessitating new algorithms that distinguish useful target information from the environmental heat signatures that swamp or clutter the scene); and <NUM>) passive ranging accuracy and 3D vision capabilities of IR cameras suffer from the errors fundamental to stereomatching, discussed below.

Referring to <FIG>, a high level schematic diagram of a HADAR system <NUM> is shown. The HADAR system <NUM> is adapted to receive heat signature in the form of IR radiation from objects and direct the radiation to a spectropolarimetric imaging system <NUM>, which is adapted to generate a plurality of frames <NUM>, each including a plurality of pixels. Each pixel is associated with metadata, discussed below. A pixel and its associated metadata is referred to herein as a thermal voxel which includes metadata information about temperature, shown as block <NUM>, emissivity spectrum ε(ω), shown as block <NUM>, degree of linear polarization (DoLP) and angle of linear polarization (AoLP), shown as block <NUM>, and thermal texture factor X (a concept introduced to identify the intrinsic object thermal photons (signal) vs. extrinsic environmental thermal photons (noise) in the spectral domain entering the spectropolarimetric imaging system <NUM> which is focused on a target), identified as block <NUM>. With the above-identified thermal data associated with each thermal voxel, a processing system (not shown) either local to the HADAR system <NUM> or remote therefrom (e.g., a cloud-based system), detects the object, as shown in block <NUM>; and further determines the range (i.e., distance) of the object from the HADAR system <NUM>, as shown in block <NUM>. Each of these blocks is described below in detail.

The optimal feature for HADAR arises from the spatio-temporal dependence (x, y, z, t) of precisely these mentioned thermal voxel properties. Temperature and spectral emissivity are intrinsic properties of the thermal voxel whereas the DoLP, AoLP and thermal texture factor X involves a subtle interplay of intrinsic and extrinsic thermal photons. Intrinsic photons governed by spectral emissivity are thermally emitted by the target while extrinsic photons are thermally emitted by the environment then reflected off the object and reach the camera which is focused on the object.

In order to determine temperature associated with each thermal voxel, a method according to the present disclosure begins with a first estimate of an average environmental temperature (assumed to be a global constant for all pixels) through an on-board thermometer or GPS-assisted weather data.

In order to determine temperature of each pixel as part of the metadata of the thermal voxel, following the decoupling of intrinsic and extrinsic signals of every pixel, in a first iteration the method of the present disclosure identifies the hottest object and coldest object in the scene. In the second iteration, the environmental temperature is updated locally for every pixel but keeping it within the hot/cold bounds of these two values since the thermal noise from nearby objects dominates the scene. A clustering approach is exploited from unsupervised learning to de-noise the data and guide the scene analysis through publicly available atmospheric models (e.g., MODTRAN). It is also possible to segment the pixels according to noise class and identify global vs. local noisy variations in emissivity/temperature.

To better elucidate these techniques, reference is now made to <FIG>, which is a high-level schematic of outputs of the spectropolarimetric imaging system <NUM>. There are several possibilities for this type of spectropolarimetric imaging system <NUM>, including a hyperspectral camera or other types of cameras known to a person having ordinary skill in the art. In any of these cases, the spectropolarimetric imaging system <NUM> provides two types of output: <NUM>) polarization frames as shown in the frames <NUM>; and <NUM>) spectral frames, as shown in the frames <NUM>. These polarization and spectral frames (<NUM> and <NUM>) are based on a plurality of different settings. For example, according to one embodiment, the polarization frames <NUM> include transmitted light from the spectropolarimetric imaging system <NUM> providing linearly polarized light at <NUM> degree, thereby generating a raw polarized image frame (I<NUM> frame). Next, the spectropolarimetric imaging system <NUM> is adapted to provide linear polarization at <NUM>°. A raw polarized image frame (I<NUM> frame) is thus obtained. Continuing, the spectropolarimetric imaging system <NUM> is further adapted to provide a linear polarization at <NUM>°. A raw polarized image frame (I<NUM> frame) is thus obtained. Next, the spectropolarimetric imaging system <NUM> is further adapted to provide linear polarization at <NUM>° (-<NUM>°). A raw polarized image frame (I-<NUM> frame) is thus obtained. These example-only four polarized image frames are shown in the frames <NUM>.

Next, according to one embodiment, the spectral frames <NUM> include transmitted light from the spectropolarimetric imaging system <NUM> providing spectral frames at a plurality of different spectral frequencies based on application of a plurality of bandpass filters each with a bandwidth (vl - vh). According to one embodiment, nine bandpass filters are applied each providing a spectral frame. These example-only spectral frames are shown in the frames <NUM>.

Reference is now made to <FIG>, where a method <NUM> of determining metadata for each pixel is shown. First, As discussed above, two types of tunability is carried out by the spectropolarimetric imaging system <NUM>, including polarization and spectral tunability. First, the polarization tunability is shown with respect to polarization frames <NUM> (including, as discussed above according to one embodiment, fames I<NUM>, I<NUM>, I<NUM>, and I-<NUM>).

Next mathematical operations are performed on these raw polarized frames (I<NUM>, I<NUM>, I<NUM>, and I-<NUM>) as shown in block <NUM> and these three Stokes parameter maps (S0, S1, S2) are calculated from these operation as shown in Block <NUM> and provided below.

Next mathematical operations are performed on these Stokes parameter maps (S0, S1, S2) as shown in blocks <NUM> and <NUM> and DoLP and AoLP are then assigned based on these mathematical operations (see below) as shown in blocks <NUM> and <NUM>. From these stokes parameters, the DoLP map and AoLP map are calculated from the three Stokes parameters maps.

Thus two of the metadata (AoLP and DoLP) are obtained based on the operations of half of the flowchart of the method <NUM>. With continued reference to <FIG>, the method <NUM> proceeds to determine the remainder of the metadata (i.e., T, emissivity: ε(ω), and thermal texture Factor X). Initially, in block <NUM> spectral frames are obtained from the spectropolarimetric imaging system <NUM>, as discussed above (i.e.,, a plurality, e.g., <NUM>, bandpass filters are applied to the spectropolarimetric imaging system <NUM>, each generating a bandlimited response from the scene). The method of obtaining the spectral frames is further discussed with reference to <FIG>. Next, in block <NUM> the radiation spectrum is reconstructed using an antighosting algorithm, discussed further below, from which in blocks <NUM>, <NUM>, and <NUM> the remaining metadata (T, ε(ω), and Thermal Lightning Factor X) are determined. The antighosting algorithm, referenced in block <NUM>, is further disclosed in <FIG>.

With reference to <FIG>, an antighosting method <NUM> is presented. Initially, in block <NUM>, as discussed above, a plurality of bandlimited bandpass filters are applied to the incoming heat radiation. In one embodiment, nine such bandpass filters are applied. Each of these filters results in a frame including intensity between a lower bandpass level and an upper bandpass level. In other words, for example, for i = <NUM> (i.e., bandpass filter number <NUM>, according to the above example), the frame includes intensity for vl1 and vh1, where vl1 is the lower bandpass level, and vh1 is the upper bandpass level. With reference back to <FIG>, these spectral image frames are shown in block <NUM>. Next in block <NUM>, the dark noise of the spectropolarimetric imaging system <NUM> (see <FIG>) and self-radiation of any bandpass filters are calibrated out, so that output of the spectropolarimetric imaging system <NUM> (see <FIG>) is purely caused by incident heat radiation Sv. To achieve this noise removal and calibration, a black body source, known to a person having ordinary skill in the art, is used to measure noise response from the spectropolarimetric imaging system <NUM> (see <FIG>).

Next, in block <NUM>, the transmittance curve <MAT> of each spectral filter i is characterized, which is a function of frequency (v). This transmittance curve <MAT> measurement is performed once using a spectrometer (i.e., frequency/wavelength bands that are generated each time a bandpass filter is applied. Measurements are made by identifying a band of frequency that passes through the filter.

Next, in block <NUM>, the response curve Zv of each sensor pixel is characterized. At a first level of approximation, a constant response curve can be assumed for each pixel across a range of frequencies (e.g., for i = <NUM>, across vl1 and vh1). Alternatively, response of each pixel with respect to the frequency range can be ascertained. The response is usually provided by the manufacturer of the spectropolarimetric imaging system <NUM> (see <FIG>), or the response be can characterized by a spectrometer, as known to a person having ordinary skill in the art. Next in block <NUM>, the incident spectrum Sv for each i is calculated. The relationship between Sv and the other parameters discussed above is provided below: <MAT>.

Once a material with the above spectrum Svm has been matched to the obtained spectrum Sv, the other metadata are obtained, as shown in blocks <NUM>, <NUM>, and <NUM>. From the thermal texture factor X, a map constituting an antighosting image can be generated.

With reference to <FIG>, one example algorithm is provided showing a method <NUM> for matching the obtained spectrum Sv against the library spectrum Svm. The method <NUM> begin in block <NUM> with the obtained spectrum Sv. A material from the library is chosen representing the material in the scene which the heat images are obtained from. The material spectrum Svm is governed by the equation provided above, as provided in block <NUM>. These two spectra (Sv and Svm) represent three dimensional curves, as discussed above. The two spectra are compared with each other, as provided in block <NUM>. Using a comparison technique, e.g., least-square-error, as shown in block <NUM>, a distinction between these two spectra is determined. This distinction is compared to a threshold, as shown in query <NUM>. If the threshold is not met, then the method <NUM> chooses another material, as provided in block <NUM> and returns to block <NUM>. If the threshold is met, then a match is ascertained and the method <NUM> proceeds to identifying the remaining metadata, as shown in block <NUM>.

According to another approach, Svm for all materials in the library are determined. Next, comparisons between Svm and Sv can be made and using a least square error method (sum of the least square errors in the aforementioned comparison) the minimum of least squares is chosen as a positive match to a selected material in the library.

To demonstrate these techniques, example graphs are provided in <FIG>. First, the spectrum Sv of the image from the scene is shown as a function of frequency in <FIG> (see block <NUM> of <FIG>). Next, for different temperature B (which is black body radiation) is determined and plotted vs. frequency, as shown in <FIG> (see block <NUM> in <FIG>) for different temperatures. Bv at T=<NUM> °K is <MAT>. Next, Svmfor different materials is obtained by plotting Svm as a function of frequency for different temperatures and different X values. The temperature can be chosen based on a priori knowledge of the ambient temperature. As discussed X values range between <NUM> and <NUM>. These graphs as generated for each material (m<NUM>. Once these Svm graphs are generated, the Sv from the scene is compared in a manner discussed according to <FIG> or an alternate plan as discussed above and a match is obtained for T, X, and a material (i.e., ε).

Referring to <FIG>, two images are shown using standard infrared imaging (<FIG> and <FIG>), while <FIG> and <FIG> are processed images utilizing the methods described above thereby generating surface textures for enhanced visualization.

Claim 1:
A method of generating object surface texture in thermal infrared images,
comprising:
receiving heat radiation from a scene by a spectropolarimetric imaging system adapted to generate a plurality of spectral frames associated within the scene;
generating the plurality of spectral frames associated with the scene, each frame having a plurality of pixels;
for each pixel from the generated plurality of spectral frames, extracting spectral information associated with the scene, including pixel-specific temperature representing an object's temperature, and thermal texture factor representing the object's texture;
for each of a plurality of materials having a specific emissivity in a library, generating reference spectral information as a function of temperature and thermal texture;
matching the extracted spectral information for each pixel from the generated plurality of spectral frames to the generated reference spectral information using a statistical method to minimize the associated variation; and
extracting spectral metadata from the matched reference spectral information for the associated material based on the match, wherein the metadata correspond to the object's temperature and thermal texture factor.