Patent Description:
Searching a wide area for rare targets presents a challenging use case for high-altitude surveillance and reconnaissance. The area to be explored can be many times greater than the platforms and imaging systems can assess in real time.

The conventional techniques have been considered satisfactory for their intended purpose. However, there is an ever present need for improved systems and methods for Image processing and analysis. This disclosure provides a solution for this need.

<CIT> discloses a method and an apparatus for extracting mountain landscape buildings based on high-resolution remote sensing images. The method comprises: segmenting a remote sensing image, and extracting non-vegetation areas from the remote sensing image by using NDVI; segmenting the non-vegetation areas, and extracting building areas by using NDBI; segmenting the building areas again, and calculating a normalized difference build shadow index NSBI of each patch; calculating NSBI separator of each patch in the non-vegetation areas and setting a separator threshold, and extracting landscape building areas based on the threshold.

According to a first aspect of the invention, a method as claimed in claim <NUM> is provided.

In certain embodiments, identifying the target includes identifying multiple instances of one type of target in one or more of the images.

In certain embodiments, the method further include directing a physical asset to move physically toward the target.

In certain embodiments, identifying includes providing output to a user showing identity and/or location of the target.

In certain embodiments, the land types include dry ground and bodies of water.

In certain embodiments, screening includes for each image, grouping pixels of like-land types into groups, one group for each land type in the image.

In certain embodiments, grouping pixels of like-land types includes grouping pixels by identical land type.

In certain embodiments, grouping pixels of like-land types includes grouping pixels in hierarchical land types.

In certain embodiments, screening includes using a look up table identifying likelihood of finding a target in each land type in the images.

In certain embodiments, the method further includes only analyzing images in the area that are within a region of interest (ROI).

In certain embodiments, the method further includes dividing the image into a grid of sub-images, determining a weight for each sub-image in a region of interest based on a land-target score, a distance traveled, and a turn penalty, directing a physical resource to move toward one or more instances of the target if the weight of one or more instances of the target is above a threshold.

In certain embodiments, the physical resource is an imaging platform, wherein obtaining the downward looking images of the area is performed by the imaging platform.

In certain embodiments, analyzing includes dividing an image into a grid of sub-images, for each sub-image, summing a product of each land type and its size within the sub-image to obtain a score, the score being representative of a likelihood of finding a target in each land type in the images, sorting the sub-images into a queue by score and analyzing the sub-images to identify instances of a target, wherein analyzing is performed in order starting with a sub-image of highest score in the queue down to sub-images with lowest score. In certain embodiments, analyzing further includes sorting new imagery into the queue and analyzing some of the new imagery before completing analysis of all original sub-images in the queue.

According to another aspect of the invention, an imaging system as claimed in claim <NUM> is provided.

In certain embodiments, screening includes for each image, grouping pixels of like-land types into groups, one group for each land type in the image. In certain embodiments, grouping pixels of like-land types can include grouping pixels by identical land type. In certain embodiments, grouping pixels of like-land types can include grouping pixels in hierarchical land types. In embodiments, screening includes using a look up table identifying likelihood of finding a target in each land type in the images.

In embodiments, the method can further include only analyzing images in the area that are within a region of interest (ROI), determining a weight for each instance of the target identified based on a land-target score, a distance traveled, and a turn penalty and directing a physical resource to move toward one or more instances of the target if the weight of one or more instances of the target is above a threshold.

In certain embodiments, analyzing includes: dividing an image into a grid of sub-images, and for each sub-image, summing a product of each type of land and its size within the sub-image to obtain a score, the score being representative of a likelihood of finding a target in each land type in the images, sorting the sub-images into a queue by score, and analyzing the sub-images to identify instances of a target, wherein analyzing is performed in order starting with a sub-image of highest score in the queue down to sub-images with lowest score. In certain such embodiments, sorting new imagery into the queue and analyzing some of the new imagery before completing analysis of all original sub-images in the queue.

These and other features of the systems and methods of the subject disclosure will become more readily apparent to those skilled in the art from the following detailed description taken in conjunction with the drawings.

Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the subject disclosure. For purposes of explanation and illustration, and not limitation, a partial view of an embodiment of a system in accordance with the disclosure is shown in <FIG> and is designated generally by reference character <NUM>. Other embodiments of systems in accordance with the disclosure, or aspects thereof, are provided in <FIG>, as will be described. The systems and methods described herein can be used to improve image processing, e.g., aboard airborne imaging platforms.

Searching a wide area for rare targets presents a challenging use case for high-altitude surveillance and reconnaissance, for example because area to be explored can be many times greater than the platforms and imaging systems can assess and can assess in real time. Performing image capture based on classified land types provides an opportunity to decrease the search space by focusing imaging on areas where targets are more commonly found (e.g. determined regions of interest (ROI)).

Focused imaging can be particularly useful in mission planning, for example, having an operator pre-select an ROI to search for a target of interest, prioritizing areas where targets are most likely to be found. In addition to mission planning, land use types can also be exploited for prioritizing which images to process first during a collection, for example screening incoming data (imagery) to prioritize areas where targets are most likely to be found in the captured images. Mission planning and image prioritization are a few non limiting examples of applications for such technology, and do not represent an exhaustive list of applications for a system (e.g. system <NUM>) using the imaging methods described herein.

An imaging system <NUM> for focused image capture and image prioritization comprises an imaging platform <NUM> (e.g. a piloted aircraft, drone, satellite, or the like), a camera <NUM> operatively connected to the imaging platform <NUM>, and a controller <NUM> operatively connected to control the imaging platform <NUM> and the camera <NUM>. The controller <NUM> includes machine readable instructions configured to cause the controller <NUM> to perform at least one method. A first method including prioritizing land types for image capture, and a second method for prioritizing captured images for analysis.

As shown in <FIG>, the imaging platform obtains downward looking images <NUM> of an area <NUM> and determining land types A, B, C, D indicated in the images. Land types as described herein can be any surface captured and recognized by the camera, including dry ground and bodies of water. Prior to a mission, land maps can be created in any suitable manner (e.g. using multispectral imaging or satellite imaging) to define and classify known land types. The existing land maps provide a priori information about the specific land type such that the controller <NUM> can exploit the a priori known land types and determine where those land types are located in an image by coordinate between the location of the imaging platform when an image is obtained, the direction (camera pose) in which the image was obtained relative to the imaging platform, and what land types from the existing land maps is in view in the image.

Once the image <NUM> is captured, the land types are mapped onto the image <NUM>, dividing the image into areas of the various land types. Next, the method includes screening the images <NUM> based on the land types. An image <NUM> will pass screening if the land type is determined to be more likely to include a desired target. As described herein, a target is a type of target, for example a type of land vehicle. In certain instances for example, the target can be a certain type of land vehicle when searching for any number of such land vehicles in the captured images <NUM>.

Screening includes, for each image <NUM>, grouping pixels of like-land types into groups, one group for each land type in the image. Grouping pixels of like-land types can include grouping pixels by identical land type, or can include grouping pixels in hierarchical land types (e.g. grouping all forest versus coniferous, dense, new growth, and the like).

More specifically, screening includes using a look up table identifying likelihood of finding a target in each land type in the images. The look up table can be generated in at least the manner described below, however, any suitable manner for generating a look up table is contemplated herein. For example, <MAT> , where X is a list of a priori detections.

It is possible to tailor target lookup table (B) to each theater of operation to dynamically assess how targets <NUM> are deployed differently, or when new targets or target variants are observed. A count of target <NUM> in a given land type can be determined using, L:X → B̂,b̂ target,land ∈ B̂,b̂ target,land = (c), where L is the lookup that maps lon/lat (X) to land types (B̂) and B̂ is a matrix where row is target type and column is land type, and value is the count (c) of targets found on that land type. B̂ is easily converted to B, a row-stochastic matrix, by <MAT> meaning each target row sums to <NUM>. B is then mapped back to the physical space A by L-<NUM> : B→ A, where A has <NUM> spatial dimensions (lon, lat) = ( jx,jy) and a channel dimension whose size is the number of targets. <MAT> and summing over all target channels (Σi aijxjy) provides the utility of jx,jy. Since <MAT> , just like the original input X, it can be easily leveraged for uses cases (e.g. as discussed below). Summing over targets provides a <NUM>-D (lon-lat) object such that A can be dynamically discretized to reduce the footprint of the representation, e.g. if 2x2 pixels are the same land type, they can be combined into a single pixel.

The method next includes further analyzing the images <NUM>, where only the portions of the screened images which include the land types likely to include the target <NUM> are analyzed. For portions of the image <NUM> which are of other land types, analysis is foregone, decreasing processing time for the image <NUM>. If the image is determined to have a land type likely to include the target <NUM>, the controller can then identify the target <NUM> in the image <NUM>, either singly, or identify multiple instances of one type of target in one or more of the images. For example, analyzing the image <NUM> can be done by creating a discretized map spanning the ROI (e.g. the land type in which the target <NUM> is detected, or likely to be detected - land type A in this example), with a channel for each target class where channel values are populated using the lookup (L-<NUM> ) that generated L : X→ B. Solving Equation <NUM> for some objective function will optimize the mission plan: ai,j ∈ A, j ∈ {lon, lat}, i ∈ {targets} where j has <NUM> spatial dimensions but can be flattened for simplicity.

In embodiments, the mission planning method can further include only capturing imagery <NUM> in the area that are within the ROI while disregarding or discarding portions of the images outside of the ROI, so that the method is performed only on those portions within the ROI. The method further includes determining a weight for the priority of each type of target <NUM> identified with respect to a land-target score, a distance traveled, and a turn penalty, where distance travelled and turn penalty relate to the physical distance travelled and amount of turn of imaging platform needed to capture the image <NUM>.

The mission plan value J can be determined by (land-target score) - (distance traveled) - (turn penalty), such that <MAT> where λ terms indicate a weight, d indicates the directive priority of each target, α is some turning function specific to the platform <NUM>, and ai,j only contributes to the score/utility once per j value, regardless of how many times it is included or how much target utility it provides (see, for example, table in <FIG>). The parameters of mission plan are represented by θ.

If the weight of one or more instances of the land-type associated with the target <NUM> is above a threshold, the method includes directing the physical resource <NUM> (e.g. the imaging platform <NUM> obtaining the downward looking images <NUM>) to move toward one or more of these land-type instances associated with target <NUM>. In this manner, the land use types can be leveraged for mission planning by directing the imaging device only to areas which will take the least amount of resources while retaining a high likelihood of imaging the target. For example, a captured image <NUM> may detect a land type with a low likelihood of containing the desired target <NUM>, resulting in a low weight and require a large physical travel and high turn penalty for the imaging device. In that case, the physical resource <NUM> will not be guided to that land type. On the other hand, a high weight can be if the captured image detects a land type with a high likelihood of containing the desired target, and the imaging platform <NUM> requires little to no physical travel or turn. In this case, the physical resource <NUM> will be guided to this land type for further image capture. Thus, the mission is optimized by avoiding areas with a low weight, and only focusing on those with high weight and relevant to the mission.

In another aspect, the captured images can be prioritized for further analysis by expectation of finding the desired target <NUM> in the given land type. Using the lookup table described above, the controller <NUM> is able to determine what land type each target was located on. Once the relative frequency of each target on each land type is generated, the look up table can be applied in reverse and mapped back to physical locations. The resulting map is a <NUM>-D matrix extending in <NUM> spatial dimensions (x,y) and one target dimension, creating a relative score of that land type for each target (where the score can be the same for each target on the same land types), the score based on the observed frequency of a target being present on that land type. Summing over the target scores each (x,y) location gives the utility of that square.

For example, as shown in <FIG>, analyzing the captured image <NUM> includes dividing the image <NUM> into a grid of sub-images <NUM> (e.g. a size convenient for the specific image controller <NUM> used). For each sub-image, a product of each type of land and its size within the sub-image are summed to obtain a score, the score being representative of a likelihood of finding a target <NUM> in each land type in the images <NUM>, based on a priori observations. Each square <NUM> has a composite utility uj = Σi ai sij, where a is the relative area of the land type and s is the score for that land type. The value and/or scores for each land type can be summed with respect to the target <NUM>. The score can be specific to the target <NUM> and theater (e.g. land vehicles in the dessert may remain on roads, while the same land vehicles in mountainous regions may be commonly observed on sandy areas.

Once a score is obtained, the method includes sorting the sub-images <NUM> into a queue by score so the image controller <NUM> can analyze the sub-images <NUM> to identify instances of the target <NUM>. Using the score, analyzing the sub-images <NUM> can then be performed in order starting with a sub-image of highest score in the queue down to sub-images with lowest score. In certain such embodiments, sorting new imagery into the queue and analyzing some of the new imagery before completing analysis of all original sub-images <NUM> in the queue. In certain embodiments, a latency term can also be added so that as new imagery becomes available it is prioritized (e.g. over similar but older imagery), allowing for real time, online prioritizing of captured images <NUM>.

As the images <NUM> are prioritized and analyzed, and the target <NUM> identified, the method can include providing output to a user showing identity and/or location of the target, for example for aiding in guiding a physical resource <NUM> or other mission planning items based on the identity and/or location of the target <NUM>.

In certain embodiments, the method can then include directing the physical resource <NUM> to move physically toward the identified target <NUM> (e.g. to gather more images, or any other suitable purpose for the mission at hand). In embodiments, the physical resource <NUM> can include, but is not limited to, guided munitions, further imaging assets, additional services (e.g. including object detection algorithms), or the like. For example, execution commands can be sent to computing device to guide the physical resource in any suitable manner based on the particular land type captured.

The methods and systems of the present disclosure, as described above and shown in the drawings, provide for improved effectiveness of mission planning by focusing on areas where targets are commonly found, and prioritized imagery that overlaps with areas where targets are commonly found. Tailoring the target lookup table (B) to each theater of operation to dynamically assesses how targets are deployed differently, or when new targets or target variants are observed. Additionally, the systems and methods provided herein are improve speed and efficiency of image capture and processing by using ubiquitous land cover maps which are already high resolution and easily accessible and by loading only the NITF header rather than the entire image data.

Claim 1:
A method comprising:
obtaining downward looking images (<NUM>) of an area (<NUM>);
determining land types indicated in the images (<NUM>);
screening the images (<NUM>) based on the land types wherein screened images pass screening based on land type more likely to include a target (<NUM>);
analyzing only portions of the screened images that include one or more land types likely to include the target (<NUM>), and foregoing analysis of other portions of the images (<NUM>);
determining a weight for the priority of each type of target (<NUM>) identified with respect to a land-target score, a distance traveled, and a turn penalty; directing a physical resource (<NUM>) to move toward one or more instance of the land type associated with the target (<NUM>) and capture further images if the weight of one or more instance of the land type associated with the target (<NUM>) is above a threshold;
wherein distance travelled and turn penalty relate to the physical distance travelled and amount of turn of the physical resource needed to capture the images; and
identifying the target (<NUM>) in one of the images.