Patent Description:
Terrain data and image data are different types of data that may represent a target area. As used herein, "terrain data" refers to elevation information in a gridded (e.g., image) format, with each pixel corresponding to a point on a coordinate system. For example, the point may include an X-value, a Y-value, and/or a Z-value in a Cartesian coordinate system. The terrain data may be captured by LIDAR, radar, stereo image triangulation, ground survey, or structure from motion image processing techniques.

As used herein, "image data" refers to an electro-optical image (e.g., picture) that is captured/acquired by a source such as a camera on a satellite or an aircraft. The image data may be in the visual or infrared spectrum. In a particular example, the target area may include a portion of the surface of the Earth. As will be appreciated, the portion of the surface of the earth may include varying shapes and elevations, such as mountains, trees, buildings, etc..

Terrain data and image data may each include a plurality of pixels. However, the pixels in the terrain data and image data are oftentimes misaligned. For example, a first pixel in the terrain data that corresponds to a particular point on a mountain may be misaligned with a corresponding first pixel in the image data that corresponds to the same point on the mountain. The terrain data and/or the image data may be shifted such that the first pixels are aligned; however, then a second pixel in the terrain data that corresponds to a particular point on the mountain may be misaligned with a corresponding second pixel in the image data that corresponds to the same point on the mountain. Thus, it may be difficult to align each corresponding pair of pixels. If the pixels are misaligned when the terrain data
and the image data are combined to produce an image (e.g., a map), this misalignment may reduce the quality and accuracy of the image.

<NPL>, in accordance with its abstract, states a cross-sensor image registration, orthorectification, and geopositioning of imagery are well-known problems whose Solutions are difficult, if not impossible, to automate. Registration of radar to optical imagery typically requires a manual solution, as does the registration of imagery over rugged terrain or urban areas, where foreshortening and layover present formidable obstacles to successful automation. We have developed an automated solution that is based on the registration of imagery to high-precision digital elevation models (DEMs) derived from Lidar data. The key idea is the generation of a simulated image using Lidar data, the image camera model and the illumination conditions. The simulated image is then registered to the actual image with normalized cross-correlation methods. The result is an effective and completely automated technique for registering imagery to DEMs. It has been shown to work with BuckEye Lidar, ALIRT Lidar, commercial satellite imagery and commercial synthetic aperture radar imagery over diverse terrain types, including mountains, cities, and forests. It provides an automated solution to many difficult geospatial problems, including cross-sensor registration of radar and optical imagery, image registration over rugged terrain, geopositioning of imagery and orthorectification. Its use of Lidar enables it to handle three-dimensional features that are foreshortened or laid over in different directions. Its use of simulated imagery enables it to bypass the problem of disparate features in cross-sensor registration. Statistical analyses of the registration accuracy are presented along with results on commercial satellite imagery and Lidar data over Iraq, Afghanistan, Haiti and the U.

<NPL>, in accordance with its abstract, states a number of image analysis tasks can benefit from registration of the image with a model of the surface being imaged. Automatic navigation using visible light or radar images requires exact alignment of such images with digital terrain models. In addition, automatic classification of terrain, using satellite imagery, requires such alignment to deal correctly with the effects of varying sun angle and surface slope. Even inspection techniques for certain industrial parts may be improved by this means. We achieve the required alignment by matching the real image with a synthetic image obtained from a surface model and known positions of the light sources. The synthetic image intensity is calculated using the reflecance map, a convenient way of describing surface reflection as a function of surface gradient. We illustrate the technique using LANDSAT images and digital terrain models.

A method for co-registering terrain data and image data according to claim <NUM> is disclosed.

In an embodiment, the method includes receiving terrain data and receiving image data. The image data is captured by a camera on an aircraft or a satellite. The method also includes determining a position of the sun at a time that the image data was captured based upon shadows in the image data. The position of the sun includes an azimuth and an altitude of the sun. The method also includes creating a hillshade representation of the terrain data based upon the terrain data and the position of the sun. The method also includes identifying a portion of the hillshade representation and a portion of the image data that correspond to one another. The method also includes comparing the portion of the hillshade representation and the portion of the image data using an image-matching technique, a pattern-matching technique, or an obj ect-matching technique to output a plurality of coordinate pairs. Each coordinate pair includes a first pixel in the image data and a second pixel in the hillshade representation. The first and second pixels each correspond to a same point on a surface of the Earth. The first and second pixels are misaligned. The method also includes determining a vector control for each coordinate pair. The vector control includes a distance and a bearing from the first pixel to the second pixel. The method also includes applying the vector control to the image data to move the first pixel the distance along the bearing to become aligned with the second pixel, thereby producing updated image data.

A computing system according to claim <NUM> is also disclosed. The image data is captured by a camera on an aircraft or a satellite. The operations also include determining a position of the sun at a time that the image data was captured based upon shadows in the image data. The position of the sun includes an azimuth and an altitude of the sun. The operations also include creating a hillshade representation of the terrain data based upon the terrain data and the position of the sun. The operations also include identifying a portion of the hillshade representation and a portion of the image data that correspond to one another. The operations also include comparing the portion of the hillshade representation and the portion of the image data using an image-matching technique, a pattern-matching technique, or an object-matching technique to output a plurality of coordinate pairs. Each coordinate pair includes a first pixel in the image data and a second pixel in the hillshade representation. The first and second pixels each correspond to a same point on a surface of the Earth. The first and second pixels are misaligned. The operations also include determining a vector control for each coordinate pair. The vector control includes a distance and a bearing from the first pixel to the second pixel. The operations also include applying the vector control to the image data to move the first pixel the distance along the bearing to become aligned with the second pixel, thereby producing updated image data.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present teachings, as claimed.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate aspects of the present teachings and together with the description, serve to explain the principles of the present teachings.

It should be noted that some details of the figures have been simplified and are drawn to facilitate understanding rather than to maintain strict structural accuracy, detail, and scale.

Reference will now be made in detail to the present teachings, examples of which are illustrated in the accompanying drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific examples of practicing the present teachings. The following description is, therefore, merely exemplary.

<FIG> illustrates a flowchart of a method <NUM> for co-registering terrain data and image data, according to an implementation. As used herein, "co-registering" refers to making the pixels in two images (e.g., the terrain data and the image data) align or match. The method <NUM> includes capturing or receiving terrain data and image data, as at <NUM>. <FIG> illustrates an example of terrain data <NUM>, and <FIG> illustrates an example of image data <NUM>, according to an implementation. The terrain data <NUM> and the image data <NUM> may both represent the same target area. For example, the terrain data <NUM> and the image data <NUM> may both represent the same portion of the surface of the Earth. As shown, the area represented in the terrain data <NUM> and the image data <NUM> is mountainous. In the particular example shown, the terrain data <NUM> includes a ridge <NUM> with one or more arms <NUM>-<NUM>, and the image data <NUM> also includes the same ridge <NUM> with the one or more arms <NUM>-<NUM>.

<FIG> illustrates the terrain data <NUM> side-by-side with (i.e., compared to) the image data <NUM>, according to an implementation. As shown, there may be a misalignment <NUM> between the terrain data <NUM> and the image data <NUM>. In this example, the misalignment <NUM> of the ridge <NUM> in the terrain data <NUM> and the ridge <NUM> in the image data <NUM> may be about <NUM> meters. The misalignment <NUM> may have a negative impact on the look and accuracy of an orthoimage generated using the terrain data <NUM> and the image data <NUM>. The negative impact on the look and accuracy of the orthoimage may in turn have a negative impact on any spatial data derived from the orthoimage source. In other words, an orthoimage may be created by correcting the image data <NUM> to the terrain data <NUM>, or vice versa. The actual quality of the orthoimage, outside of spatial corrections, may not yet be assessed and may depend upon the transform that is used to warp the image once vector controls have been created, as discussed below. An orthoimage, also referred to as an orthophoto, is a remotely sensed image, such as an aerial or satellite photo, that has been corrected to remove distortions caused by topographic relief (e.g., terrain) and collection parameters (e.g., sensor lens distortion, collection obliquity). Orthoimages preserve scale, allowing for accurate distance measurement.

Returning to <FIG>, the method <NUM> includes determining a position of a light source (e.g., the sun) from the image data <NUM>, as at <NUM>. More particularly, this may include determining a sun azimuth, a sun altitude (also referred to as a sun elevation), a sun zenith, or a combination thereof based at least partially upon metadata of the image data <NUM>.

The method <NUM> also includes creating a hillshade using the terrain data <NUM> and the position of the sun, as at <NUM>. <FIG> illustrates a hillshade <NUM> created using the terrain data <NUM> and the position of the sun, according to an implementation. The hillshade <NUM> is a two-dimensional (2D) or three-dimensional (3D) representation of the terrain data <NUM> that takes the position of the sun into account for shadows and shading. More particularly, the hillshade <NUM> simulates the determined position of the sun to create shadows and shading, which adds a relief-like look to the terrain data <NUM>. To match the terrain data <NUM> to the image data <NUM> as closely as possible, the sun azimuth, sun altitude, and/or sun zenith may be used to create the hillshade <NUM>.

Returning to <FIG>, the method <NUM> also includes determining or identifying a portion of the hillshade <NUM> that corresponds to a portion of the image data <NUM>, as at <NUM>. <FIG> illustrates a portion <NUM> of the hillshade <NUM>, and <FIG> illustrates a corresponding (e.g., overlapping) portion <NUM> of the image data <NUM>, according to an implementation. Determining the portion <NUM> of the hillshade <NUM> that corresponds to the portion <NUM> of the image data <NUM> may first include placing bounding coordinates (e.g., a border) <NUM> around the hillshade <NUM> (see <FIG>) and bounding coordinates <NUM> around the image data <NUM> (see <FIG>).

Then, a portion <NUM> of the hillshade <NUM> (e.g., inside the bounding coordinates <NUM>) and a portion <NUM> of the image data <NUM> (e.g., inside the bounding coordinates <NUM>) are determined/identified that correspond to (e.g., overlap with) one another. This may be performed by calculating a spatial overlap, also known as clipping. In at least one implementation, even though the portions <NUM>, <NUM> overlap, one or more of the pixels in the portion <NUM> of the hillshade <NUM> may be misaligned with one or more pixels in the portion <NUM> of the image data <NUM>.

Then, bounding coordinates <NUM> may be placed around the portion <NUM> of the hillshade <NUM>, as shown in <FIG>, and bounding coordinates <NUM> may be placed around the portion <NUM> of the image data <NUM>, as shown in <FIG>. The remainder of the hillshade <NUM>, between the bounding coordinates <NUM> (in <FIG>) and the bounding coordinates <NUM> (in <FIG>) may be removed and discarded. Similarly, the remainder of the image data <NUM>, between the bounding coordinates <NUM> (in <FIG>) and the bounding coordinates <NUM> (in <FIG>) may be removed and discarded.

Returning to <FIG>, the method <NUM> also includes comparing the portion <NUM> of the hillshade <NUM> and the portion <NUM> of the image data <NUM>, as at <NUM>. <FIG> illustrates the portion <NUM> of the hillshade <NUM> side-by-side with (i.e., compared to) the portion <NUM> of the image data <NUM>, according to an implementation. <FIG> is similar to <FIG>, except that the terrain data <NUM> has been replace with the hillshade <NUM>. Thus, <FIG> may show the misalignment <NUM> more clearly than <FIG>. The comparison of the portion <NUM> of the hillshade <NUM> to the portion <NUM> of the image data <NUM> (or the portion <NUM> of the image data <NUM> to the portion <NUM> of the hillshade <NUM>) may be performed using an image-matching technique, a pattern-matching technique, or an object-matching technique.

The matching technique may output one or more coordinate pairs. As used herein, a coordinate pair refers to a first coordinate in the portion <NUM> of the hillshade <NUM> and a second coordinate in the portion <NUM> of the image data <NUM>. The first and second coordinates may each correspond to the same 2D or 3D point of the target area (e.g., the same point on the surface of the Earth). The coordinate pairs may be in geographic space or image space. If the coordinate pairs are in image space (e.g., image coordinates), they may be converted to geographic coordinates. In one example, the coordinate pairs may be or include arrows (e.g., from the first coordinate to the second coordinate, or vice versa).

The method <NUM> also includes determining vector controls between the portion <NUM> of the hillshade <NUM> and the portion <NUM> of the image data <NUM> based at least partially upon the comparison, as at <NUM>. The vector controls represent the distance and/or direction (i.e., bearing) between the first and second coordinates in each coordinate pair. For example, a first vector control may represent the distance and direction from a coordinate in the portion <NUM> of the hillshade <NUM> to a corresponding coordinate in the portion <NUM> of the image data <NUM>. The vector controls may be created using GeoGPM.

Table <NUM> below represents some of the coordinates in <FIG> that are used to determine the vector controls. In Table <NUM>:.

The method <NUM> also includes applying the vector controls to the image data <NUM> to produce updated image data, as at <NUM>. <FIG> illustrates the terrain data <NUM> side-by-side with (i.e., compared to) the updated image data <NUM>, according to an implementation. <FIG> illustrates the hillshade <NUM> side-by-side with (i.e., compared to) the updated image data <NUM>, according to an implementation. The updated image data <NUM> may be used to produce a map. Instead of, or in addition to, applying the vector controls to the image data <NUM>, the vector controls may be applied to the terrain data <NUM> to produce updated terrain data (or an updated hillshade <NUM>).

Applying the vector controls may co-register (and/or geo-register) the terrain data <NUM> and/or the image data <NUM>, which may reduce or eliminate the misalignment <NUM> seen in <FIG> and <FIG>. As used herein, geo-register" refers to making the pixels in a control image (e.g., the terrain data or the image data) align or match with the correct location/position on the surface of the Earth. In other words, applying the vector controls may transform different sets of data (e.g., the terrain data <NUM> and the image data <NUM>) into a single, common coordinate system where the pixels are aligned to correlated locations in image space and/or geographic space. For example, the image data <NUM> can be adjusted to the terrain data <NUM> (or hillshade <NUM>) using the vector controls, or the terrain data <NUM> (or hillshade <NUM>) can be adjusted to the image data <NUM> using the vector controls. The vector controls may be applied using a transform to align the pixels. The transform may be or include a polynomial transform. The transform may be <NUM>st order, <NUM>nd order, <NUM>rd order, <NUM>th order, <NUM>th order, or higher. Instead of, or in addition to, using a transform, the vector controls may be applied using a warp to align the pixels.

The method <NUM> also includes generating a plot including the vector controls, as at <NUM>. <FIG> illustrates a plot <NUM> including the distance and bearing values for each vector control, according to an implementation. For example, each dot (e.g., dot <NUM>) in the plot <NUM> represents a distance and a bearing from a coordinate in <FIG> to a corresponding coordinate in <FIG> (or from a coordinate in <FIG> to a corresponding coordinate in <FIG>). As described above, the coordinates in <FIG> may each correspond to the same point (e.g., 2D or 3D coordinate) on the surface of the Earth. Thus, the dots in the plot <NUM> may represent the misalignment between <FIG>. The distance and bearing values may be plotted to represent statistics such as a circular error probability (CEP) of <NUM>% (referred to as CEP90), average bearing, and differentials.

The plurality of dots (e.g., including dot <NUM>) in the plot <NUM> provide the following exemplary statistics when comparing the coordinates in <FIG>:.

Although the plot <NUM> shows the CEP of the portion <NUM> of the image data <NUM> compared to the portion <NUM> of the hillshade <NUM>, in other implementations, the plot <NUM> may show the CEP of the image to image, image to terrain, terrain to image, and/or terrain to terrain. In addition, the plot <NUM> may illustrate data that may enable a user to quickly see the type of shift between the sources (e.g., the terrain data <NUM> and the image data <NUM>). For example, if the dots are clustered in a single quadrant, this may indicate a shift (e.g., misalignment) between the sources. However, if the dots are distributed across multiple quadrants of the plot <NUM>, this may indicate a scale and/or rotation issue.

At least a portion of the method <NUM> is not a mental process and cannot be performed in the human mind. For example, in one implementation, one or more (e.g., all) of the steps are performed by a computing system, such as the one described below.

<FIG> illustrates a schematic view of a computing system <NUM> for performing at least a portion of the method <NUM>, according to an implementation. The computing system <NUM> may include a computer or computer system 1001A, which may be an individual computer system 1001A or an arrangement of distributed computer systems. The computer system 1001A includes one or more analysis module(s) <NUM> configured to perform various tasks according to some implementations, such as one or more methods disclosed herein. To perform these various tasks, the analysis module <NUM> executes independently, or in coordination with, one or more processors <NUM>, which is (or are) connected to one or more storage media <NUM>. The processor(s) <NUM> is (or are) also connected to a network interface <NUM> to allow the computer system 1001A to communicate over a data network <NUM> with one or more additional computer systems and/or computing systems, such as 1001B, 1001C, and/or 1001D (note that computer systems 1001B, 1001C and/or 1001D may or may not share the same architecture as computer system 1001A, and may be located in different physical locations, e.g., computer systems 1001A and 1001B may be located in a processing facility, while in communication with one or more computer systems such as 1001C and/or 1001D that are located in one or more data centers, and/or located in varying countries on different continents).

The storage media <NUM> can be implemented as one or more computer-readable or machine-readable storage media. Note that while in the example implementation of <FIG> storage media <NUM> is depicted as within computer system 1001A, in some implementations, storage media <NUM> may be distributed within and/or across multiple internal and/or external enclosures of computing system 1001A and/or additional computing systems. Storage media <NUM> may include one or more different forms of memory including semiconductor memory devices such as dynamic or static random access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs) and flash memories, magnetic disks such as fixed, floppy and removable disks, other magnetic media including tape, optical media such as compact disks (CDs) or digital video disks (DVDs), BLURAY® disks, or other types of optical storage, or other types of storage devices. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.

In some implementations, computing system <NUM> contains one or more co-registration module(s) <NUM> that may perform at least a portion of the method <NUM> described above. It should be appreciated that computing system <NUM> is only one example of a computing system, and that computing system <NUM> may have more or fewer components than shown, may combine additional components not depicted in the example implementation of <FIG>, and/or computing system <NUM> may have a different configuration or arrangement of the components depicted in <FIG>. The various components shown in <FIG> may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.

Claim 1:
A computer implemented method (<NUM>) for co-registering terrain data and image data, comprising:
receiving terrain data (<NUM>) and image data (<NUM>);
determining a position of a light source based upon the image data (<NUM>);
creating a hillshade representation of the terrain data (<NUM>) based upon the terrain data (<NUM>) and the position of the light source;
identifying a portion of the hillshade representation and a portion of the image data that correspond to one another;
comparing the portion of the hillshade representation and the portion of the image data;
determining a vector control between the portion (<NUM>) of the hillshade representation and the portion (<NUM>) of the image data based upon the comparison; and
applying the vector control to the image data to produce updated image data,
wherein comparing the portion (<NUM>) of the hillshade representation and the portion (<NUM>) of the image data is performed using at least one of an image-matching technique, a pattern-matching technique, or an object-matching technique,
wherein comparing the portion (<NUM>) of the hillshade representation and the portion (<NUM>) of the image data outputs a coordinate pair,
wherein the coordinate pair comprises a first pixel in the hillshade representation and a second pixel in the image data, wherein the first and second pixels correspond to a same point on a surface of the Earth,
wherein the first and second pixels are misaligned,
wherein the vector control comprises a distance between the first pixel and the second pixel and a bearing from the first pixel to the second pixel,
wherein applying the vector control to the image data includes moving the first pixel the distance along the bearing to become aligned with the second pixel, thereby producing updated image data,
wherein the method further comprises generating a plot (<NUM>) to show the distance and the bearing for each vector control,
wherein the distance and the bearing are represented by dots (<NUM>) in the plot (<NUM>),
wherein the terrain data (<NUM>) is captured by a source, and further comprising determining that a scale of the source is not equivalent to a scale of a camera, a rotation angle of the source is not equivalent to a rotation angle of the camera, or both, based at least partially upon the vector controls being located in at least two different quadrants of the plot (<NUM>).