Patent Description:
The present invention relates to an image correction device, an image correction method, and a program.

As a device for acquiring images of a ground surface from an aircraft or a satellite, a pushbroom-type image acquisition device has been widely adopted. A device of this type is configured so as to acquire a line-shaped image extending in the X axis direction by using a one-dimensional array sensor as an image sensor. Then, with translation of the entire image acquisition device in the perpendicular direction (Y axis direction) with respect to the line-shaped acquired image by the movement of the aircraft or the satellite, a two-dimensional image is formed. Further, in the case of acquiring images of a plurality of wavelength bands by using an image acquisition device of this type, the device is configured to image a subject with a plurality of filters, in each of which the band that is a wavelength band of light to be transmitted is different, attached to each of the one-dimensional array sensors. An image of each wavelength band is called a band image.

For example, Patent Literature <NUM> discloses a technology of reducing a color shift caused in an image acquisition device of this type, by means of a combination of band image shifting corresponding to the position shift quantity and general interpolation processing such as a linear interpolation method.

When distortion is caused by the characteristics of the optical system, a phase difference is generated between bands, and a color shift may be caused by the phase difference. A color shift caused by a phase difference in this context means that in the case of imaging the same subject by a plurality of bands such as RGB (red, green, blue) for example, the color of the same portion of the subject may be different from that of the subject depending on the position of the portion in the pixel of each band. Such a color shift caused by a phase shift is difficult to be reduced by a combination of band image shifting corresponding to the position shift quantity and general interpolation processing such as a linear interpolation method at the time of correcting the color shift.

An exemplary object of the present invention is to provide an image correction device that solves the above-described problem, that is, a problem that it is difficult to reduce a color shift, caused by a phase shift, by means of a combination of band image shifting and general interpolation processing such as a linear interpolation method.

Dependent claims describe preferred embodiments.

The present invention enables reduction of a color shift caused by a phase shift.

Next, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

Referring to <FIG>, an image correction device <NUM> according to a first exemplary embodiment of the present invention includes a plurality of one-dimensional array sensors <NUM>, a communication interface (I/F) unit <NUM>, an operation input unit <NUM>, a screen display unit <NUM>, a storage unit <NUM>, and an arithmetic processing unit <NUM>.

The one-dimensional array sensors <NUM> include, for example, a one-dimensional charge-coupled device (CCD) sensor, a one-dimensional complementary MOS (CMOS) sensor, or the like, and constitute a pushbroom-type image acquisition device that images a subject <NUM>. The one-dimensional array sensors <NUM> are provided with a plurality of filters <NUM> whose bands that are wavelength bands of light to be transmitted are different. The number of band images and the wavelength bands are determined according to the combinations and the number of sets of the one-dimensional array sensors <NUM> and the filters <NUM> to be used. For example, in a multiband sensor mounted on ASNARO-<NUM> that is a high-resolution optical satellite, the following six band images are acquired:.

Further, ASNARO-<NUM> is equipped with a panchromatic sensor that acquires a panchromatic image that is a black and white image having a higher resolution than that of a multiband image. A panchromatic image is a single band image having a wavelength of <NUM> to <NUM>.

The communication I/F unit <NUM> is configured of, for example, a dedicated data communication circuit, and is configured to perform data communication with various devices connected via wired or wireless communication. The operation input unit <NUM> includes operation input devices such as a keyboard and a mouse, and is configured to detect an operation by an operator and output it to the arithmetic processing unit <NUM>. The screen display unit <NUM> includes a screen display device such as a liquid crystal display (LCD) or a plasma display panel (PDP), and is configured to display, on the screen, a corrected band image and the like in accordance with an instruction from the arithmetic processing unit <NUM>.

The storage unit <NUM> includes storage devices such as a hard disk and a memory, and is configured to store processing information and a program <NUM> necessary for various types of processing to be performed in the arithmetic processing unit <NUM>. The program <NUM> is a program that is read and executed by the arithmetic processing unit <NUM> to thereby realize various processing units. The program <NUM> is read, in advance, from an external device (not illustrated) or a storage medium (not illustrated) via a data input and output function such as the communication I/F unit <NUM>, and is stored in the storage unit <NUM>.

The main processing information to be stored in the storage unit <NUM> includes a multiband image <NUM>, a high-resolution image <NUM>, position difference information <NUM>, and a corrected multiband image <NUM>.

The multiband image <NUM> is a set of a plurality of band images acquired by a pushbroom-type image acquisition device. The multiband image <NUM> may be a set of all band images acquired by a pushbroom-type image acquisition device, or a set of some band images. In the present embodiment, the multiband image <NUM> is assumed to be configured of three bands namely an R-band image <NUM>-<NUM>, a G-band image <NUM>-<NUM>, and a B-band image <NUM>-<NUM>. For example, in the case of ASNARO-<NUM> mentioned above, a band <NUM> may be assigned as the R-band image <NUM>-<NUM>, a band <NUM> may be assigned as the G-band image <NUM>-<NUM>, and a band <NUM> may be assigned as the B-band image <NUM>-<NUM>, respectively.

The high-resolution image <NUM> is an image that is acquired by a pushbroom-type image acquisition device and has a higher resolution than that of the multiband image <NUM>. For example, in the case of ASNARO-<NUM> described above, a panchromatic image may be used as the high-resolution image <NUM>.

In the case where each of the band images constituting the multiband image <NUM> is an object band image and one of the band images is a reference band image, the position difference information <NUM> is information about the position difference between the reference band image and an object band image. In the present embodiment, the G-band image <NUM>-<NUM> is used as a reference band image. Therefore, the position difference information <NUM> is configured of position difference information <NUM>-<NUM> in which the position difference of the R-band image <NUM>-<NUM> relative to the G-band image <NUM>-<NUM> is recorded, position difference information <NUM>-<NUM> in which the position difference of the G-band image <NUM>-<NUM> relative to the G-band image <NUM>-<NUM> is recorded, and position difference information <NUM>-<NUM> in which the position difference of the B-band image <NUM>-<NUM> relative to the G-band image <NUM>-<NUM> is recorded.

In the case where distortion is caused by the characteristics or the like of the optical system in the pushbroom-type image acquisition device, when the subject <NUM> is imaged, a phenomenon that the same portion of the subject <NUM> is imaged at different locations in pixels of the respective bands occurs. <FIG> is a schematic diagram for explaining such a phenomenon. In <FIG>, a solid-line rectangle shows an imaging range of a pixel (x, y) of the reference band image, a broken-line rectangle shows an imaging range of a pixel (x, y) of an object band image, and a black circle represents a part of the subject <NUM>. In the example of <FIG>, the black circle that is a part of the subject <NUM> is located at almost the center in the pixel (x, y) of the reference band, while it is located at the lower right in the pixel (x, y) of the object band. In this case, as illustrated in <FIG>, when the pixel (x, y) of the object band is moved rightward on the sheet by s pixels (<NUM> ≤ s < <NUM>) and moved downward on the sheet by t pixels (<NUM> ≤ t < <NUM>), respectively, the imaging ranges of the pixel (x, y) of the reference band image and the pixel (x, y) of the object band image match. When the imaging ranges of a plurality of band images match in as described above, it is called that pixel boundaries of the band images match. Further, (s, t) at that time is referred to as a position difference. The position difference may differ in each object band and in each pixel. In the present embodiment, the position difference information <NUM> is recorded for each object band and for each pixel. Note that in the present embodiment, the G-band image is a reference band and also an object band. Therefore, the position difference of each pixel of the G-band image is zero.

<FIG> is a table illustrating an exemplary configuration of the position difference information <NUM>-<NUM> of the R-band image <NUM>-<NUM>. The position difference information <NUM>-<NUM> of this example is configured of items of the object band ID and information of each pixel. In the item of object band ID, identification information for uniquely identifying the object band ID is stored. The item of information of each pixel is configured of a combination of an item of a pixel position of the object band image and an item of position difference in that pixel position. In the item of pixel position, xy coordinate values (x, y) specifying the position of the pixel on the object band image is stored. In the item of position difference, (s, t) described with reference to <FIG> is stored. Although not illustrated, the position difference information <NUM>-<NUM> of the B-band image <NUM>-<NUM> also has a configuration similar to that of the position difference information <NUM>-<NUM>.

The corrected multiband image <NUM> is a multiband image obtained by applying correction to the multiband image <NUM> so as not to cause a color shift. The corrected multiband image <NUM> is configured of a corrected R-band image <NUM>-<NUM>, a corrected G-band image <NUM>-<NUM>, and a corrected B-band image <NUM>-<NUM>.

The arithmetic processing unit <NUM> has a microprocessor such as an MPU and the peripheral circuits thereof, and is configured to read, from the storage unit <NUM>, and execute the program <NUM> to allow the hardware and the program <NUM> to cooperate with each other to thereby realize the various processing units. The main processing units realized by the arithmetic processing unit <NUM> include a multiband image acquisition unit <NUM>, a high-resolution image acquisition unit <NUM>, a position difference acquisition unit <NUM>, a corrected multiband image creation unit <NUM>, and a corrected multiband image output unit <NUM>.

The multiband image acquisition unit <NUM> is configured to acquire the multiband image <NUM> from the pushbroom-type image acquisition device configured of the one-dimensional array sensors <NUM>, and store it in the storage unit <NUM>. Further, the high-resolution image acquisition unit <NUM> is configured to acquire the high-resolution image <NUM> from the image acquisition device, and stores it in the storage unit <NUM>. However, the multiband image acquisition unit <NUM> and the high-resolution image acquisition unit <NUM> are not limited to have the configuration of acquiring the multiband image <NUM> and the high-resolution image <NUM> from the image acquisition device. For example, when the multiband image152 and the high-resolution image <NUM> acquired from the image acquisition device are accumulated in an image server device not illustrated, the multiband image acquisition unit <NUM> and the high-resolution image acquisition unit <NUM> may be configured to acquire the multiband image <NUM> and the high-resolution image <NUM> from the image server device.

The position difference acquisition unit <NUM> is configured to acquire the position difference information <NUM> of the multiband image <NUM> acquired by the multiband image acquisition unit <NUM>, and store it in the storage unit <NUM>.

The corrected multiband image creation unit <NUM> is configured to read the multiband image <NUM>, the high-resolution image <NUM>, and the position difference information <NUM> from the storage unit <NUM>, create the corrected multiband image <NUM> therefrom, and store it in the storage unit <NUM>.

The corrected multiband image output unit <NUM> is configured to read the corrected multiband image <NUM> from the storage unit <NUM>, display the corrected multiband image <NUM> on the screen display unit <NUM>, or/ and output it to an external device via the communication I/F unit <NUM>. The corrected multiband image output unit <NUM> may down-sampling each of the corrected R-band image <NUM>-<NUM>, the corrected G-band image <NUM>-<NUM>, and the corrected B-band image <NUM>-<NUM> constituting the corrected multiband image <NUM> as required, and display and output it singly. Alternatively, the corrected multiband image output unit <NUM> may display a color image obtained by synthesizing the corrected R-band image <NUM>-<NUM>, the corrected G-band image <NUM>-<NUM>, and the corrected B-band image <NUM>-<NUM> on the screen display unit <NUM>, or/and output it to an external device via the communication I/F unit <NUM>. Alternatively, the corrected multiband image output unit <NUM> may generate a pansharpened image by superimposing the corrected multiband image <NUM> and the high-resolution image <NUM>, and display and output the pansharpened image.

<FIG> is a flowchart of an exemplary operation of the image correction device <NUM> according to the present embodiment. Referring to <FIG>, first, the multiband image acquisition unit <NUM> acquires the multiband image <NUM> imaged by the image acquisition device configured of the one-dimensional array sensors <NUM>, and stores it in the storage unit <NUM> (step S1). Then, the high-resolution image acquisition unit <NUM> acquires the high-resolution image <NUM> imaged by the image acquisition device, and stores it in the storage unit <NUM> (step S2). Then, the position difference acquisition unit <NUM> acquires the position difference information <NUM> of the multiband image <NUM> acquired by the multiband image acquisition unit <NUM>, and stores it in the storage unit <NUM> (step S3). Then, the corrected multiband image creation unit <NUM> reads the multiband image <NUM>, the high-resolution image <NUM>, and the position difference information <NUM> from the storage unit <NUM>, creates the corrected multiband image <NUM> on the basis thereof, and stores it in the storage unit <NUM> (step S4). Then, the corrected multiband image output unit <NUM> reads the corrected multiband image <NUM> from the storage unit <NUM>, displays it on the screen display unit <NUM>, or/and outputs it to an external device via the communication I/F unit <NUM> (step S5).

Next, main constituent elements of the image correction device <NUM> will be described in detail. First, the position difference acquisition unit <NUM> will be described in detail.

<FIG> is a block diagram illustrating an example of the position difference acquisition unit <NUM>. The position difference acquisition unit <NUM> of this example is configured to include a position difference information creation unit <NUM>.

The position difference information creation unit <NUM> is configured to read the multiband image <NUM> from the storage unit <NUM>, create the position difference information <NUM>-<NUM> of the R band from the G-band image <NUM>-<NUM> and the R-band image <NUM>-<NUM>, and create the position difference information <NUM>-<NUM> of the B band from the G-band image <NUM>-<NUM> and the B-band image <NUM>-<NUM>. The position difference information creation unit <NUM> creates the position difference information <NUM>-<NUM> of the G-band image <NUM>-<NUM> such that the position difference of each pixel is (<NUM>, <NUM>).

<FIG> is a flowchart illustrating exemplary processing by the position difference information creation unit <NUM>. First, the position difference information creation unit <NUM> divides respective images of the reference band and the object bands into a plurality of small regions each having a predetermined shape and size, as illustrated in <FIG>, for example (step S11). In <FIG>, the small region is a rectangle, but it may be in a shape other than rectangle. It is desirable that the size of a small region is sufficiently larger than one pixel.

Then, the position difference information creation unit <NUM> focuses on one of the object bands (for example, R band) (step S12). Then, the position difference information creation unit <NUM> initializes the position difference information of the focused object band (step S13). For example, when the position difference information of the object band has a format illustrated in <FIG>, the position difference (s, t) at each pixel position (x, y) is initialized to a NULL value, for example.

Then, the position difference information creation unit <NUM> focuses on one small region of the focused object band (step S14). Then, the position difference information creation unit <NUM> focuses on one small region of the reference band corresponding to the focused small region of the object band (step S15). In the present embodiment, it is assumed that the position difference of the object band is one pixel or smaller. Therefore, the one small region of the reference band corresponding to the focused small region of the object band is a small region located at the same position as that of the small region of the object band. That is, when the small region of the focused object band is a small region at the upper left corner in <FIG>, the focused small region in the reference band is also a small region at the upper left corner in <FIG>.

Then, the position difference information creation unit <NUM> calculates the shift quantity (s, t) in which the focused small region of the object band most closely matches the focused small region of the reference band (step S16). For example, in the case where the focused small region of the object band most closely matches the focused small region of the reference band when it is shifted by <NUM> pixels in the X axis direction and <NUM> pixels in the Y axis direction for example, the X-axial shift quantity s = <NUM> pixels and the Y-axial shift quantity t = <NUM> pixels are the obtained shift quantity. Such shift quantity may be calculated by using a subpixel matching method that enables calculation of shift quantity with the accuracy of less than <NUM> pixel, such as a phase limiting correlation method or an SSD parabola fitting method. Then, the position difference information creation unit <NUM> updates the position difference information of the focused object band by using the calculated shift quantity (s, t) as the position difference of every pixel included in the focused small region of the object band (step S17). Note that the position difference may be a real number or an integer.

Then, the position difference information creation unit <NUM> moves the focus to another small region of the focused object band (step S18), and returns to the processing of step S15 to execute the processing similar to that described above on the newly focused small region of the object band. Then, upon completion of focusing on all small regions in the focused object band (YES at step S19), the position difference information creation unit <NUM> moves the focus to one of the other object bands (for example, B band) (step S20), and returns to the processing of step S13 to execute the processing similar to that of the processing described above on the newly focused object band. Then, upon completion of focusing on all object bands (that is, R and B bands) (YES at step S21), the position difference information creation unit <NUM> stores the created position difference information of the respective object bands in the storage unit <NUM> (step S22). Then, the position difference information creation unit <NUM> ends the processing illustrated in <FIG>.

<FIG> is a flowchart illustrating another example of processing by the position difference information creation unit <NUM>. The processing illustrated in <FIG> differs from the processing illustrated in <FIG> in that step S17 is replaced with step S17A and that new step S23 is provided between step S19 and step S20. The rest are the same as those illustrated in <FIG>. At step S17A of <FIG>, the position difference information creation unit <NUM> updates the position difference information of the focused object band by using the shift quantity (s, t) calculated at step S16 as a position difference of the pixel at the center position in the focused small region of the object band. Therefore, at step S17A of <FIG>, the position difference of the pixels other than the pixel at the center position in the focused small region is not updated, and the NULL value that is the initial value remains. At step S23 of <FIG>, the position difference information creation unit <NUM> calculates the position difference of the pixels other than the pixel at the center position in each small region of the focused object band, by interpolation from the position difference of the pixel at the center position in each small region calculated at step 17A, and updates the position difference information of the focused object band. The interpolation method may be interpolation by a weighted average according to the distance from the center of an adjacent small region, for example.

When the shift quantity calculated for each small region is used as the position difference of all pixels in the small region as illustrated in <FIG>, the position difference may not continue at the boundary of small regions. Meanwhile, in the method illustrated in <FIG>, position difference changes continuously according to the pixel positions, which can prevent discontinuous position difference. Consequently, the method illustrated in <FIG> has an effect of preventing generation of a level difference in the color of the corrected image at the boundary of small regions.

Further, in the method illustrated in <FIG> and <FIG>, the entire images of the reference band and the object band are thoroughly divided into small regions. However, as illustrated in <FIG>, the position difference information creation unit <NUM> may divide them such that there is a gap (hatched portion in the figure) between small regions. Then, the position difference information creation unit <NUM> may obtain the position difference of a pixel included in the gap by interpolation from the position difference of the pixel at the center position calculated at step S17 or S17A. The interpolation in that case may be interpolation by a weighted average according to the distance from the center of the adjacent small region, for example. According to the method of creating the position difference by dividing the reference band and the object band into small regions so as to have a gap between small regions as described above, the calculation time can be reduced compared with the case of dividing them so as to not to have any gap.

In the method illustrated in <FIG> and <FIG>, the position difference information creation unit <NUM> creates the position difference of each pixel of the object band image from the multiband image <NUM>. However, the position difference information creation unit <NUM> may create the position difference of each pixel of the object band image from the position and posture information of the platform (artificial satellite or aircraft) that acquires the multiband image. In general, an image captured from an artificial satellite or an aircraft is projected to a map by using the position, posture, and the like of the platform at the time of acquiring the image to thereby be processed into an image product. In the case of a multispectral image, since it is projected to a map by each band, position difference information is also obtained in the process of map projection. Therefore, the position difference information creation unit <NUM> may create the position difference information of each pixel of the object band image by using a general map projection method.

<FIG> is a block diagram illustrating another example of the position difference acquisition unit <NUM>. The position difference acquisition unit <NUM> of this example is configured to include a position difference information input unit <NUM>.

The position difference information input unit <NUM> is configured to input the position difference information <NUM> therein from an external device not illustrated via the communication I/F unit <NUM>, and store it in the storage unit <NUM>. Alternatively, the position difference information input unit <NUM> is configured to input therein the position difference information <NUM> from an operator of the image correction device <NUM> via the operation input unit <NUM>, and store it in the storage unit <NUM>. That is, the position difference information input unit <NUM> is configured to input therein the position difference information <NUM> calculated by a device other than the image correction device <NUM>, and store it in the storage unit <NUM>.

As described above, the position difference acquisition unit <NUM> is configured to create by itself the position difference information <NUM> of the multiband image <NUM>, or input it therein from the outside, and store it in the storage unit <NUM>.

Next, the corrected multiband image creation unit <NUM> will be described in detail.

<FIG> is schematic diagram for explaining the principle that the corrected multiband image creation unit <NUM> creates a corrected multiband image from a multiband image, a high-resolution image, and position difference information. Here, as illustrated in <FIG>, a subject <NUM> in which a smaller whiteboard 18b overlaps a blackboard 18a is considered. In <FIG>, a reference numeral <NUM>-<NUM> denotes an R-band image obtained by imaging the subject <NUM> and is configured of <NUM> × <NUM> = <NUM> pieces of pixels. Meanwhile, in <FIG>, a reference numeral <NUM> denotes a high-resolution image obtained by imaging the subject <NUM> and is configured of <NUM> × <NUM> = <NUM> pieces of pixels. That is, there are the R-band image <NUM>-<NUM> and the high-resolution image <NUM> in which the same subject <NUM> is imaged. Each of the four pixels of the R-band image <NUM>-<NUM> shows part of the blackboard 18a and part of the whiteboard 18b in its imaging range. Therefore, assuming that a maximum value of the pixel value of a pixel of the R-band image is <NUM>, the pixel value of the four pixels of the R-band image <NUM>-<NUM> is an intermediate pixel value corresponding to the ratio of the blackboard and the whiteboard included in the imaging range. On the other hand, in the high-resolution image <NUM>, there is a pixel showing only the blackboard 18a and a pixel showing only the whiteboard 18b in the imaging range. Therefore, the pixel value of a pixel of the high-resolution image <NUM> showing only the blackboard 18a is zero, while the pixel value of a pixel of the high-resolution image <NUM> showing only the whiteboard 18b is <NUM>. As described above, distribution of luminance values and brightness within the imaging range of pixels of the R-band image <NUM>-<NUM> in which the blackboard 18a and the whiteboard 18b are included in the imaging range can be estimated from the pixel values of <NUM> × <NUM> = <NUM> pieces of pixels of the high-resolution image <NUM> corresponding to the one pixel of the R-band image <NUM>-<NUM>.

That is, the luminance values and the brightness have a strong correlation between the multiband image and the high-resolution image, that is, between the bands. Therefore, assuming that each of the four pixels of the R-band image is configured of <NUM> × <NUM> = <NUM> pieces of pixels that is similar to the high-resolution image as denoted by <NUM>-<NUM>' in <FIG>, it is considered that the relationship between the pixel values of the pixels of the <NUM> pieces of pixels becomes the same as the relationship between the pixel values of the pixels of the <NUM> pieces of pixels of the corresponding high-resolution image <NUM>. The present invention focuses on such a point, and determines a pixel value of a plurality of sub regions obtained by vertically and horizontally dividing the imaging region of one pixel of the multiband image into a plurality of pieces, on the basis of the pixel value of such a pixel and a relationship between the pixel values of a plurality of pixels of the high-resolution image corresponding to such a pixel.

Then, by shifting the pixel of the object band image by the position difference, the present invention determines the pixel position of the reference band image, and determines the total sum of the pixel values of a plurality of sub regions on the object band image included in the determined pixel position to be a pixel value of light on the object band at the pixel position.

An example of a relationship between pixel values of pixels that can be used in the present invention is a ratio of pixel values. Instead of a ratio of pixel values, a difference between pixel values can also be used. The ratio of pixel values between pixels is the same between the object band image and the high-resolution image means that, in the case of four pixels as an example, m1:m2:m3:m4 = r1:r2:r3:r4 is established, where m1 to m4 represent pixel values of the four pixels of the object band image and r1 to r4 represent pixel values of the four pixels of the high-resolution image. Further, a difference between pixels of a plurality of pixels is the same between the object band image and the high-resolution image means that pixel values of the four pixels of the object band image are expressed as gg+r1-av, gg+r2-av, gg+r3-av, and gg+r4-av, where av represents an average of pixel values of the four pixels of the high-resolution image, and gg represents the value of the original pixel.

As a relationship between pixel values, whether to use a ratio of the pixel values or use a difference between the pixel values may be determined arbitrarily. For example, in the environment where a condition that the brightness ratio is the same between the corresponding pixels of the multiband image and the high-resolution image is established, the ratio of pixel values may be used. That is, in order to enable comparison of the brightness ratio, if the pixel value <NUM> serving as the reference is in a state of not applied with light so that it is in an environment where a pixel value is determined in comparison with the brightness entering each band, the ratio of pixel values may be used. However, in an image obtained by capturing the ground from an artificial satellite in particular, not only light reflected at the ground surface that is a desirable signal but also light scattered in the atmosphere also enters the sensor. Therefore, the pixel value becomes larger by the light scattered in the atmosphere. In the light scattered in the atmosphere, since a shorter wavelength has a larger value, how the pixel value becomes larger differs according to the band. Therefore, in an image capturing the ground from an artificial satellite, the ratio of pixel values may not show the brightness ratio. Accordingly, in such an environment, it is preferable to use a difference between pixel values as the relationship between the pixel values. This is because the difference between pixel values is not changed even if a certain quantity of pixel value of each band is added. By using the difference between pixel values, with respect to an image of the ground captured from an artificial satellite, it is possible to remove the effect of adding the output by the light scattered in the atmosphere or the like. Therefore, by using the difference between pixel values, even if the ratio of pixel values does not show the brightness ratio, it is possible to create a corrected image with no color shift.

The example of processing the multiband image with the precision of <NUM>/<NUM> pixel has been described above. It is also possible to enlarge the multiband image and the high-resolution image by interpolation to thereby perform processing with the precision of <NUM>/<NUM> pixel or higher (for example, <NUM>/<NUM> pixel). Hereinafter, a specific example by the corrected multiband image creation unit <NUM> will be described in detail.

<FIG> is a flowchart illustrating exemplary processing by the corrected multiband image creation unit <NUM>. Referring to <FIG>, the corrected multiband image creation unit <NUM> first enlarges each band image constituting the multiband image <NUM> so as to have the same resolution as that of the high-resolution image <NUM> (step S31). For example, when the resolution of the multiband image <NUM> is <NUM>/<NUM> of that of the high-resolution image <NUM>, the corrected multiband image creation unit <NUM> enlarges the multiband image <NUM> by four times. Alternatively, the corrected multiband image creation unit <NUM> may enlarge the multiband image <NUM> by eight times, and enlarge the high-resolution image <NUM> by two times. In this way, both the multiband image and the high-resolution image may be enlarged if the resolution of the multiband image and that of the high-resolution image become the same. Enlargement of an image is performed using interpolation such as bilinear interpolation or bicubic interpolation, for example. In this example, it is assumed that the multiband image <NUM> is enlarged by eight times and the high-resolution image <NUM> is enlarged by two times by interpolation so as to allow the multiband image <NUM> to be processed with precision of <NUM>/<NUM> pixel. Hereinafter, a multiband image and a high-resolution image after the enlargement may be referred to as an enlarged multiband image and an enlarged high-resolution image.

Then, the corrected multiband image creation unit <NUM> focuses on one of the object bands (for example, R band) (step S32). Then, the corrected multiband image creation unit <NUM> uses an enlarged object band image and an enlarged high-resolution image to create an image (referred to as a subpixel object band image) in which the pixel value of each pixel of the focused object band image is allocated to each sub region when each pixel is divided into a plurality of sub regions (step S33). Then, the corrected multiband image creation unit <NUM> uses position difference information of the created subpixel object band image and the object band to create a correction object band image (step S34). Then, the corrected multiband image creation unit <NUM> moves the focus to another object band (step S35), returns to step S33 through step S36 to repeat the processing similar to the processing described above. Then, upon completion of creating a correction object band image for all object bands (YES at step S36), the corrected multiband image creation unit <NUM> stores the created correction object band images in the storage unit <NUM> (step S37). Then, the corrected multiband image creation unit <NUM> ends the processing illustrated in <FIG>.

<FIG> is a flowchart illustrating the details of step S33 of <FIG>, that is, exemplary processing of creating a subpixel object band image. Referring to <FIG>, the corrected multiband image creation unit <NUM> first converts the enlarged high-resolution image so as to overlap the enlarged object band image to thereby create a reference image (step S41). For example, when the object band is an R band, the corrected multiband image creation unit <NUM> estimates affine transformation (A, b) (A represents a matrix, b represents a vector) to superimpose the enlarged R band image on the enlarged high-resolution image, and applies inverse transformation of the obtained affine transformation (A, b) to the enlarged high-resolution image to thereby create a reference image. Note that since a pixel position after the affine transformation is not an integer generally, interpolation processing is performed to obtain it.

Then, the corrected multiband image creation unit <NUM> creates a subpixel object band image in an initial state (step S42). The subpixel object band image in an initial state has a pixel corresponding to the pixel of an enlarged band image of the focused object band one by one, which is a pixel in which the pixel value of the original one pixel is set to all pixels (in this example, <NUM> pieces of pixels) after the enlargement corresponding to one pixel of the object band image before the enlargement.

Then, the corrected multiband image creation unit <NUM> focuses on one pixel (referred to as an object pixel) of the object band image (step S43). Then, from the pixel value of the focused object pixel and the relationship between the pixel values of the pixels of the <NUM> pieces of pixels of the reference image corresponding to the <NUM> pieces of pixels of the enlarged object band image corresponding to the focused object pixel, the corrected multiband image creation unit <NUM> determines the pixel values after correction of the pixel values of the <NUM> pieces of pixels of the enlarged object band image corresponding to the focused object pixel (step S44). For example, when (i, j) represents the object pixel, R(i, j) represents the pixel value of the object pixel (i, j), (u, v) (u, v = <NUM>, <NUM>,. <NUM>) represents the <NUM> pieces of pixels of the enlarged object band image corresponding to the object pixel (i, j), P(i, j, u, v) represents the pixel value of the <NUM> pieces of pixels of the reference image corresponding to the <NUM> pieces of pixels of the enlarged object band image corresponding to the object pixel (i, j), and R'(i, j, u, v) represents the pixel value after correction, the corrected multiband image creation unit <NUM> calculates the pixel value R'(i, j, u, v) after correction by using Expression <NUM> shown in <FIG>. However, in Expression <NUM>, <P(i, j, u, v)> represents an average value of the pixel values of the <NUM> pieces of pixels of the reference image corresponding to the <NUM> pieces of pixels of the enlarged object band image corresponding to the object pixel (i, j).

The average value of the pixel values R'(i, j, u, v) after the correction, calculated according to Expression <NUM>, is the same as the pixel value R(i, j) of the object pixel (i, j). Further, allocation of the pixel values R'(i, j, u, v) after the correction, calculated according to Expression <NUM>, becomes the same as the pixel values P(i, j, u, v) of the <NUM> pieces of pixel of the reference image corresponding to the <NUM> pieces of pixels of the enlarged object band image corresponding to the object pixel (i, j). In Expression <NUM>, a difference between pixel values is used as a relationship between the pixel values. However, it is also possible to determine the pixel values after the correction of the pixel values of the <NUM> pieces of the enlarged object band image corresponding to the focused object pixel by using a ratio as a relationship between the pixel values.

Then, the corrected multiband image creation unit <NUM> updates the pixel value of the subpixel object band image with the determined pixel value after the correction (step S45). Then, the corrected multiband image creation unit <NUM> moves the focus to another pixel (object pixel) of the object band image (step S46), returns to step S44 through step S47 to repeat processing similar to the processing described above. Then, upon completion of focusing on all pixels of the object band image (YES at step S47), the corrected multiband image creation unit <NUM> ends the processing illustrated in <FIG>.

<FIG> is a flowchart illustrating the details of step S34 of <FIG>, that is, exemplary processing of creating a correction object band image from a subpixel object band image and position difference information. Referring to <FIG>, the corrected multiband image creation unit <NUM> first focuses on one pixel (object pixel) of the focused object band image (step S51). Then, the corrected multiband image creation unit <NUM> determines the pixel position of an enlarged reference band image by shifting the <NUM> pieces of pixels of the enlargement object band image corresponding to the focused object pixel by the position difference of the object pixel (step S52). Here, in the present example, shifting is performed with a <NUM>/<NUM> pixel being a minimum unit. Therefore, when the position difference is not an integral multiple of <NUM>/<NUM> pixel, shifting is performed after correcting the difference to the closest integral multiple of <NUM>/<NUM> pixel. For example, when the <NUM> pieces of pixels of the enlargement object band image corresponding to the focused object pixel are a pixel group within a bold solid line in <FIG> and the position difference (s, t) is (<NUM>/<NUM> pixel, <NUM>/<NUM> pixel), the corrected multiband image creation unit <NUM> determines a pixel group within a broken line in <FIG> to be a pixel position of the enlargement reference band image.

Then, the corrected multiband image creation unit <NUM> calculates the total sum of the pixel values of the <NUM> pieces of pixels at the determined pixel position as a pixel value of light on the enlargement object band image at the determined pixel position, and stores in the correction object band image (step S53). Then, the corrected multiband image creation unit <NUM> moves the focus to another pixel (object pixel) of the object band image (step S54), and returns to step S52 through step S55 to repeat processing similar to the processing described above. Then, upon completion of focusing on all pixels of the object band image (YES at step S55), the corrected multiband image creation unit <NUM> ends the processing illustrated in <FIG>.

As described above, the image correction device <NUM> according to the present embodiment first acquires a plurality of band images obtained by imaging a subject, and a high-resolution image obtained by imaging the subject and having higher resolution than that of the band images. Then, using at least one of the band images as a reference band image and at least one of the rest as an object band image, the image correction device <NUM> acquires a position difference between the object band image and the reference band image. Then, by using a pixel of the object band image as an object pixel, for each pixel, the image correction device <NUM> determines a pixel value of each of sub regions obtained by dividing the imaging region of the object pixel into a plurality of regions, on the basis of the pixel value of the object pixel and a relationship between pixel values of a plurality of pixels of the high-resolution image corresponding to the object pixel, and creates a corrected band image that holds a pixel value of light on the object band image at the pixel position of the reference band image from the determined pixel value of the sub region and the position difference. Thereby, the image correction device <NUM> of the present embodiment can reduce a color shift caused by a phase difference.

The configuration, operation, and effects of the image correction device <NUM> according to the first exemplary embodiment has been described above. Next, some modifications of the first exemplary embodiment will be described.

In Modification <NUM>, as illustrated in the flowchart of <FIG>, the processing differs from that illustrated in <FIG> in that the corrected multiband image creation unit <NUM> is configured to further execute step S48 of adjusting the pixel value of an image.

The sensitivity or offset may differ between the multiband image to be processed and a reference image created from a high-resolution image (for example, panchromatic image). In that case, when a pixel value of each of the sub regions obtained by dividing the pixel region of the object pixel of the object band image into a plurality of regions is determined on the basis of the pixel value of the object pixel and the relationship between the pixel values of a plurality of pixels of the high-resolution image corresponding the object pixel, an error becomes large.

Therefore, in the case of using a difference between the pixel values as the relationship between the pixel values, the corrected multiband image creation unit <NUM> adjusts the pixel values of the reference image at step S48 by using Expression <NUM> shown in <FIG>. Further, in the case of using a ratio of pixel values as the relationship between the pixel values, the corrected multiband image creation unit <NUM> adjusts the pixel values of the object band image and the reference image at step S48 by using Expressions <NUM> and <NUM> shown in <FIG>. In Expressions <NUM>, <NUM>, and <NUM>, VM(x, y) represents a pixel value of an object band image at a pixel (x, y), VR(x, y) represents a pixel of a reference image created from a high-resolution image, VRC(x, y) represents a pixel value of the reference image after the adjustment, VMC(x, y) represents a pixel value of the object band image after the adjustment, VR(x, y) with an overline represents an average of VR(x, y), σ(VM(x, y)) and σ(VR(x, y)) represent standard deviation of VM(x, y) and standard deviation of VR(x, y), respectively, and minVM(i, j) and minVR(i, j) represent minimum values of the object band image and the reference image, respectively.

In the example illustrated in <FIG>, the position difference information <NUM> is recorded in a list of sets of pixel position and position difference of each pixel of the object band image. However, the recording method of the position difference information <NUM> is not limited to that described above. For example, the position difference information <NUM> may be recorded in such a manner that the object band image is divided into a plurality of sub regions consisting of a plurality of pixels having the same position difference, and the position difference information <NUM> is recorded as a list of sets of pixel position and position difference of each sub region. The shape of a sub region may be a rectangle for example. Further, the pixel position of a sub region may be a set of pixel positions of an upper left pixel and a lower right pixel if it is a rectangle. Further, if the position differences of all pixels of the object band image are almost the same, only one position difference may be recorded.

Moreover, the position difference information <NUM> may be recorded as a mathematical expression or a coefficient of a mathematical expression, instead of being recorded as numerical information. For example, when the position difference is caused by optical distortion, the position difference is determined by the positional relationship between the object band and the reference band on the focus surface or optical characteristics. Therefore, the position difference of each pixel can be expressed by an expression defined by optical characteristics using the pixel position as an argument. Accordingly, such an expression or a coefficient thereof may be recorded as the position difference information <NUM>. The position difference of each pixel can be calculated from the aforementioned expression.

Further, the position difference acquisition unit <NUM> may, for each object band, calculate an approximate plane from the calculated position difference of each pixel position, and record an expression representing the calculated approximate plane or a coefficient thereof as the position difference information <NUM>. For example, when each pixel of the object band is three-dimensional point group data consisting of three-dimensional data (x, y, (s, t)) of a position difference (s, t) of the pixel position x and the pixel position y for example, the approximate plane may be a plane in which the sum of the square distance from the point group becomes minimum. For example, in the case of using a plane given by Expression <NUM> of <FIG> as an approximate plane, the position difference acquisition unit <NUM> calculates a matrix A and a vector b that fit best by using the calculated position difference of each pixel position, to thereby able to obtain an expression representing the position difference information <NUM> of all pixels. While a coefficient of Expression <NUM> is obtained by using the position difference of each pixel position of the object band in the above description, it is possible to obtain the matrix A and the vector b that fit best by using the position difference of a pixel at the center position of each sub region described with reference to <FIG>.

In the above description, the multiband image <NUM> is an image of three bands namely RGB. However, the multiband image <NUM> may be one other than that. For example, the multiband image <NUM> may be a four band image having three bands, namely RGB, and a near-infrared band. As described above, the number of bands of the multiband image <NUM> is not limited, and any number of bands having any wavelength bands may be used.

In the above description, it is described that the position difference (s, t) between the pixel (x, y) of the object band and the pixel (x, y) of the reference band at the same pixel position is <NUM> or larger and less than <NUM>. However, the position difference (s, t) may be less than <NUM> or <NUM> or larger. With respect to any position difference (s, t), it is assumed that s' = s-s0, t' = t-t0 are established, where s0 represents a maximum integer not exceeding s, and t0 represents a maximum integer not exceeding t. Then, with respect to the pixel (x, y) of the reference band, when x' = x-s0 and y' = y-t0, the position difference between the pixel (x, y) of the object band and the pixel (x', y') of the reference band becomes (s', t') that is <NUM> or larger and less than <NUM>. Therefore, by replacing the pixel (x, y) of the reference band with the pixel (x', y'), and replacing the position difference (s, t) with the position difference (s', t'), it is possible to obtain the pixel value of the corrected band image by the processing that is the same as the above-described processing.

<FIG> is a block diagram illustrating an image correction device <NUM> according to a second exemplary embodiment of the present invention. Referring to <FIG>, the image correction device <NUM> is configured to include a band image acquisition means <NUM>, a high-resolution image acquisition means <NUM>, a position difference acquisition means <NUM>, a corrected band image creation means <NUM>, and a corrected band image output means <NUM>.

The band image acquisition means <NUM> is configured to acquire a plurality of band images obtained by capturing a subject. The band image acquisition means <NUM> may be configured similarly to the multiband image acquisition unit <NUM> of <FIG> for example, but is not limited thereto.

The high-resolution image acquisition means <NUM> is configured to acquire a high-resolution image obtained by imaging the subject and having a higher resolution than that of the band image. The high-resolution image acquisition means <NUM> may be configured similarly to the high-resolution image acquisition unit <NUM> of <FIG> for example, but is not limited thereto.

The position difference acquisition means <NUM> is configured to, by using at least one of the band images as a reference band image and at least one of the rest as an object band image, acquire a position difference between the object band image and the reference band image. The position difference acquisition means <NUM> may be configured similarly to the position difference acquisition unit <NUM> of <FIG> for example, but is not limited thereto.

The corrected band image creation means <NUM> is configured to, by using a pixel of the object band image as an object pixel, for each object pixel, determine a pixel value of each of sub regions obtained by dividing imaging region of the object pixel into a plurality of regions, on the basis of the pixel value of the object pixel and the relationship between the pixel values of a plurality of pixels of the high-resolution image corresponding the object pixel. Further, the corrected band image creation means <NUM> is configured to create a corrected band image that holds a pixel value of light on the object band image at the pixel position of the reference band image, from the determined pixel value of each sub region and the position difference. The corrected band image creation means <NUM> may be configured similarly to the corrected multiband image creation unit <NUM> of <FIG> for example, but is not limited thereto.

The corrected band image output means <NUM> is configured to output the corrected band image. The corrected band image output means <NUM> may be configured similarly to the corrected multiband image output unit <NUM> of <FIG> for example, but is not limited thereto.

The image correction device <NUM> configured as described above operates as described below. First, the band image acquisition means <NUM> acquires a plurality of band images obtained by imaging a subject, and the high-resolution image acquisition means <NUM> acquires a high-resolution image obtained by imaging the subject and having higher resolution than that of the band images. Then, the position difference acquisition means <NUM> acquires, by using at least one of the band images as a reference band image and at least one of the rest as an object band image, a position difference between the object band image and the reference band image. Then, the corrected band image creation means <NUM> determines, by using a pixel of the object band image as an object pixel, for each object pixel, a pixel value of each of sub regions obtained by dividing the pixel region of the object pixel into a plurality of regions, on the basis of the pixel value of the object pixel and the relationship between the pixel values of a plurality of pixels of the high-resolution image corresponding the object pixel. Further, the corrected band image creation means <NUM> creates a corrected band image that holds a pixel value of light on the object band image at the pixel position of the reference band image, from the determined pixel value of each sub region and the position difference. Then, the corrected band image output means <NUM> outputs the corrected band image.

According to the image correction device <NUM> that is configured and operates as described above, it is possible to reduce a color shift caused by a phase difference. This is because the image correction device <NUM> acquires a plurality of band images obtained by imaging a subject and a high-resolution image obtained by imaging the subject and having a higher resolution than that of the band images, acquires, by using one of the band images as a reference band and using at least one of the rest as an object band image, a position difference between the object band image and the reference band image, and by using a pixel of the object band image as an object pixel, for each of the object pixel, determines a pixel value of each of sub regions obtained by dividing the imaging region of the object pixel into a plurality of regions, on the basis of the pixel value of the object pixel and the relationship between pixel values of the pixels of the high-resolution image corresponding to the object pixel, and creates a corrected band image that holds a pixel value of light on the object band image at the pixel position of the reference band image from the determined pixel value of each sub region and the position difference.

While the present invention has been described with reference to the exemplary embodiments described above, the present invention is not limited to the above-described embodiments. The form and details of the present invention can be changed within the scope of the claims in various manners that can be understood by those skilled in the art.

Claim 1:
An image correction device (<NUM>) comprising:
a wavelength band image acquisition means (<NUM>) for acquiring a plurality of wavelength band images obtained by imaging a subject;
a high-resolution image acquisition means (<NUM>) for acquiring a high-resolution image (<NUM>) obtained by imaging the subject, the high-resolution image (<NUM>) having a resolution that is higher than a resolution of the wavelength band images;
a position difference acquisition means (<NUM>) for, by using at least one of the wavelength band images as a reference band image and at least one of rest of the wavelength band images as an object band image, acquiring a position difference between the object band image and the reference band image;
a corrected band image creation means (<NUM>) for creating a corrected band image from the high-resolution image, the reference band image, the object band image, and the position difference; and
a corrected band image output means (<NUM>) for outputting the corrected band image, wherein
the corrected band image creation means (<NUM>) is configured to:
create an enlarged object band image and an enlarged reference band image by enlarging the object band image and the reference band image so as to have a same resolution as the resolution of the high-resolution image;
create a reference image in which the high-resolution image is converted so as to overlap the enlarged object band image;
by using a pixel of the object band image as an object pixel, for each object pixel, determine pixel values of a plurality of pixels corresponding to the object pixel in the enlarged object band image, on a basis of a pixel value of the object pixel and a relationship between pixel values of a plurality of pixels of the reference image corresponding to the object pixel; and
for each object pixel, determine a total sum of pixel values of a plurality of pixels of the enlarged object band image overlapped when the plurality of pixels of the enlarged object band image corresponding to the object pixel are shifted according to the position difference, as a pixel value of a pixel corresponding to the object pixel in the corrected band image.