Image processing method and device interpolating G pixels

According to one embodiment, an image processing method includes storing, in a frame memory, first and second images of continuously taken N (however, N is not less than 2) images, cutting an image as an image for correction from a plurality of reference areas of the first and second images stored in the frame memory, calculating such a hand shake correction amount that the first image and the second image overlap with high accuracy with the use of a plurality of images for correction of the first and second images and correcting the first image with the calculated hand shake correction amount, overwriting the corrected first image in the first image stored in the frame memory, composing the corrected first image and the second image to obtain a composite image, and overwriting the composited the composite image in the second image stored in the frame memory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2010-069321, filed on Mar. 25, 2010; the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processing method and device.

BACKGROUND

In a camera module used in a digital camera and so on, a method of correcting hand shake includes an electronic method (for example, see Japanese Patent Laid-Open Publication No. 11-75105) of performing calculation from image data, received from a light-receiving element at the time of recording, and performing correction and an optical method of adjusting an optical axis physically.

In a camera module, it is preferable that the thickness of a lens module is reduced in accordance with requirements for reduction of the thickness and size. In the electronic method, a module having a dedicated function is not required to be mounted, and therefore, the electronic method is more excellent than the optical method in terms of the reduction of the thickness and size.

In a general electronic hand shake correction, a plurality of images are continuously taken, and a large number of images are stored in a frame memory. Thereafter, blur due to hand shake is reduced in the post-stage signal processing.

However, in the related art electronic hand shake correction, since a high-capacity frame memory is used, cost may be significantly increased.

DETAILED DESCRIPTION

In general, according to one embodiment, an image processing method includes storing, in a frame memory, first and second images of continuously taken N (however, N is not less than 2) images, cutting an image as an image for correction from a plurality of reference areas of the first and second images stored in the frame memory, calculating such a hand shake correction amount that the first image and the second image overlap with high accuracy with the use of a plurality of images for correction of the first and second images and correcting the first image with the calculated hand shake correction amount, overwriting the corrected first image in the first image stored in the frame memory, composing the corrected first image and the second image to obtain a composite image, and overwriting the composited the composite image in the second image stored in the frame memory.

Exemplary embodiments of an image processing method and device will be explained below in detail with reference to the accompanying drawings. The present invention is not limited to the following embodiments.

The components in the following embodiments include ones easily assumed by those skilled in the art or substantially the same ones.

First Embodiment

FIG. 1is a block diagram showing one example of a configuration of an image recording apparatus provided with an image processing device according to a first embodiment. As shown inFIG. 1, the image recording apparatus is provided with an imaging lens1, an IR cut filter2, an image sensor unit3, an image processing device4, and a recording unit5. In the example shown inFIG. 1, although the image sensor unit3and the image processing device4are separated, the image sensor unit3may include the image processing device4.

The imaging lens1constitutes an optical system for taking in light from a subject and forms a subject image in the image sensor unit3. The IR cut filter2removes infrared light from the light taken in by the imaging lens1. The image sensor unit3converts the light taken in by the imaging lens1into a signal charge to thereby image the subject image. The image sensor unit3takes in pixel values of R (red), G (green), and B (blue) in an order corresponding to a Bayer array to thereby generate an analog image signal. The image sensor unit3further converts the obtained analog image signal into a digital image signal to output the digital image signal to the image processing device4.

The image processing device4applies various image processing to the digital image signal input from the image sensor unit3to output the processed digital image signal to the recording unit5. The image processing device4is provided with a frame memory6used as a working region when hand shake correction and so on are performed and an image processing circuit11including a hand shake correction function, a shading correction function, and a white balance adjustment function. The recording unit5records image data, input from the image processing device4, in a memory and a recording medium.

Next, hand shake correction processing in the image processing device4will be described with reference toFIGS. 2 to 8.FIG. 2is a view for explaining a reference area of an image for hand shake correction and a moving direction of an image.FIGS. 2A and 28are views for explaining the reference area of the image for hand shake correction.FIG. 2Bis a view for explaining the moving direction of the image.FIG. 3is a view for explaining a processing flow of the hand shake correction in the image processing device4. Especially,FIG. 3shows a case where two images are processed continuously.FIG. 4is a schematic view for explaining a usage of the frame memory6. As shown inFIG. 4, the frame memory6is provided with two image memories21and22and two reference memory groups23and24which store images of a plurality of reference areas of images stored in the two image memories21and22.

In a hand shake correction mode, if a shutter speed is not reduced, when a shutter operation is performed under such a dark taking condition that sufficient sensitivity is not obtained, and noise is increased, two images are continuously taken at high speed so that a subject is not blurred. A first taken RAW image is referred to as an image1, and a second taken RAW image is referred to as an image2.

As shown inFIG. 2A, a plurality of (nine in the example shown inFIG. 2A) the reference areas for hand shake correction (hereinafter referred to as “window”) are set in an image frame. The number of the window is not limited to this example. The window may be set around the center of the angle of view in a region where an image height is up to 70% as a region with a tendency to suppress deterioration of lens performance. According to this constitution, a memory capacity and a calculation time are reduced, and, at the same time, a hand shake correction amount can be detected with high accuracy.

InFIG. 3, the images1and2input from the sensor images unit3are stored in the frame memory6(see, (A) inFIG. 4). In crop processing111and112, a plurality of the windows shown inFIG. 2Aare cut from the images1and2stored in the frame memory6. The images cut from the images1and2are images for hand shake correction used when the hand shake correction amount is calculated and referred to respectively as window images1and window images2. The window images1and the window images2are stored in the frame memory6(see, (A) inFIG. 4). When the calculation time is reduced, or when the capacity of the frame memory is reduced, the number of the windows may be reduced, or the area of the window may be reduced.

Next, in demosaicing (image restoration) processing121and122, the demosaicing processing is applied to a plurality of the window images1and2while paying attention to a G signal, and a G image constituted only of the G signal is generated. The window images1and2of the G image are used in the following processing. In the present embodiments, the demosaicing processing means that all the G images are restored by interpolation etc from an image in which R and B signals are omitted and there is only a G signal.

In a frame matching processing13, as shown inFIG. 2B, the image1is moved every constant amount in pitch and yaw directions. The pitch means a Y-axis (longitudinal) direction, and the yaw means an X-axis (lateral) direction. A difference of a signal value between the window images1after movement and the window images2is calculated for each window movement by movement, and moving directions and movement amounts of the pitch and the yaw in which the difference is minimum are calculated. The window images1are cut from the image1every movement of the image1to be overwritten in the frame memory6. The difference of the signal value is calculated using the window images1and the window images2stored in the frame memory6. When the images1and2overlap with high accuracy, the difference of the signal value between the window images1and the window images2is minimum.

The moving directions and the movement amounts correspond to the hand shake correction amount. The image1is moved by the calculated moving directions and movement amounts of the pitch and the yaw, and, namely, the hand shake correction of the image1is performed with the calculated hand shake correction amount. The image1after movement (hereinafter referred to as a “post-movement image1”) is overwritten in the image1in the frame memory6(see, (B) inFIG. 4).

Finally, in a merge processing14, the post-movement image1and the image2are composed to generate a composite image2. The composite image2is overwritten in the image2in the frame memory6(see, (C) inFIG. 4).

According to the above constitution, the electronic hand shake correction can be performed using the frame memory6having a small capacity, and a high-definition image with a high image resolution and with little noise can be obtained.

FIG. 5is a flow chart for explaining a flow of the frame matching processing14in the image processing device4. InFIG. 5, the image1is first moved every constant amount in the pitch and yaw directions within a specified range (step S1). As the image1is moved more finely, blur can be corrected with higher accuracy, and thus it is preferable that the image1is moved every one pixel.

Next, the difference of the signal value between the window images1of the post-movement image1and the window images2is calculated movement by movement (step S2). It is preferable that the difference of the signal value is the sum of squares of differences. When the calculation time is reduced, the difference of the signal value may be an average value of differences.

The moving directions and movement amounts of the pitch and the yaw in which the difference of the signal value calculated movement by movement is minimum are calculated as the hand shake correction amount (step S3). The image1is moved to the moving directions and movement amounts of the pitch and the yaw calculated in step S3(step S4), and, namely, the image1is corrected by the hand shake correction amount.

In the above embodiment, although the two images are processed continuously, three or more images may be processed continuously.FIG. 6is a view for explaining a processing flow of the hand shake correction in the image processing device4when three images are processed continuously.FIG. 7is a schematic view for explaining a usage of the frame memory6when three images are processed continuously.

When a shutter is operated, three images are continuously photographed and taken in. Since the processing of the first and second images is similar to that inFIGS. 2 and 3, the processing of the third image will be described. A RAW image taken as the third image is referred to as an image3.

InFIG. 6, the image3is overwritten in the post-movement image1in the frame memory6(see, (D) inFIG. 7). In crop processing113and114, processing of cutting each window shown inFIG. 2Ais applied to the composite image2and the image3stored in the frame memory6. The image cut from the composite image2and the image cut from the image3are referred to respectively as a window composite image2and a window image3. The window composite image2and the window image3are stored in the frame memory6(see, (D) inFIG. 7).

Next, in demosaicing processing123and124, the demosaicing processing is applied to the window composite image2and the window image3while paying attention to the G signal, and the G image constituted only of the G signal is generated.

In frame matching processing132, the window composite image2is moved every constant amount in the pitch and yaw directions. The difference of the signal value between the window composite images2of the composite image2after movement (hereinafter referred to as a “post-movement composite image2”) and the window images3of the image3is calculated movement by movement, and the moving directions and movement amounts of the pitch and the yaw in which the difference is minimum are calculated. The composite image2is moved by the calculated moving directions and movement amounts in the pitch and yaw directions. The post-movement composite image2is overwritten in the composite image2in the frame memory6(see, (E) inFIG. 7).

Finally, in merge processing142, the post-movement composite image2and the image3are composed to generate a composite image3. The composite image3is overwritten in the image3in the frame memory6(see, (F) inFIG. 7). When four or more images are processed continuously, a similar method can be used. As the number of images processed continuously increases, an image with a higher image resolution and with less noise is obtained.

As described above, according to the first embodiment, the continuously taken images1and2are stored in the frame memory6, and images are cut as the images for correction from the reference areas of the images1and2stored in the frame memory6. Such a hand shake correction amount that the images1and2overlap with high accuracy is calculated using the images for correction of the plurality of images1and2. The image1is corrected by the calculated hand shake correction amount, and the corrected image1is overwritten in the image1stored in the frame memory6. The corrected images1and2are composed to generate a composite image, and the composite image is overwritten in the image2stored in the frame memory6. Therefore, the electronic hand shake correction can be executed with high accuracy, using a minimum frame memory, and a high-definition image with a high image resolution and with little noise can be obtained.

Further, according to the first embodiment, a G pixel of the images for correction of the images1and2is interpolated, and a G image for correction is restored. The hand shake correction amount is calculated using the restored G image for correction. Consequently, a high accuracy hand shake correction amount can be obtained with a small calculation amount.

Furthermore, according to the first embodiment, when the image1is moved, the difference of the signal value between the images for correction of the image1after movement and the images for correction of the image2is calculated movement by movement, and the moving direction and moving distance in which the difference is minimum are calculated as the hand shake correction amount. Therefore, the high accuracy hand shake correction amount can be calculated.

Furthermore, according to the first embodiment, the reference areas are arranged around the center of the angle of view in the region where the image height is up to 70% as the region with a tendency to suppress deterioration of lens performance. Therefore, the memory capacity and the calculation time are reduced, and, at the same time, the hand shake correction amount can be calculated with high accuracy.

When the continuously taken images include a subject which has moved at a higher speed than the shutter speed, blur (motion blur) of the subject occurs in a screen, and the hand shake sometimes cannot be completely corrected by the above hand shake correction processing. Thus, image in which the motion blur occurs due to the subject having moved at a higher speed than the shutter speed is removed, and a composite image may be generated from only images with no motion blur.FIG. 8is a view for explaining a method of calculating the motion blur.

Specifically, for example, inFIG. 7, the motion blur is detected from the window images1and2of the images1and2. In the detection of the motion blur, a difference value of the brightness or G signal between adjacent pixels is obtained in each of the window images1and2. For example, as shown inFIG. 8, the difference value between the right and lower pixels of a target pixel is obtained. This processing is performed in the window images1and2, and a value obtained by adding up all the difference values is calculated for each of the images1and2.

The obtained values are compared between the images1and2. When the larger value is 100%, and the smaller value is not more than a predetermined value (for example, 80%), it is determined that there is the motion blur, and the image is removed to generate a composite image. For example, when the image2includes the motion blur, the composite image is generated by the images1and3.

As described above, the hand shake correction is performed using an image with no motion blur of a plurality of images, whereby even when a taken image includes a subject which has moved at a higher speed than the shutter speed, the high accuracy hand shake correction can be performed.

Second Embodiment

FIG. 9is a block diagram showing one example of a configuration of an image recording apparatus provided with an image processing device4according to a second embodiment. InFIG. 9, the components having functions equivalent to those inFIG. 1are assigned the same reference numerals, and only the different points will be described. As shown inFIG. 9, the image processing device4according to the second embodiment has a constitution in which a frame memory8and an image processing circuit9are added to the image processing device4according to the first embodiment (see,FIG. 1). Since the image processing device4according to the second embodiment has two sets of the frame memory and the image processing circuit, the hand shake correction processing is performed in parallel, whereby the processing time can be reduced.

In the first embodiment, the crop processing11to the merge processing14shown inFIG. 2are repeatedly executed, whereby the hand shake correction processing is performed using the frame memory with a minimum capacity. Meanwhile, in the second embodiment, two sets of the frame memory and the image processing circuit are provided, so that the crop processing11to the merge processing14shown inFIG. 2are executed in parallel, whereby the processing time can be reduced to half.FIG. 10shows a flow in which a shutter is operated twice in a short time to take two images, and, thus, to perform the hand shake correction. When a circuit size, cost, and a size have allowances, a plurality of sets of the frame memory and the image processing circuit are provided, whereby the processing speed can be increased by times of the number of sets.

In the first and second embodiments, a recording medium in which a program code of software realizing the functions of the above image processing device is recorded is supplied to a system or a device, and a computer (or CPU, MPU, or DSP) of the system or the device may execute the program code stored in the recording medium, whereby the functions of the image processing device can be realized.

In that case, the program code itself read from the recording medium realizes the functions of the image processing device, and the program code or the recording medium storing the program constitutes the present embodiments. As the recording medium for supplying the program code, an FD, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory, an optical recording medium such as a ROM, a magnetic recording medium, a magneto-optical recording medium, and a semiconductor recording medium may be used.

The program code read by the computer is executed, whereby the functions of the above image processing device are realized. As another case in which the functions of the image processing device is realized, an OS (operating system) operating on the computer performs a part of or whole actual processing based on an instruction of the program code, and the processing realizes the functions of the image processing device.

As still another case in which the functions of the image processing device is realized, the program code read from the recording medium is written in a memory of a function enhancement board inserted in the computer or a function enhancement unit connected to the computer; thereafter, a CPU of the function enhancement board or the function enhancement unit performs a part of or whole actual processing based on an instruction of the program code, and the processing realizes the functions of the image processing device.