Image processing apparatus that performs an alignment process on a plurality of images, a related control method, and a related storage medium that stores a control program

An image processing apparatus includes a memory configured to store instructions, and a processor in communication with the memory. The processor is configured to execute the instructions to set up a plurality of image sets according to a predetermined rule using a plurality of images obtained by continuous photography, to calculate a first conversion coefficient corresponding to a moving amount of an object between images in each of the plurality of image sets, to calculate a second conversion coefficient used for an alignment process about a correction target image other than a base image included in the plurality of images by multiplying a plurality of first conversion coefficients, and to generate an alignment image in which the object in the correction target image is aligned to the object in the base image by applying a conversion process to the correction target image using the second conversion coefficient.

This application claims the benefit of Japanese Patent Application No. 2016-116119, filed Jun. 10, 2016, which is hereby incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing apparatus, a related control method, and a related storage medium that stores a control program for an image processing apparatus, and, in particular, relates to an image retrieval apparatus that performs an alignment process on a plurality of images.

Description of the Related Art

In general, an image pickup apparatus, such as a digital camera, performs an alignment process for correcting a deviation between images that occurs due to a user's camera shake or a panning operation (for example, panoramic photographing or panning photographing). For example, there is a known image pickup apparatus that detects moving amounts of an object among a plurality of images during a panning operation, and generates a panoramic image by performing a composition process so as to align the position of the object according to the moving amounts (see WO 2011/162227).

Furthermore, there is a known technique for taking a plurality of images while focusing on objects of which distances from a camera differ (objects of which focusing distances differ) in order, and for aligning and compositing these images to generate a full focus image in which all the objects are focused.

Incidentally, when moving amounts of a background become large among a plurality of images, a subsequent alignment process is performed using an image after alignment as a reference image in the above-mentioned publication. Accordingly, when alignment accuracy is poor in a past frame (i.e., an image), deviation occurs in the reference image itself.

As a result, subsequent alignment accuracy will also be lowered. Furthermore, when a full focus image would be generated, it is necessary to align images of which focusing distances differ largely while the moving amounts of the background are small.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, a control method for controlling an image processing apparatus, and a storage medium storing a control program for controlling an image processing apparatus, each of which is capable of performing an alignment process with sufficient accuracy, even if a moving amount of an object is large among a plurality of images.

A first aspect of the present invention provides an image processing apparatus including a setting unit configured to set up a plurality of image sets according to a predetermined rule using a plurality of images obtained by continuous photography, a first calculation unit configured to calculate a first conversion coefficient corresponding to a moving amount of an object between images in each of the plurality of image sets, a second calculation unit configured to calculate a second conversion coefficient used for an alignment process about a correction target image other than a base image included in the plurality of images by multiplying a plurality of first conversion coefficients, and a generation unit configured to generate an alignment image in which the object in the correction target image is aligned to the object in the base image by applying a conversion process to the correction target image using the second conversion coefficient.

A second aspect of the present invention provides a control method for an image processing apparatus, the control method including a setting step of setting up a plurality of image sets according to a predetermined rule using a plurality of images obtained by continuous photography, a first calculation step of calculating a first conversion coefficient corresponding to a moving amount of an object between images in each of the plurality of image sets, a second calculation step of calculating a second conversion coefficient used for an alignment process about a correction target image other than a base image included in the plurality of images by multiplying a plurality of first conversion coefficients, and a generation step of generating an alignment image in which the object in the correction target image is aligned to the object in the base image by applying a conversion process to the correction target image using the second conversion coefficient.

A third aspect of the present invention provides a non-transitory computer-readable storage medium storing a control program causing a computer to execute the control method of the second aspect.

According to the present invention, the alignment process is performed with sufficient accuracy, even if a moving amount of an object is large among a plurality of images.

DESCRIPTION OF THE EMBODIMENTS

Hereafter, image processing apparatuses according to embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1is a block diagram schematically showing a configuration of an image pickup apparatus as an image processing apparatus according to a first embodiment of the present invention.

The illustrated image pickup apparatus is a digital camera (also simply referred to as a “camera”), for example, and has a control unit101, such as a central processing unit (CPU). The control unit101reads an operation program for the camera100from a read-only memory (ROM)102, develops it to a random access memory (RAM)103, and controls the camera100by running the operation program. The ROM102is a rewritable nonvolatile memory, for example, and stores parameters required for the operation of the camera100, etc., in addition to the operation program. The RAM103is a rewritable volatile memory, and is used as a temporary storage area storing data obtained in the operation of the camera100.

An optical system104forms an object image (optical image) on an image pickup unit105. The image pickup unit105is provided with an image pickup device, such as a couple-charged device (CCD) sensor, or a complementary metal-oxide semiconductor (CMOS) sensor. The image pickup unit105photoelectrically converts the optical image formed on the image pickup device, and outputs an analog image signal to an analog/digital (A/D) convertor106. The A/D convertor106stores image data obtained by applying an A/D conversion process to the analog image signal to the RAM103.

An image processing unit107applies predetermined image processing, such as white balance adjustment, color interpolation, and filtering, to the image data stored in the RAM103. A storage medium108is a memory card, etc., that is detachable from the camera100. The image data obtained through the image processing by the image processing unit107and the image data obtained through the A/D conversion process by the A/D convertor106are recorded on the storage medium108as recorded images. A display unit109is a display device, such as an LCD. An image corresponding to image data obtained by the A/D convertor106is displayed on the display unit109.

A motion detection unit110is a device, such as a gyro sensor, that detects a motion, and detects motions of the camera100in a yaw direction and a pitch direction on the basis of angle variations of the camera100per unit time (i.e., angular velocities). In the illustrated example, images are taken while panning or tilting the camera100. Then, positions of background parts (background objects) are aligned among a plurality of images as described herein.

FIG. 2A,FIG. 2B, andFIG. 2Care views for describing an alignment process performed by the camera shown inFIG. 1. That is,FIG. 2Ais a view showing a plurality of images obtained by the camera.FIG. 2Bis a view showing the images of which positions have been corrected according to object positions in order to composite the images shown inFIG. 2A. Moreover,FIG. 2Cis a view showing a composite image obtained by compositing the corrected images shown inFIG. 2B.

The images401through407shown inFIG. 2Aare obtained as a result of photography while panning the camera100. Then, these images402through407are corrected according to the object positions to become the post-correction images412through417as described herein. It should be noted that the position of the image411corresponding to the image401is not corrected. Moving amounts421through426due to the panning operation are found, and the positions of the images402through407are corrected according to the moving amounts421through426to obtain the post-correction images412through417(seeFIG. 2B). Then, the composite image (panoramic image) shown inFIG. 2Cis obtained using the post-correction images411through417.

FIG. 3is a flowchart showing an image processing process performed by the camera100shown inFIG. 1. It should be noted that the process concerning the illustrated flowchart is performed under the control of the control unit101. Moreover, the process concerning the illustrated flowchart is performed when the camera100obtains a second image. Furthermore, a user shall take a plurality of images while performing a panning operation in the description.

The control unit101controls the image pickup unit105to obtain a plurality of images with the image pickup unit105(step S201). In the step S201, the control unit101sends an image acquisition request to the image pickup unit105, and receives and obtains the image that the image pickup unit105sends in response to the request.

Subsequently, the control unit101sets up an image set of a pair of images as a group for measuring a positional relationship between an alignment target image (a correction target image) and a temporally last image (referred to as a frame) in step S202. An image set consists of two images, and a last image (past image) becomes a reference image for alignment. For example, when the image402shown inFIG. 2Ais the correction target image, the image401becomes the reference image. Similarly, the image402becomes the reference image in the image set consisting of the images402and403, and the image403becomes the reference image in the image set consisting of the images403and404. Then, the control unit101finds an alignment coefficient between the image sets (step S203).

FIG. 4is a view for describing the alignment coefficient calculated by the camera shown inFIG. 1.

A reference image501and a correction target image502are an image set obtained by the process in the step S202. The correction by the alignment does not aim at only a translation component corresponding to the camera motion by the panning operation shown inFIG. 2B. In addition to the translation component, there are a rotation component and a tilting component due to a camera shake. As a result, an image, like the image502, that was subject to the effects of rotation and tilt may be obtained. Accordingly, the control unit101finds a conversion coefficient for correcting a translation component, a rotation component, and a tilting component by geometric deformation. The conversion coefficient for performing this geometric deformation is called the alignment coefficient.

For example, as shown inFIG. 4, it is necessary to deform the correction target image503that was subject to the effects of the translation component, the rotation component, and the tilting component to the image504geometrically. When performing this geometric deformation, the alignment coefficient511is used. When the alignment coefficient511is denoted by a matrix A, the matrix A is denoted by the following formula (1):

When a coordinate in the correction target image shall be I (x, y) and a coordinate in the image after the geometric deformation shall be I′ (x′, y′), the image503is geometrically deformed to the image504by performing the geometric deformation according to the following formula (2):

Subsequently, the control unit101determines whether the alignment coefficient was found at least once in the past (step S204). When the alignment coefficient was found at least once in the past (YES in the step S204), the control unit101finds a product of the alignment coefficients that were found for image sets in the past and the current alignment coefficient found (step S205).

FIG. 5is a view for describing a process for finding the product of the alignment coefficients in the step S205inFIG. 3.

Images301,302, and303are obtained by panning the camera100. InFIG. 5, a horizontal axis shows horizontal positional relationship between images. The images301,302, and303are taken in order while moving the camera100rightward. An image set321includes the images301and302, and an image set322includes the images302and303.

In the image set321, the alignment coefficient for correcting the image302to the position of the image301(a post-correction image311corresponds to the image301) shall be A1. Similarly, in the image set322, the alignment coefficient for correcting the image303to the position of the image302(a post-correction image312corresponds to the image302) shall be A2. Since there is no image set before the image set321, the product of the alignment coefficients is not found about the image set321. Since there is the previous image set321about the image set322, the product of the alignment coefficient found about the previous image set321and the alignment coefficient of the current image set322is found according to the following formula (3):

The image303is corrected to the position with reference to the image301(a post-correction image313corresponds to the image301) by finding the product of the alignment coefficients according to the formula (3).

Next, the control unit101geometrically deforms the correction target image with the image processing unit107using the alignment coefficient found as described above (step S206). The geometric deformation is performed using the alignment coefficient found by the process up to the step S205according to the formula (2), described above. When no alignment coefficient has been found in the past (NO in the step S204), the control unit101proceeds with the process to the step S206without calculating a product.

Subsequently, the control unit101determines whether the alignment of images is finished (step S207). In the step S207, the control unit101determines the end of the alignment of images on the basis of whether the following image is obtained by the image pickup unit105. When the following image is not obtained (YES in the step S207), the control unit101finishes the image processing process. On the other hand, when the following image is obtained (NO in the step S207), the control unit101returns the process to the step S201.

FIG. 6A,FIG. 6B, andFIG. 6Care views for describing the process for calculating the alignment coefficient in the step S203inFIG. 3. That is,FIG. 6Ais a view showing feature points set in an image, andFIG. 6Bis a view showing a correction target image and a reference image. Moreover,FIG. 6Cis a view showing a group of vectors corresponding to the feature points.

In order to calculate the alignment coefficient, a reference image and a correction target image are set up as an image set. In the image set concerned, the reference image is a last frame of the correction target image in photography time. Then, vectors (a vector group) are calculated as feature amounts for respective areas in the two images of the image set by finding moving directions and moving amounts of feature points (or feature regions) in the respective areas. After that, a geometric conversion coefficient that indicates the positional relationship (for example, translation, rotation, reduction, and enlargement) between the two images based on the vectors concerned is found as the alignment coefficient.

As shown inFIG. 6A, the feature points611are uniformly set up in the image601. For example, blocks may be set up in the image at regular intervals as the feature points611. In such a case, when each block preferably includes an edge part and a texture containing frequency components from a low frequency to a high frequency, a vector is accurately found in the image set.

As shown inFIG. 6B, positions of feature points in the correction target image602corresponding to the feature points611in the reference image601are found. And then, the moving directions and the moving amounts of the feature points in the correction target image602with respect to the feature points611in the reference image601are calculated. For example, the feature amounts621,622,623,631,632, and633are vectors that indicate the directions and the moving amounts. Similarly, vectors are calculated also to the remaining feature points. It should be noted that a vector group641inFIG. 6Cis calculated for the predetermined feature points.

Generally, what is called a block matching method is used as a method of finding correlation between feature points of two images. The block matching method is able to detect a hierarchy vector by performing block matching about not only equal-magnification images but also gradually reduced images. The hierarchical detection of a vector enables reduction of the processing time required to detect a vector, even if a search area spreads.

Subsequently, the geometric conversion coefficient is found using the vector group found as mentioned above. For example, a coordinate (x′1, y′1, 1) shall indicate a feature point in a reference image, and a coordinate (x1, y1, 1) shall indicate a feature point in a correction target image. In this case, the vector X1 has information including the coordinate (x′1, y′1, 1) and the coordinate (x1, y1, 1) in a three-dimensional coordinate system. Then, a coordinate X is found by multiplying a predetermined conversion coefficient A to a coordinate X of a feature point in a correction target image according to formula (4), below. Then, an actual conversion coefficient A that minimizes the difference between the coordinate X′ concerned and the coordinate X of the feature point in the reference image is found.

A known optimization method, such as a Newton's method or a Gaussian Newton's method, is used for finding the conversion coefficient A. Then, this conversion coefficient A is used as the alignment coefficient.

Thus, since the conversion coefficient is found with respect to the last image, the alignment coefficient is found by images between which the deformation amount is relatively small even in a scene with a large panning operation of the camera100. This enables a high-precision alignment process.

Although the alignment method that uses a correction target image and a past image (last frame) in time series as an image set has been described in the embodiment, other images may be selected as an image set. For example, two images between which brightness variation, a moving amount, a change of a blur of an object image, or a change of defocus of an object image is small may be selected as an image set.

When a plurality of image sets is set up about a plurality of images, there is a method of setting an image set by selecting two images that are temporally close to each other from the plurality of images arranged in the order of image pickup, for example. Furthermore, there is a method of changing the order of the images according to the amount of change between images, and setting an image set by selecting two images that are close to each other in the changed order. This method corresponds to setting of an image set according to a predetermined rule.

FIG. 7AandFIG. 7Bare views for describing examples of selections of image sets.FIG. 7Ashows an example in which image sets are selected from images arranged in order of photography.FIG. 7Bshows an example in which image sets are selected from images arranged in ascending order of brightness variation and a moving amount with reference to a first image.

Moving amounts721through726from a head image (first image) for aligning the images with the same object are shown inFIG. 7A. Moreover, images701through707shown inFIG. 7AandFIG. 7Bare taken while a user pans the camera100. InFIG. 7A, the order of photography and the moving amounts of the object in the images have correlation, and the images are arranged so that a moving amount becomes greater as image pickup time becomes later. As mentioned above, the image sets741through746are set up in the order of photography.

On the other hand, when the brightness varies between frames, the accuracy of alignment may improve by selecting an image set in consideration of brightness and moving amount. For example, in the case of panoramic photography, an AE value is fixed to a value for a first image in order not to enlarge brightness variation for each image.

When clouds cover the sun suddenly or when the user brings the camera100to a shadow area from a sunny area during photography, a scene becomes dark for a moment in a series of images taken continuously. Then, the camera is set so as to be suitable for the scene concerned, and a series of images are taken. Since the brightness varies between frames in such a series of scenes, the alignment accuracy for every image set may deteriorate due to the brightness variation when image sets are set up in order of photography. Accordingly, in the example shown inFIG. 7B, an image set is set up so that the brightness variation becomes small with respect to the head image701.

InFIG. 7B, the order of the images702,703, and704is changed so that the brightness variation becomes small with respect to the head image701. In the description, brightness of an image is indicated within a range of “0” through “10”.

FIG. 8is a flowchart showing a process at a time of rearranging images shown inFIG. 7B. In the description, the first image in the order of photography is used as a base image for alignment.

The control unit101detects the brightness of the second image (step S1701). Subsequently, the control unit101detects an overlapped area between the base image (the head image in this example) and the second image (step S1702). Then, the control unit101determines whether there is an overlapped area between the base image and the second image (step S1703).

When there is an overlapped area (YES in the step S1703), the control unit101finds luminance variation (brightness variation) between the base image and the second image concerned to rearrange the images (step S1704). After that, the control unit101determines whether the rearrangements for all the images have been completed (step S1705). When the rearrangements for all the images have been completed (YES in the step S1705), the control unit101finishes the rearrangement process. On the other hand, when the rearrangements for not all the images have been completed (NO in the step S1705), the control unit101returns the process to the step S1701and detects brightness of the following image. When there is no overlapped area (NO in the step S1703), the control unit101proceeds with the process to the step S1705without rearranging images.

As shown inFIG. 7B, since the brightness of the image702is “5” and the brightness of the head image701is “10”, the brightness variation is large. Accordingly, in the example inFIG. 7B, the image702with the brightness “5” is rearranged to the fourth position and the image704with the brightness “8” is rearranged to the second position.

On the other hand, although the brightness variations of the images705,706, and707with respect to the head image701are small, they do not have an overlapped area including the same object. Accordingly, since the alignment coefficient is incomputable, the images705,706, and707are not rearranged.

It should be noted that the moving amount may be used for determining whether there is an overlapped area. That is, when the moving amount is greater than a predetermined amount, it may be determined that there is no overlapped area.

Furthermore, when the overlapped area between the last image and the base image is narrower than the overlapped area between the following image and the base image, and when the difference between both the overlapped areas is greater than a predetermined threshold, it may be determined that there is no overlapped area between the following image and the base image.

Thus, the image sets751through756are obtained by rearranging the images using the brightness variations and the moving amounts of the images. Similarly, a blur or defocus of an image may be used as a determination factor instead of brightness when an image set is determined. In such a case, images between which difference in a blur or a defocus is small are set up as an image set.

Moreover, although the head image is used as the base image for alignment in the description, the base image may be changed according to an alignment coefficient found between image sets (i.e., according to a predetermined condition).

FIG. 9AthroughFIG. 9Eare views for describing change of the base image for alignment in the camera100shown inFIG. 1.FIG. 9Ais a view showing an example in a case of aligning other images with a first image as the base image.FIG. 9Bis a view showing an example in a case of aligning other images with a second image as the base image. Moreover,FIG. 9Cis a view showing an example in a case of aligning other images with a third image as the base image.FIG. 9Dis a view showing an example in a case of aligning other images with a fourth image as the base image. Furthermore,FIG. 9Eis a view showing an example of change of the base image.

InFIG. 9A, numbers “1” through “4” show the order of photography of images801through804. Moreover, an arrow in the drawings shows a direction of alignment of an image. Then, a tip of an arrow shows a base image for alignment and a root of an arrow shows an alignment target image. Furthermore, the alignment coefficients corresponding to the arrows are shown by A1, A2, and A3(inverse matrices are A1−1, A2−1, and A3−1).

In the example shown inFIG. 9A, the images802,803, and804are aligned to the first image801as the base image. Transformation matrices (alignment coefficients) A801through A804for aligning the images801through804are shown by the following formulas (5):
A801=1
A802=A1
A803=A1A2
A804=A1A2A3(5).

In the formulas (5), the alignment coefficient A801of the image801is equal to “1”. Since the image801is the base image for alignment, it is not converted. The alignment coefficients concerning the other images are found by multiplying the alignment coefficients of the image sets.

InFIG. 9B, images811,813, and814are aligned to a second image812as the base image for alignment after finding the alignment coefficients A1, A2, and A3of the image sets. Transformation matrices A811through A814of the images811through814are shown by the following formulas (6):
A811=A1−1
A812=1
A813=A2
A814=A2A3(6).

Since the image811is aligned to the second image812, the alignment process is needed. Since the alignment coefficient A1in the image set is obtained assuming that the image812will be converted into the image811, the actual alignment coefficient becomes an inverse matrix A1−1.

InFIG. 9C, images821,822, and824are aligned to a third image823as the base image for alignment after finding the alignment coefficients A1, A2, and A3of the image sets. Transformation matrices A821through A824of the images821through824are shown by the following formulas (7):
A821=A2−1A1−1
A822=A2
A823=1
A824=A3(7).

Since the images821and822are aligned to the image823, it is necessary to find inverse matrices of the alignment coefficients A1and A3. Moreover, the alignment coefficient A821of the image821is found by multiplying the alignment coefficient A1−1for aligning the image821to the image822by the alignment coefficient A2−1for aligning the image822to the image823. The relationship between the coordinates I821through I823of the images821through823is shown by the following formula (8):
I823=A2−1I822=A2−1A1−1I821(8)

InFIG. 9D, images831,832, and833are aligned to a fourth image824as the base image for alignment after finding the alignment coefficients A1, A2, and A3of the image sets. Transformation matrices A831through A834of the images831through834are shown by the following formulas (9):
A831=A3−11A2−1A1−1
A832=A3−1A2−1
A833=A3−1
A834=1  (9).

For example, when a first image shifts in a horizontal direction or a vertical direction due to inclination of the camera100during photography of the first image, or when a first image is affected by defocus or a camera shake, a base image is selected by changing the reference of alignment (the predetermined condition). Accordingly, even if the photography of the first image fails, a satisfactory image will be used as the base image for alignment. Moreover, a user's favorite image can be used as the base image for alignment.

In the foregoing description, the example for setting up a last image as the reference image used for calculating the alignment coefficient of an image set was described. On the other hand, when a feature amount, such as a moving amount, or a change of brightness, blur, or defocus, between images is less than a predetermined threshold, one or more last images may be set up as a reference image.

InFIG. 9E, a head image841is set up as a base image for alignment. In an image set of images841and842, the head image841is set up as a reference image. Next, when an image set of an image843is set up, the feature amount that is a difference between the head image841and the image843is found. For example, at least one of a moving amount, and changes of brightness, blur, and defocus, between the images is used as the feature amount concerned. Then, the feature amount concerned is compared with the predetermined threshold.

When the feature amount is less than the threshold, the images841and843are set as an image set, and the image841becomes the reference image. On the other hand, when the feature amount of an image844to the base images841is equal to or greater than the threshold, the image844forms an image set with the image843as the reference image. Transformation matrices A841through A844of the images841through844are shown by the following formulas (10):
A841=1
A842=A1
A843=A′2
A844=A′2A3(10).

Thus, the first embodiment of the present invention enables the alignment of images with sufficient accuracy, even if a moving amount between the images becomes large in continuous photography. Furthermore, even if change between images becomes large because of a factor other than the moving amount, the images are aligned easily.

Subsequently, a camera900according to a second embodiment of the present invention will be described.

FIG. 10is a block diagram schematically showing a configuration of the camera900according to the second embodiment of the present invention. It should be noted that the same reference numerals are assigned to components inFIG. 10that are the same as the components of the camera100shown inFIG. 1.

The illustrated camera900performs panoramic composition using the alignment process described in the first embodiment. Accordingly, since a process in an image processing unit differs from the process in the image processing unit107of the camera100shown inFIG. 1, a reference numeral907is assigned to the image processing unit inFIG. 10.

FIG. 11is a flowchart showing an image processing process performed by the camera900shown inFIG. 10. Since a process in steps S1001through S1006inFIG. 11is identical to the process in the steps S201through S206inFIG. 3, a description of steps S1001to S1006is omitted.

After the process in the step S1006, the control unit101expands a field angle by performing the composition process near a boundary between images after the geometric deformation with the image processing unit907(step S1007). Then, the control unit101determines whether the panoramic composition is terminated (step S1008). When determining that the panoramic composition is not terminated (NO in the step S1008), the control unit101returns the process to the step S1001. On the other hand, when determining that the panoramic composition is terminated (YES in the step S1008), the control unit101finishes the image process.

When the process of the flowchart shown inFIG. 11is performed, the images401through407shown inFIG. 2Aform a panoramic image of which the field angle is expanded, as shown inFIG. 2C.

FIG. 12is a view for describing the composition process (the panoramic composition process) in the step S1007inFIG. 11.

When the alignment by the geometric deformation described in the step S1006, shown inFIG. 11, is performed, images1101,1102, and1103are obtained. Then, when the boundaries between the images1101,1102, and1103are composited in order, a panoramic composite image is obtained.

The images1101and1102are composited at a center line1121of the image1101in the horizontal direction as the boundary. For example, the image1101is allocated to the left area of the line1121, and the image1102is allocated to the right area of the line1121. On the line1121, a process of mixing pixel data of the images1101and1102is performed in order to make a joint natural. For example, the pixel data of the images1101and1102are composited at 1:1 on the line1121. Then, the ratio of the image1101on the left side of the line1121increases with distance from the line1121. On the other hand, the ratio of the image1102is enlarged on the right side of the line1121. As a result, a composite image1111is obtained.

Subsequently, the composite image1111and the image1103are composited. In this case, they are composited at a center line1122of the last image1102in the horizontal direction. As a result, a composite image1112is obtained.

Thus, a panoramic composite image is generated by compositing images at boundaries, in order, after alignment. Then, when the images1102and1103are panoramically composited to the image1101, an area1131is added to the image1101that expands the field angle.

As mentioned above, in the second embodiment of the present invention, the panoramic composite image is generated by connecting the images to which the alignment process is performed to the images that are continuously taken during a moving operation of the camera900, such as a panning operation.

Subsequently, one example of a camera1200according to a third embodiment of the present invention will be described.

FIG. 13is a block diagram schematically showing a configuration of the camera1200according to the third embodiment of the present invention. It should be noted that the same reference numerals are assigned to components inFIG. 13that are the same as the components of the camera100shown inFIG. 1.

The illustrated camera1200generates a full focus image that expands a depth of field by compositing a plurality of images that are taken while focusing on a plurality of objects of which distances from the camera1200are different (objects of which focusing distances differ) in order with using the alignment process described in the first embodiment. Accordingly, since a process in an image processing unit differs from the process in the image processing unit107of the camera100shown inFIG. 1, a reference numeral1207is assigned to the image processing unit inFIG. 13.

FIG. 14is a flowchart showing an image processing process performed by the camera1200shown inFIG. 13.

The control unit101photographs the objects while changing a distance to an object to be focused from the optical system104with the image pickup unit1205, and obtains a plurality of images (step S1301).

FIG. 15AandFIG. 15Bare views showing images obtained by the camera1200shown inFIG. 13while changing a focusing distance.FIG. 15Cis a view showing a full focus image that is obtained by compositing the images inFIG. 15AandFIG. 15B.

FIG. 15Ashows an image that is taken while focusing on a front person as a target object. Accordingly, since the focusing distance to a back plant, as a background object, is deviated, the background object concerned is out of focus. On the other hand,FIG. 15Bshows an image that is taken while focusing on the back plant as the background object. Accordingly, the person that is the target object is out of focus.

Referring back toFIG. 14, the control unit101performs a process in steps S1302through S1306about an image of which the focusing distance differs. The process in the steps S1302through S1306is the same as the process in the steps S202through S206that was described inFIG. 3.

It should be noted that an image of which a defocus amount or a camera shake is small may be used as a base image for alignment instead of an image obtained by first photography when a product of alignment coefficients is found in the step S1305. Moreover, an image that focuses on a target object may be used as a base image.

There is a known method that estimates a Point Spread Function (PSF) that is a defocus function as a method of detecting a defocus amount or a camera shake (see Japanese Laid-Open Patent Publication (Kokai) No. 2014-219725 (JP 2014-219725A)). In the estimate of the PSF, a luminance value that is obtained by differentiating image data before performing a correction process is compared with a threshold. Then, an area where the luminance value is less than the threshold and where a sign of a pre-correction luminance value of image data is reverse to a sign of a luminance value of post-correction image data is extracted. Furthermore, when a reverse edge component of the extracted area is obtained, the PSF that indicates a defocus state (amount) is obtained.

After the process in the step S1306, the control unit101composites the area focused on the main object and the area focused on the background object in the post-alignment images with the image processing unit1207(step S1307). As a result, an image of which a depth of field is expanded about the composite area is obtained.

For example, a contrast composition is employed in the process in the step S1307. Details of the contrast composition will be described later.

Subsequently, the control unit101determines whether a series of the full-focus-image compositions are terminated (step S1308). When determining that the full-focus-image compositions are not terminated (NO in the step S1308), the control unit101returns the process to the step S1301. On the other hand, when determining that the full-focus-image compositions are terminated (YES in the step S1008), the control unit101finishes the image process.

When the images shown inFIG. 15AandFIG. 15Bare composited by performing the process described inFIG. 14, the full focus image that focuses on both the target object and the background object is generated, as shown inFIG. 15C. It should be noted that the number of images that are composited is not limited to two. A full focus image may be generated by compositing three or more images that were taken while changing the focusing distance by performing the contrast composition.

The contrast composition noted above will be described below. Images that are subjected to the contrast composition (also referred to as composition target images) are post-alignment images. Moreover, when the contrast composition was performed in the last frame, the composite image of the last frame becomes a reference image of the composition (also referred to as a composition reference image). Then, when the contrast composition was not performed, the last frame becomes the composition reference image.

In the contrast composition, the focusing degrees about the composition reference image and the composition target image are found. A high frequency component is obtained from a pixel area around a target pixel, and the absolute value of the high frequency component concerned becomes the focusing degree.

As mentioned above, the focusing degrees are respectively found about the composition reference image and the composition target image (referred to as a reference-image focusing degree and a target-image focusing degree). Then, a focusing degree difference Δ between the reference-image focusing degree and the target-image focusing degree is found (Δ=“target-image focusing degree”−“reference-image focusing degree”). After that, weights of the composition reference image and the composition target image are adjusted according to the focusing degree difference Δ, and then the composition process is performed.

FIG. 16Ais a view showing the composition reference image obtained with the camera shown inFIG. 13, andFIG. 16Bis a view showing the composition target image. Moreover,FIG. 16Cis a view showing the full focus image that is obtained by compositing the images inFIG. 16AandFIG. 16B.

Areas1601and1603inFIG. 16AandFIG. 16Bare front person areas in the composition reference image and the composition target image, respectively. Since the person area is focused in the composition reference image shown inFIG. 16A, the greater frequency component is detected in the person area1601of the composition reference image as compared with the person area1603of the composition target image shown inFIG. 16B. Then, the focusing degree in the area1601of the composition reference image is greater than the focusing degree in the area1603of the composition target image. For example, assuming that the reference-image focusing degree is “10” and the target-image focusing degree is “5”, the focusing degree difference Δ will become “−5”.

FIG. 17is a graph showing an example of a composition ratio of the composite target image used when a full focus image is generated in the camera1200shown inFIG. 13.

InFIG. 17, a horizontal axis indicates the focusing degree difference Δ, and a vertical axis indicates the composition ratio w of the composition target image. The composition ratio w corresponding to the focusing degree difference Δ in the areas1601and1603, shown inFIG. 16AandFIG. 16B, respectively, is “0” as shown by a reference numeral1511, and the signal values of the pixels included in the area1603of the composition target image are not used for composition. In this case, an area1605of the full focus image shown inFIG. 16Cconsists of only signal values of the pixels included in the area1601of the composition reference image.

Thus, the smaller the composition ratio w is, the lower the focusing degree of the composition target image to the composition reference image is. When the contrast composition is performed with using the composition ratio w, the following formula (11) is used:
Bc=wBt+(1−w)Bs(11).

It should be noted that Bc, Bt, and Bs indicate signal values of pixels or a pixel area in the contrast-composition image, a composition target image, and a composition reference image, respectively.

Areas1602and1604shown inFIG. 16AandFIG. 16Bare background plant areas in the composition reference image and the composition target image, respectively. Then, the focusing degree in the area1604of the composition target image is greater than the focusing degree in the area1602of the composition reference image. For example, assuming that the reference-image focusing degree is “5” and the target-image focusing degree is “10”, the focusing degree difference Δ will become “5”.

The composition ratio win this case is “1” (equivalent to 100%) as shown by a reference numeral1512. The composition process is performed so that the ratio of the signal values in the area1604of the composition target image becomes 100%. That is, an area1606inFIG. 16Cconsists of only signal values of the pixels included in the area1604.

Furthermore, the composition reference image and the composition target image is composited by weighting both the images in a section between a lower limit threshold1501and an upper limit threshold1502of the focusing degree difference Δ. The lower limit threshold and the upper limit threshold are set up on the basis of experimental values, for example. It should be noted that the lower limit threshold and the upper limit threshold may be changed according to object distance or settings of the camera1200.

As mentioned above, in the third embodiment of the present invention, a full focus image is obtained by performing the contrast composition after aligning images that are obtained as a result of photography while changing the focusing distance.

As is clear from the above description, the control unit101and the image processing unit107function as a first calculation unit, a second calculation unit, a processing unit, and a composition unit in the example shown inFIG. 1.

Other Embodiments

For example, the functions of the above-mentioned embodiments may be achieved as a control method that is executed by the image processing apparatus. Moreover, the functions of the above-mentioned embodiments may be achieved as a control program that is executed by a computer with which the image processing apparatus is provided.