Patent Description:
The image stitching or panoramic image technology refers to splicing a series of images with overlapping areas into one image with a wide viewing angle, closing to the original image, with little distortion and no obvious stitching lines, in order to meet the demand of people for browsing the panoramic image.

At present, there are already many image fusion algorithms by which the overlapping parts in the same scene image can be extracted and images can be quickly stitched into one panoramic image, and a satisfactory fusion result is obtained. One method is to use wavelet to perform multi-scale fusion. But this method only performs local fusion processing in the overlapping areas, and cannot guarantee the global naturalness of the image. Another method is to use the gradient domain fusion method to stitch the images, which can reduce the influence of the color difference between the source images on the fusion result. However, for images with obvious differences in color and brightness, the above two methods may result in distortion in the merged panoramic image.

The document "<NPL>" discloses a fast stitching approach for combining a set of source images into a panoramic image using little memory, and implemented on mobile phones. In this approach, color correction reduces color differences of source images and balances colors and luminance in the whole image sequence, dynamic programming finds optional seams in overlapping areas between adjacent images and merges them together, and image blending further smoothens color transitions and hides visible seams and stitching artifacts. A sequential panorama stitching procedure constructs panoramic images. The advantages include fast processing speed using dynamic programming for optimal seam finding, reducing memory needs by using the sequential panorama stitching and improved quality of image labeling and blending due to the use of color correction. The approach has been tested with different image sequences and it works well on both indoor and outdoor scenes.

The document "<NPL>" discloses several formal cost functions for the evaluation of the quality of stitching. In these cost functions, the similarity to the input images and the visibility of the seam are defined in the gradient domain, minimizing the disturbing edges along the seam. A good image stitching will optimize these cost functions, overcoming both photometric inconsistencies and geometric misalignments between the stitched images.

The document "<NPL>; [<NPL>" discloses a method for dealing with objects that move between different views of a dynamic scene. If such moving objects are left in, they will appear blurry and 'ghosted', treating such regions as nodes in a graph, a vertex cover algorithm is utilized to selectively remove all but one instance of each object. The document further discloses a method for continuously adjusting exposure across multiple images in order to eliminate visible shifts in brightness or hue. Exposure corrections are computed on a block-on-block basis, then the parameters are smoothly interpolated using a spline to get spatially continuous exposure adjustment.

The present application provides an image method, a computer-readable storage medium and a portable terminal, which aims to eliminate the chromatic aberration of stitched image and achieve a good image fusion effect.

In the first aspect of the present application, a computer implemented method for image fusion is provided, which includes:.

acquiring a plurality of aligned images;
calculating gradient information of each image respectively;
setting a mask image of each image and generating a target gradient image;
performing a gradient operation on the target gradient image to obtain a target Laplacian image;
performing a deconvolution transform on the target Laplacian image to generate a fused panoramic image;
the plurality of aligned images are capable of being stitched into one panoramic image, and a number of the aligned images is equal to n, with n≥<NUM>;
the aligned images represent that any pixel in the n aligned images corresponds to a pixel in the panoramic image obtained by stitching;
for each pixel of the panoramic image, only one mask image has a corresponding pixel value of <NUM>;
the target Laplacian image is obtained by formulas (<NUM>) and (<NUM>): <MAT> <MAT>
where N is a total number of images, Ii is the i-th original image, Mi is a mask image of the i-th image, <MAT> <MAT> <MAT> <MAT>
where (u, v) is a coordinate of the pixel, I is the pixel value, g is the gradient information of the pixel, Width is a width of an image, and Height is a height of an image;
the deconvolution transformation is performed with a template:.

In the second aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program, when executed by a processor, implements the steps of the image fusion method in the first aspect of the present application.

In the third aspect of the present application, a portable terminal is provided, which includes:.

In the present application, the Laplacian image is generated by calculating the gradient information of a number of images, and then the deconvolution transform is performed to generate the fused panoramic image, which eliminates the color difference in the stitched images and achieves a good fusion effect of the images.

In order to make the objectives, technical solution, and advantages of the present application clearer, the present application will be described in detail with reference to the accompanying drawings and embodiments. It should be appreciated that the specific embodiments described here are only used for explaining the present application, rather than limiting the present application.

In order to illustrate the technical solution of the present application, specific embodiments are used for description below.

Referring to <FIG>, an image fusion method provided by the embodiment I of the present application includes the following steps.

S101: a number of aligned images are acquired.

A number of aligned images may be images taken by an ordinary camera from multiple angles of view, or video frames of a video;
There are n images, and n≥<NUM>. The n images can be stitched into one panoramic image, and the images are aligned to indicate that in the n images, any pixel has a corresponding pixel in the stitched panoramic image.

The stitching can be implemented by camera external parameter calibration, feature point matching or optical flow alignment methods.

S102: gradient information of each image is calculated respectively.

Each image is down-sampled sequentially to obtain a corresponding down-sampled image; the down-sampling is <NUM>/m times of the original image, with m≥<NUM>.

Horizontal forward gradient information, vertical forward gradient information, horizontal backward gradient information and vertical backward gradient information of each down-sampled image are calculated respectively.

Calculation formulas for the gradient information of pixel coordinate (u, v) are as follows.

As for the horizontal forward gradient information gx, a calculation formula is: <MAT>.

As for the vertical forward gradient information gy, a calculation formula is:
<MAT>.

As for the horizontal backward gradient information gxb, a calculation formula is: <MAT>.

As for the vertical backward gradient information gyb, a calculation formula is: <MAT>.

(u, v) is the coordinate of a pixel, I is a value of the pixel, g is gradient information of the pixel, Width is a width of the image, and Height is a height of the image.

S103: a mask image of each image is set, and a target gradient image is generated.

The mask image of each image is set according to an area of effective information of the image in the panoramic image; a specific mask image can be selected according to a specific stitching scheme. For the stitching scheme with fixed stitching seams, for example, a video sequence shot by a fixed camera, a fixed mask can be selected due to the same the stitching method. For a scheme of dynamically adjusting the stitching seams, such as using images taken by different cameras to perform the stitching and fusion, the stitching method may change, accordingly a corresponding mask can be calculated respectively, which is referred to as a dynamic mask.

A new four channels image with the same length and width as the panoramic image is created. The four channels are respectively configured to store the horizontal forward gradient information, the vertical forward gradient information, the horizontal backward gradient information, and the vertical backward gradient information.

The gradient information of the four channels of an area covered by a mask image of each image is copied to the newly created four channels image to generate the target gradient image.

S104: a gradient operation is performed on the target gradient image to obtain a target Laplacian image.

The Laplacian image is calculated according to the four channels gradient information of the target gradient image, and a calculation formula is:
<MAT>.

For the Laplacian image, a small component of a brightness value of the original image is added, and a calculation formula is:.

<MAT>
where, N is a total number of images, Ii is the i-th original image, Mi is a mask image of the i-th image, and the pixel value is <NUM> when the pixel is effective pixel, otherwise the pixel value is <NUM>; Mi ensures that for each pixel (u, v), only one of N images has the pixel value of <NUM>, and the rest have the pixel value of <NUM>.

S105: a deconvolution transformation is performed on the Laplacian image to obtain a fused image.

The Laplacian image L' can be regarded as generated by convolving the panoramic image with the following template, the template is:.

Therefore, the above template is utilized to perform the Fourier transform on the Laplacian image L'. According to the convolution theorem, this step can be completed in the frequency domain. A Fourier spectrum of L' is divided by a Fourier spectrum of the above template, and then an inverse Fourier transform is performed thereon; a ratio graph is obtained according to a ratio difference between the original image and the fused image; then the ratio graph is up-sampled by m times, and then multiplied by image points of multiple corresponding original images to obtain a number of aligned images; then a number of aligned images are fused with corresponding mask images together to obtain a final panoramic image.

The deconvolution transform is performed on the Laplacian image L' to obtain the fused image, which can be converted in the frequency domain of the image. Because the above two Fourier spectra are both plural, in order to reduce the calculation and storage, discrete cosine transform is utilized to simplify the calculation.

Referring to <FIG>, the specific steps of deconvolution transform of the images include following steps.

S1051: discrete cosine transform is performed on the Laplacian image L' to obtain DL'.

S1052: a width Width and a height Height of the Laplacian image L' are acquired, an image is created and is denoted as T, a width and a height of the image are respectively 2Width and 2Height, a value of (-<NUM>+ε) is assigned to a center position (Width, Height) of the image, a value of <NUM> is assigned to positions (Width+<NUM>, Height), (Width-<NUM>, Height), (Width, Height+<NUM>), (Width, Height-<NUM>) respectively, and a value of <NUM> is assigned to other positions.

S1053: discrete Fourier transform is performed on T to obtain an image T<NUM>.

S1054: an image of a real part of an upper left quarter of T<NUM> (<NUM>≤u<Width, <NUM>≤v<Height) is acquired and is denoted as T<NUM>.

S1055: T<NUM>(u,v) is changed to -T<NUM>(u,v) at a position where u+v is an odd number to obtain T<NUM>'(u,v), as shown in a following formula (<NUM>):
<MAT>.

S1056: DL' is divided by T<NUM>' to obtain DL, as shown in a following formula (<NUM>):
<MAT>.

S1057: inverse discrete cosine transform is performed on DL to obtain the fused image resL at a low resolution.

S1058: a relationship between the resL image and multiple down-sampled images is utilized to obtain multiple corresponding ratio images; the ratio images are up-sampled by m times, and then have a point multiplication operation with inputted original images, and then fused through corresponding mask images to obtain the panoramic image which is clear and has no color difference and brightness difference.

It should be noted that for an image with less details, because the image is relatively smooth, the discrete Fourier transform data is concentrated in the middle low-frequency signal area, and the discrete cosine transform data is concentrated in the upper left corner. It is almost impossible to see the advantages of the discrete cosine transform. However, for an image with rich details, the data after the discrete Fourier transform is very divergent, and the data after the discrete cosine transform is still relatively concentrated. If the original image is also restored from the frequency spectrum, it is more reasonable to choose the discrete cosine transform, because the discrete cosine transform only needs to store fewer data points, the calculation and storage capacity can be reduced.

The second embodiment of the present application provides a computer-readable storage medium which stores a computer program, and the computer program, when executed by a processor, performs the steps of the image fusion method as provided in the embodiment I of the present application. The computer-readable storage medium can be a non-transitory computer-readable storage medium.

<FIG> shows a specific structure block diagram of a portable terminal provided in embodiment III of the present application. The portable terminal <NUM> includes: one or more processors <NUM>, a memory <NUM>, and one or more computer programs; the processor <NUM> is connected to the memory <NUM> by a bus; the one or more computer programs are stored in the memory <NUM>, and are configured to be executed by the one or more processors <NUM>; and the processor <NUM>, when executing the computer programs, implements the steps of the image fusion method provided in the embodiment I of the present application.

Claim 1:
A computer implemented method for image fusion, comprising:
acquiring (S101) a plurality of aligned images; characterized by
calculating (S102) gradient information of each image respectively;
setting (S103) a mask image of each image and generating a target gradient image;
performing (S104) a gradient operation on the target gradient image to obtain a target Laplacian image;
performing (S105) a deconvolution transform on the target Laplacian image to generate a fused panoramic image;
wherein,
the plurality of aligned images are capable of being stitched into one panoramic image, and a number of the aligned images is equal to n, with n≥<NUM>;
the aligned images represent that any pixel in the n aligned images corresponds to a pixel in the panoramic image obtained by stitching;
for each pixel of the panoramic image, only one mask image has a corresponding pixel value of <NUM>;
wherein the target Laplacian image is obtained by formulas (<NUM>) and (<NUM>): <MAT> <MAT>
wherein, N is a total number of images, Ii is the i-th original image, Mi is a mask image of the i-th image, <MAT> <MAT> <MAT> <MAT>
wherein, (u, v) is a coordinate of the pixel, I is the pixel value, g is the gradient information of the pixel, Width is a width of an image, and Height is a height of an image;
wherein the deconvolution transformation is performed with a template:

<TAB>