Patent Publication Number: US-2016247285-A1

Title: Image processing device and image depth processing method

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The instant disclosure relates to an image processing device and an image processing method; in particular, to an image processing device and an image processing method than can generate a simulation image having a depth of field according to a depth image and a reference image. 
     2. Description of Related Art 
     As technology develops, the volume of the smart phone or the digital camera becomes smaller and smaller, which makes them portable so the user can take photos any time. Generally speaking, the aperture of the smart phone and the digital camera is smaller, so the smart phone and the digital camera can generate a clear photo no matter whether the captured scene is far or near; however, it also makes the smart phone and the digital camera unable to generate photographic images having depth of field to emphasize a certain object. 
     On the other hand, the digital single lens reflex camera (DSLR) has a bigger aperture, so it can blur the image of an element not within the focal area and make another specific element within the focal area clear. However, the digital single lens reflex camera having a bigger aperture has big volume, is costly and not as portable. 
     Therefore, in recent years, many image depth processing methods have been applied to the smart phone and the digital camera, so as to blur part of the photographic image taken by the smart phone and the digital camera, in order to emphasize a certain element in the photographic image. However, the physics principles of geographic optics are not considered in the traditional image depth processing method, so it is hard to estimate the blurring degree of the element within and not within the focal area, which makes the blurred photographic images look not natural or not continuous. 
     SUMMARY OF THE INVENTION 
     The instant disclosure provides an image processing device and an image depth processing method based on geometric optics. The image processing device executes the image depth processing method, so that it can generate a simulation image having a depth of field that looks natural and continuous according to a reference image and a depth image corresponding to the reference image. 
     The instant disclosure also provides an image depth processing method. The image depth processing method comprises: each time obtaining a background image and a foreground image from a reference image according to an order indicating a depth of field shown in a depth image (from far to near), wherein the depth image corresponds to the reference image and a depth value of the background image is larger than a depth value of the foreground image; blurring the foreground image and a local image of the blurred background image that is near the margin of the foreground image; and after blurring the foreground image and the local image, forming a simulation image according to the foreground image and the background image. 
     In one of the embodiments of the instant disclosure, after blurring the foreground image and the local image and before forming the simulation image, the image depth processing method further comprises: executing a fading process so as to fade the margin image of the background image. 
     The instant disclosure also provides an image processing device, and the image processing device comprises a memory module and a processing module. The processing module is coupled to the memory module. The processing module is configured to execute the above mentioned image depth processing method. The memory module is configured to store the reference image and the depth image. 
     The instant disclosure further provides an image processing device, and the image processing device comprises a memory module and a processing module. The image capturing module is configured to capture an image of a scene so as to generate a plurality of photographic images. The processing module is coupled to the image capturing module, to generate a reference image according to the photographic images and to generate a depth image corresponding to the reference image. The processing module each time obtains a background image and a foreground image from a reference image according to an order indicating a depth of field shown in a depth image (from far to near), wherein a depth value of the background image is larger than a depth value of the foreground image. The processing module blurs the background image according to the depth value of the background image, and blurs the foreground image and a local image of the blurred background image that is near the margin of the foreground image according to the depth value of the foreground image. After the processing module blurs the foreground image and the local image, the processing module forms a simulation image according to the foreground image and the background image. 
     To sum up, via the image processing device and the image processing method provided by the embodiment of the instant disclosure, after the image processing device blurs the background image according to the depth value of the background image, the image processing device blurs the foreground image and a local image of the background image according to the depth value of the foreground image, such that the connection between the foreground image and the background image can be explained by the geometrical optics, which makes an image look natural and continuous. In addition, the image processing device can further execute a fading process, so as to fade the margin image of the background image and thereby to generate a simulation image having a depth of field and a large aperture value. 
     For further understanding of the instant disclosure, reference is made to the following detailed description illustrating the embodiments and examples of the instant disclosure. The description is only for illustrating the instant disclosure, not for limiting the scope of the claim. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which: 
         FIG. 1  shows a block diagram of an image processing device of one embodiment of the instant disclosure; 
         FIG. 2  shows a flow chart of an image depth processing method of one embodiment of the instant disclosure; 
         FIG. 3  shows a reference image stored in a memory module of the image processing device; 
         FIG. 4  shows a schematic diagram of a background image obtained according to the reference image shown in  FIG. 3 ; 
         FIG. 5  shows a schematic diagram of another background image obtained according to the reference image shown in  FIG. 3 ; 
         FIG. 6  shows a schematic diagram of still another background image obtained according to the reference image shown in  FIG. 3 ; and 
         FIG. 7  shows a schematic diagram of a simulation image generated according to the reference image shown in  FIG. 3 . 
     
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     The aforementioned illustrations and following detailed descriptions are exemplary for the purpose of further explaining the scope of the instant disclosure. Other objectives and advantages related to the instant disclosure will be illustrated in the subsequent descriptions and appended drawings. In the drawings, the size and relative sizes of layers and regions may be exaggerated for clarity. 
     It will be understood that, although the terms first, second, third, and the like, may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are only to distinguish one element, component, region, layer or section from another region, layer or section discussed below and could be termed a second element, component, region, layer or section without departing from the teachings of the instant disclosure. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. 
     Please refer to  FIG. 1 .  FIG. 1  shows a block diagram of an image processing device of one embodiment of the instant disclosure. As shown in  FIG. 1 , an image processing device  1  comprises a memory module  11 , a processing module  12  and a display module  13 , and the processing module  12  is connected to the memory module  11  and the display module  13 . In this embodiment, the image processing device  1  is a smart phone, a laptop, a personal computer, a tablet computer, a digital camera, a digital photo frame or other electric device capable of calculating and displaying, and it is not limited therein. 
     The memory module  11  is a storage medium, storing a reference image, a depth image (that is a gray scale image presenting the image with a gray scale range 0˜255, wherein the brighter the color of the image is (the greater the gray scale is) means the nearer the position is, and vice versa) and at least an aperture simulation parameter. The depth image corresponds to the reference image. The memory module  11  is, for example, an embedded temporal memory, a physical memory or an external storage device (such as an external memory card). In one embodiment, the reference image is an image of the full depth of field having a clear foreground image and a clear background image, but it is not limited thereto. In other words, the reference image can also be an image which is partly clear. The method for generating the depth image can be realized via laser distance measurement, binocular vision, structured light or the optical field effect, which is well known by those skilled in the art and not repeated herein. The aperture simulation parameter is, for example, the shape of aperture, the size of aperture or the focal length. 
     The processing module  12  obtains the reference image, the depth image and the aperture simulation parameters from the memory module  11 . The processing module  12  determines the depth value of each pixel in the reference image according to the depth image, and accordingly separates the reference image into a plurality of layer images having different depth values. The processing module  12  implements a blurring process for the layer images according to the depth values of the layer images and the aperture simulation parameters, so as to generate a simulation image having a larger aperture and the depth of field and the like photographed by a single lens camera. In one embodiment, the processing module  12  is an application specific integrated circuits (ASIC), a programmable microprocessor, a digital signal processor (DSP), a programmable logic device (PLD) or a CPU with a software module, and it is not limited thereto. 
     The display module  13  is, for example, a liquid crystal display screen that displays the reference image for a user to click any position of the reference image. Thereby, the processing module determines the reference depth value according to the depth value of the pixel corresponding to the click position. It should be noted that, the display module  13  can also be a digital display screen with the touch-sensing function or other general digital display screen, and it is not limited therein. In this embodiment, the user can click any position of the reference image via a mouse, a keyboard or other input module, such that the processing module  12  determines a reference depth value according to the depth value of the pixel corresponding to the clicked position. The user can also directly input a value via the input module, and the processing module  12  can use the input value as the reference depth value, but is not limited therein. 
     In this embodiment, after the processing module  12  determines a reference depth value, the processing module  12  calculates the differences between the depth value of each layer image and the reference depth value, and determines the blurring degrees of the layer images according to the differences. Moreover, the processing module  12  makes the display module  13  display different aperture shapes for the user to choose according to the aperture simulation parameter. After that, after the user chooses an aperture shape via a display module  13  with the touch-sensing function, a mouse, a keyboard or other input modules, the processing module  12  generates a simulation image having a specific Bokeh shape (such as the star-shaped, heart-shaped, circle, pentagon or other shapes) according to the aperture shape. It should be noted that, if the user does not choose an aperture shape, the processing module  12  would automatically load in a predetermined aperture simulation parameter so as to generate a simulation image of which the Bokeh shape is like a circle Gaussian function. 
     In addition, the way for the image processing device  1  to obtain the reference image and the depth image is that, the user saves the reference image and the corresponding depth image in the memory module  11  in advance, but it is not limited herein. In other words, the image processing device  1  can also generate the reference image and the depth image itself. In detail, the image processing device  1  comprises an image capturing module (not shown), and the image capturing module is connected to the processing module  12 . The image capturing module comprises a lens, a light-sensing element and an aperture, which is used to capture images of a scene so as to generate a plurality of photographic images. The light-sensing element is, for example, a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). In this embodiment, the processing module  12  generates the reference image and the depth image according to the photographic images. It should be noted that, how the processing module  12  generates the reference image and the corresponding depth image according to the photographic images is well-known by those skilled in the art, and not repeated herein. 
     Please refer to  FIGS. 1-3 .  FIG. 2  shows a flow chart of an image depth processing method of one embodiment of the instant disclosure, and  FIG. 3  shows a reference image stored in a memory module of the image processing device. The image depth processing method can be applied to the image processing device  1  shown in  FIG. 1 , and can generate a simulation image having a larger aperture and the depth of field and like photographed by a single lens camera according to the reference image shown in  FIG. 3 . Thus, the image depth processing method in this embodiment is described as follows via the image processing device  1  shown in  FIG. 1  and the reference image shown in  FIG. 3 . It should be noted that, the reference image shown in  FIG. 3  is merely for describing the working principle of the image depth processing method, which does not mean that the image depth processing method merely can be applied to the reference image shown in  FIG. 3 . In other words, the image depth processing method can generate simulation images according to other reference images. 
     In Step S 201 , the processing module  12  obtains N layer images from the reference image according to the depth value shown in the depth image (from far to near), wherein N is an integer greater than 1 and the depth values of the layer images are different. As shown in  FIG. 3 , the processing module  12  obtains four layer images from the reference image P 1  according to the depth image corresponding to the reference image P 1 , which is a first layer image N 1 , a second layer image N 2 , a third layer image N 3  and a fourth layer image, wherein the higher the layer number of the layer image is, the greater the depth value of the layer image would be (further depth of field). It should be noted that, before the processing module  12  obtains N layer images from the reference image, the user can set the value of N via the display module  13  having touch-sensing function, a mouse or a keyboard to further set the numbers of layer images and the numbers of re-focus, wherein N can be a predetermined value. 
     In Step S 203 , after the user chooses a reference depth value via a display module  13  having the touch-sensing function, a mouse or a keyboard, the processing module  12  respectively calculates the differences between the depth values of the layer images and the reference depth value, so as to obtain the difference values corresponding to each layer image. The processing module  12  determines the simulation degree of each layer image according to the difference values, wherein if the difference value is large the blurring degree of the layer image would be large. Also, the memory module  11  stores a look-up table (not shown), and the look-up table records a plurality of blurring parameters. Assuming that the user clicks the first layer image N 1  via the display module  13  having the touch-sensing function, the processing module  12  would use the depth value of the first layer image N 1  as the reference depth value, and respectively calculate difference values between the depth values of the layer images and the reference depth value so as to correspondingly obtain the first, the second, the third and the fourth difference values. The processing module  12  obtains the corresponding blurring parameters from the look-up table according to the difference values, and then determines the blurring degree of each layer image according to the blurring parameters. 
     In Step S 205 , the processing module  12  uses the N th  layer image having the greatest depth value as a background image (such as the fourth layer image N 4 ), and uses the N−1 th  layer image as a foreground image (such as the third layer image N 3  wherein the depth value of the N−1 th  layer image is smaller than the N th  layer image. After that, the processing module  12  blurs the background image according to the predetermined aperture simulation parameter (or the aperture simulation parameter chosen by the user) and the corresponding difference value of the N th  layer image. 
     In Step S 206 , the processing module  12  composes the foreground image and a local image in the blurred background image, which is near the edge of the foreground image. Please refer to  FIG. 4 .  FIG. 4  shows a schematic diagram of a background image obtained according to the reference image shown in  FIG. 3 . In this step, the processing module  12  obtains a local image P 11  from the fourth layer image N 4 , and the local image P 11  corresponds to a part of the fourth layer image N 4 , which is near the edge of the third layer image N 3 . The processing module  12  composes the local image P 11  and the third layer image N 3  into a composite image. 
     In Step S 207 , the processing module  12  blurs the composite image formed by the foreground image and the local image of the background image, such that the depth of field across from the fourth layer image N 4  to the third layer image N 3  would look continuous and natural. 
     In Step S 209 , after Step S 207  the blurring degree of the local image and the blurring degree of the background image are different, so there would be a margin image existed in the background image. Thus, in this step, the processing module  12  implements a fading process to fade a margin image in the background image. Please refer to  FIG. 5 .  FIG. 5  shows a schematic diagram of another background image obtained according to the reference image shown in  FIG. 3 . As shown in  FIG. 5 , after Step S 207 , the blurring degree of the local image P 11  and the blurring degree of the fourth layer image N 4  are different, and thus there would be a margin image L 1  existed in the fourth layer image N 4 . Therefore, in this step, the processing module  12  would fade in the fourth layer image N 4  along the margin image L 1  (that is, to make the fourth layer image N 4  near the margin image L 1  gradually clear) and fade out the local image P 11  along the margin image L 1  (that is, to make the local image P 11  of the margin image L 1  gradually blur), so as to fade the margin image L 1 . 
     In Step S 211 , the processing module  12  determines whether (N−1) equals to 1. If yes, it goes to Step S 213  to form a simulation image according to the current background image and foreground image. If no, it goes to Step S 215  to subtract 1 from N and then it goes to Step S 217 . 
     In Step S 217 , the processing module  12  forms a new background image according to the current background image and foreground image to replace the old background image, and uses the N−1 th  layer image as a new foreground image to replace the old foreground image. After Step S 215 , N is 3 (that is, N=4−1), and thus the processing module  12  uses the second layer image N 2  as a new foreground image, and forms a new background image according to the fourth layer image N 4  and the third layer image N 3  which are processed by Steps S 203 ˜S 209  (as shown in  FIG. 5 ). 
     After Step S 217 , the processing module  12  again implements Step S 206  to compose the foreground image (that is, the second layer image N 2 ) and a local image of the background image (that is, a local image P 12  of the background image, which is near the edge of the second layer image N 2 ) into a composite image. 
     After that, the processing module  12  implements Step S 207  to blur the composite image generated in Step S 206 , according to the aperture simulation parameter and the depth value corresponding to the second layer image N 2 . 
     Please refer to  FIG. 6 .  FIG. 6  shows a schematic diagram of still another background image obtained according to the reference image shown in  FIG. 3 . As shown in  FIG. 6 , after Step S 207 , there would be a margin image L 2  existing in the background image, and thus the processing module would again execute Step S 209  so as to fade in the background image (that is, to make the background image of the margin image L 2  gradually clear) and to fade out the local image P 12  along the margin image L 2  (that is, to make the local image P 12  of the margin image L 2  gradually blur), so as to fade the margin image L 2 . 
     After that, the processing module  12  again executes Step S 211  to determine that (N−1) is not equal to 1 (that is, N−1=2). In addition, the processing module  12  again executes Step S 217  to use the first layer image N 1  as a new foreground image and to form a new background image according to the background image and foreground image processed via Steps S 207 ˜S 209  (as shown in  FIG. 6 ). 
     After that, the processing module  12  again executes Step S 206  to compose the foreground image (that is, the first layer image N 1 ) and a local image of the background image (that is, a local image P 13  of the background image, which is near the edge of the first layer image N 1 ) into a composite image. 
     After that, the processing module  12  executes Step S 207  and Step S 209  to blur the composite image generated in Step S 206  according to the aperture simulation parameter and the corresponding depth value of the first layer image N 1 . The processing module  12  fades in the background image along a margin image (not shown) in the background image and fades out the local image P 13  along the margin image, so as to fade the margin image. 
     After that, the processing module  12  again executes Step S 211  to determine whether N−1 is equal to 1 (that is, 2−1=1), and then executes Step S 213  to generate a simulation image. Specifically, when N−1 equals to 1, all of the layer images have been processed. Therefore, please refer to  FIG. 7 ,  FIG. 7  shows a schematic diagram of a simulation image generated according to the reference image shown in  FIG. 3 . As shown in  FIG. 7 , the processing module generates a simulation image having a larger aperture and the depth of field like photographed by a single lens camera according to the current background image and foreground image. 
     From the above, after the image processing device  1  blurs the background image, the image processing device  1  would blur the foreground image according to the difference value corresponding to the foreground image and the aperture simulation parameters, and the image processing device  1  would also blur a local image of the background image according to the difference value corresponding to the foreground image and the aperture simulation parameters. After that, the image processing device  1  implements a fading process along a margin image in the background image, so as to generate a simulation image satisfied with the physics principles of geometrical optics so that the simulation image would have the graduated depth of field and look natural and continuous. 
     Moreover, the depth value of the first layer image N 1  is taken as the reference depth value in the above embodiment to generate a simulation image having near depth of field, but it is not limited herein. In other words, the image processing device  1  can also take the depth value of other layer images as the simulation image according to the position clicked by the user so as to generate a simulation image having other kinds of depth of field. However, it should be noted that, no matter which layer image the image processing device  1  takes the depth value of as the reference depth value, the image processing device  1  processes images in an order from the layer image having the greatest depth value to the layer image having the smallest depth value during the image process (that is, from the farthest layer image to the nearest layer image), so as to generate a simulation image satisfying the physics principles of geometrical optics. 
     Additionally, in another embodiment, before the image processing device  1  blurs the layer images, the image processing device  1  would increase the image brightness value of at least one bright area in the reference image P 1 . Specifically, there may be images of a bright area in the reference image P 1 , such as a light-concentrating point or reflective surface. Usually, the image of the bright area has the greatest brightness which is 255 (the brightness of the general images ranges from 0 to 255). However, the brightness of the image of the bright area is usually presented as 255, so if the reference image P 1  is blurred without increasing the brightness of the image of the bright area in advance, the brightness of the image of the bright area of the blurred reference image P 1  would decrease. Thereby, the simulation image generated by the image processing device  1  would not satisfy the principles of optics. Therefore, before the image processing device  1  executes Step S 205 , the image processing device  1  would increase the brightness of the image of the bright area in advance (for example, from 255 to 500), so as to prevent decreasing the brightness of the image of the bright area during the process of blurring each layer image. However, it is not limited herein, and those skilled in the art could choose to skip this step based on need. 
     It should be also mentioned that, the image depth processing method can be applied to still images and also to simulated dynamic images (that is, animation) to have depth of field, but is not limited herein. 
     It is clarified that, the sequence of steps in  FIG. 2  is set for a need to instruct easily, but the sequence of the steps is not used as a condition for demonstrating embodiments of the instant disclosure. 
     To sum up, the image processing device and the image depth processing method allow the user to set the numbers of layer images and the numbers of re-focus, to simulate the depth of field of any practical lens according to the aperture simulation parameters and also to increase the brightness of the image of the bright area in advance so as to strengthen at least one image of the bright area of the simulation image. In the image processing device and the image depth processing method, via blurring the foreground image and blurring a local image of the background image, which is near the edge of the foreground image, and implementing a fading process to fade the margin image in the background image, it a simulation image can be generated that satisfies the physics principles of geographic optics (that is, a simulation image having graduated depth of field, which looks natural and continuous). 
     The descriptions illustrated supra set forth simply the preferred embodiments of the instant disclosure; however, the characteristics of the instant disclosure are by no means restricted thereto. All changes, alterations, or modifications conveniently considered by those skilled in the art are deemed to be encompassed within the scope of the instant disclosure delineated by the following claims.