Abstract:
A method of image processing is provided for separating an image object from a captured or provided image according to a three-dimensional (3 D) depth and generating a synthesized image from the image portions identified and selectively modified in the process. The method retrieves or determines a corresponding three-dimensional (3D) depth for each portion of an image, and enables capturing a selective portion of the image as an image object according to the 3D depth of each portion of the image, so as to synthesize the image object with other image objects by selective processing and superimposing of the image objects to provide synthesized imagery.

Description:
This application claims the benefit of Taiwan application Serial No. 100115846, filed May 5, 2011, the subject matter of which is incorporated herein by reference. 
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The invention relates in general to a method of image processing and associated apparatus, and more particularly, to a method for separating an image object from an image according to a three-dimensional (3D) depth and associated apparatus. 
     2. Description of the Related Art 
     With the ability to enhance quality, data amount, data content, and amusement of static and/or dynamic images, image processing and synthesis have become prevalent in modern society. For example, background removal is a common image processing performed on digital images. The image background removal process captures significant foregrounds of the image as independent image objects, so that the foregrounds can be separated from the remaining, less important background portions. 
     In the prior art, background removing is generally performed with chroma keying. During chroma keying, main substances forming the foregrounds are placed in front of a monochromatic (blue or green) background to be captured with the monochromatic background as an image. Subsequently, with post-image processing, the monochromatic background is removed from the image and the foregrounds are obtained as foreground image objects, which are then synthesized with a separately formed background image to form a new image. In another conventional background removing technique, the foreground image objects are separated from the background according to edges and/or color differences between the foreground objects and the background. 
     However, the above solutions associated with the prior art suffer from various application drawbacks. For example, when colors of certain parts of the foreground objects approximate or are identical to a color of the background, the parts are improperly removed during the background removal, and incomplete foreground objects are then formed. Furthermore, for a rather complicated foreground and/or a foreground with less apparent edges between the foreground and the background, difficulties may arise when applying the prior art techniques to correctly acquire the foreground image objects from the image. 
     SUMMARY OF THE INVENTION 
     The invention is directed to a method of image processing according to a determined three-dimensional (3D) depth. A foreground of a static or dynamic image is captured as an image object according to the 3D depth, so that the foreground image object may be utilized in a subsequent image processing, such as image synthesis. 
     According to an embodiment of the present invention, a method of image processing is provided. The method comprises receiving an image, providing a 3D depth corresponding to a portion of the image, capturing the portion of the image as a distinct image object (first image object) according to the 3D depth corresponding to the portion, providing a second image object, and superimposing the first image object and the second image object to generate a synthesized image. 
     In an embodiment, the first image object and/or the second image object is pre-processed, and the pre-processed first image object (i.e. the first superimposing object), and/or a pre-processed second image object (i.e. the second superimposing object), are superimposed to generate the synthesized image. The pre-processing may include scaling, color, brightness, contrast, and/or sharpness adjustment of the image objects, and/or adjustment of the corresponding 3D depths and/or distances. 
     The superimposing step comprises a superimpose prioritization processing, a superimpose layering processing, and a post-superimposing processing. During the superimpose prioritization processing, a corresponding superimpose sequence is respectively provided according to the 3D depths and/or distances corresponding to the first superimposing object and the second superimposing object. During the superimpose layering processing, the first superimposing object and the second superimposing object are superimposed into a synthesized image according to the superimpose sequence corresponding to the first superimposing object and the second superimposing object. For example, when the distance of the first superimposing object is smaller than the distance of the second superimposing object, the first superimposing object is reserved, an overlapping part of the second superimposing object with the first superimposing object is removed, and the first superimposing object is then superimposed on the second superimposing object. 
     During the post-superimpose processing, detailed processing is performed on a superimposed result of the first superimposing object and the second superimposing object. For example, superimposed edges of the first and second superimposing objects are processed by blending, anti-aliasing, and/or feathering, so as to render a more natural-looking superimposed image. 
     The first and/or second image object is captured from an image. The image may be a 3D image comprising portions respectively corresponding to a visual deviation. According to the visual deviation of each portion of the image, a 3D depth is obtained for each portion of the image. 
     According to another embodiment of the present invention, an image processing apparatus comprising a depth module, a separation module, an optional pre-processing module, and a superimpose module is provided. The depth module respectively provides a 3D depth for portions of an image. The separation module captures a portion of the image as an image object according to the 3D depth corresponding to each portion of the image. The pre-processing module pre-processes the image object to be the superimposing object. The superimpose module superimposes the image object, and comprises a superimpose prioritization module for superimpose prioritization processing, a superimpose layering module for superimpose layering processing, and a post-superimpose module for post-superimpose processing. 
     The above and other aspects of the invention will become better understood with regard to the following detailed description of the preferred but non-limiting embodiments. The following description is made with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of a 3D image captured by a video camera according to an embodiment of the present invention. 
         FIG. 2  is a block diagram of an image processing apparatus according to an embodiment of the present invention. 
         FIG. 3  is a schematic diagram of image processing according to an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     When the same object is observed by both eyes, images presented by the left and right eyes have subtle differences, and the human brain establishes a make-shift three-dimensional (3D) image according to the images perceived by the both eyes.  FIG. 1  shows three distinct schematic diagrams of a 3D image, where section  1 A details a schematic diagram of a 3D image photographed by a camera MS according to an embodiment of the present invention. The camera MS comprises a left camera lens CL and a right camera lens CR. For an object OB 1  located at a distance Y 1  from the camera MS, the left camera lens CL captures an image IL 1  of the object OB 1  in a left image picture PL, and the right camera lens CR captures an image IR 1  of the object OB 1  in a right image picture PR. According to the left image picture PL and the right image picture PR, a 3D image is formed. A distance from the image IL 1  of the left image picture PL to the image IR 1  of the right image picture PR is defined as a visual deviation X 1 . To playback the 3D image, the left image picture PL and the right image picture PR are respectively presented to the left and right eyes of a viewer, such that the 3D image of the object OB 1  is observed with the corresponding visual deviation X 1  between the images IL 1  and IR 1 . 
     Similarly, referring to section  1 B of  FIG. 1 , for an object OB 2  located at a distance Y 2  from the camera MS, the left camera lens CL captures an image IL 2  of the object OB 2  in a left image picture PL, and the right camera lens CR captures an image IR 2  of the object OB 2  in a right image picture PR. Likewise, a distance from the image IL 2  of the left image picture PL to the image IR 2  of the right image picture PR is defined as a visual deviation X 2 . It is to be noted that, the object OB 2  is located farther away from the camera MS than the object OB 1  (i.e., Y 2 &gt;Y 1 ), such that the visual deviation X 2  between the images IL 2  and IR 2  is smaller than the visual deviation X 1  between the images IL 1  and IR 1 . 
     According to the above characteristics, a concept of 3D depth is developed. It is observed from section  1 A and section  1 B that, by comparing the visual deviations of portions of the left image PL and the right image PR, a distance between an object and the camera MS may be obtained. Supposing the left image picture PL in section  1 A is defined as a reference image, the visual deviation X 1  between the image IL 1  of the left image picture PL and the IR 1  of the right image picture PR of the object OB 1  is a positive value, which is a 3D depth of the images IL 1  and IR 1 . Similarly, in section  1 B, when the left image picture PL is again defined as the reference image, the visual deviation X 2  between the image IL 2  of the left image picture PL and the IR 2  of the right image picture PR of the object OB 2  is a positive value, which is a 3D depth of the images IL 2  and IR 2 . In contrast, supposing the right image picture PR is defined as a reference image in section  1 A and section  1 B, the 3D depth of the images IL 1  and IR 1  is the negative value of the visual deviation X 1 , and the 3D depth of the images IL 2  and IR 2  is the negative value of the visual deviation X 2 . As shown in section  1 C, the distance Y 1  between the object OB 1  and the camera MS may be acquired by obtaining the 3D depth of the images IL 1  and IR 1  of the left and right images IL 2  and IR 2  from comparison. Similarly, the distance Y 2  between the object OB 2  and the camera MS may be acquired by obtaining the 3D depth of the images IL 2  and IR 2  of the left and right images IL 1  and IR 1  from comparison. In other words, it may be concluded which of the objects OB 1  and OB 2  is closer to or farther away from the camera MS according to these determined 3D depths. 
     According to the principles of 3D depth shown in section  1 C, the image of the object OB 1  and the image of the object OB 2  in the images PL and PR may be separated into a foreground and a background. More specifically, according to the 3D depths, it is concluded that the object OB 1  is located closer to the camera MS and the object OB 2  is located farther away from the camera MS, so that the images IL 1  and IR 1  of the object OB 1  are determined as the foreground, and the images IL 2  and IR 2  of the object OB 2  are determined as the background. Therefore, the images IL 1  and IR 1  of the object OB 1  are captured from the images PL and PR to become a foreground image object to accomplish effective background removing of the image. 
       FIG. 2  shows a functional block diagram of an apparatus  10  according to an embodiment of the present invention. The apparatus  10  is an image processing apparatus, which comprises at least one depth module  12  (and optionally an additional depth module  26 ), a separation module  14 , at least one pre-processing module  16  (and optionally an additional pre-processing module  28 ), and a superimpose module  18 . The superimpose module  18  comprises a superimpose prioritization module  20 , a superimpose layering module  22 , and a post-superimpose module  24 . 
     In the apparatus  10 , the depth module  12  receives an image data input Pi_A comprising a dynamic or static, 2D or 3D image divided into a plurality of portions. The depth module  12  provides a 3D depth and a distance corresponding to each portion of the image, and the separation module  14  captures a portion of the image as an image object according to the 3D depth and the distance corresponding to each portion of the image. Referring to  FIG. 3 , for example, the image data Pi_A comprises a 3D image formed by the left image picture PL and the right image picture PR; the depth module  12  provides a corresponding 3D depth for each portion of the 3D image to provide a distance for each portion; and the separation module  14  separates a foreground according to the 3D depths and the distances, e.g., the foreground images IL 1  and IR 1  are captured as an image object lob 1  according to the principles illustrated in section  1 C of  FIG. 1 . The image object lob 1  corresponds to a distance Yob 1  associated with the distance Y 1 . 
     According to the principles illustrated in section  1 C of  FIG. 1 , the left image picture PL and the right image picture PR are captured by the camera MS to obtain the 3D depths of the objects OB 1  and OB 2 . In another embodiment, the depth module  12  may also generate the 3D depth with reference to a distance detected by a distance detection device supposing the image data Pi_A is associated with the distance detected by the distance detection device, so as to separate the foreground from the image data Pi_A. Referring to  FIG. 3 , in an embodiment, when distributed positions of an xy plane of the objects OB 1  and OB 2  are captured as images I 1  and I 2  in the image data Pi_A, a distance detection device  30  also detects distances Ys 1  and Ys 2  of the objects OB 1  and OB 2  on a normal line perpendicular to the xy plane, (i.e. the z-axis), so that the images I 1  and I 2  respectively associate with the distances Ys 1  and Ys 2 . The depth module  12  then provides the 3D depths of the images I 1  and I 2  according to the distances Ys 1  and Ys 2 , and the separation module  14  separates the foreground image I 1  as the image object lob 1 . For example, the distance detection device  30  can be a laser, infrared, sound-wave, ultrasonic, and/or electromagnetic wave distance detection device. 
     In another embodiment, the image data Pi_A is obtained with computer graphics rendering from a virtual 3D model. The computer graphics provided may also indicate a distance (e.g., a depth map) of the 3D model or a parameter associated with the distance, and the depth module  12  may generate the 3D depth with reference to the distance provided by the computer graphics or the parameter associated with the distance, enabling the separation module  14  to capture the foreground of the image data Pi_A as an independent image object. 
     In this embodiment, after the separation module  14  captures out the image object, the image object is pre-processed by the pre-processing module  16 . The pre-processing may include scaling, color, brightness, contrast, and/or sharpness adjustment of the image object, and/or corresponding 3D depth/distance adjustment. 
     Similar to operations of the depth module  12 , a depth module  26  may also be included in an embodiment of the present invention to provide a distance corresponding to another image data input Pi_B, which may, for example, be a superimposed image object. Referring to  FIG. 3 , the image data Pi_B may be an image object lob 2  corresponding to a distance Yob 2 , and the pre-processing module  28  is utilized for pre-processing the image data Pi_B, wherein the pre-processing is similar to that performed by the pre-processing module  16  on the image object lob 1 . 
     After obtaining image objects of the image data Pi_A and Pi_B and the corresponding distances, the superimposing module  18  superimposes the image objects of the image data Pi_A and Pi_B according to a superimpose parameter into a synthesized image. For example, the synthesized image can be a 2D or 3D, static or dynamic image. During the superimposing processing, the superimpose prioritization module  20  performs a superimpose priority processing to provide a corresponding superimpose sequence according to the 3D depths and distances corresponding to the image objects. The superimpose layering module  22  superimposes the image objects according to the superimpose sequence to a synthesized image. The post-superimpose module  24  performs post-superimpose processing on the superimposed result of the superimpose layering module  22  by performing detailed processing, such as blending, anti-aliasing, and/or feathering superimposed edges of the image objects to render a more natural looking synthesized object. 
       FIG. 3  shows operations of the superimpose processing. Upon obtaining the 3D depths and corresponding distances Yob 1  and Yob 2  of the image objects lob 1  and lob 2 , the superimpose prioritization module  20  defines superimpose priorities and sequence according to the 3D depth. For example, by comparing the 3D depth and the distance Yob 1  of the image object lob 1  and the 3D depth and the distance Yob 2  of the image object lob 2 , it is concluded that the image object lob 1  is located farther in the front of perceived image than the image object lob 2 , so that the image object lob 1  is given a higher superimpose priority than the image object lob 2 . When the superimpose layering module  22  superimposes the image objects lob 1  and lob 2  to a 2D synthesized image Po, the image object lob 1  with the higher superimpose priority is entirely preserved, whereas an overlapping part (e.g., an overlapping part lovlp) of the superimposed image object lob 2  with the superimposed image object lob 1  is removed, so as to superimpose the image object lob 1  on the image object lob 2 . 
     Similarly, an output of the superimposed image may also be a 3D image, e.g., a 3D image comprises left and right images PLo and PRo. The superimposing module  18  respectively performs superimpose layering and post-superimpose processing on the left and right images PLo and PRo. In an embodiment of the present invention, the image object lob 1  in the left image picture PLo comes from the image IL 1  in the left image picture PL, and the image object lob 1  in the right image picture PRo comes from the image IR 1  in the right image picture PR. Similarly, if another image data Pi _B also comprises left and right images of a 3D image, the image object lob 2  in the left image picture PLo is formed by the left image in the image data Pi_B, and the image object lob 2  in the right image picture PRo is formed by the right image in the image data Pi_B. 
     In another embodiment, the image data Pi_B may also be a 2D image. Under such conditions, to output a 2D image, the image object lob 1  may be directly superimposed on the 2D image object lob 2  without considering the 3D depth and the distance Yob 2 . Therefore, as previously stated, the depth module  26  is an optional element, which can be omitted in some cases. In another embodiment, the image data Pi_B is also a 2D image. However, to output a 3D image from the 2D image, the image object lob 2  is given a predetermined 3D depth, and the image data Pi_B is utilized as both left and right images of the 3D image, so as to superimpose the left and right images with other image objects to form the 3D image output. The predetermined 3D depth may be defined with reference to the 3D depth of the reference image data Pi_A, or may be given by a user. In detail, as the value of the 3D depth gets greater, the more front the image object is located; thus, when the image data Pi_B serves as a background, the corresponding image object lob 2  is given a 3D depth smaller than those corresponding to all other image objects. Alternatively, the 3D depth of the image data Pi_B may also be given by the user to adjust relative distances between the image object lob 2  and the image object lob 1  on a 3D image output. In another embodiment, the image data Pi_A and/or Pi_B may be a dynamic image formed by a plurality of frames, and the apparatus  10  performs capturing and superimposing of image objects with respect to images of each frame. 
     It is to be noted that, the superimposing of the two image data inputs Pi _A and Pi _B is merely an example, and the present invention is not limited to processing only two image data inputs. That is, a plurality of image data inputs may be simultaneously processed and superimposed to output various types of image data. 
     In the apparatus  10 , the pre-processing modules  16  and  28  adjust image objects according to superimpose and synthesis requirements of the image objects. For example, the pre-processing module  16  may reduce the distance Yob 1  and correspondingly enlarge the image object lob 1 , so that the image object lob 1  is positioned more to the front in the synthesized image. Alternatively, the pre-processing module  16  may also increase the brightness of the image object lob 1  to emphasize the image object lob 1 . In contrast, the pre-processing module  28  may reduce the brightness of the image object lob 2  and decrease the sharpness with blurring, so as to present shallow depth of field effects in the synthesized image. It is to be noted that the implementation of pre-processing modules  16  and  28  may include optionally functional elements that are provided according to actual application requirements. 
     Please note that each of the modules in the apparatus  10  can be implemented by hardware, firmware, software, or any combination thereof. The depth module  12  and the depth module  26  may be a single module, and the pre-processing module  16  and the pre-processing module  28  may also be a single module. 
     For example, the present invention can be applied to filming of movies, where the image data Pi_A and Pi_B are individually filmed and then superimposed according to operational principles of the apparatus  10 . Furthermore, the present invention is also applicable to image processing and editing of photographs as well as to playbacks of video telephone calls, video conference, and/or Internet video. For example, video conference participants are captured by the 3D video camera ( FIG. 1 ) to form the image data Pi_A, and images of the participants are separated from the background according to techniques of the present invention and then superimposed and synthesized with the background of another image data Pi_B. In addition, the present invention is particularly applicable to the travel industry, multimedia applications, sports, educations, entertainment, and games. For example, a user is captured by the 3D video camera MS, and the user image is separated from the background and then superimposed and synthesized with a virtual background of a game. 
     With the embodiments of the present invention, it is illustrated that by separating a foreground from a background according to the 3D depth, the foreground may be independently utilized so as to facilitate not only more accurate and convenient image processing but also more diversified image contents. 
     While the invention has been described by way of example and in terms of the preferred embodiments, it is to be understood that the invention is not limited thereto. On the contrary, it is intended to cover various modifications and similar arrangements and procedures, and the scope of the appended claims therefore should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements and procedures.