Systems and methods for image depth map generation

Systems and methods which provide generation of image depth maps which more accurately represent the local depth discontinuity within images through use of image global depth maps adapted based upon image global motion and/or localized depth analysis utilizing relative relationships of attributes across depth discontinuities in the image are disclosed. Embodiments utilize a full global depth map which is larger than or equal to the image being converted in order to accommodate image global motion, in generating an image global depth map. In operation according to embodiments, an image global depth map is identified within the full global depth map, such as based upon global motion within the image. Localized depth analysis, using pixel attribute relative relationships, is applied with respect to the image global depth map according to embodiments to generate an image depth map which more accurately reflects the local depth discontinuities within the image.

TECHNICAL FIELD

The invention relates generally to image processing and, more particularly, to generation of depth maps, such as for use in rendering three-dimensional images from two-dimensional images.

BACKGROUND OF THE INVENTION

With the development of improved three-dimensional (3D) projection technologies, as used in theatres, amusement rides, etc., and the more recent introduction of 3D television sets, the demand for 3D image content is rapidly increasing. Accordingly, there has been considerable interest in converting 2D images (e.g., feature length movies, television shows, etc.) captured using mono-view devices into 3D images.

Some of the conventional mono-view 2D-to-3D conversion techniques utilize computer vision based technologies, such as segmentation, vanishing line detection, etc. Likewise, some mono-view 2D-to-3D conversion techniques utilize motion information, such as to capture content obscured by an object moving in the foreground. These technologies, however, are generally not practical for real time 2D-to-3D conversion. In particular, such computer vision based technologies require significant computing resources and are not well suited either for real time operation or for low cost applications.

Other techniques used to convert 2D images to 3D images generate a depth map (i.e., an image or image channel that contains information relating to the distance of the surfaces of scene objects from a viewpoint), then use the depth map to create the left and right view (i.e., parallax views) from the image in accordance with the depth map. For example, various techniques utilize a global depth model and more localized depth analysis to generate a local depth map with which a 3D image may be generated. The global depth model provides a generalized depth model for the image, such as may be based on a planar model or spherical model, which does not accurately reflect the local depth discontinuity. Accordingly, more localized analysis, such as image texture analysis, is used with the global depth model to generate a local depth map for the image which more accurately reflects the local depth discontinuity.

The image depth maps generated using the foregoing global depth model and localized depth analysis techniques are often less than ideal, such as due to the use of the global depth models not dynamically representing the images to be converted and the localized depth analysis techniques used not accurately representing local depth. For example, a global depth model chosen as a central symmetric model may be used by scene analysis, wherein the model will keep central symmetry for all the frames of the scene irrespective of changes within the scene (e.g., movement or motion not rising to the level of a scene change). Likewise, local depth assignments made using typical localized depth analysis techniques are particularly inaccurate under poor light conditions.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to systems and methods which provide generation of image depth maps which more accurately represent the local depth discontinuity within images through use of image global depth maps adapted based upon image global motion and/or localized depth analysis utilizing relative relationships of attributes across depth discontinuities in the image. Accordingly, embodiments of the invention utilize a full global depth map in generating an image global depth map. Localized depth analysis, using pixel attribute relative relationships, is applied with respect to the image global depth map according to embodiments to generate an image depth map which more accurately reflects the local depth discontinuities within the image.

A full global depth map utilized according to embodiments of the invention comprises a global depth map which is larger than or equal to the image for which 2D-to-3D conversion is to be provided in order to accommodate image global motion (e.g., general movement of the image, such as due to panning and/or zooming the camera, general movement of the object groupings within the image, etc., which does not amount to a scene change). In operation according to embodiments, an image global depth map is identified within the full global depth map (e.g., a sub-region of the full global depth map), such as based upon global motion within the image. Accordingly, techniques to track image global motion may be implemented according to embodiments of the invention. For example, embodiments of the invention can change the central symmetric model through tracking the global motion of the scene of the image. Moreover, embodiments of the invention may update the depth directly based on saliency area detection.

The localized depth analysis to refine an image global depth map and provide an image depth map according to embodiments utilizes relative relationships of different color components of the various pixels. Accordingly, inaccuracies resulting from independent analysis of one or more color component, such as may be associated with poor light conditions, may be avoided according to embodiments of the invention by introducing the relative relationship of different color channels.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1shows a system adapted to provide image depth map generation according to embodiments of the invention. Specifically, system100is adapted to accept input of image data101(e.g., a series of mono-view images comprising scenes from a moving video) and generate image depth map102(e.g., images that contain information relating to the distance of the surfaces of scene objects in the moving video from a viewpoint) for output through use of image global depth maps adapted based upon image global motion and/or localized depth analysis utilizing relative relationships of attributes across depth discontinuities in the image. Accordingly, system100of the illustrated embodiment comprises attribute relative relationship based localized depth analysis logic (also referred to herein as depth analysis logic)110, global motion based image global depth map generation logic (also referred to herein as global depth map generation logic)120, and depth fusion logic130, as will be described in further detail below.

It should be appreciated that the foregoing logic may be implemented in various ways, including hardware logic circuits, software, firmware, and combinations thereof. When implemented in software, elements of embodiments of the present invention may comprise code segments operable upon a processor-based system, such as computer system200ofFIG. 2, to perform the tasks described herein. The code segments can be stored in a computer readable medium, such as random access memory (RAM)203, read only memory (ROM)204, and/or storage device206. Additionally or alternatively, the code segments may be downloaded via computer networks, such as network212.

Computer system200adapted for use according to an embodiment of the present invention may comprise a general purpose processor-based system configurable with program code (e.g., including the aforementioned code segments) to provide functionality as described herein. Accordingly, computer system200of the illustrated embodiment includes central processing unit (CPU)201coupled to system bus202. CPU201may be any general purpose CPU, such as a processor from the PENTIUM or CORE family of processors available from Intel Corporation or a processor from the POWERPC family of processors available from the AIM alliance (Apple Inc., International Business Machines Corporation, and Motorola Inc.). However, the present invention is not restricted by the architecture of CPU201as long as CPU201supports the inventive operations as described herein.

Bus202of the illustrated embodiment is coupled to RAM203, such as may comprise SRAM, DRAM, SDRAM, flash memory, and/or the like. ROM204, such as may comprise PROM, EPROM, EEPROM, and/or the like, is also coupled to bus202of the illustrated embodiment. RAM203and ROM204hold user and system data and programs as is well known in the art. Bus202is also coupled to input/output (I/O) controller205, communications adapter211, user interface adapter208, and display adapter209.

I/O controller205connects to storage device206, such as may comprise one or more of a hard disk, an optical disk (e.g., compact disk (CD) or digital versatile disk (DVD)), a floppy disk, and a tape, to the computer system. I/O controller205of the illustrated embodiment is also connected to printer214, which would allow the system to print information such as documents, photographs, etc. Such a printer may be a traditional printer (e.g. dot matrix, laser, etc.), a fax machine, a copy machine, and/or the like.

Communications adapter211is adapted to couple computer system200to network212to provide communications to and/or from external systems, devices, networks, etc. Network212may comprise the public switched telephone network (PSTN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), an extranet, an intranet, the Internet, a cellular network, a cable transmission network, and/or the like.

User interface adapter208of the illustrated embodiment couples various user input devices to the computer system. For example, keyboard213, pointing device207, and microphone216may be coupled through user interface adapter to accept various forms of user input. Similarly, speakers215may be coupled through user interface adapter to provide user interface output.

Display adapter209provides an interface to display210. Accordingly, CPU201may control display of various information, including text, graphics, and images upon display210through display adapter209. Display210may comprise a cathode ray tube (CRT) display, a plasma display, a liquid crystal display (LCD), a projector, and/or the like. Although not expressly shown in the illustrated embodiment, display210may provide for input of data as well as output of data. For example, display210may comprise a touch screen display according to embodiments of the invention.

Although illustrated as a general purpose processor-based system inFIG. 2, it should be appreciated that a processor-based system adapted to provide depth analysis logic110, global depth map generation logic120, and depth fusion logic130, or portions thereof, may comprise a special purpose processor-based system, such as may provide hardware logic circuits to implement the foregoing functionality. For example, some or all of the foregoing logic may be implemented in an application specific integrated circuit (ASIC) or field programmable gate array (FPGA) according to embodiments of the invention.

Referring again toFIG. 1, as described above, system100of the illustrated embodiment comprises depth analysis logic110, global depth map generation logic120, and depth fusion logic130cooperating to accept input of image data101and generate image depth map102. Image data101may, for example, comprise a series of mono-view images of a scene from a moving video. Correspondingly, image depth map102may provide images that contain information relating to the distance of the surfaces of scene objects in the moving video from a particular viewpoint (e.g., the vantage point of the viewer or camera).

Detail with respect to operation of global depth map generation logic120in providing image global depth map generation for use in generating image depth map102according to embodiments of the invention is shown in flow300ofFIG. 3. It should be appreciated that the functions of flow300of the illustrated embodiment may be performed, for example, by code segments implemented as global depth map generation logic120.

At block301of the illustrated embodiment, an image of input image101for which an image depth map is to be generated is input to global depth map generation logic120. For example, where input image101comprises a moving video scene, a single color image of the moving video scene may be input to global depth map generation logic120. It should be appreciated that processing according to flow300of embodiments may be performed with respect to each such single image of the moving video scene.

Also at block301of the illustrated embodiment, a global depth model for the image is input to global depth map generation logic120. The global depth model may comprise a global depth model configuration as is well known in the art, such as a planar model, a spherical model, etc., selected for the particular corresponding image input to global depth map generation logic120.

Global motion with respect to the image content is tracked at block302of the illustrated embodiment. For example, global motion of the image may be tracked in the horizontal and vertical directions according to embodiments of the invention. Global motion tracking in the horizontal and vertical directions according to embodiments herein may utilize analysis of changes in luminance in the center area of the image, as described in detail with reference toFIGS. 4 and 5below. Irrespective of the particular techniques utilized, global motion tracking provided according to embodiments of the invention preferably tracks general movement of the image, such as due to panning of the camera, general movement of the object groupings within the image, etc., rather than tracking the movement of a particular individual object within the image. Accordingly, global motion tracking may be utilized according to embodiments to locate or map the image to a corresponding portion of a full global depth map (FGDM) for generation of an image global depth map (IGDM) herein.

At block303of the illustrated embodiment global motion with respect to the image content is further tracked, such as to provide global motion tracking in an additional direction. For example, where global motion tracking is provided in the horizontal and vertical directions at block302, global motion tracking at block303may be provided in the Z direction (i.e., the direction orthogonal to the image plane). Global motion tracking in the Z direction according to embodiments herein may utilize analysis of changed of a saliency area size in the center area of the image, as described in detail with reference toFIGS. 6 and 7below. As with the global motion tracking provided at block302, global motion tracking provided at block303of embodiments herein preferably tracks general movement of the image, such as due to zooming in and/or zooming out the camera, etc., rather than tracking the movement of a particular individual object within the image.

It should be appreciated that operation as illustrated in flow300includes functionality that is optionally implemented depending upon the particular circumstances of the implementation. As will be described in further detail below, embodiments herein may implement various techniques (e.g., mapping based or calculating based) for generating a FGDM and for generating an IGDM from the FGDM by global depth map generation logic120. Accordingly, where the particular techniques implemented do not utilize a particular function, such as the global motion tracking of block302and/or the global motion tracking of block303, that functionality may be omitted from the operation of global depth map generation logic120(as indicated by the dotted arrows providing processing flows around block302and block303), if desired. It should be appreciated, however, that global motion tracking as provided at block302may nevertheless be utilized by embodiments implementing both mapping based and calculating based techniques. For example, embodiments of mapping based and calculating based techniques herein utilize global motion tracking in the horizontal and vertical directions (block302) and embodiments of calculating based techniques herein utilize global motion tracking in the Z direction (block303).

At block304of the illustrated embodiment a FGDM is generated. The FGDM of embodiments of the invention comprises a global depth map which is larger than or equal to the image for which 2D-to-3D conversion is to be provided in order to accommodate image global motion (e.g., motion within an image which does not amount to a scene change). For example, the FGDM may comprise a global depth map corresponding to a scene of a motion video (i.e., corresponding to a series of images forming the scene), and thus sized to accommodate the global motion of each image of the scene.

As shown in block304of the illustrated embodiment, various techniques may be utilized to generate a FGDM herein, such as a mapping based technique or a calculating based technique. A mapping based technique as may be utilized to generate a FGDM according to embodiments may comprise utilization of a pre-generated FGDM, as discussed in detail with reference toFIGS. 9A-9Dbelow. A calculating based technique as may be utilized to generate a FGDM according to embodiments may comprise utilization of global motion information to stretch or shrink a FGDM according to the global motion detected within the image, as discussed in detail with reference toFIGS. 8A-8Dbelow.

At block305of the illustrated embodiment the IGDM is determined using the FGDM. The IGDM of embodiments of the invention comprises a global depth map which is the size of the image for which 2D-to-3D conversion is to be provided and which may be used with various depth refining techniques in generating an image depth map herein. It should be appreciated that, due to the analysis performed (e.g., FGDM sub-region selection, such as using motion tracking) according to embodiments herein, the IGDM provides a global depth map which is more accurate to the image than the global depth models traditionally utilized in depth map generation.

As with block304above, block305of the illustrated embodiment illustrates that various techniques may be utilized to determine an IGDM herein, such as the aforementioned mapping based technique or calculating based technique. In operation of a mapping based technique of embodiments, a mapping area for the IGDM may be determined within the FGDM, as discussed in detail with reference toFIGS. 9A-9Dbelow. In operation of a calculating based technique of embodiments, a calculating area for the IGDM may be determined within the FGDM, as discussed in detail with reference toFIGS. 8A-8Dbelow.

At block305of the illustrated embodiment the IGDM is generated. For example, where a mapping based technique is being utilized, a sub-region within the FGDM may have been identified as the IGDM for the current image and thus the IGDM generated from this sub-region. Where a calculating based technique is being utilized, a sub-region within the FGDM may have been identified as the IGDM for the current image and the global depth model may be utilized to calculate the IGDM for that sub-region. The IGDM generated by global depth map generation logic120in accordance with flow300described above may be provided to depth fusion logic130, along with local depth adjustment information generated by depth analysis logic110, for generating image depth map102according to embodiments herein.

Detail with respect to operation of global depth map generation logic120to provide global motion tracking functionality (e.g., as provided at block302ofFIG. 3) in providing image global depth map generation according to embodiments of the invention is shown in flow400ofFIG. 4. It should be appreciated that the functions of flow400of the illustrated embodiment may be performed, for example, by code segments implemented as global depth map generation logic120.

At block401of the illustrated embodiment the center area of the image for which an image depth map is to be generated is set. For example, the center area of a single color image of a moving video scene may be set.FIG. 5shows center area501set within image500. The height and width of the center area (shown here as center height=image height−delta_y_0−delta_y_1 and center width=image width−delta_x_0−delta_x_1) may be selected as some percentage or relative portion of the overall image height and width, such as based upon the global movement within the image expected/detected, the image content, the global depth model used, etc.

Having set the center area of the image, at block402of the illustrated embodiment the center area is separated into sub-areas. For example, the center area may be separated into equal size blocks forming columns and rows. Center area501of the embodiment illustrated inFIG. 5is separated into 6 columns and 4 rows.

At block403of the illustrated embodiment, global depth map generation logic120analyses the luminance change in each column and each row. For example, the statistic counts of pixels which have large luminance changes (e.g., larger than a pre-defined threshold) with the adjacent pixels for each block row and each block column may be determined to detect changes in the luminance of the image.

At block404of the illustrated embodiment the image global motion is determined using the detected luminance change information. For example, the global motion in the horizontal direction may be tracked through finding the block column of the center area with the minimum statistics counts of the luminance change. Similarly, the global motion in vertical direction may be tracked through finding the block row of the center area with the maximum statistics counts of the luminance change. If more than one minimum columns or maximum rows are detected, the one closest to the image center will be chosen according to embodiments of the invention.

As an example of the above global motion determination operation, the global center point for a previous image in the scene may have been determined to be at position (c_x, c_y) as denoted by center point511. Using the foregoing analysis of luminance change the updated global center point for the image may be determined to be at position (u_c_x, u_c_y) as denoted by updated center point512. A vector from the center point (e.g., center point511) to the updated center point (e.g., updated center point512) represents the global motion of the image.

Although the foregoing operation determines image global motion as may be utilized for image global depth map generation in accordance with the concepts herein, embodiments provide further refinement of the global motion determination. Accordingly, block405of the illustrated embodiment refines the image global motion as determined at block404, such as to maintain temporal consistency, to avoid the effects caused by erroneous global motion detection, etc. For example, the image global motion determined at block404may be refined by thresholds, which means the absolute difference of u_c_x and c_x and the absolute difference of u_c_y and c_y should be less than pre-defined or auto-generated thresholds, at block405.

Continuing with the foregoing example, the updated global center point determined to be at position (u_c_x, u_c_y) as denoted by updated center point512may be refined to be at position (r_c_x, r_c_y) as denoted by refined center point513. A vector from the center point (e.g., center point511) to the refined center point (e.g., refined center point513) may thus represent the global motion of the image as tracked in block302ofFIG. 3by global depth map generation logic120.

Detail with respect to operation of global depth map generation logic120to provide global motion tracking (e.g., as provided at block303ofFIG. 3) in an additional direction (e.g., providing global motion tracking in the Z direction), as may be used to enhance 3D effect and 3D image rendering according to embodiments is shown in flow600ofFIG. 6A. It should be appreciated that the functions of flow600of the illustrated embodiment may be performed, for example, by code segments implemented as global depth map generation logic120.

At block601of the illustrated embodiment the center region of the image for which an image depth map is to be generated is set. For example, the center region of a single color image of a moving video scene may be set.FIG. 7shows center region701set within image700(i.e., center region701aset in image700arepresenting image700having a relatively small saliency area size, center region701bset in image700brepresenting image700having a relatively medium saliency area size, and center region701cset in image700crepresenting image700having a relatively large saliency area size). The height and width of the center region may be selected as some percentage or relative portion of the overall image height and width, such as based upon the global movement within the image expected/detected, the image content, the global depth model used, etc.

Having set the center region of the image, at block602of the illustrated embodiment the size of a saliency area within the center region is detected. Global depth map generation logic120of embodiments may analyze the frequency and spatial information to detect a change in size of the saliency area. For example, the image for which an image depth map is currently being generated may be compared to a previous and/or subsequent image from the scene to detect changes in the frequency and spatial domain of the image to detect a change in the size of the saliency area.

At block603of the illustrated embodiment a determination is made as to whether a change in the size of the saliency area is indicated by the detected saliency area size. For example, the saliency area of the current image may have changed from the relatively small saliency area size of saliency area711ato the relatively medium saliency area size of saliency area711b. Conversely, the saliency area of the current image may have changed from the relatively medium saliency area size of saliency area711bto the relatively small saliency area size of saliency area711a. Similarly, the saliency area of the current image may have changed from the relatively medium saliency area size of saliency area711bto the relatively large saliency area size of saliency area711c. Conversely, the saliency area of the current image may have changed from the relatively large saliency area size of saliency area711cto the relatively medium saliency area size of saliency area711b.

If the saliency area size is determined to have increased (e.g., saliency area711ato saliency area711bor saliency area711bto saliency area711c) at block603, processing according to the illustrated embodiment proceeds to block604. At block604of the illustrated embodiment the global depth model's parameters are changed (e.g., the parameters of the depth model for calculating the FGDM are changed) to make sure the depth value of the center area of the FGDM is increased compared with the depth value of the center area of the previous or subsequent image's FGDM in correspondence with the image view having moved (e.g., zoomed in). In operation according to embodiments, all the parameters changing step size are constrained by thresholds. Such thresholds may be utilized to ensure that the depth value of current image's FGDM is not changed too much compared with the previous/subsequent image's FGDM. The step of changing the parameters is constrained, according to embodiments, by thresholds to keep the FGDM temporal consistent and avoid FGDM sudden changing caused by saliency area error detection.

If, however, the saliency area size is determined to have decreased (e.g., saliency area711cto saliency area711bor saliency area711bto saliency area711a) at block603, processing according to the illustrated embodiment proceeds to block605. At block605of the illustrated embodiment the global depth model's parameters are changed to make sure the depth value of the center area of FGDM is decreased compared with depth value of the center area of the previous or subsequent image's FGDM in correspondence with the image view having moved (e.g., zoom out). As discussed above, all the parameters changing step size are constrained by thresholds according to embodiments of the invention.

FIG. 6Bshows exemplary operation of global depth map generation logic120to modify the parameters of sphere model to change the FGDM based on saliency area detection consistent with the foregoing. It should be appreciated that in the exemplary embodiment ofFIG. 6B, in implementing the depth model parameter changes to increase the depth value of the center region (e.g., block604ofFIG. 6A), u_c_d>c_d (increased) and u_t_d<t_d && u_b_d<b_d (decreased) Likewise, in implementing the depth model parameter changes to decrease the depth value of the center region (e.g., block605ofFIG. 6A), u_c_d<c_d (decreased) and u_t_d>t_d && u_b_d>b_d (increased).

The global depth model, as changed at either bock604or605to correspond with the image view having moved, may therefore more closely represent the image being processed. Global depth map generation logic120thus use this global depth model in generating the FGDM and IGDM used to provide generation of an image depth map which more accurately represents the local depth discontinuity within image.

FIGS. 8A-8Dillustrate operation of global depth map generation logic120implementing a calculating based technique for generating a FGDM and for generating an IGDM from the FGDM (e.g., blocks304and305ofFIG. 3) according to embodiments of the invention. In particular,FIGS. 8A-8Dshow stretching (FIGS. 8A and 8C) and shrinking (FIGS. 8B and 8D) the FGDM in association with 4 different global motion directions, wherein the respective detected global motion directions are indicated by arrows801a-801d.

In operation according to the illustrated embodiments, the FGDM is stretched or shrunk in different size and direction according to the global motion to make the updated center point position to be the center of the updated FGDM. For example, in the example ofFIG. 8A, the global motion direction detected in association with movement of center point811a(position c_c_x, c_c_y) to center point812a(position u_c_x, u_c_y) stretches previously calculated FGDM821a(e.g., calculated from the global depth model, as may have been stretched or shrunk in accordance with global movement in other images of the scene) having center point811ato FGDM822ahaving center point812a. In the example ofFIG. 8B, the global motion direction detected in association with movement of center point811b(position c_c_x, c_c_y) to center point812b(position u_c_x, u_c_y) shrinks previously calculated FGDM822bhaving center point811bto FGDM821bhaving center point812b. In the example ofFIG. 8C, the global motion direction detected in association with movement of center point811c(position c_c_x, c_c_y) to center point812c(position u_c_x, u_c_y) stretches previously calculated FGDM821chaving center point811cto FGDM822chaving center point812c. In the example ofFIG. 8D, the global motion direction detected in association with movement of center point811d(position c_c_x, c_c_y) to center point812d(position u_c_x, u_c_y) stretches previously calculated FGDM822dhaving center point811dto FGDM821dhaving center point812d.

As can be appreciated from the foregoing, the FGDM size will be larger or equal to image size according to embodiments herein. However, only the depth values for positions in a sub-region of the updated FGDM corresponding to the current image are calculated according to embodiments of the invention. Accordingly, embodiments of global depth map generation logic120operates to determine a sub-region of the FGDM corresponding to the current image, whereby the identified FGDM sub-region is selected as the IGDM for use in calculating an image depth map.

For example, the following calculations may be utilized for identifying the sub-region within the FGDM. Derive the (c_x, c_y) of the new FGDM according to the updated coordinate (u_c_x, u_c_y). If u_c_y is less than half of the image height, the image should correspond to the bottom sub-region of the FGDM (If u_c_y<img_height/2, =>u_h=(img_height−u_c_y)*2=>c_y=u_h/2). Otherwise, the image should correspond to the top sub-region of the FGDM (If u_c_y>img_height/2, =>u_h=u_c_y*2=>c_y=u_h/2). If u_c_x is less than half of the image width, the image should correspond to the right sub-region of the FGDM (if u_c_x<img_width/2, =>u_w=(img_width−u_c_x)*2=>c_x=u_w/2). Otherwise, the image should correspond to the left sub-region of the FGDM (if u_c_x>img_width/2, =>u_w=u_c_x*2=>c_x=u_w/2).

Using calculations, such as those of the exemplary embodiment above, global depth map generation logic120operates to determine a sub-region of the FGDM corresponding to the current image. For example, sub-region831amay be selected as the IGDM from FGDM822a, sub-region831bmay be selected as the IGDM from FGDM821b, sub-region831cmay be selected as the IGDM from FGDM822c, and sub-region831dmay be selected as the IGDM from FGDM821d. The global depth value for each position in the image may be calculated using the corresponding sub-region according to the updated parameters c_x, c_y, c_d, t_d, b_d and the pre-defined formula.

FIGS. 9A-9Dillustrate operation of global depth map generation logic120implementing a mapping based technique for generating a FGDM and for generating an IGDM from the FGDM (e.g., blocks304and305ofFIG. 3) according to embodiments of the invention. In particular,FIGS. 9A-9Dshow mapping the IGDM from the FGDM.

In operation according to embodiments, the FGDM may be pre-generated using techniques, such as deriving the global depth map model based on scene analysis (wherein the model is an input parameter for operation of global depth map generation logic of embodiments of the invention), to generate a FGDM for the scene of which the image is a part and is thus larger than or equal to the size of the image. For example, after scene analysis and the deriving the global depth map model, by using the conventional techniques, the global depth map with width equal to image width and height equal to image height is generated according to the model. In operation of embodiments of the present invention, the FGDM may be generated with the width equal to (image_width+delta_w) and the height equal to (image_height+delta_h), and delta_w>=0, delta_h>=0. In order to derive the larger FGDM, the parameters for the model (sphere, planar or others) should be changed.

The following provides exemplary embodiment parameters and model (formula) for calculating a FGDM. The parameters utilized may, for example, comprise c_x (horizontal coordinator of the depth map center (W/2)), c_y (vertical coordinator of the depth map center (H/2)), c_d (depth value of the depth map center), t_d (depth value of the top left pixel and the top right pixel in the depth map), and b_d (depth value of the bottom left pixel and bottom right pixel in the depth map), as represented by the embodiment illustrated inFIG. 9D. A formula for generating depth value at position (x, y) may, for a two sphere global depth model, be expressed as:
delta_x=x−c_x,delta_y=y−c_y;dis_max=sqrt(c_x*c_x+c_y*c_y)  (1)
top_d=(delta_x*delta_x+delta_y*delta_y)/(dis_max*dis_max)*(t_d−c_d)+c_d(2)
bot_d=(delta_x*delta_x+delta_y*delta_y)/(dis_max*dis_max)*(b_d−c_d)+c_d(3)
The two sphere models may be fused together according to the pixel position in vertical direction as represented below:
alpha=c_y>y?(c_y−y)/(H/2):0  (4)
Such that:
FGDM(x,y)=(int)(alpha*top_d+(1.0-alpha)*bot_d).

Irrespective of the particular way in which the FGDM is generated, embodiments of global depth map generation logic120implementing a mapping based technique generate an IGDM from the FGDM by mapping the IGDM from a sub-region of the FGDM. For example, a global depth map may be obtained by shifting the corresponding image sub-region inside the FGDM to make the updated coordinate (u_c_x, u_c_y) in image coordinate system to be the center of the FGDM. Accordingly, the IGDM of embodiments is mapped to a sub-region of the FGDM according to the global motion detected in the image. As shown inFIG. 9A, FGDM900is larger than the image size (represented by the size of IGDM901mapped into the FGDM in association with a previously processed image), and thus is adapted to accommodate global motion of the images within a scene. The global motion detected with respect to the current image in the exemplary embodiment illustrated inFIG. 9Ais to the left and up. Thus, IGDM902for the current image is offset from IGDM901for the previous image within the scene to the right and bottom (offset horizontally by (c_c_x−u_c_x) and vertically by (c_c_y−u_c_y)). That is, the sub-region selected for the IGDM for the image currently being processed is shifted in a direction according to the global motion to get the image global depth map. As can be appreciated from the global depth map representations ofFIGS. 9B and 9C, IGDM902generated for the current image from FGDM900is adjusted, as compared to IGDM901, for the image currently being processed based upon the global movement detected.

Detail with respect to operation of depth analysis logic110in providing localized image depth information for use in generating image depth map102according to embodiments of the invention is shown inFIG. 10. It should be appreciated that functionality provided as represented inFIG. 10of the illustrated embodiment may be performed, for example, by code segments implemented as depth analysis logic110.

In providing localized depth map refinement, embodiments of depth analysis logic110utilize a relative relation between color components to adjust the local depth map and improve the 3D visual perception. For example, assume that R and B are two different color components in a color space, where R represent the warm pigment and B represents the cool pigment. Local depth map adjustment metrics may be generated by depth analysis logic110, using a relative relation of the R and B color components, for use with respect to the foregoing IGDM in generating image depth map102by depth fusion logic130. In operation according to embodiments, the local depth adjustment will be made only for the pixels with the value of R color component larger than the value of B color component. The local depth adjustments implemented according to embodiments herein will be proportional to the relative relation between the color components. For example, for the pixels with the same R color component, the adjustment ratio implemented according to embodiments will be in direct ratio with the difference between R and B. For the pixels with the same difference between R and B, the adjustment ratio implemented according to embodiments will be in inverse ratio with the value of R color component.

The graph ofFIG. 10illustrates defining the relative relation between R and B by ratio for use in localized depth map refinement according to embodiments herein, wherein R1 and R2 shown inFIG. 10are different values of R color components. The relative relationship between the color components may be quantized as a ratio computed, for example, according to the following equation:
Ratio=(R−B)/(R+B)
where, (R>=B), and(R+B)!=0.  (5)
R1 and R2 in theFIG. 10show that for the pixels with the same difference between R and B, the adjustment ratio implemented according to embodiments will be in inverse ratio with the value of R color component. That means, if R1>R2, and (R1−B1)==(R2−B2), then the ratio value ratio 1 calculated by R1 should be less than the ratio value ratio 2 calculated by R2.

Depth analysis logic110of embodiments herein may utilize such ratios representing the relative relationship between color components through various techniques to refine depth values in a depth map corresponding to the respective pixels. For example, a refining technique utilizing such ratios may operate to compute a depth adjustment constraining metric, m_ratio, as m_ratio=1.0+k*Ratio, where 0<k<=1, used to constrain the refining step size such that local_depth=local_depth*m_ratio. Alternatively, a refining technique utilizing such ratios may operate to compute local_depth=local_depth+Ratio*CONST_DEPTH_INCREMENT, where CONST_DEPTH_INCREMENT is a metric used to constrain the refine step size (e.g., CONST_DEPTH_INCREMENT may be an integer value, such as 32).

It should be appreciated that the use of color component relative relation in accordance with embodiments of the invention differs significantly from the techniques heretofore employed for localized depth map refinement. For example, many prior implementations use only a separate color component to derive the local depth map through a heuristic method (e.g., the warm color component used to indicate a near object while the cool color component used to indicate a far object).

Having generated local depth map adjustment metrics by operation of depth analysis logic110and generated an IGDM through operation of global depth map generation logic120, depth fusion logic130operates to generate image depth map102using these two inputs. For example, depth fusion logic130may utilize the local depth map adjustment metrics to adjust the depth values at positions in the IGDM corresponding to their respective pixel positions, thereby refining the IGDM into an image depth map which more accurately represents the local depth discontinuity within the image.