Abstract:
A method for converting a 2D images and videos to 3D includes applying a set of pre-defined heuristic rules to assign a depth value for each pixel of a two-dimensional (2D) image source based on pixel attributes to generate an initial default depth map, refining the pre-defined heuristic rules to produce customized heuristic rules, applying the customized heuristic rules to the initial default depth map to produce a refined depth map, and rendering a three-dimensional (3D) image in a predefined format using the refined depth map.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims benefit of U.S. provisional patent application Ser. No 61/897,106, filed Oct. 29, 2013, which is herein incorporated by reference. 
     
    
     FIELD OF THE INVENTION 
       [0002]    Embodiments here relate generally to the field of 2D to 3D video and image conversion performed either in real time or offline, with application in consumer image/video editing software, consumer 3D display devices such as TVs, game consoles, mobile devices etc., consumer satellite and cable boxes, electronic billboards and displays for commercial advertisement, and post-production professional video editing software or solution for converting existing 2D movies and videos to 3D. More particularly, embodiments relate to a method and apparatus for extracting depth information automatically and/or semi-automatically from various visual cues in a monocular image and using the said depth information to render the image in 3D for different 3D display technologies and formats. 
       BACKGROUND 
       [0003]    The rising sale of 3D-enabled TVs and personal devices in the consumer segment, releasing of new and old movies in 3D and increasing use of large screen electronic billboards which can display attention grabbing 3D-images for advertising or informational purposes, has increased the need for creating 3D-content. The ability to convert existing 2D content to 3D content automatically or with limited manual intervention can result in large cost and time saving and will grow the 3D-content creation market even further. 
         [0004]    Traditionally, converting 2D videos to 3D for professional application consists of very labor intensive process of roto-scoping where objects in each frame are manually and painstakingly traced by the artist and depth information for each object is painted by hand. This traditional 2D to 3D conversion suffers from disadvantages. Depending on the complexity of the scene in each frame, it may take several hours to several days to generate a depth map of a single frame. A 2-hour movie at 24 frames per second may contain up to one hundred thousand unique frames and this manual depth map creation can cost upwards of $200 per frame. Consequently, this method is very expensive and slow. 
         [0005]    On the low end of the 2D to 3D conversion, consumer 3D-TV sets have built in hardware that can automatically convert 2D video or image into 3D in real time. However, the 3D quality is extremely poor with hardly any depth effect in the converted 3D-image. Such fully automated method is obviously not acceptable by professional movie post-production houses. 
         [0006]    There have been numerous research publications on methods of automatically generating depth map from a mono-ocular 2D-image for the purpose of converting the 2D-image to 3D-image. The methods range from very simplistic heuristics to very complicated and compute intensive image analysis. Simple heuristics may be suitable for real time conversion application but provides poor 3D quality. On the other hand, complex mathematical analysis may provide good 3D-image quality but may not be suitable for real time application and hardware implementation. 
         [0007]    A solution to this quality versus difficulty dilemma is to start with an automated default lower quality 3D-image and provide ability to add additional manual editing capabilities to enhance the 3D image quality. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]      FIG. 1  shows an exemplary block diagram of the system, according to one embodiment of the invention 
           [0009]      FIG. 2  shows an exemplary transformation of an image frame as it is processed in the system pipeline. 
           [0010]      FIG. 3  and  FIG. 4  illustrate two exemplary graphical user interfaces (GUI) for user to add or modify rules for depth map estimation, according to one software embodiment of the invention. 
           [0011]      FIG. 5  illustrates a graphical user interface (GUI) for user to control depth map filters, according to one embodiment of the invention. 
           [0012]      FIG. 6  illustrates an exemplary method for generating depth map from the left and right eye views of a stero-3D image by finding disparity between left and right views for each object, according to one embodiment of the invention. 
           [0013]      FIG. 7  illustrates a flow chart for computing depth map from a 2D image source, according to one embodiment of the invention. 
           [0014]      FIG. 8  illustrates a flow chart for additional processing and filtering depth map to enhance and/or exaggerate 3D-effects, according to one embodiment of the invention. 
           [0015]      FIG. 9  illustrates a flow chart for computing depth map from a 3D-stero image source which contains a left eye view and a right eye view of the scene, according to one embodiment of the invention. 
           [0016]      FIG. 10  illustrates a system diagram, according to one embodiment of the invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0017]    Embodiments of the present invention relate to a method, apparatus, system, and computer program for generating automatically using set of pre-defined heuristic rules to generate a depth map from mono-ocular (2D) image source. Optionally, in a semi-manual mode, user can augment or replace pre-defined heuristic rules with user defined rules to generate superior quality depth map. The said depth map in conjunction with the original 2D image source can be used to generate 3D image in any format desired. The embodiments of the invention can take advantage of the computing power of general purpose CPU, GPU or dedicated FPGA or ASIC chip to process sequence of images from video frames of a streaming 2D-video to generate 3D video frames. Depending on the available processing capabilities of the processing unit and complexity and size of pre-defined rules, the conversion of 2D video frames to 3D can be done in real time in automatic mode. 
         [0018]    In one embodiment, the 2D to 3D conversion algorithm is implemented as a software application running on a computing device, such as a personal computer, tablet computer or smart-phone. A user receives a streaming 2D-video from the Internet or from a file stored on a local storage device and uses the application to automatically convert the 2D video to 3D and display it on the attached 3D display in real time. In one embodiment, the converted 3D-video can be stored back on the local or network storage device. In one embodiment, the user can modify or augment the pre-defined heuristic rules for depth map estimation and depth map filters to produce user-desired quality and format of 3D-image. In one embodiment, the user can save the custom heuristic rules for each 2D-image or a sequence of 2D-images in a control file using some pre-defined syntax such as XML and can play the said control file together with the 2D-image or 2D-image sequence to reproduce the 3D-image or image sequences or the depth map for the image or image sequences. 
         [0019]    In one embodiment, the 2D to 3D conversion process is implemented as a software application running on a computing device such as a personal computer, tablet computer or smart-phone. A user loads a video from a file stored on a local or network attached storage device and uses the application to automatically or in an interactive mode convert the 2D video to 3D and store it back offline on the local or network attached disk. In one embodiment, the user can adjust or augment the pre-defined heuristic rules for depth map estimation and depth map filters to produce user-desired quality and format of 3D-image. In one embodiment the user can adjust existing rules and add new rules through graphical user interface (GUI) of the application. In one embodiment, the user modified or added rules can be stored in a control file using some pre-defined syntax such as XML and can be read in by the 2D to 3D conversion application and applied in the conversion. 
         [0020]    In one embodiment, the 2D to 3D conversion algorithm is implemented in dedicated hardware such as an FPGA (field programmable gate array) or custom ASIC (application specific integrated circuit) chip. In one embodiment, the entire 2D to 3D video conversion system is implemented as a stand-alone converter box. In one embodiment, the entire 2D to 3D video conversion system is implemented a circuit board or a daughter card. In one embodiment, a stand-alone implantation of the conversion system can be attached to the output of a streaming video receiver, broadcast TV receiver, satellite-TV receiver or cable-TV receiver and the output of standalone converter box can be connected to 3D-displays. 
         [0021]    In one embodiment, the 2D to 3D conversion algorithm is implemented as a software application utilizing on the graphics processing unit (GPU) of a computing device such as a personal computer, tablet computer or smart-phone to enhance performance. 
         [0022]      FIG. 1  shows an exemplary block diagram of the 2D to 3D conversion process, according to one embodiment of the invention. In one embodiment, the process comprises of receiving single or a sequence of image frames. Each pixel of the image frame, singularly or as a group, is analyzed. Based upon either default depth rules or user specified depth rules, the process assigns a depth value to the pixels. In one embodiment, the depth value of the entire frame is stored as grey scale depth map image. In one embodiment, the raw depth map image is further processed and filtered according to default rules and/or user defined rules. In one embodiment, the processed depth map image is applied to the original 2D-image to calculate pixel displacements in by the render engine. Default and or user adjustments are applied to fine tune the 3D-rendering of the original 2D-image for the 3D-display device. 
         [0023]    Referring back to  FIG. 1 , in one embodiment the system comprises a 2D-video source  101  that can stream video either from local or remote source. The depth estimator  102  estimates the depth of each pixel in the image frame using default set of rules stored in a rules database  104 . An example of a default rule will be “if the position of the pixel is in upper third of the image frame and the color is within certain range of blue, and the intensity is greater than 60% then assign depth for this pixel the value for sky.” In one embodiment, the user can input additional rules interactively at  103  or through a file as illustrated by  104 . In one embodiment, output raw depth map  112  from  104  can be further refined, filtered and processed by depth enhancer  106  using default rule sets from  105  or user defined rule sets from  107 . In one embodiment, the output refined depth map  113  from  106  is used with the original 2D-image  111  by the render engine  108  to produce a 3D-image  119 . The rendering may be controlled by the user at  110 . 
         [0024]      FIG. 2  illustrates one embodiment of the images  111 ,  112 ,  113  and  119  as they go through transformation from one processing block to the next. The original image  111  comes from the 2D video source  101 . The image  112  results from the depth estimator  102 . The depth map enhancer  106  produces the image  113 . The process then renders the image  119  on the display  109 . 
         [0025]      FIG. 3  illustrates one embodiment of graphical user interface (GUI)  201  to enable the user to enter depth rules consisting of color, intensity and location of the pixel within the image frame. Block  202  illustrates one embodiment of specifying pixel color range as RGB values with offsets and intensity value. Block  203  illustrates one embodiment of bounding box region for the rule to apply.  201  also illustrates an embodiment of a preview window showing the result of applying the rules on depth map. 
         [0026]      FIG. 4  illustrates one embodiment of graphical user interface (GUI)  204  to enable the user to enter depth rules consisting of hue, saturation and intensity of the pixel within the image frame. The user makes these inputs through a series of sliders, or other user interface devices in  205 . 
         [0027]      FIG. 5  illustrates one embodiment of graphical user interface (GUI)  206  to enable the user to enter depth map filtering and processing. The GUI  206  also illustrates an embodiment of a preview window showing the result of applying the rules on depth map. 
         [0028]      FIG. 6  illustrates one embodiment of graphical user interface (GUI)  207  to enable the user to identify and associate similar objects in the left and right eye views manually using mouse selection operation. The user input region  208  also illustrates an embodiment showing the disparity between the same object in left and right eye views and the process uses this disparity to calculate depth value for pixels within the object. 
         [0029]      FIG. 7  shows a flowchart of one embodiment of a method to calculate the depth of each pixel with in the image frame. The process starts with the received 2D video frame at  301 . At  302  the process takes a pixel from the image, initialized a counter i and compares the pixel attributes against some or all the depth map rules in  304 . If the rule specified attributes are found in the pixel, the pixel depth is calculated using the matching rule, as shown in block  305 . If no rule matches pixel, the counter is incremented, checked to ensure it is less than a threshold count N, and a default depth value is assigned as shown in  308 . This process continues until all of the pixels in the frame are processed at  309 , producing the enhanced depth map at  310 . 
         [0030]      FIG. 8  shows a flow chart of one embodiment of a method to enhance a depth map image. Various default and or user specified filter operations can be applied to post process the raw depth map generated. The depth map is received, such as from  309  in the previous process, although the depth map may be produced by other means. Again, a counter is initialized at  402 . If the counter is below a previously decided count at  403 , the process moves to applying the filter for that iteration to the depth map at  405 . The counter is then incremented at  406  and the process returns to  403 . If the counter reaches its final count at  403 , he generated depth map can be optionally saved as a grey-scale image, as shown in block  407 . The 3D image is then rendered at  408 . 
         [0031]      FIG. 9  illustrates one embodiment of a block diagram for estimating a depth map from a stereo 3D-image, which may result from a process other than that discussed with regard to  FIG. 3 . The stereo 3D image consists of a left eye view and a right eye view of the scene and is received at  501 . Initially, the depth may is assumed to have some default depth at  502 . Similar objects, referred to here as ‘blobs’ from the left and right eye views are identified either automatically using some attributes such as color, intensity, size and location or manually by user defined instructions. These blobs are added to the blob list. The user defined blob matches if they exist at  506 , result in an update to the blob list at  508 . The process then generates a depth map value for that pixel at  509 , which eventually results in the entire depth map used at  401  in  FIG. 8 . The disparity between and left and right eye view for the same object is a direct measure of the depth of the object, and this disparity data is used to estimate the depth of each pixel with in the object. 
         [0032]      FIG. 10  illustrates a system diagram, according to one embodiment of the invention. The instructions such as  614  for the method flow charts described above are stored on a memory  612  as machine-readable instructions that when executed cause a processor such as  608  in a specific system to execute the instructions. In one embodiment, the system is a mobile device. In another embodiment, the system is the stand alone computer. In another embodiment the system is an embedded processor in a larger system. Elements of embodiments are provided as a machine-readable storage medium for storing the computer-executable instructions. The machine-readable storage medium may include, but is not limited to, flash memory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, or other type of machine-readable storage media suitable for storing electronic or computer-executable instructions including disk storage  610 . For example, embodiments of the invention may be downloaded as a computer program which may be transferred from a remote computer to a requesting computer by way of data signals via a communication link  602  coupled to a network interface  604  for the requesting computer. The processor  608  executes the instructions to render the 3D image on the display  616 . 
         [0033]    Reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments. The various appearances of “an embodiment,” “one embodiment,” or “some embodiments” are not necessarily all referring to the same embodiments. If the specification states a component, feature, structure, or characteristic “may,” “might,” or “could” be included, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to “a” or “an” element, that does not mean there is only one of the element. If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element. 
         [0034]    While the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of such embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description.