Abstract:
Provided are three-dimensional (3D) data processing apparatus and method. 3D stereoscopic image common format data preserving common 3D image information is defined by converting 3D stereoscopic image data of each format to apply the same 3D image processing algorithm to 3D data of various formats, and native format data of each format is maintained to maintain merits of each format, and a format-specific algorithm table is maintained with respect to a common 3D image processing function, whereby the same 3D image processing algorithm may be applied to various 3D data and a unique algorithm allowing for maintaining the merits of each 3D data may be applied.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority under 35 U.S.C. §119 to Korean Patent Application No. 10-2014-0006675, filed on Jan. 20, 2014, the disclosure of which is incorporated herein by reference in its entirety. 
     TECHNICAL FIELD 
     The present invention relates to an apparatus and method for processing three-dimensional (3D) data obtained through various sensors, and more particularly, to a 3D data processing apparatus and method for applying a 3D stereoscopic image processing algorithm to various types of 3D stereoscopic image data. 
     BACKGROUND 
     With the advancement of sensors for recording data having 3D information such as laser scanning, a stereoscopic camera, a structured light scanner, the Microsoft Kinect™, a light field camera, and the like, various types of 3D stereoscopic image data may be obtained. The respective sensors generate 3D data in different formats, and such data in different formats have the merits and demerits, which, thus, are stored in different manners and different algorithms are applied thereto to be processed according to purposes. 
     A framework of software for 3D stereoscopic image processing performs 3D stereoscopic image processing using various types of data formats. Since data formats are different, when the same stereoscopic image processing algorithm is applied, data formats are converted into intermediate data in a common format and the stereoscopic image processing algorithm is applied thereto. Thus, obtained 3D stereoscopic image data is converted into intermediate formats and the 3D image processing algorithm is subsequently applied to the converted intermediate format, not to the original obtained format. 
     However, there may be a native algorithm that may be used only in a data format, not a common algorithm, according to types of 3D image processing scheme and types of data. In this case, the native algorithm applied only to the corresponding format should be used to obtain better results in terms of speed and quality. 
     Also, data storing 3D information since it was obtained or generated (for example, a laser-scanned point cloud, 3D mesh generated through modeling by a user, and the like) has a significant difference in data characteristics from 3D image data based on a 2D image, so conversion of such data into a common format and application of a common algorithm thereto causes huge data loss. 
     SUMMARY 
     Accordingly, the present invention provides a three-dimensional (3D) data processing apparatus and method capable of applying the same 3D image processing algorithm to a 3D stereoscopic image data obtained by using various types of sensors and 3D stereoscopic image data generated by a designer regardless of data formats, and capable of applying a unique 3D image processing algorithm according to formats to utilize the merits of respective formats. 
     In one general aspect, a three-dimensional (3D) data processing apparatus includes: a 3D data obtaining unit configured to obtain 3D data; an integrated format data generating unit configured to integrate native format data of the 3D data and common format data generated by using the 3D data to generate integrated format data; and an integrated format data processing unit configured to process the integrated format data according to an image processing algorithm. 
     The integrated format data processing unit may check whether a native algorithm with respect to the native format data of the integrated format data exists, and when a native algorithm exists, the integrated format data processing unit may process the native format data of the integrated format data according to the native algorithm, and when a native algorithm does not exist, the integrated format data processing unit may process the common format data of the integrated format data according to a common algorithm. 
     When a native algorithm and a common algorithm for processing the integrated format data do not exist, the integrated format data processing unit may return an error. 
     The integrated format data processing unit may generate the common format data by using 2D image information and 2D depth information of the 3D data, and when the 3D data does not include depth information, the integrated format data processing unit may restore depth information of the 3D data through a depth restoration algorithm, and generate common format data. 
     In another general aspect, a three-dimensional (3D) data processing method includes: obtaining 3D data; generating common format data by using the 3D data; integrating native format data of the 3D data and the common format data to generate integrated format data; processing the integrated format data according to an image processing algorithm; and processing the integrated format data according to the image processing algorithm to output a generated stereoscopic image. 
     Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram illustrating a structure of a three-dimensional (3D) data processing apparatus according to an embodiment of the present invention. 
         FIG. 2  is a flow chart illustrating a process of a 3D data processing method according to an embodiment of the present invention. 
         FIG. 3  is a block diagram showing a computer system implemented according to an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. 
     The advantages, features and aspects of the present invention will become apparent from the following description of the embodiments with reference to the accompanying drawings, which is set forth hereinafter. The present invention may, however, be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. 
     The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of example embodiments. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. 
     Terms used in the present invention are defined as follows.
         3D stereoscopic image data: Data that has both 2D color or gray image information such as a stereo, light field, range scan data, and the like, and 3D information or that may be extracted or converted through a unique algorithm   3D data: Data that includes 3D stereoscopic image data and that includes 3D information such as 3D mesh, point cloud, or the like, not stereoscopic image data, and that may be converted into an image   3D stereoscopic image common format: Format of data converting 3D data to store 2D image information and 2D depth information required in a common algorithm   3D stereoscopic image native format: Format of data as the sum of original data such as stereo, light field, range scan, and the like, a state before 3D data is converted into a stereoscopic image and additional data such as depth data, or the like, obtained by processing the original data so as to be easily used.   3D stereoscopic image integrated format: Format of data obtained by adding a 3D stereoscopic image native format to 3D stereoscopic image common format data   3D image processing function, algorithm: Function or algorithm providing 3D conversion, or the like, such as a 3D special effect such as a fog impact, or the like, or view conversion by using a depth value in addition to a color value of a color or gray image   3D depth restoration algorithm: Algorithm of estimating or calculating a depth of each pixel from 3D stereoscopic image data not having a depth value of each pixel       

       FIG. 1  is a block diagram illustrating a structure of a 3D data processing apparatus according to an embodiment of the present invention. 
     The 3D data processing apparatus according to the present invention includes a 3D data obtaining unit  100 , an integrated format data generating unit  110 , and an integrated format data processing unit  120 . 
     The 3D data obtaining unit  100  obtains 3D data through a sensor or obtains 3D data generated by software. 
     The 3D data obtaining unit  100  may obtain a 3D stereoscopic image of an actual subject through a sensor according to types of 3D data, and in case of a subject that does not actually exists or a subject difficult to obtain by a sensor, data virtually created by using software such as a 3D modeling tool by a designer may be obtained as 3D data. 
     A sensor refers to any type of device capable of obtaining a stereoscopic image such as a stereoscopic camera, a light field camera, a laser scanner, and the like, and 3D data includes 3D stereoscopic image data and data that includes 3D information such as a 3D mesh, a point cloud, or the like, and that may be converted into an image, besides 3D stereoscopic image data. 
     The 3D data obtaining unit  100  designates original data of obtained 3D data, as native format data or designates data obtained by adding depth information to obtained 3D data, as native format data. 
     Native formats of data output from sensors are determined by types of sensor to be used. In case of a stereoscopic camera, left and right color images and a depth image calculated by using the left and right color images are a format, and in case of a light field camera, 4D light field data and a calculated depth value are a native format. In case of using a laser scanner, point cloud data or a 2D color image and a depth image are a native format. 
     Unique 3D algorithms are provided by native formats of respective 3D data, so types of algorithms provided for each format are defined. In case of a stereoscopic image, a left/right view conversion function is provided, and in case of a light field image, an upper/lower left/right view conversion function is provided. 
     In a case in which obtained 3D data is data other than 3D stereoscopic image data, the 3D data obtaining unit  100  may generate image information and depth information by applying rendering. 
     The 3D data obtaining unit  100  delivers information regarding the obtained 3D data and native format data of the 3D data to the integrated format data generating unit  110 , and in this case, the 3D data obtaining unit  100  may deliver image information and depth information generated from the 3D data together. 
     The integrated format data generating unit  110  generates integrated format data by integrating common format data generated by using the native format data of the 3D data and the 3D data delivered from the 3D data obtaining unit  100 . 
     In detail, the integrated format data generating unit  110  generates common format data by using 2D image information and 2D depth information of the 3D data. In a case that the 3D data includes 2D image information and 2D depth information, the integrated format data generating unit  110  may generate common format data by using the information included in the 3D data, or otherwise, the integrated format data generating unit  110  may generate 2D image information or 2D depth information from the 3D data and generate common format data. 
     Meanwhile, in a case that the 3D data does not include depth information, the integrated format data generating unit  110  may restore depth information of the 3D data through a depth restoration algorithm and generate common format data by using the restored depth information. 
     The obtained 3D data may include depth data (for example, a laser scanner, or the like) or may not (for example, stereoscopic data, or the like) according to types of data. Since the common format data requires depth data, if data does not have depth data, the integrated format data generating unit  110  calculates depth data by applying a 3D depth restoration algorithm to the data without depth data. When the depth data is restored, the integrated format data generating unit  110  integrates the restored depth data and the 2D image information to generate data of a 3D stereoscopic image common format, and combines the generated data with 3D stereoscopic image native format data to generate 3D stereoscopic image integrated data. 
     After generating the integrated format data by integrating the native format data of the 3D data and the common format data, the integrated format data generating unit  110  delivers the generated integrated format data to the integrated format data processing unit  120 . 
     The integrated format data processing unit  120  processes the integrated format data delivered from the integrated format data generating unit  110  according to an image processing algorithm, and outputs the processed stereoscopic image. 
     In detail, in processing the integrated format data, the integrated format data processing unit  120  checks whether a unique native algorithm with respect to the native format data of the integrated format data exists. When a unique native algorithm exists, the integrated format data processing unit  120  processes the native format data by applying the native algorithm, rather than using the common format data of the integrated format data to be processed. 
     If a native algorithm with respect to the native format data of the integrated format data does not exist, the integrated format data processing unit  120  processes the common format data of the integrated format data by applying a common algorithm and outputs a processed stereoscopic image. 
     Meanwhile, both a native algorithm and a common algorithm with respect to the 3D stereoscopic image processing function to be applied do not exist, the integrated format data processing unit  120  returns an error. 
     In case of applying a focus changing function as an example of 3D stereoscopic image processing to a stereo-based 3D stereoscopic image data and light field-based 3D stereoscopic image data, when focus changing is applied to stereoscopic data, since a native algorithm does not exist, a focus changing common algorithm is applied to common format data to create a result. In case of performing focus changing on light field-based data, a light field-dedicated native focus changing algorithm, without using the common algorithm, is applied to create a result. 
     By applying native algorithms by respective formats, better results in terms of speed and quality may be obtained, compared to a case of performing a common algorithm on common format data causing data loss by processing native data once. 
       FIG. 2  is a flow chart illustrating a process of a 3D data processing method according to an embodiment of the present invention. 
     The 3D data processing apparatus according to the embodiment of the present invention obtains 3D data obtained through a sensor or generated by software in operation S 200 . When the 3D data is obtained, common format data including 2D image information and 2D depth information of the 3D data is generated in operation S 210 , and native format data and common format data of the 3D data are integrated to generate integrated format data in operation S 220 . 
     When processing the integrated format data in operation S 230 , the 3D data processing apparatus checks whether a unique native algorithm with respect to the native format data of the integrated format data exists in operation S 240 . 
     When a native algorithm exists, the 3D data processing apparatus applies the native algorithm to the native format data of the integrated format data to process the same in operation S 250 , and outputs a processed stereoscopic image in operation S 260 . 
     When a native algorithm does not exist, the 3D data processing apparatus checks whether a common algorithm supporting an image processing function to be applied exists in operation S 241 , and applies the common algorithm to the common format data of the integrated format data to process the data in operation S 251 . 
     If both the native algorithm and the common algorithm with respect to the image processing function to be applied do not exist, the 3D data processing apparatus returns an error in operation S 252 . 
     In the embodiments of the present invention, 3D stereoscopic image data obtained through various types of sensors may be processed by applying the same algorithm and if each data format has different merits, a native algorithm for each format may be applied to utilize the merits thereof within a single framework. 
     Also, in the case of applying a common algorithm, since conversion into a common format has already been performed in advance, a process time taken for data conversion may be saved in processing a stereoscopic image. 
     An embodiment of the present invention may be implemented in a computer system, e.g., as a computer readable medium. As shown in  FIG. 3 , a computer system  300  may include one or more of a processor  301 , a memory  303 , a user input device  306 , a user output device  307 , and a storage  308 , each of which communicates through a bus  302 . The computer system  300  may also include a network interface  309  that is coupled to a network  310 . The processor  301  may be a central processing unit (CPU) or a semiconductor device that executes processing instructions stored in the memory  303  and/or the storage  308 . The memory  303  and the storage  308  may include various forms of volatile or non-volatile storage media. For example, the memory  303  may include a read-only memory (ROM)  304  and a random access memory (RAM)  305 . 
     Accordingly, an embodiment of the invention may be implemented as a computer implemented method or as a non-transitory computer readable medium with computer executable instructions stored thereon. In an embodiment, when executed by the processor, the computer readable instructions may perform a method according to at least one aspect of the invention. 
     A number of exemplary embodiments have been described above. Nevertheless, it will be understood that various modifications may be made. For example, suitable results may be achieved if the described techniques are performed in a different order and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Accordingly, other implementations are within the scope of the following claims.