Automatic 2D to 3D Video Conversion based on Global Nearest Neighbor Depth Learning

The conversion of existing 2D videos to 3D is proving commercially viable and fulfills the growing need for high quality stereoscopic videos. Most successful 2D-to-3D image and video conversion methods involve human operators, which is timeconsuming and costly. Automatic methods, which typically make use of a deterministic 3D scene model, have not yet achieved the same level of quality for they rely on assumptions that are often violated in practice. In this paper, there is a new method that is based on the radically different approach of learning the 2D-to3D video conversion from examples. The method is based on globally estimating the entire depth map of each frame that we are selected for the conversion. The approach proposes here is built upon a key observation and an assumption. The key observation is that among millions of 3D frames (images) available on-line, there likely exist many whose 3D content matches that of a 2D input (query) we wish to convert to 3D. We are also making an assumption that two frames that are photo metrically similar also have similar 3D structure (depth). Keywords— 3D frames, stereoscopic video, nearest neighbor classification, cross-bilateral filtering. The concept of stereoscopy has existed for a long time . But the breakthrough from conventional 2D broadcasting to realtime 3D broadcasting is still pending. However, in recent years, there has been rapid progress in the field’s image capture, coding and display which brings the realm of 3D closer to reality than ever before. The existing 2D to 3D conversion algorithms developed in the past years by various computer vision research communities across the world. Each algorithm has its own strengths and weaknesses. Most conversion algorithms make use of certain depth cues to generate depth maps. Among 2D-to-3D image or video conversion methods, those involving human operators have been most successful but also time-consuming and costly. Fully automatic methods typically make strong assumptions about the 3D scene. Although such methods may work well in some cases, in general it is very difficult to construct a deterministic model that covers all possible background and foreground combinations. In practice, such methods have not achieved the same level of quality as the semi-automatic methods. The last few years have seen a dramatic increase in the demand for stereo content. This has largely been driven by the commercial availability of multi viewer auto stereoscopic displays. Stereoscopy also called stereoscopic or is a technique for creating or enhancing the illusion of depth in an image or video by means of stereopsis for binocular vision. There are two basic approaches to 2D-to-3D conversion: one That requires a human operator’s intervention and one that does not. In the former case, the so-called semi-automatic methods have been proposed where a skilled operator assigns depth to various parts of an image or video. Based on this sparse depth assignment, a computer algorithm estimates dense depth over the entire image or video sequence. The involvement of a human operator may vary from just a few scribbles to assign depth to various locations in an image or frames to a precise delineation of objects and subsequent depth assignment to the delineated regions. 3D capable hardware like 3D TVs, Blu-Ray players, handheld gaming consoles, cell phones, still and video cameras are widely available in the market. But this hardware availability is not yet matched 3D content production. Current available 2D to 3D conversion methods has not achieved a high quality level. The most successful approaches are interactive. That means it involve human operators. The method proposes in this paper; carry the ―big data‖ philosophy of machine learning. In consequence, they apply to arbitrary scenes and require no manual annotation. The data driven approach to 2D-to-3D conversion has been inspired by the recent trend to use large image databases for various computer vision tasks, such as object recognition [8] and image saliency detection. In particular, this proposes a new method that is based on the radically different approach of learning the 2D-to-3D conversion from examples. The method is based on globally estimating the entire depth map of a query frame directly from a repository of 3D frames or images (frame+ depth pairs or stereo pairs) using a nearest-neighbor regression type idea. Early versions of learning-based approach to 2D-to-3D image or video conversion, suffered from high computational complexity . The method demonstrate the improved quality of the depth maps produced by our global method relative to state ofthe-art methods together with up to 4 orders of magnitude reduction in computational effort. While 2D-to-3D conversion based on learning a local point Transformation has the undisputed advantage of computational efficiency – the point transformation can be learned off-line and applied basically in real time – the same transformation is applied to images or frames with potentially Automatic 2D to 3D Video Conversion based on Global Nearest Neighbor Depth Learning