High-Fidelity 3D Structure Recovery Using Hybrid Image-based Modelling

The design of 3D digital models has become a prominent and integral part of many applications ranging from science, engineering, education and architecture to biomedical visualisation and medicine. Traditionally, 3D models are created using modelling tools such as Maya, 3D Max or Blender. Although these applications enable graphic designers to construct highly realistic and complex 3D models, they have a steep learning curve and often require a considerable amount of training and artistic skills to use. These limitations restrict their usability for non-professionals. The introduction of specialised hardware, such as laser scanners, has simplified the creation of digital models from real physical objects. While many of these devices are capable of delivering highfidelity results, they are inherently expensive and typically pose stringent limitations on the dimensions, colours and surface properties of the object to be reconstructed. These fundamental drawbacks hinder their widespread use in practice. This thesis addresses the practical aspects associated with the image-based reconstruction of an intricate 3D scene from an unconstrained and unannotated image sequence. These image sequences can be acquired by a video camera or a handheld digital camera without the need for camera calibration. Once supplied with the input images, our system automatically processes and produces a 3D model. We propose a novel approach, which integrates uncalibrated Structure from Motion (SfM), shape-from-silhouette and shape-from-correspondence, to create a dense scene geometry of the observed scene. The advantage of combining silhouette-based and correspondence-based reconstruction approaches is that the new hybrid system is able to deal with both featureless objects and objects with concave regions. These classes of objects usually pose great difficulty for shape-from-correspondence and shape-from-silhouette approach. As a result, our approach is capable of producing satisfactory results for a large class of objects. Our approach does not require any a priori information about camera and image acquisition parameters. In the second stage, a high quality texture atlas is produced by parameterising the reconstructed objects using a segmentation approach, projecting suitable sections of input images onto the model, and combining them using a graph-cut technique. Holes in the texture atlas due to surface patches without projected input images are filled using a novel exemplar-based inpainting method which exploits appearance space attributes to improve patch search, and blends patches using Poisson-guided interpolation. The reconstructed 3D models sometimes exhibit unsatisfactory and erroneous mesh regions due to inherent properties of typical 3D scenes such as occluded or inaccessible parts, unfavourable lighting conditions, as well as insufficient and poor-quality visual inputs. In order to circumvent these challenges, inaccurate mesh regions must be classified and reconstructed. Mesh classification is achieved through the fusion of three uncertainty measures: geometric distances, normal direction distributions and texture discrepancies. Once successfully identified, these regions are segmented through a boundary-energy minimisation process. The objective here is to ensure that the extracted mesh regions enclose the previously identified low-confidence mesh segments, while at the same time exhibit the lowest possible genus and boundary length as well as demonstrate the highest possible degree of boundary smoothness. Establishing mesh segments with simple shapes enfolding the low-confidence mesh regions plays a vital role for the subsequent step, during which 3D geometric information of these regions is constructed by means of sketch-based modelling. Several evaluations using real and synthetic data demonstrate the robustness and effectiveness of our algorithm in enabling novice users to effortlessly and efficiently create complex 3D content using a standard off-the-shelf consumer-level camera. Our results compare favorably to those acquired by existing techniques.