3D Image Understanding for Urban Scenes

Throughout human history, replicas of the real world have played an invaluable role in providing spatial information vital for the exploration and understanding of our environment. Starting with simple maps, recent technological advancements have culminated in sophisticated virtual models, e.g., Apple Maps, swissTLM3D from Swisstopo, etc. These reality capture solutions have great implications for the continuing challenges of 3D mapping and scene reconstruction, and allow for complex applications such as navigation, urban planning, and virtual reality, to name a few. Despite a high level of automation, the generation of such products still relies on human interaction, using great cost and effort. Contemporary technologies in photogrammetry, computer vision, and mapping typically employ image or range scan data, and recreate human habitats based on either recognizing geometry and shapes, or by semantic understanding, whereby objects are classified into semantically meaningful entities, differentiating between a house, tree, etc. However, the dynamic and complex nature of the real world often leads to mapping inaccuracies, even when well-established techniques are applied. For example, a computer can misinterpret a facade with many windows when they reflect objects such as cars or trees. In the geometric realm, occlusions such as a tree in front of a facade can lead to wrong or incomplete shapes. The clean-up of such artifacts is essential and embodies the demand for manual input. To overcome such problems, the work in this thesis creates synergy between the geometric reconstruction and semantic understanding approaches. Revisiting the above example, a correct classification of the building facade can be achieved through geometric awareness, i.e. if the computer understands that vertical and flat surfaces are buildings, rather than cars or trees. Alternatively, knowing the semantic context – a building would be situated behind a tree – allows for a superior geometric reconstruction with proper spatial distinction between the two objects. Addressing these two problems jointly leads to superior interpreted virtual 3D models, and, a reduction in the need for human assistance when annotating 3D reconstructions with semantic classes. As such, this combination is similar to a human operator who also holistically interprets images when extracting measurements from them. Exploiting semantics for 3D reconstruction and vice-versa can be achieved through so called class-specific regularization of the virtual model, which enables mutual influence between these modalities. This concept is explored in this thesis. Existing semantic 3D reconstruction methods have been limited to small scenes and low resolution so far, due to their large memory footprint. To push the boundaries of their application range, an adaptive volumetric multi-resolution formulation is proposed in §4, which scales to much larger regions. §5 combats the problem of retrieving fine details in semantic 3D models. Towards this end, the method alternates between updating shapes and classes of a semantic triangular mesh. Beyond reconstruction and visualization, virtual 3D models, with or without semantics, should support a wide variety of practical tasks. In this light, §6 investigates 3D representations from an application perspective by assessing their utility in performing specific reasoning tasks.