Advanced image processing for immersive media applications using sparse light-fields

Starting by the year 2013, head-mounted displays in combination with virtual reality or augmented reality have become a mass phenomenon in the area of computer games . In contrast to classical displays, head-mounted displays enable a user to move around in a virtual world and to review the embedded objects from different perspectives. Therefore, the scene and all objects need to be represented as a 3D model or in a comparable representation (e.g. as a light-field). Creating and replaying such content is especially challenging for dynamic live-action scenes, possibly with human actors. Firstly, many cameras need to capture the scene simultaneously from many perspectives. Afterwards, specialized software processes the footage before a user can review the scene. Integrating a live-action scene captured with a multi-camera array into a virtual environment forms the central application of this work. To reach this goal, this thesis reviews the whole processing pipeline including camera setup, camera calibration, depth-reconstruction and novel view synthesis and optimizes the processing pipeline at critical points. A special focus is on the necessary software. Here, this thesis contributes a series of novel software plugins that enable a user in a creative environment (e.g. in the area of media production) to perform such a task. Such a software needs to be robust, easy to use and needs to integrate smoothly into existing workflows. In the area of multi-camera calibration (chapter 4) this thesis contributes a novel, specialized and reliable calibration technique. The proposed self-calibration procedure does not rely on special calibration objects. Instead, the method exploits the high mechanical stability and manufacturing precision of the underlying camera array. In contrast to classical self-calibration techniques, the proposed method uses a new optimization concept that significantly reduces the number of unknown parameters. This concept is also beneficial for existing calibration procedures. Multi-camera disparity estimation and filtering forms the second major chapter and adds two contributions. Firstly, a new algorithm for near-real time depth reconstruction from planar camera arrays. With this algorithm (and the view-synthesis method from chapter 6), users e.g. on a film set can directly review the captured footage with proper motion parallax. This way, a creative director can detect potential problems with the artistic performance at an early stage. Secondly, this chapter presents a new concept for off-line multi-camera depth reconstruction. Instead of a monolithic algorithm, the concept builds up on modular elements where each module implements a specific function such as stereo-matching or disparity filtering. A user in a post-production environment can then setup and parametrize a scene-specific depth-reconstruction pipeline. The plugin suite Realception for Nuke is therefore a direct result of this thesis. Comparisons show that an adapted pipeline based on block-matching as initial depth-reconstruction technique competes with or even outperforms highly complex, monolithic depth-reconstruction methods. Chapter 6 finally introduces the real-time rendering method which is based on depth-image based rendering. Here, the combination of light-field rendering and classical computer graphics pipelines forms the major contribution. In the absence of an explicit 3D model, this chapter proposes an intermediate screen element that transfers pixels coming from a light-field representation into a 3D environment that is essentially based on explicit geometric models (e.g. meshes). Experiments show that this approach yields high image quality. A realistic test shot and a demo setup finally demonstrate the functionality and quality of the whole capture and processing pipeline.