Defocus Modelling for 3D Reconstruction and Rendering
Creators: name: Bailey, Matthew James
Publisher: name: University of Surrey
Year: 2023
Subjects: Depth from Defocus
Thick Lens Model
Multi-View Reconstruction
Neural Rendering

description: Defocus formation from a finite aperture is a well-known phenomenon, occurring in many forms
description: of photographic media. Although defocus is often exploited for artistic reasons, a surprising
description: amount of information about the scene structure is encoded in the camera’s blurring function.
description: The aim of this work is to explore how this can be leveraged to recover 3D geometry from
description: scenes with complex reflectance.<br>
description: <br>
description: Depth from defocus (DFD) is a well-established field that aims to reconstruct scene geometry
description: from analysis of the defocus appearance, usually by modelling the camera as a thin lens.
description: While many existing methods achieve approximate depth maps suitable for some applications,
description: the majority are limited to geometrically inconsistent single-view reconstructions. The first
description: contribution generalises image formation to a thick lens, and proposes a novel calibration
description: procedure for accurate defocus modelling. This approach is shown to significantly outperform
description: traditional thin lens assumptions in macro-scale scene reconstruction.<br>
description: <br>
description: The second contribution generalises reconstruction to multiple views, and evaluates the com-
description: plementary properties of defocus and stereo information in a novel reconstruction framework.
description: Unlike conventional multi-view stereo (MVS), which depends on photometric consistency be-
description: tween views, DFD requires only a single viewpoint for reconstruction. This makes defocus-based
description: approaches naturally robust to view-dependent materials considered challenging for traditional
description: MVS. Conversely, textures which are invariant to defocus can be suitable for correspondence.
description: This complementary relationship is investigated to determine the benefits of combining defocus
description: information with stereo cues; with performance evaluated on per-viewpoint depth maps as well
description: as complete 3D reconstructions. The results demonstrate an improvement over DFD alone even
description: in specular and reflective datasets, and outperform state-of-the-art MVS.<br>
description: <br>
description: The third contribution explores the novel application of neural rendering to defocus modelling.
description: Specifically, the recent advances in deep learning are leveraged to solve for three latent variables
description: encoded as pixel intensities in a focal stack: the scene depth and radiance, and the camera point
description: spread function. This is in contrast to the majority of existing defocus-based literature, which
description: assume at least one of these variables is known. These quantities are disentangled by modelling
description: each as a multilayer perception network, and trained end-to-end on the appearance of each pixel
description: under different camera settings. This approach allows novel refocused images to be rendered
description: that accurately capture the bokeh produced by specular highlights with arbitrary aperture shapes.
description: Since the networks are trained according to a convolutional defocus model, the synthesised
description: images can generalise to unconstrained aperture diameters, and achieve depth of field effects
description: that exceed what was observed during training.<br>