# Iterative Refinement of Uniformly Focused Image Set for Accurate Depth from Focus
## 1 Introduction
For the encoder part, they use VGG-16, a popular deep neural network for object recognition [23], without fully connected layers, and for decoder network they simply use flipped structure of the encoder network. It provides sharpness map (feature map) for each frame separately with single output depth map. However, such deep learning approach has limitations, such as fixed number of input frames (it depends on training samples). However, traditional DFF methods suffers from unstable approximation results with texture-less regions and object boundary regions. Indeed, texture-less region shows limited clue for the estimation of focus amount from only its visual appearance and object boundary contains sudden changes in depth that making patch based focus estimation difficult. In order to resolve such limitations, we suggest maximizing depth inference from neighbor pixels and neighbor frames globally optimizing overall changes of focus amount along the spatial and frame domains. In this work, we propose a new DFF method using single color camera. We investigate the appearance changes in spatial and frequency domains over differently focused image frames. In spatial domain, visual appearance change of low-textured region along the frames is difficult to be observed. In frequency domain, however, low-textured region gives slightly better observation over frames. Using both spatial and frequency domain observations, we can get robust depth estimation at each pixel even in low-textured regions. In order to achieve sub-frame level accuracy in depth estimation at each pixel, optimal location of best focused frame is estimated by fitting a second order (quadratic) polynomial curve on the dissimilarity measurements. Based on the estimated initial depth, we build a uniformly focused image set by pixel-wise adjustment of depth level. In other words, all-in-focus image and following uniformly out-focused images are created. We perform our depth estimation iterative manner on this uniformly focused image set that refine our depth estimation. During depth estimation process we create confidence map that helps to fill erroneous depth points caused by textureless uniform regions. After filling erroneous points, we use guided filter [24] using predicted depth to build its all-in-focus image with which we finally create cleaner depth image. Each estimated depth point is considered as 3D point, and surface normal of the point is computed from its 8 neighbors [25]. We interpolate them using Algebraic Point Set Surface(APSS) [26], and, finally, using marching cubes algorithm [27], we create polygonal surface representation. Our main contributions are the proposal of a new focus measure that uses both spatial and frequency domain information, creating a uniformly focused image set (UFIS) to achieve more accurate depth, as well as hole-filling process that fixes erroneous depth values caused by textureless region. Figure 1 illustrates the pipeline of our proposed method. We tested our proposed method with Light Field dataset (LFSD) [28], dataset from smartphone, synthetic, and teeth datasets (medical). We compare our method with existing focus-measure [29; 30; 31; 32; 33; 34; 35; 36; 37], and state-of-the-art DFF [1; 18; 19; 20; 22; 38; 39] methods quantitatively (Table 1) and qualitatively.