# Monocular Real Time Full Resolution Depth Estimation Arrangement with a Tunable Lens
## 2 Materials and Methods
### Depth Estimation Algorithm  
Dense depth estimation algorithms must obtain the non textureless zones, and fill the textureless zones with the neighbour information, in this paper the result of obtaining the high frequency zones are defined as a sparse estimation of depth. The sparse estimation of depth is done by composing the focal stack using a defocus operator [29]. The amount of information estimated will heavily depend on the scene; if the image is lacking high frequencies, depth cannot be estimated. Once the sparse depth map has been computed, the unknown values must be filled. To this purpose, a sparse depth map and a composition of the focal stack is used as a starting point to fill the algorithm. Figure 4 below shows the steps in the algorithm pipeline. Let \(I(z)\) be the focal stack with shape \(W\times H\times N\) with \(W\), \(H\), \(N\) as height, width and the number of planes of the stack, respectively, the sparse depth map is defined as follows:  
Figure 4: Algorithm pipeline. \[SD=\max_{z}G(I)\bigcap_{i=2}^{n}\max_{z}(U_{i}(G(R_{\frac{1}{i}}(I)))) \tag{2}\]  
where \(G_{z}(I)\) is the 2D gradient magnitude function for each image of the stack, \(U_{i}\) is an area upsample function and \(R_{\frac{1}{i}}\) is an area downsample function, where \(i\) is the factor to resize and \(n\) the maximum number of pyramids, for our results \(n\) value is equal to 3. The multiscale approach avoids noise artifacts due to the elimination of small noisy pixels, the maximum gradients that exceed a tolerance factor with respect to the other gradient values of the stack will be considered. As only gradients are used, the resulting sparse depth map will have information only along the edges. However, a complete depth map is needed. Figure 5 shows a real sparse depth map obtained with our camera. The next step prior to generating a depth map is to evaluate the movement intra-stack, the idea of capture in non-static scenes generates artifacts due to the displacement occurred between each stack plane, the procedure is to eliminate the artifacts from the sparse depth-map using a Z-entropy function as named in the Equation (3), the regularization procedure does not enhance the movement because of the technique. \[H(S)+\sum_{z}^{n}I(z)\log_{2}I(z) \tag{3}\]  
where \(S\) is the stack, \(H\) the entropy and z is the index of each plane, to this procedure the input image is selected by bins, the idea is to have, at most, 2 bit to represent the different values in the same pixel along the z axis, avoiding big changes due to the movement. Another part of the algorithm implies inferring the unknown values in the sparse depth map. The final solution should satisfy the following constraints:  
* All the data points in the existing sparse depth map have to be present in the final solution. * There are no missing depth values. * The resulting depth map should be edge preserving and should be smooth within the map. Following the previous statement, the solution will be the one that minimizes the error function:  
Figure 5: Sparse depth map represented in JET color space (blue-near, red-far), black means unknown information. \[\lambda S(D,R)+\sum_{x,y}M(x,y)(D(x,y)-SD(x,y))^{2} \tag{4}\]  
where \(D(x,y)\) is the resulting depth map, \(M(x,y)\) is the binary mask of Equation (5), this parameter ensures that the original values of the sparse depth-map are not modified and \(S(D,R)\) is the smooth term weighted with the parameter \(\lambda\) which ensures that the depth map is smooth at the same time that preserves the edges present in the reference \(R\), the idea of \(S(D,R)\) is to fill the depth map values using the sparse depth-map, which is the first iteration but also using the color distance provided by the reference. \[M(x,y)= \begin{cases}0&\text{if SD(x,y) is uncertain}\\ 1,&\text{else}\end{cases}  \tag{5}\]  
The reference \(R\) is a simple composition of the input stack presented in Equation (6) and the smoothing term defined by Equation (7).