## 4 Depth Estimation Algorithm  
The used single recording plenoptic camera based depth estimation algorithm is based on [6]. However, it is modified to completely exploit the advantages of the hardware realization. The modifications include the usage of a non-parametric transform and a respective cost function for finding the correspondence with minimum error.  
In order to model the main camera thin lens camera model can be used:  
\[\frac{1}{f_{m}}=\frac{1}{a_{m}}+\frac{1}{b_{Lm}+b_{L}}. \tag{1}\]  
In an MLA of the plenoptic camera, micro-lenses are arranged in a specific order, such as hexagonal. Due to such formation, a classic stereo camera paradigm can be considered for a pair of adjacent micro-lenses to calculate the depth of the scene, as shown in the Figure 3. Using the thin lens model, plenoptic camera is represented in 2D space as parallel planes: object plane \(\rm{O}_{P}\), main lens plane \(\rm{M}_{P}\), MLA plane \(\rm{M}_{LP}\), sensor plane \(\rm{S}_{P}\) and image plane \(\rm{I}_{VP}\). Two micro-lenses \(\rm{L}_{i}\) project points \(\rm{P}_{si}\) onto \(\rm{S}_{P}\) that is placed at the distance of \(b_{Ls}\) from the \(\rm{M}_{LP}\). Rays from \(\rm{L}_{1}\) and \(\rm{L}_{2}\) intersect at virtual point \(\rm{P}_{V}\) onto the \(\rm{I}_{VP}\). Moreover, \(d_{i}\) refers to the distance onto the \(\rm{S}_{P}\) from the principal axis of the respective \(\rm{L}_{i}\), and \(b\)  
Figure 2: (**a**) Raw image captured by plenoptic camera (**b**) part of the image displaying micro-images (**c**) resulting image after applying refocusing (**d**) estimated sparse depth map of the scene.  
represents the baseline distance between the micro-lenses. The triangulation of this setup results in:  
\[\frac{\Delta d}{b}=\frac{b_{Ls}}{b_{L}}. \tag{2}\]  
Whereby \(\Delta d\) represents disparity that equals to: \(d_{2}-d_{1}\). The distance of \(\mathrm{P_{v}}\), represented as virtual depth (\(v\)), is estimated with respect to intrinsic parameter \(b_{Ls}\) of the camera that is calculated during the calibration process. Thus, \(v\) is computed by the relative distance of \(b_{L}\) with respect to \(b_{Ls}\), which is only possible if the projected point is focused by minimum two micro-lenses.  
\[v=\frac{b_{L}}{b_{Ls}}=\frac{b}{\Delta d}. \tag{3}\]  
The baseline distance between two adjacent micro-lenses equals to its diameter. So Equation (3) becomes:  
\[v=\frac{d_{L}}{\Delta d}. \tag{4}\]  
By considering \(\mathrm{M_{LP}}\) as a reference plane, 3D points can be represented as: center of a micro-lens \(\mathrm{L_{i}}(c_{xi},c_{yi},0)\), projection of a point \(\mathrm{P_{s}}(x_{si},y_{si},b_{Ls})\) onto \(\mathrm{S_{p}}\), and the virtual point \(\mathrm{P_{v}}(x_{v},y_{v},v)\) on \(\mathrm{I_{vp}}\). With the help of [PERSON]'s theorem:  
\[\frac{\|\mathrm{P_{v}}-\mathrm{P_{s1}}\|}{\|\mathrm{P_{s1}}-\mathrm{L_{1}}\|}= \frac{b_{L}-\mathrm{b_{Ls}}}{\mathrm{b_{Ls}}}. \tag{5}\]  
\[\mathrm{P_{v}}=\mathrm{L_{i}}+v\cdot(\mathrm{P_{si}}-\mathrm{L_{i}}). \tag{6}\]  
Similarly:  
\[x_{v}=c_{xi}+v\cdot(x_{si}-c_{xi}). \tag{7}\]  
\[y_{v}=c_{yi}+v\cdot(y_{si}-c_{yi}). \tag{8}\]  
To calculate the actual distance of the object \(a_{m}\) from the main lens, Equation (1) can be used by substituting \(b_{m}=b_{Lm}+vb_{Ls}\), which results in:  
\[a_{m}=\frac{f_{m}(b_{Lm}+vb_{Ls})}{b_{Lm}+vb_{Ls}-f_{m}}. \tag{9}\]  
Figure 3: Plenoptic camera model with two micro-lenses and corresponding planes.