## 2 Light Field Camera Structure and Depth Extraction
### Depth Extraction Based on Optical Flow for Light Field Camera
In case of LFC that is shown in Figure 2b, the depth of optical flow value is represented as \(D_{\mathrm{OF}}\). Figure 2 shows only the integer values of the optical flow for the sake of convenience though this algorithm has sub-pixel accuracy. The relationship between depth information and optical flow in LFC image should be analyzed to determine DOLF. To solve the depth quantization of extraction algorithm, the accuracy of depth map is secured by applying a median filter or averaging process to each depth map that was obtained by comparing each sub-image with the center sub-image. Equation (1) describes the relationship between the depth information and the optical flow in an LFC. Once a sub-image to be compared and optical flow of corresponding points are determined, the location of the point object is calculated simply by triangle proportionality. Let \(L\) be the pitch of the magnified microlens and \(P\) be the pixel pitch of the magnified image sensor, i.e., \(L=l_{p}\times m_{1}\) and \(P=p_{p}\times m_{2}\). When the resolution of one lens image is \(n_{x}\times n_{y}\), center index of lens is represented (\(i_{c}j_{c}\)), respectively. The distance between two optical elements can be calculated as \(\mid b_{2}-b_{1}\mid\) by lens formula. In this paper, it is assumed that optical flow only uses the result as compared with the center sub-image. The variable \(O_{h}\) means optical flow in the \(x\)-axis, and \(O_{v}\) means that in the \(y\)-axis. \(O_{h(i,j)}(x,y)\) or \(O_{v(i,j)}(x,y)\) denotes a 2D input function where the output is an optical flow value for a given lens index comparing the center sub-image. Note that input variable of \(O_{h}\) and \(O_{v}\) are omitted for simplicity in Equation (1). If \(x\) and \(y\) are coordinates in the center sub-image, then the depth for the center sub-image can be obtained as:  
\[ \begin{array}{ll}D_{OF}(x,y)=&\frac{b_{2}-b_{1}}{2}\left[\frac{1}{n_{x}-1} \sum\limits_{i=1,j\
eq k_{i}}^{n_{x}}\left|\frac{O_{h(i,j)\cdot L}}{O_{h(i,k) \cdot P\cdot n_{x}+(l_{x}-l)P-O_{h(i,k)\cdot L}}}\right|\right.\\ &+\left.\frac{1}{n_{y}-1}\sum\limits_{j=1,j\
eq k_{i}}^{n_{y}}\left|\frac{O_{h (i,j)\cdot L}}{O_{h(i,j)\cdot P\cdot n_{y}+(l_{x}-l)P-O_{h(i,j)\cdot L}}} \right|\right].\end{array}  \tag{1}\]  
The DOLF method is described in Equation (1). Using the above equation, one can generalize the case of extraction using the integral imaging technique. In integral imaging, the lens pitch and the image sensor assigned to it are assumed to have the same size. In other words, in the above equation, \(P\times n\) is equal to \(L\), which results in erasing the \(O\)-related term of the denominator. The value of \(b_{2}-b_{1}\) simply becomes the focal length of the lens array. This result is consistent with previous studies using integral imaging [24]. The above depth calculation method can be an example of the extraction method using optical flow. In the above equation, the optical flow for the center sub-image is calculated in the horizontal and the vertical directions and the values are averaged. Depth maps that were obtained from several sub-images are averaged again. However, depending on the situation and user preferences, various statistical techniques other than averaging can be adopted, such as the median filter. There is another difference between two pickup methods shown in Figure 2. In 3D object pickup using focused mode, InIm, there is no theoretical limit on the maximum value of optical flow for captured light field. If the sensor format of pickup system is large enough, that is, if the overall size of the lens array and image sensor are large enough, the maximum optical flow value that can be captured also increases. This is because the central optical axes are still parallel, even though a  
Figure 2: Depth extraction from captured light field using optical flow in **(a)** integral imaging and (b) light field camera. microlens is located at the periphery.