## 2 Light Field Camera Structure and Depth Extraction
### Depth Extraction Based on Optical Flow for Light Field Camera  
To obtain the depth information of an object by processing the 4D light field image, we propose depth extraction based on the optical flow for light field camera. This method extracts the local disparity between the sub-images using the optical flow method. Optical flow is a distribution of relative motion difference according to the distance between an imaging system and a subject [27]. By finding a corresponding pixel to a given pixel, the method calculates how quickly the pixel flows in the images [28]. Conventional methods of extracting the depth of the object by analyzing the elemental image is the sum of absolute difference (SAD), sum of squared differences (SSD), and sum of sum of squared differences (SSSD) while using  
Figure 1: Geometry structure of light field camera. MLA: micro lens array. the minimum difference function [18; 19]. These methods, however, have problems that the extracted depths are quantized and are not able to measure in a sub-pixel unit. To solve these problems, a method has been proposed for averaging the depth information that was extracted by comparing optical flows between sub-images and applying a median filter [24]. However, an optical flow method for LFC has not been suggested yet. Captured light field information of LFC also has very narrow baseline to compare the position of corresponding points [20; 21]. Therefore, sub-pixel-level depth extraction algorithms are essential for the LFC. The corresponding points between the images can be found by comparing the sub-image at the very center of the light field information and periphery sub-images. According to the specifications of the optical system, the depth information of certain objects is encoded with the disparity between the sub-images calculated by the optical flow value. Figure 2 shows the geometrical relationship between optical flow data and depth information in InIm and LFC pickup system. Optical flow (OF) means the value of optical flow between two corresponding points. The main difference of two optical systems is that, in the focused mode InIm, the axes of each microlens and image sensor are both parallel and perpendicular to the optical system. In case of pickup by the LFC, however, the MLA and the image sensor are differently located and are magnified depending on distances from the main lens to the optical elements. Since the magnification values of both optical elements are different, the microlens and the part of image sensors assigned to them are tilted out of the main optical axis depending on their positions. Because two capturing systems have different optical structures, the depth extraction method for light field camera should be reestablished. It is supposed that lens array in InIm and magnified MLA in the LFC have the same size. \(m_{1}\) and \(m_{2}\) are magnifications of MLA and image sensor, respectively. The variable \(p_{p}\) is the pixel pitch of the image sensor. As shown in checked pixels in Figure 2a, a point object in space can be repeatedly recorded on the image sensor by MLA. The core task of depth extraction is to reveal the relationship between the location of repeating points and depth information. In focused mode, InIm pickup, for example, let two checked pixels on the Figure 2a have a corresponding point relation. The rays recorded at two pixels start from the same point but pass different centers of microlenses. Because the distance between two corresponding points on image sensor is two times larger than the lens pitch, the optical flow is calculated as 2. In Figure 2a, the black pixel on a center lens image and other black pixels are supposed to be corresponding points to each other. Depending on the optical flow values, the depth of each corresponding point is determined. The position of the depth according to the calculated optical flow is represented as \(d_{\mathrm{OF}}\) in Figure 2a, and the depth increases linearly with the value of optical flow.