# Depth Estimation from a Hierarchical Baseline Stereo with a Developed Light Field Camera
###### Abstract  
This paper presents a hierarchical baseline stereo-matching framework for depth estimation using a novelly developed light field camera. The imaging process of a micro-lens array-based light field camera is derived. A macro-pixel map is constructed by treating each micro-lens as one macro-pixel in the light field's raw image. For each macro-pixel, a feature vector is represented by leveraging texture and gradient cues over the surrounding ring of neighboring macro-pixels. Next, the micro-lenses containing edges are detected on the macro-pixel map. Hierarchical baseline stereo-matching is performed by macro-pixel-wise coarse matching and pixel-wise fine matching, effectively eliminating matching ambiguities. Finally, a post-processing step is applied to improve accuracy. The lab-designed light field camera's imaging performance is evaluated in terms of accuracy and processing speed by capturing real-world scenes under studio lighting conditions. And an experiment using rendered synthetic samples is conducted for quantitative evaluation, showing that depth maps with local details can be accurately recovered. light field camera; macro-pixel map; hierarchical baseline; stereo-matching +
Footnote †: journal: _Sci._  
0000-0001-6500-0000-0000-0000-0000-0000-0000-0000-0000-00000-0  
Light field imaging has some distinct advantages: (a) for micro-lens array-based light field cameras, the FoV (field of view) and depth range can be dynamically adjusted by manipulating the focal length of the main lens, well-suited for capturing small-sized objects at a distance; (b) LF imaging can be easily integrated with other imaging systems [8]. Classical stereo-matching methods are only valid on viewpoints with wide baselines (such as a baseline of 32, 64, or 128 pixels), which are simple in theory and execute rapidly. Due to the large baseline and sparse angular sampling, stereo-matching encounters challenges leading to depth ambiguities and matching inconsistencies in occlusions, repeated textures, or textureless regions, as well as extreme lighting conditions. Existing methods often enforce a consistency constraint and utilize neighboring depth distributions to fill in gaps [9]. However, the sub-aperture images decoded from light field imaging have an extremely narrow baseline, typically ranging from 3 to 5 pixels for Lytro Illum. This results in inconsistent and sparse matching, accompanied by blurriness and ambiguities. Therefore, directly estimating depth information with a light field camera is challenging due to the inherent narrow baseline and the trade-off between spatial and angular resolution. In the past two decades, many algorithms have been proposed to improve the accuracy of depth estimation from light field imaging. These include machine learning methods [10; 11; 12; 13; 14] and deep learning methods [15; 16; 17; 18; 19]. Considering the input data structure, depth estimation methods can be categorized into three types: EPIs (epipolar plane images), SAIs (sub-aperture images), and focal stacks. The main workflow involves feature extraction, cost construction, cost aggregation, and disparity regression. Both [PERSON] et al. [12] and [PERSON] et al. [10] exploited the phase shift theorem in the Fourier domain to represent sub-pixel displacement. [PERSON] et al. [11] proposed an occlusion-aware framework using a modified angular photo-consistency for occlusion prediction, and they fused depth, correspondence, and refocus cues for robust depth estimation. [PERSON] et al. [13] proposed a spinning parallelogram operator based on EPIs, using a confidence metric to integrate information from the LF structure, thus reducing occlusions and ambiguities. [PERSON] et al. [15] proposed a view selection module that generated an attention map, indicating the importance of each view for accurate depth estimation. [PERSON] et al. [16] developed an attention-based multi-level fusion network, designing intra-branch and inter-branch strategies for hierarchically fusing effective features from different views. [PERSON] et al.