<figure><img src="image_1.png" /></figure>

HAL
open science

# Blur aware metric depth estimation with multi-focus plenoptic cameras

Mathieu Labussière, Céline Teulière, Omar Ait Aider

► To cite this version:

Mathieu Labussière, Céline Teulière, Omar Ait Aider. Blur aware metric depth estimation with multi-focus plenoptic cameras. Computer Vision and Image Understanding, inPress, 10.1016/j.cviu.2023.103802. hal-04178873

HAL Id: hal-04178873

https://hal.science/hal-04178873

Submitted on 8 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Blur aware metric depth estimation with multi-focus plenoptic cameras

Mathieu Labussière, Céline Teulière, Omar Ait-Aider

► To cite this version:

Mathieu Labussière, Céline Teulière, Omar Ait-Aider. Blur aware metric depth estimation with multi-focus plenoptic cameras. Computer Vision and Image Understanding, 2023, pp.103802. 10.1016/j.cviu.2023.103802. hal-04178873

HAL Id: hal-04178873

https://hal.science/hal-04178873

Submitted on 8 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>
# Blur aware metric depth estimation with multi-focus plenoptic cameras

Mathieu Labussière<sup>a,**</sup>, Céline Teulière<sup>a</sup>, Omar Ait-Aider<sup>a</sup>

<sup>a</sup>Université Clermont Auvergne, Clermont Auvergne INP, CNRS, Institut Pascal, F-63000 Clermont-Ferrand, France

## ABSTRACT

While a traditional camera only captures one point of view of a scene, a plenoptic or light-field camera, is able to capture spatial and angular information in a single snapshot, enabling depth estimation from a single acquisition. In this paper, we present a new metric depth estimation algorithm using only raw images from a multi-focus plenoptic camera. The proposed approach is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. The main goal of our blur aware depth estimation (BLADE) approach is to improve disparity estimation for defocus stereo images by integrating both correspondence and defocus cues. We thus leverage blur information where it was previously considered a drawback. We explicitly derive an inverse projection model including the defocus blur providing depth estimates up to a scale factor. A method to calibrate the inverse model is then proposed. We thus take into account depth scaling to achieve precise and accurate metric depth estimates. Our results show that introducing defocus cues improves the depth estimation. We demonstrate the effectiveness of our framework and depth scaling calibration on relative depth estimation setups and on real-world 3D complex scenes with ground truth acquired with a 3D lidar scanner.

## 1. Introduction

Plenoptic or light-field cameras are imaging systems able to capture both *spatial* and *angular* information about a scene in a single exposure. These systems are usually built upon a micro-lenses array (MLA) placed between a main lens and a sensor (Ng et al., 2005; Perwass and Wietzke, 2012; Georgiev and Lumsdaine, 2012). Information from light is multiplexed onto the sensor in the form of a micro-images array (MIA) as illustrated in Figure 1. With *unfocused* plenoptic cameras (1.0), the micro-lenses are focused at infinity, and each pixel captures a specific orientation. With *focused* plenoptic cameras (2.0), the imaging process can be modeled as the projection of object points by the main lens into a virtual intermediate space which are then re-imaged by each micro-lens onto the sensor. To improve the spatial resolution and depth of field (DoF) of the plenoptic camera, a multi-focus configuration has been proposed by Perwass and Wietzke (2012); Georgiev and Lumsdaine (2012). In this setup, the MLA is composed of

several micro-lenses with different focal lengths. The same part of a scene is projected into multiple observations on the sensor with different amounts of blur according to the micro-lens' type, as shown in the zoom of Figure 1. In this paper, we especially focus on this latter setup, i.e., multi-focus plenoptic cameras. Plenoptic or light-field data can designate different quantities: either it is referring to the multi-views array of reconstructed sub-aperture images (SAIs) as by Wu et al. (2017), or to the micro-images array from raw images. The term *view* is applied either to a SAI or to a micro-image, depending on the context.

From this redundant information, depth estimation and 3D reconstruction can be performed directly from a single acquisition, with scale information. First, plenoptic or light-field depth estimation can be seen as a standard multi-view stereo matching problem. Depth can be inferred for a reference view, as in *Depth from Stereo* approaches (Scharstein and Szeliski, 2002), using *correspondence* cues. Second, as the plenoptic camera captures part of the scene at different focus settings (either using refocused SAIs or different micro-images' type), *defocus* cues can be used to estimate depth like in *Depth from Focus/Defocus* approaches (Pentland, 1987; Grossmann, 1987; Subbarao, 1989; Lai et al., 1992).

<footnote>**Corresponding author: 
e-mail: mathieu.labussiere@uca.fr (Mathieu Labussière)</footnote>
In the focused plenoptic camera case, working with SAIs is prone to error as depth is usually required to reconstruct the light-field or the SAIs (Hog et al., 2017; Liang and Ramamoorthi, 2015; Wanner et al., 2011). To overcome this issue, algorithms can work directly with the raw plenoptic images, at micro-images level. However, usually only micro-images with the smallest amount of blur are used, or alternatively, specific patterns are designed to exploit the information (Fleischmann and Koch, 2014; Ferreira and Goncalves, 2016; Palmieri and Koch, 2017). Using the model of Labussière et al. (2020) extended by Labussière et al. (2022), we relate the camera parameters to the amount of blur in the image, and all information can be used simultaneously, without distinction between types of micro-lenses. We propose then to leverage blur information where it was previously considered a drawback. Indeed defocus cues are complementary to correspondence cues, and can improve the quality of depth estimation as shown by Tao et al. (2013).

**Contributions.** As our main contribution, we introduce in this paper a metric depth estimation framework for plenoptic cameras, named blur aware depth estimation (BLADE), leveraging both spatially-variant blur and disparity cues between micro-images, using only raw images from plenoptic cameras. It is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. The method uses the camera model presented in Labussière et al. (2022) which explicitly includes the defocus blur in the projection model. We derive its inverse which provides depth estimates up to a scale factor. Thus, we empirically take into account depth scaling to achieve precise and accurate metric depth estimates. For that purpose, calibration is required. Especially, we propose a method to calibrate the inverse model. To validate our framework, we present a new dataset of 3D real-world scenes with ground truths acquired with a 3D lidar scanner, and a methodology to calibrate the extrinsic parameters. We make our source code and datasets publicly available to the community on our github page's https://github.com/comsee-research/.

**Paper Organization.** The remainder of this paper is organized as follows. First, we review the existing methods for depth estimation based on light-field data in section 2. Second, we briefly recall the blur aware plenoptic (BAP) camera model and present the derivation of the inverse projection model in section 3, as well as, how depth scaling is taken into account and calibrated in section 4. Third, we explain in section 5 how we can link the disparity in image space to the defocus blur information. Then, we detail our blur aware depth estimation (BLADE) framework in section 6. Our experimental setup is presented in section 7. Finally, our results are presented and discussed in section 8.

**Notations.** The camera notations used in this paper are shown in Figure 2. Pixel counterparts of metric values are denoted in lower-case Greek letters. Bold font represents vectors (usually in lower-case letters) and matrices (usually in upper-case letters). Scalars are given by light letters.

<figure><img src="image_4.png" /><figcaption>Fig. 1: Example of raw plenoptic image multiplexing both angular and spatial information onto the sensor in the form of a Micro-Images Array (MIA) with several types of micro-lenses, thus different amounts of blur.</figcaption></figure>

## 2. Related work

First, we briefly review classical paradigms for depth estimation and how they apply to plenoptic imaging. Then, we extensively analyze the different approaches to estimate depth from light-field.

### 2.1. *Depth from Stereo*

Depth from stereo is achieved by triangulating rays passing through corresponding regions or features from two images. Correspondences are generally obtained based on local photometric similarity measurement. The global consistency is also achieved by taking into account epipolar geometry constraint. To make the process simpler and faster, the stereo pair can be rectified in order to handle non-parallel image planes (Scharstein and Szeliski, 2002). In the case of plenoptic cameras, micro-images share the same plane. Thus, there is no need for rectification. However, in the multi-focus case, the similarity constraint is violated due to the difference of focus between two micro-images types. To satisfy the constraint, one might only compare images of the same type, or restrict the working range to the DoF of the camera (as at least two micro-images are in-focus simultaneously). We propose an alternative approach that exploits all the available information by including blur within the depth estimation process.

### 2.2. *Depth from Focus/Defocus*

The depth from focus/defocus approach aims to estimate the spatially variant spread parameter of the blur kernel, by acquiring two images of the same scene with different camera settings (Pentland, 1987; Grossmann, 1987; Subbarao, 1989; Lai et al., 1992). The blur radius is linked to the inverse distance, and once the blur is estimated, depth can be retrieved. The spread parameter is usually estimated in the frequency domain or in the spatial domain (Subbarao and Surya, 1994). Depth from focus/defocus works better on short ranges and can thus be seen as a solution for these distances. Indeed, depth from stereo methods are less effective in a short range due to part of the scene being not visible by both cameras. However, the main hypothesis is that the two images represent the same scene viewed from the same point of view and with the same view angle, which is not the case with a plenoptic camera. Therefore, the micro-images have to be matched according to parallax.
### 2.3. Depth from Light-Field

Most light-field depth estimation processes are divided into two steps: 1) initial depth map estimation from SAIs or epipolar plane images (EPIs), and 2) depth refinement with global methods. These kinds of methods usually operate with unfocused plenoptic cameras (1.0).

#### 2.3.1. Based on sub-aperture images (SAIs)

One category of approaches estimates depth from reconstructed SAIs. A framework is proposed by Perez Nava and Luke (2009) to simultaneously estimate an all in-focus image along with the depth map, based on focal stack analysis for the focused plenoptic camera. Bishop and Favaro (2012) applied traditional multi-view stereo methods along with an explicit image formation model and antialiasing to reconstruct both scene depth and its super-resolved texture in a Bayesian framework. In Hahne et al. (2018), light field triangulation has been proposed to determine depth distances and baselines in unfocused plenoptic cameras. However their model is not applicable to the focused plenoptic camera. Correspondence and defocus cues has been analyzed by Kim et al. (2014) to select reliable pixels for depth estimation using a cost volume reconstructed from the light-field. In Jeon et al. (2015), a depth map estimation algorithm is presented using a multi-label optimization of a cost volume for lenslet-based light-field image. They improved their depth estimation in Jeon et al. (2018) by combining different matching costs and learned to automatically determine which combination performs better on the given input. Several aggregation costs have also been tested and evaluated in Williem and Park (2018). Built upon the work of Tao et al. (2013), the depth estimation is conducted in Wang et al. (2015) by treating occlusions explicitly in a photo-consistency model using the reconstructed central view. To better account for correlation and dependencies within angular patches and spatial images, Zhang et al. (2019) proposed a two-step light-field depth estimation based on graph spectral analysis.
In another direction, recent work of Anisimov and Wasem (2019) aimed at reducing computational cost by leveraging a semi-global matching strategy, instead of focusing on improving depth estimation. Their method is based on pixel matching in SAIs with different similarity measurements for estimation of a dense depth map.

Another category of methods aims at leveraging the 4D light-field structure to extract features to be matched. Several descriptors have been proposed such as LiFF (Dansereau et al., 2019) built upon SIFT, the binary descriptor introduced by Alain and Smolic (2020) built upon BOOM, or FDL-HSIFT (Xiao et al., 2021) built upon Harris and SIFT in the scale-disparity space.

All the previous methods operate either on the light-field or on reconstructed SAIs. Those are easily available with a camera-array setup, sequential acquisition, or unfocused plenoptic cameras. However, this is not the case for focused plenoptic cameras. The latter setup leads to an ill-posed problem because depth information is required to reconstruct the light-field or the SAIs (Hog et al., 2017; Liang and Ramamoorthi, 2015; Wanner et al., 2011).

#### 2.3.2. Based on epipolar plane images (EPIs)

EPI usually represents a 2D slice (in the spatial and angular dimensions) of the 4D light-field. From analysis of variations within this representation, we can infer depth information. An overview and taxonomy of dense light-field depth estimation algorithms is available in Johannsen et al. (2017), mostly including methods working on reconstructed images or EPIs.

This structure was first analyzed by Bolles et al. (1987) to retrieve depth by estimating lines in the EPI, as the slope is inversely proportional to depth. For focused plenoptic cameras, an algorithm to generate EPI representation was proposed by Wanner et al. (2011). They computed the full depth of field view for each lens type independently and then applied a merging algorithm to include the multi-focal aspect. The latter representation was used by Wanner and Goldluecke (2012) who proposed a globally consistent framework using structure tensors to estimate the directions of feature pixels in the 2D EPI. In Yu et al. (2013), geometric structures of 3D lines in ray space extracted from EPI were explored. They encoded the line constraints to further improve the reconstruction quality. An algorithm that computes dense depth estimation by combining both defocus and correspondence depth cues was introduced by Tao et al. (2013). They latter included shading as a third clue to improve their depth estimation Tao et al. (2015). A method was proposed by Tosic and Berkner (2014) to detect ray geometry in EPIs based on light-field scale and depth (Lisad) space transform. Xu (2016) implemented a three-step depth estimation using EPIs. Latter methods are vulnerable to occlusions as they generate inconsistencies in the EPIs. Chen et al. (2014) explicitly tackled this issue by presenting a new light-field stereo matching algorithm that is capable of handling occlusion based on analysis of the angular statistics of the light-field. More recently, Zhang et al. (2016) introduced a spinning parallelogram operator (SPO) to locate lines and calculate their orientations in an EPI for local robust depth estimation. According to the authors, SPO has been demonstrated to be insensitive to occlusions, noise, spatial aliasing, or limited angular resolution. A new framework using SPO was developed by Sheng et al. (2018) to locate lines in multi-orientation EPIs.

Similarly to SAIs-based methods, the EPI representation can easily be retrieved for unfocused plenoptic cameras, but needs prior depth to be generated from focused plenoptic cameras.

#### 2.3.3. Based on learning

Deep learning methods have also been applied on light-field images, in particular in context of super-resolution. The first deep convolutional neural network (CNN) that jointly optimizes angular and spatial super-resolution images from a pair of SAIs was proposed in Yoon et al. (2015). Generated SAIs were then used in a stereo matching-based depth estimation built-upon the method of Jeon et al. (2015). An end-to-end network was introduced by Ma et al. (2018) using all SAIs allowing to capture both local and global features to generate a disparity map. To reduce the amount of input data, Shin et al. (2018) developed a multi-stream fully CNN using only SAIs stacked in four angular directions to produce a disparity map. Liu
et al. (2019) presented a three-part neural network architecture that allows to take into account both parallax and ambiguity cues. Recently, a new depth estimation based on unsupervised learning was proposed in Jin and Hou (2021) to overcome the necessity of having depth maps as ground truth and to reduce the gap between simulated and real data. Using EPI representation, Johannsen et al. (2016) proposed a novel approach for depth estimation based on a learned dictionary which codes for disparity from EPI. Heber et al. (2016) introduced a U-shaped fully CNN with skipped connections where inputs are EPI representations and output is a disparity map. Recently, Li et al. (2020) proposed a pseudo-siamese neural network to estimate depth at each pixel, taking as input the vertical and the horizontal EPI at this location.

Instead of explicitly including vision cues, Huang (2018) proposed to model the light-field matching problem using an empirical Bayesian framework which better generalizes to different light-fields (dense, sparse, color, gray-scaled, etc.) to achieve better depth quality.

To the best of our knowledge, no learning-based methods are able to directly operate on raw plenoptic images, and therefore required SAIs or EPIs generation.

### 2.3.4. *Based on raw images*

To overcome the issues related to reconstruction of the SAIs or EPIs, several methods work directly with raw images. This is particularly suited for *focused* plenoptic cameras (2.0), as each micro-image captures more spatial information than its unfocused counterpart.

With the arrival of commercial focused plenoptic cameras, Perwass and Wietzke (2012) proposed a methodology to estimate depth directly from raw images, later improved based on the camera model of Heinze et al. (2016). Their method was based on triangulation from micro-images views employing a correlation technique just as in standard stereo-matching approaches. But it required contrasted micro-images and sharp micro-images. If the camera is calibrated and once each pixel has a depth estimate, sparse metric depth can be retrieved. An automatic method is presented in Custodio (2014) to estimate the depth of a scene based on multi-view geometry and ray back-tracing from detected salient points. Use of points of interest matched between micro-images (SURF, SIFT and Harris) has also been investigated by Konz et al. (2016) to estimate virtual depth by triangulation. However, the accuracy of the estimation was limited, and the number of features were too low to obtain a proper depth map.

In Noury (2019), metric depth estimation per micro-image was conducted. Their method is inspired from Scharstein and Szeliski (2002) but applied to micro-images from the raw plenoptic image. It relies on a minimization process of the dense re-projection error of the reconstructed neighbors micro-images given a depth hypothesis following their projection model of the camera developed in Noury et al. (2017).

Depth estimation for the multi-focus plenoptic camera has been explicitly considered in Fleischmann and Koch (2014). Their method, based on one depth estimation per micro-image, operates by regularizing a cost volume computed from a similarity

measure between micro-images at different disparity hypotheses. Disparities were then converted to virtual depths, according to the MLA parameters. To take into account the varying amount of defocus blur between micro-images of different types, they developed an adaptive strategy to select only certain candidates micro-images. Similar to the previous work, Ferreira and Goncalves (2016) used salient points detected with SIFT to select micro-images in which a search along epipolar line is performed. A specific lens selection scheme to improve robustness was proposed. Finally, matched points were back-projected into virtual space to form a point cloud. Virtual points are then reprojected and an average depth is attributed for every micro-image. Lens selection strategies have also been addressed in Palmieri and Koch (2017). Another depth estimation for the multi-focus plenoptic camera was proposed by Cunha et al. (2020). Their method operates by first detecting edges in micro-images, which are second matched with neighbors micro-images. Finally, matched points were triangulated into virtual space, and then reprojected into metric space with calibration parameters available. They addressed the issue of the different amount of blur during the matching by proposing a switching mechanism between intensity and LPQ domains.

Zeller et al. (2016) introduced the first probabilistic depth estimation from raw plenoptic images obtained from a focused plenoptic camera. They addressed depth estimation as a multi-view stereo problem. For each pixel having a sufficient gradient, virtual depth hypothesis is obtained by finding correspondences along epipolar lines in neighbors micro-images with local intensity error minimization. Multiples hypotheses are merged in a Kalman-like fashion, allowing to associate a variance to the estimation. To deal with the multi-focus aspect, they incorporated a term modeling the focus uncertainty.

All previous solutions from the state-of-the-art using raw images considered blur as a drawback and designed specific strategies to select micro-images. In contrast, leveraging the new camera model introduced by Labussière et al. (2022), we explicitly use the defocus information in our depth estimation process, taking into account both correspondence and defocus cues. Micro-images are matched like in multi-views stereo matching problems to estimate disparities, which are then reconverted to metric depths thanks to the camera projection model explicitly including the defocus blur.

## 3. Blur Aware Plenoptic camera model

We first briefly present the model of the focused plenoptic camera, especially the multi-focus case in subsection 3.1. The camera is composed of a main lens and photo-sensitive sensor with a micro-lenses array (MLA) in between, as illustrated in Figure 2. In the multi-focus configuration, the micro-lenses array consists of *l* different types of thin-lenses. Each micro-lens creates a micro-image (MI) onto the sensor. We consider the model presented by Labussière et al. (2020) and extended in Labussière et al. (2022), explicitly including the micro-lenses focal lengths *f*⁽ᵏ⁾ and the defocus blur *ρ* in the direct projection model. In order to relate image information to world information, we naturally derive here in subsection 3.2 the inverse projection
<figure><img src="image_5.png" /><figcaption>Fig. 2: Blur Aware Plenoptic camera model (Labussière et al., 2022) with the notations used in this paper. Object points are projected by the main lens behind the micro-lenses array (MLA) into a virtual intermediate space, and then re-imaged by each micro-lens onto the sensor.</figcaption></figure>

model. The inverse projection provides metric results up to a scale factor. We will detail in section 4 the proposed depth scaling mode and a calibration method to retrieve accurate depth.

### 3.1. Direct projection model

The blur aware plenoptic camera model links a scene point $\mathbf{p}_w = [x \ y \ z \ 1]^T$ to a blur aware plenoptic (BAP) feature $\mathbf{p}_{k,l} = [u \ v \ \rho \ 1]^T$ in homogeneous coordinates through each micro-lens $(k, l)$ of type $(i)$. The $(u, v)$ part encodes the point position in the image and the $\rho$ variable encodes the radius of the defocus blur. More details of the defocus model will be given in subsection 5.1. The direct projection $\Pi_{k,l}$ from Labussière et al. (2022) is then given by

$$ \begin{bmatrix} u \\ v \\ \rho \\ 1 \end{bmatrix} \propto \mathcal{P}(i, k, l) \cdot \mathbf{T}_{\mu}(k, l) \cdot \varphi(\mathbf{K}(F) \cdot \mathbf{T}_{c} \cdot \mathbf{p}_{w}), \quad (1) $$

where $\mathcal{P}(i, k, l)$ is the blur aware plenoptic projection matrix through the micro-lens $(k, l)$ of type $(i)$, and computed as

$$ \mathcal{P}(i, k, l) = \mathbf{P}(k, l) \cdot \mathbf{K}(f^{(i)}). \quad (2) $$

$\mathbf{P}(k, l)$ is a matrix that projects the 3D virtual point onto the sensor and takes into account the defocus blur. $\mathbf{K}(f)$ is the thin-lens projection matrix for the given focal length. $\mathbf{T}_c$ is the pose of the main lens with respect to the world frame and $\mathbf{T}_{\mu}(k, l)$ is the pose of the micro-lens $(k, l)$ expressed in the camera frame. The function $\varphi(\cdot)$ models the lateral distortion. The reader can refer to Labussière et al. (2020, 2022) and supplementary materials for more details about the direct projection model.

### 3.2. Inverse projection model

In order to relate image information to world information, we now inverse the projection model. We back-project through a micro-lens $(k, l)$ of type $(i)$ a BAP feature $\mathbf{p}_{k,l} = [u \ v \ \rho \ 1]^T$

into a point $\mathbf{p}_w = [x \ y \ z \ 1]^T$ in object space. The inverse projection $\Pi_{k,l}^{-1}$ is given by

$$ \begin{bmatrix} x \\ y \\ z \\ 1 \end{bmatrix} \propto \mathbf{T}_c^{-1} \cdot \mathbf{K}^{-1}(F) \cdot \varphi^{-1}(\mathbf{T}_{\mu}^{-1}(k, l) \cdot \mathcal{P}^{-1}(i, k, l) \cdot \mathbf{p}_{k,l}). \quad (3) $$

**Blur radius computation.** In practice, the blur radius $\rho$ is linked to the virtual depth $v$. Virtual depth refers to relative depth value with respect to the MLA. It is defined as the ratio between the signed object distance $a'$ (as indicated on Figure 2) and the sensor distance $d$, i.e.,

$$ v = -\frac{a'}{d}. \quad (4) $$

The relation between $\rho$ and $v$ is given by

$$ \rho = m \cdot v^{-1} + q_i, \quad (5) $$

where $m$ and $q_i$ are known coefficients computed from the intrinsic parameters and depending on the micro-lens' type $(i)$ (Labussière et al., 2022, see Eq. 32).

In the end, we are now able to back-project a BAP feature, i.e., a pixel having a virtual depth into a point in object space. It allows us then to convert virtual depth to metric depth up to a scale factor.

**Inverse distortion model.** Since the lateral distortion model of Labussière et al. (2022) is not invertible, we add in this paper the inverse distortion $\varphi^{-1}(\cdot)$ explicitly in the camera model. Inverse distortion is used to create an undistorted point $\mathbf{p}_u = \varphi^{-1}(\mathbf{p}_d) = [x_u \ y_u \ z \ 1]^T$ from a distorted point $\mathbf{p}_d = [x_d \ y_d \ z \ 1]^T$. Note that distortions are applied in the 3D space to take into account the deviation from the thin-lens model, not in image space. An efficient way to characterize the inverse distortion is to use a high order version of the Brown's model as shown in de Villiers et al. (2008). In particular, we use a model of the same order as the direct distortion model, and the mapping is
expressed as

$$
\left\{
\begin{array}{ll}
x_u = x_d (1 + Q_{-1}\mathcal{S}^2 + Q_{-2}\mathcal{S}^4 + Q_{-3}\mathcal{S}^6) & \text{[radial]} \\
\qquad + P_{-1} (\mathcal{S}^2 + 2x_d^2) + 2P_{-2}x_d y_d & \text{[tangential]} \\[2ex]
y_u = y_d (1 + Q_{-1}\mathcal{S}^2 + Q_{-2}\mathcal{S}^4 + Q_{-3}\mathcal{S}^6) & \text{[radial]} \\
\qquad + P_{-2} (\mathcal{S}^2 + 2y_d^2) + 2P_{-1}x_d y_d & \text{[tangential]}
\end{array}
\right.
\qquad (6)
$$

where $\mathcal{S}^2 = x_d^2 + y_d^2$. The three coefficients for the radial component are given by $\{Q_{-1}, Q_{-2}, Q_{-3}\}$, and the two coefficients for the tangential by $\{P_{-1}, P_{-2}\}$.

**Calibration of the coefficients.** The optimization of the inverse coefficients is done only once, as a post-calibration step, such that

$$
\arg\min_{\{Q_{-1}, Q_{-2}, Q_{-3}, P_{-1}, P_{-2}\}} \sum_{\mathbf{p}} \|\mathbf{p} - \varphi^{-1}(\varphi(\mathbf{p}))\|^2, \quad (7)
$$

for a large number of samples **p** uniformly distributed in the virtual intermediate space (i.e., after projection by the main lens). The optimization is conducted with the Levenberg-Marquardt algorithm. In practice, coefficients are initialized from the direct distortion coefficients, and less than fifteen iterations are sufficient to converge, with a root mean square error (RMSE) of approximately $10^{-5}$ mm.

4. Depth scaling model and calibration

When mapping from the virtual space to the object space
with the inverse projection model, the reconstructed objects are
scaled up to a certain factor. The scale factor grows approxi-
mately linearly as function of the distance with respect to the
focus distance. This phenomenon is present both on real and
simulated data. It is due to the limitations of the thin-lens model.
The proposed model describes efficiently the projective geome-
try for a point whose all rays inside the projection cone attain
the retina, but this is not always the case. Given specific con-
figurations and with aperture corresponding to the *f*-number
matching principle (Perwass and Wietzke, 2012), not all rays
from the cone of light reach the sensor, inducing a shift in the
radiance. See supplementary materials for more comments. In-
deed, when simulating the imaging process with a large aperture,
the retrieved depths from the simulated images do not suffer
from a scale error.

Although not explicitly pointing out this issue, Zeller et al. (2016) calibrated the mapping between virtual depth and metric depth in object space, by proposing three different models. Heinze et al. (2016) also noted the presence of a systematic error in the depth estimation process, even after correcting the offset induced by the thick-lens model.

In the following, we show, first, how we empirically model
the depth scaling, second, how to measure and characterize the
scaling error, and thus how to correct it in a post-calibration
process. Once the depth scaling has been calibrated, coordinates
of the back-projected point (i.e., from virtual to metric space) are
corrected according to our proposed model, to achieve precise
and accurate depth estimation.

4.1. Scale correction model

From experimental data (both from several physical configura-
tions and in simulation), we infer that the scaling error is function
of the distance z, and that polynomial function is sufficient to fit
the results. We proposed then to model the scaling correction
as a polynomial, noted Γ(·). The function Γ takes as input the
z-component of the 3D point obtained by back-projection, and
the point is then re-scaled by

$$
\gamma = \frac{\Gamma(z)}{z}, \qquad (8)
$$

i.e., the corrected point **p**\* from a pixel **p**<sub>k,l</sub> having a virtual depth **v** is

$$
\mathbf{p}^* = \gamma \cdot \begin{bmatrix} x \\ y \\ z \end{bmatrix}, \text{ where } \begin{bmatrix} x \\ y \\ z \end{bmatrix} \propto \Pi_{k,l}^{-1} \left( \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} + q_i \right). \quad (9)
$$

We found a quadratic model to be the best compromise between
model simplicity and performances. It allows to efficiently cor-
rect the scale error as shown in section 8.

4.2. Scaling error measurement

To quantify the scale error, we use the relative mean bias
error (MBE) to measure the relative difference of the distance
between each pair of back-projected points $\|\bar{\mathbf{p}}_i - \bar{\mathbf{p}}_j\|$ and the
known distance $\|\mathbf{p}_i - \mathbf{p}_j\|$ between these points. In practice, we
use points corresponding to corners of a known-sized checker-
board.

First, we perform BAP detection on raw image of a checker-
board. Second, we back-project each BAP feature **p**<sub>k,l</sub> having a
virtual depth **v** of the same cluster **C**<sub>i</sub>, i.e., corresponding to the
same corner **p**<sub>i</sub>, using Eq. 3. Third, we compute the centroid **p**&#773;<sub>i</sub>
corresponding to the checkerboard corner **i**, as

$$
\bar{\mathbf{p}}_i = \frac{1}{\#C_i} \cdot \sum_{\mathbf{p}_{k,l} \in C_i} \Pi_{k,l}^{-1}(\mathbf{p}_{k,l}), \qquad (10)
$$

where #C_i is the number of observations in the cluster C_i. Finally,
the scale error ε_scale, for a frame having I×J corners, is computed
as

$$
\epsilon_{\text{scale}} = \frac{1}{I \cdot J} \cdot \sum_{(i,j) \in I \times J} \left( 1 - \frac{\| \bar{\mathbf{p}}_i - \bar{\mathbf{p}}_j \|}{\| \mathbf{p}_i - \mathbf{p}_j \|} \right). \quad (11)
$$

4.2.1. Depth scaling calibration

At this point, the camera intrinsic parameters, the relative
blur coefficient and the inverse distortion coefficients have been
calibrated. We propose then a post-calibration process for the
scaling correction based on non-linear optimization of the scale
error over several checkerboard raw plenoptic images. Let Ξ =
{γ₀, γ₁, γ₂} be the set of parameters to optimize, such that Γ(z) =
γ₂z² + γ₁z + γ₀. The cost function Θ(Ξ) is expressed as the sum
over each frame *n* of the scale errors εscale, i.e.,

$$
\Theta(\Xi) = \frac{1}{IJN} \cdot \sum_n \sum_{(i,j) \in I \times J} \left( 1 - \frac{\| \gamma_i \cdot \bar{\mathbf{p}}_i^n - \gamma_j \cdot \bar{\mathbf{p}}_j^n \|}{\| \mathbf{p}_i^n - \mathbf{p}_j^n \|} \right), \quad (12)
$$
where N is the number of frames and I · J is the number of checkerboard corners. Each point $\bar{p}_i^p$ is re-scaled by $\gamma_i$ as defined in Eq. 8 and Eq. 9. The optimization is conducted using the Levenberg-Marquardt algorithm.

## 5. Linking disparity and defocus blur

In order to obtain metric depth, we first estimate the virtual depth from disparity, and defocus is a complementary cue to resolve the inherent ambiguity of stereo matching as shown in Held et al. (2012); Schechner and Kiryati (2000). The goal is then to improve disparity estimation for defocus stereo images via compensating the mismatch of focus thus integrating both correspondence and defocus cues.

### 5.1. Defocus and relative blur models

We first recall how blur and relative blur between micro-images are modeled as presented in Labussière et al. (2022).

**Defocus blur models.** We modeled the geometric blur using the circle of confusion (CoC). The blur radius $\rho$ and its metric counter part $r$ as in (Labussière et al., 2022, Eq. 6). From a signal processing point of view, the response of an imaging system to an object out-of-focus can be modeled by the point-spread function (PSF). Let $\mathcal{I}(x, y)$ be the observed blurred image of an object at a constant distance. The image is the result of the convolution of the PSF, noted $h(x, y)$, with the in-focus image, $\mathcal{I}^*(x, y)$, as define in (Labussière et al., 2022, Eq. 7). The spread parameter $\sigma$ of the PSF is proportional to the blur circle radius $\rho$. Therefore, we can write

$$ \sigma \propto \rho \Leftrightarrow \sigma = \kappa \cdot \rho \quad (13) $$

where $\kappa$ is a camera constant that is determined by calibration, using the method presented in Labussière et al. (2022). Note that the spatially-variant spread parameter $\sigma$ thus depends on the object distance, i.e., blur and depth are linked.

**Micro-lenses relative blur.** A point imaged by two different micro-lenses of type (i) and (j) will have different blur radii, i.e., the resulting images will have different spread parameters for the PSF model, such that

$$ \left\{ \begin{array}{l} \mathcal{I}_{(i)}(x, y) = h_{(i)} * \mathcal{I}^{*}(x, y) + n_{(i)}(x, y) \\ \mathcal{I}_{(j)}(x, y) = h_{(j)} * \mathcal{I}^{*}(x, y) + n_{(j)}(x, y), \end{array} \right. \quad (14) $$

where $\mathcal{I}^*(x, y)$ is the latent in-focus image, and $n(x, y)$ is the image noise<fnref n="1" />. We then use the equally-defocused representation (Labussière et al., 2022, Eq. 36) by applying additional blur to the relatively in-focus micro-image,

$$ \left\{ \begin{array}{ll} \mathcal{I}_{(i)}(x, y) \approx h_{(i)} * \mathcal{I}_{(j)}(x, y) & \text{if } \sigma_{(i)} \ge \sigma_{(j)} \\ h_{(i)} * \mathcal{I}_{(i)}(x, y) \approx \mathcal{I}_{(j)}(x, y) & \text{if } \sigma_{(i)} < \sigma_{(j)}. \end{array} \right. \quad (15) $$

$h_r$ is the relative blur kernel, of spread parameter $\sigma_r$, applied to either one of the views such that both are equally-defocused.

### 5.2. Link between disparity and the relative blur

As highlighted in Chen et al. (2015), while estimating depth from a defocus stereo configuration, both spatially-variant blur and disparity provide the inference for depth information. Establishing the visual correspondence across two images must take both disparity and blur into account.

**Defocus stereo images configuration.** In standard stereo matching, two focused rectified stereo images are used to determine disparity. The left image $\mathcal{I}_{(i)}$ and right image $\mathcal{I}_{(j)}$ are related with the spatially-variant disparity $\delta \in \mathbb{R}^2$, and thus the correspondence between the two images can be modeled by

$$ \mathcal{I}_{(i)}(\mathbf{p}) = \mathcal{I}_{(j)}(\mathbf{p} + \delta) \quad (16) $$

where $\mathbf{p} = (x, y)$ is the spatial index of a pixel. Taking into account blur, we model each image as the convolution of the in-focus image with a blur kernel as in Eq. 14 and we consider the equally defocused images as in Eq. 15. Therefore, injecting the disparity in the equally defocused model, the correspondence is given by

$$ \left\{ \begin{array}{ll} \mathcal{I}_{(i)}(\mathbf{p}) \approx h_r * \mathcal{I}_{(j)}(\mathbf{p} + \delta) & \text{if } \sigma_{(i)}(\mathbf{p}) \ge \sigma_{(j)}(\mathbf{p} + \delta) \\ h_r * \mathcal{I}_{(i)}(\mathbf{p}) \approx \mathcal{I}_{(j)}(\mathbf{p} + \delta) & \text{if } \sigma_{(i)}(\mathbf{p}) < \sigma_{(j)}(\mathbf{p} + \delta). \end{array} \right. \quad (17) $$

**Relative blur as function of the virtual depth.** From the blur radius formula, the relative blur can be approximated by a linear function of the disparity (i.e., of the inverse virtual depth up to a factor), such that

$$ \Delta r^2 = r_{(i)}^2 - r_{(j)}^2 \approx m_{i,j} \cdot \nu^{-1} + q_{i,j} \quad [\text{mm}^2] \quad (18) $$

with

$$ m_{i,j} = \frac{B^2 d}{2} \cdot \left( \frac{1}{a_0^{(i)}} - \frac{1}{a_0^{(j)}} \right) \quad (19) $$

and

$$ q_{i,j} = \frac{B^2 d^2}{4} \cdot \left( \left( \frac{1}{a_0^{(i)}} \right)^2 - \left( \frac{1}{a_0^{(j)}} \right)^2 \right), \quad (20) $$

where B is the micro-lens diameter and $a_0^{(i)}$ is the distance to the plane of focus of the type (i) micro-lens, computed as

$$ a_0^{(i)} = \frac{df^{(i)}}{d - f^{(i)}}. \quad (21) $$

All the parameters are known thanks to the camera calibration. Finally, the spread parameter $\sigma_r$ is computed using Eq. 18 and Eq. 13, such that

$$ \sigma_r = \kappa \cdot \frac{1}{s} \cdot \left| \Delta r^2 \right|^{\frac{1}{2}}. \quad (22) $$

Note that the approximation of Eq. 18 is exact when considering that micro-lenses are parallel to the sensor plane. When dealing with micro-lenses in the same local neighborhood, the z-shift inducing a slight difference between the virtual depths can be neglected, and with orthogonal approximation, the relation stands.

<footnote n="1">¹ Noise can be omitted under the assumption that, $n_{(i)}(x, y) * h_{(j)} \approx n_{(j)}(x, y) * h_{(i)} \approx 0$. This assumption does not hold when exactly one of the images is in focus. However if we blur both images by a small amount then we can expect the assumption to be valid. The approximation error is small when both images have large blur.</footnote>
## 6. Blur Aware Depth Estimation (BLADE)

We make the hypothesis that the camera is calibrated and we have access to the intrinsic parameters. We propose a process based on area matching techniques to estimate a raw depth map $\mathcal{D}$ directly from raw plenoptic images. Two variations are considered: 1) coarse estimation, i.e., one depth per micro-image; and 2) refined estimation, i.e., one depth per pixel. Figure 3 summarizes the estimation process in the refined case. A new residual error is formulated to leverage blur information for depth estimation using a multi-focus plenoptic camera. The computation is illustrated in Figure 4. Example of depth maps obtained by our method are illustrated in Figure 5. Let an observation be a pair of micro-images such that the reference $\mathcal{I}_{(i)}$ is the most defocused and the target $\mathcal{I}_{(j)}$ is the micro-image to be equally defocused.

**Disparity.** The disparity is given by $\delta = \|\delta\|$ where $\delta \in \mathbb{R}^2$ is obtained at a virtual depth hypothesis $v$ usually using the following relation

$$ \delta = \frac{1}{v} \cdot \mathbf{B} \quad \text{with} \quad \mathbf{B} = (\mathbf{C}^* - \mathbf{C}), \qquad (23) $$

where $\mathbf{C}^*, \mathbf{C}$ are respectively the centers of the reference and target micro-lenses in the MLA plane, and $\mathbf{B}$ is the baseline. This relation gives the disparity in case of orthogonal projection of micro-lens center to micro-image center. To take into account the deviation of the micro-image centers, the corrected disparity $\delta'$ in micro-image space is given by

$$ \delta' = \frac{(1 - \lambda) \cdot v + \lambda}{v} \cdot \mathbf{B}' \qquad (24) $$

with

$$ \mathbf{B}' = \lambda \cdot \mathbf{B} = (\mathbf{c}^* - \mathbf{c}), \qquad (25) $$

where $\lambda = D/(D+d)$, and $\mathbf{c}^*, \mathbf{c}$ are respectively the centers of the reference and target micro-images defining the baseline $\mathbf{B}'$ in image space.

**Matching problem.** Before triangulation, we need to know which pixel in neighbors micro-images are images of the same object point. The correspondence is modeled by an affine warping function $\omega(\mathcal{I}, \delta)$ at the disparity hypothesis $\delta \in \mathbb{R}^2$ along the epipolar line that is applied to rectify the image $\mathcal{I}$, such that

$$ \omega(\mathcal{I}, \delta)(\mathbf{p}) = \mathcal{I}(\mathbf{p} + \delta). \qquad (26) $$

Pixel intensities are interpolated using bilinear-interpolation. Pixels which are not reprojected are set to 0. We can compare the warped target image with the reference image to estimate if the disparity hypothesis is correct. This is done by calculating the sum of absolute differences (SAD).

**Mask.** To deal with circular micro-image of center $\mathbf{c}$ and radius $\varrho$, we define a mask image $\mathcal{M}$ by

$$ \mathcal{M}(\mathbf{p}) = \begin{cases} 0 & \text{if } \|\mathbf{p} - \mathbf{c}\| > \varrho - b \\ 1 & \text{if } \|\mathbf{p} - \mathbf{c}\| \le \varrho - b \end{cases} \qquad (27) $$

<table><caption>Fig. 3: Process of computing a refined depth map $\mathcal{D}(x, y)$ using our Blur Aware Depth Estimation (BLADE) framework.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>Raw image, Camera model</td></tr><tr><td>Output:</td><td>Refined depth map $\mathcal{D}(x, y)$</td></tr><tr><td>1:</td><td>for all micro-image $\mathcal{I}$ do</td></tr><tr><td>2:</td><td>retrieve default neighborhood $\mathcal{N}(\mathcal{I})$</td></tr><tr><td>3:</td><td>for all pixel $(x, y)$ with enough texture in $\mathcal{I}$ do</td></tr><tr><td>4:</td><td>compute initial virtual depth $v_0$</td></tr><tr><td>5:</td><td>$\mathcal{D}(x, y) \leftarrow v_0$</td></tr><tr><td>6:</td><td>end for</td></tr><tr><td>7:</td><td>update neighborhood $\mathcal{N}(\mathcal{I}, v_0)$</td></tr><tr><td>8:</td><td>for all pixel $(x, y)$ with enough texture in $\mathcal{I}$ do</td></tr><tr><td>9:</td><td>compute virtual depth $\hat{v}$</td></tr><tr><td>10:</td><td>$\mathcal{D}(x, y) \leftarrow \hat{v}$</td></tr><tr><td>11:</td><td>end for</td></tr><tr><td>12:</td><td>end for</td></tr><tr><td>13:</td><td>convert virtual to metric using $\Pi_{k,l}^{-1}$</td></tr></tbody></table> ▶ Eq. 3

where $b$ is the margin border of the micro-image. In our experiments, we used $b = 1.5$ pixel to minimize the vignetting effect. The final mask $\mathcal{M}^*$ is given as the intersection of the circular mask and the warped circular mask (see Figure 4), representing the common pixels between the reference and the target at the given disparity hypothesis, i.e.,

$$ \mathcal{M}^* = \mathcal{M} \circ \omega(\mathcal{M}, \delta), \qquad (28) $$

where $\circ$ is the element-wise matrix multiplication.

**Blur equalization.** One specificity of the multi-focus plenoptic camera is that for a same portion of a scene observed in micro-images using different focal lengths, these micro-images will demonstrate different amounts of blur. To compensate for the blur mismatch between the reference and the target, we use the equally defocused representation by adding supplemental blur to the target image. The spread parameter $\sigma_r$ is obtained from Eq. 22. To avoid dealing with micro-image borders while adding the relative blur, we use the S-Transform formulation from Subbarao and Surya (1994) to compute the convolution with the blur kernel. The equally defocused target image $\tilde{\mathcal{I}}$ is then computed as

$$ \tilde{\mathcal{I}} = h_r * \mathcal{I} \approx \mathcal{I} + \frac{\sigma_r^2}{4} \nabla^2 \mathcal{I}, \qquad (29) $$

where $\sigma_r$ is the spread parameter of the blur kernel, and $\nabla^2$ is the Laplacian operator.

**Similarity error computation.** The similarity residual error is computed as the normalized SAD between the masked reference image and the masked equally-defocused matched target image. It is expressed as

$$ \begin{aligned} \varepsilon_{\text{sim}}(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}, \delta) &= \eta \cdot \sum_{p} |\mathcal{I}_{(i)}(\mathbf{p}) - \omega(\tilde{\mathcal{I}}_{(j)}, \delta)(\mathbf{p})| \cdot \mathcal{M}^*(\mathbf{p}) \\ &= \eta \cdot \left\| (\mathcal{I}_{(i)} - \omega(\tilde{\mathcal{I}}_{(j)}, \delta)) \circ \mathcal{M}^* \right\|_1, \end{aligned} \qquad (30) $$
<figure><img src="image_6.png" /><figcaption>Fig. 4: Process of computing the similarity residual error in our BLADE framework. First, the target is equally-defocused given the virtual depth hypothesis. Second, the mask and the target are warped at the corresponding disparity. Third, the reference and the target are masked and compared to compute the similarity error.</figcaption></figure>

with $\eta$ being the normalization factor defined as the sum of the common pixels between the target and the reference. This factor is given by

$$ \eta^{-1} = \sum_{\mathbf{p}} \mathcal{M}^*(\mathbf{p}) = \|\mathcal{M}^*\|_1, \quad (31) $$

where $\|\cdot\|_1$ is the entry-wise matrix $\ell_1$-norm, i.e., $\|A\|_1 = \sum_{i,j} |a_{i,j}|$. Normalization of the error is required as the number of pixels to take into account varies with the disparity, and therefore with the virtual depth and the pair of micro-images. For a same virtual depth hypothesis, according to the pair of micro-images, the disparity will change.

**Cost computation.** For a micro-image $\mathcal{I}_{(i)}$, the cost is the weighted sum of all the errors computed for each micro-image $\mathcal{I}_{(j)}$ in its neighborhood $\mathcal{N}(\mathcal{I}_{(i)}, \nu)$ at the given virtual depth hypothesis $\nu$, i.e., the cost $\Theta(\mathcal{I}_{(i)}, \nu)$ is

$$ \frac{1}{W} \cdot \sum_{\mathcal{I}_{(j)} \in \mathcal{N}(\mathcal{I}_{(i)}, \nu)} w(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}) \cdot \varepsilon_{\text{sim}}(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}, \delta), \quad (32) $$

with $w(\mathcal{I}_{(i)}, \mathcal{I}_{(j)})$ being a weight function, and $W$ being the total weight computed as

$$ W = \sum_{\mathcal{I}_{(j)} \in \mathcal{N}(\mathcal{I}_{(i)}, \nu)} w(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}). \quad (33) $$

In our experiment, we define the weight as constant.

**Initialization.** The number of MIs that see the same scene points in the neighborhood $\mathcal{N}(\mathcal{I}, \nu)$ of the considered MI depends on the virtual depth hypothesis. So, we first coarsely initialize $\nu_0$ from MIs of the same type (here, at baseline $B = 2 \cdot \sin \frac{\pi}{3}$ in case of a hexagonal MLA organization with three types of micro-lenses). We use then $\nu_0$ to retrieve the correct neighborhood, and we restrict the search of the optimal value to $\nu_0 \pm N$, with $N = 1.96$ in our experiments.

**Coarse depth estimation.** Under the hypothesis of one depth per micro-image, corresponding to locally planar approximation,

the coarse depth map $\mathcal{D}(k, l)$ is estimated as follows. For each micro-image $\mathcal{I}$, virtual depth estimation is conducted by a minimization of the latter cost function in an optimization process, such that

$$ \hat{v} = \arg \min_{\nu} \Theta(\mathcal{I}, \nu). \quad (34) $$

As the function is in 1-D, we use the Golden Search Section (GSS) algorithm (Kiefer, 1953) to find the minimum with the desired precision. To improve time computation, only MIs with sufficient amount of texture are considered, such that

$$ \text{std}(\mathcal{I}, \mathcal{M}) > t_c, \quad (35) $$

where $\text{std}(\cdot, \cdot)$ is the standard deviation of the pixels intensity $\mathbf{p} \in \mathcal{I} | \mathcal{M}(\mathbf{p}) \neq 0$, and $t_c$ is a threshold to reject non-textured area. In our experiments, we set $t_c = 5$. Example of coarse virtual depth map is given in Figure 5(a).

**Refined depth estimation.** Under the hypothesis of one depth per pixel, a refined depth map $\mathcal{D}(x, y)$ is computed. The virtual depth estimation is conducted in a similar fashion as for the coarse estimation. For a pixel $\mathbf{p} = (x, y)$, errors and costs are computed the same way as previously but considering only the result within a window $\mathcal{W}$ extracted around $\mathbf{p}$. In our experiments, we use a window of size $5 \times 5$ pixel. The similarity residual error $\varepsilon_{\text{sim}}(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}, \delta, \mathbf{p})$ is then given by

$$ \|\mathcal{W}(\mathbf{p}, \mathcal{M}^*)\|_1^{-1} \cdot \|\mathcal{W}(\mathbf{p}, (\mathcal{I}_{(i)} - \omega(\bar{\mathcal{I}}_{(j)}, \delta)) \circ \mathcal{M}^*)\|_1. \quad (36) $$

The cost at a pixel $\mathbf{p}$ having a sufficient contrast, i.e., such that it verifies

$$ \text{std}(\mathcal{W}(\mathbf{p}, \mathcal{I}), \mathcal{W}(\mathbf{p}, \mathcal{M})) > t_c, \quad (37) $$

is given by $\Theta(\mathcal{I}_{(i)}, \nu, \mathbf{p})$ as

$$ \frac{1}{W} \cdot \sum_{\mathcal{I}_{(j)} \in \mathcal{N}(\mathcal{I}_{(i)}, \nu)} w(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}) \cdot \varepsilon_{\text{sim}}(\mathcal{I}_{(i)}, \mathcal{I}_{(j)}, \delta, \mathbf{p}), \quad (38) $$

where the weighs are defined as in Eq. 33. Finally, we compute the virtual depth $\hat{v}$ at each pixel $\mathbf{p}$ such that

$$ \hat{v} = \arg \min_{\nu} \Theta(\mathcal{I}, \nu, \mathbf{p}). \quad (39) $$
<figure><img src="image_7.png" /><figcaption>Fig. 5: Examples of raw virtual depth maps obtained by our BLADE framework with a zoom on an occluded area. (a) Coarse virtual depth map $D(k, l)$, with one estimation per micro-image. (b) Refined virtual depth map $D(x, y)$, with one estimation per pixel. More details can be captured per micro-image, but the map is sparser.</figcaption></figure>

Example of refined virtual depth map is given in Figure 5(b).

**Converting virtual to metric depth.** At the end of the depth estimation process, we have a virtual depth estimate associated to each pixel. To obtain a metric depth, we use the inverse projection model given in Eq. 3 taking into account depth scaling as in Eq. 9, where $\rho$ is computed from the virtual depth by Eq. 5.

# 7. Experimental setup

To validate our blur aware depth estimation framework, we proceeded as follows: first, we analyzed the scaling error modelings; second, we compared our method on relative depth estimation with state-of-the-art methods, including the Raytrix software, corresponding to the model of Heinze et al. (2016), and estimation with the model of Noury et al. (2017) using only the disparity; finally, we evaluated the depth estimation on real-world 3D complex scenes with ground truths acquired with a lidar. Our experimental setup is illustrated in Figure 6.

## 7.1. Hardware environment

**Camera setup.** For our experiments we used a Raytrix R12 color 3D-light-field-camera, with a MLA of F/2.4 aperture. The camera is in Galilean configuration, i.e., the micro-lens focal lengths are greater than the distance MLA-sensor. The mounted lens is a Nikon AF Nikkor F/1.8D with a 50 mm focal length. The MLA organization is hexagonal

<figure><img src="image_8.png" /><figcaption>Fig. 6: Our Raytrix R12 multi-focus plenoptic camera in our experimental setup, capturing a scene for 3D reconstruction. A Leica ScanStation P20 allows us to capture a colored point cloud that can be used as ground truth data.</figcaption></figure>

row-aligned, and composed of 176 × 152 (width × height) micro-lenses with $I = 3$ different types. The sensor is a Basler beA4000-62KC with a pixel size of $s = 0.0055$ mm. The raw image resolution is 4080 × 3068 pixel. We used four focus distance configurations, with $h \in \{450, 1000, 2133, \infty\}$ mm. Note that when changing the focus setting, the main lens moves with respect to the block MLA-sensor. Therefore, we can consider that is equivalent to four physical different cameras, allowing us to validate our methodology on multiple setups.

**Relative depth setup.** The camera is mounted on a linear motion table with micro-metric precision as presented in Labussière et al. (2020). The target plane is orthogonal to the translation axis, and the camera optical axis is aligned with this axis. Images with known relative translation between each frame are then used to estimate depths and compared to the ground truth, for quantitative evaluation.

**3D scenes setup.** We used a 3D lidar scanner, a Leica ScanStation P20 (LP20), that allowed us to capture a color point cloud with high precision that can be used as metric ground truth 3D data. The LP20 is configured with no HDR and with a resolution of 1.6 mm at 10 m.

## 7.2. Software environment

All images have been acquired using the MultiCamStudio free software (v6.15.1.3573) of the Euresys company. We set the shutter speed to 5 ms. For Raytrix data, we use their proprietary software RxLive (v4.0.50.2) to calibrate the camera, and compute the depth maps used in the evaluation. Our source code has been made publicly available at https://github.com/comsee-research/libpleno, and https://github.com/comsee-research/blade.

## 7.3. Datasets

**R12-A, B, C.** For the relative depth evaluation, we used the datasets presented in Labussière et al. (2020). We evaluated then our depth estimation framework for focus distances $h \in \{450, 1000, \infty\}$ mm (i.e., datasets R12-A, B, C respectively), representing then three different working depth ranges.
**R12-E, ES, ELP20.** For the 3D scenes evaluation, we introduced a new dataset, namely R12-E, corresponding to a new focus distance *h* = 2133 mm. The camera has been calibrated using Compote (Labussière et al., 2022). From this configuration, we created two sub-datasets: 1) a simulated dataset built upon our own simulator based on raytracing<fnref n="2" /> to generate images with known absolute position, named R12-ES; 2) a dataset composed of several real-world 3D scenes with metric ground truth acquired with the LP20, for object distances ranging from 400 mm to 1500 mm, exploiting then the whole distance range offered by the camera setup. The latter dataset, named R12-ELP20, includes five scenes: 1) a scene for extrinsic calibration, containing checker corner targets, named Calib; 2) two scenes containing textured planar objects, named Plane-1 and Plane-2; 3) and two scenes containing various figurines, named Figurines-1 and Figurines-2. Each scene is composed of: a colored point cloud (with spatial (x, y, z) information, color information (r, g, b), and intensity information) in format .ptx, .pts and .xyz; 3D positions of the targets in the lidar reference frame; two raw plenoptic images in rgb color and two raw plenoptic images in bayer; finally, photos and labels of the scene.

Our datasets have been made publicly available, and can be
downloaded from our github repository with their descriptions.
We believe it is a first step toward helping future research to
evaluate their method on metric space.

7.4. Lidar-camera calibration

In order to transform the lidar point cloud data **P**<sub>*l*</sub> in the same reference frame as the camera, we calibrate the extrinsic parameters between those frames, i.e., the transformation <sup>c</sup><b>T</b><sub>*l*</sub> ∈ SE (3) such that **P**<sub>*c*</sub> = <sup>c</sup><b>T</b><sub>*l*</sub><b>P</b><sub>*l*</sub>, where **P**<sub>*c*</sub> is the point cloud data expressed in the camera frame. The calibration is a four-steps process. First, using the Leica station, we acquired a point cloud of a scene containing calibration targets, and associated the 3D coordinates **p**<sub>*l*</sub> of the corners manually from the point cloud. This set of corners forms a points constellation, noted **C**<sub>*l*</sub>. Second, a raw plenoptic image of the same scene is acquired with the plenoptic camera, and BAP features {**p**<sub>*k*,*l*</sub>} are extracted. The features are clustered and associated to the points constellation. For each cluster of observations, the barycenter is computed. Those barycenters are initial estimates of the projections of the points constellation through the main lens using a standard pinhole model. Third, the initial transformation <sup>c</sup><b>T</b><sub>*l*</sub> is estimated using the Perspective-n-Point (PnP) algorithm Kneip et al. (2011), like in classic pinhole imaging system. Finally, the transformation <sup>c</sup><b>T</b><sub>*l*</sub> is refined by minimizing the reprojection error of the points constellation, such that

$$
\arg\min_{^c T_l} \sum_{p_l \in C_l} \sum_{k,l} \| p_{k,l} - \Pi_{k,l} (^c T_l p_l) \|^2 . \quad (40)
$$

The optimization is conducted using the Levenberg-Marquardt
algorithm. The point cloud data **P**<sub>*l*</sub> can now be expressed in the
camera frame. It is thus used as ground truth for quantitative
evaluations.

**8. Results and discussions**

In the following, a relative error εz for a known displacement δz is computed as the mean absolute relative difference between the estimated displacement δ̂z and the ground truth, for each pair of frames (n, m) separated by a distance δz, i.e.,

$$
\epsilon_z(\delta_z) = \eta^{-1} \sum_{(n,m) | z_m - z_n = \delta_z} \frac{|\delta_z - \delta_z|}{\delta_z}, \quad (41)
$$

where $\hat{\delta}_z = \hat{z}_m - \hat{z}_n$, and $\eta$ is a normalization constant corresponding to the number of frames pairs. Except if indicated otherwise, $\hat{z}$ is the median of the depth estimates of the considered frame.

8.1. Depth scaling error analysis

**Depth scaling correction.** Coarse depth estimation for datasets R12-A, B, C, ES is performed on images corresponding to planar checkerboards orthogonal to the optical axis and uniformly distributed in the range of distances, whilst for R12-E, depth estimation is performed on hand-held checkerboards, leading to noisier evaluation of depth estimates. The reported depth associated to each frame is the median of the depth estimates. Calibrated depth scaling coefficients are reported in Table 1, along with their median scale error after correction for the evaluation datasets. Depths before and after correction are illustrated in Figure 7. A positive error means the estimated object is smaller than the ground truth, and a negative error means that the estimated object is bigger than the ground truth. The absolute error grows when getting farther from the focus distance. All corrected distances have a nearly null scale error. For all datasets, with real data from four configurations and also in simulation, our methodology successfully corrects the scale, with a final median scale error εscale of less than 0.05 %, indicating thus nearly no scale error.

**Analysis on simulated data.** We investigated scale error on simulated images of the R12-ES dataset. Depths are estimated based on the coarse depth estimation framework, for ground truth distances from 500 mm to 1900 mm with a step of 100 mm. Without scale correction, depths are estimated from 610.9 mm to 1917.4 mm with a mean relative error εz = 8.06 %. After scale correction, depths are closer to the ground truth and are estimated from 495.7 mm to 1841.8 mm with a mean relative error reduced by a factor two, εz = 3.78 %. As shown by Figure 7, depth scaling error appears even in simulated data, showing that this phenomenon must be added to the inverse projection model to reach precise and accurate depth measurements.

**Scaling model comparison.** Secondly, we evaluated the choice of the empirical scaling model and presented the results for the dataset R12-C. As illustrated in Figure 7, the quadratic model performs slightly better than the linear model, as the slope of the fitted line is closer to zero compared to the slope of the linear fitted model. This is confirmed by the median scale error after correction reported in Table 1 which is reduced by a factor two with the quadratic model. In the following, depths will be corrected with the quadratic model.

<footnote n="2">²Available at https://github.com/comsee-research/prism</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 7: Scale errors before and after correction as function of the distance for the datasets R12-A (a), R12-B (b), R12-C (c), R12-E (e) and R12-ES (f) with their associated fitting functions. A positive error means the object is smaller than the reference, and a negative error means that the objects is bigger than the reference. The absolute error grows when getting farther from the focus distance. The corrected distance has a nearly null error. Sub-figures (c) and (d) give a comparison of the linear model versus the quadratic model scale correction on the dataset R12-C. The quadratic model performs slightly better than the linear model, as the slope of the fitted line is closer to zero compared to the slope of the linear fitted model.</figcaption></figure>

<table><caption>Table 1: Depth scaling coefficients for datasets R12-A, B, C and R12-E, ES with the median scale error after correction. For dataset R12-C, the error for the linear model is also reported.</caption><thead><tr><th></th><th>γ<sub>2</sub> (×10<sup>-5</sup>)</th><th>γ<sub>1</sub></th><th>γ<sub>0</sub></th><th>ε<sub>scale</sub> (%)</th></tr></thead><tbody><tr><td>R12-A</td><td>79.709</td><td>0.625</td><td>31.512</td><td>-0.014</td></tr><tr><td>R12-B</td><td>23.017</td><td>0.796</td><td>10.913</td><td>0.022</td></tr><tr><td>R12-C</td><td>2.736</td><td>0.883</td><td>10.910</td><td>0.024</td></tr><tr><td>R12-C</td><td>-</td><td>0.929</td><td>-6.101</td><td>0.043</td></tr><tr><td>R12-ES</td><td>14.431</td><td>0.667</td><td>35.910</td><td>-0.023</td></tr><tr><td>R12-E</td><td>4.617</td><td>0.912</td><td>-2.004</td><td>0.047</td></tr></tbody></table>

## 8.2. Relative depth evaluation

We compared our method with (BLADE) and without (BLADEu) depth scaling on relative depth estimation with state-of-the-art methods, including the Raytrix software (Perwass and Wietzke, 2012), corresponding to the model of Heinze et al. (2016) (denoted RTRX), and depth estimation with the model of Noury et al. (2017) using only the disparity (denoted DISP) in our framework. The latter provides a baseline using a classic block matching technique but that does not leverage blur information. The relative depth errors along the z-axis with respect to the ground truth displacement from the closest frame are reported in Figure 8 for datasets R12-A (a), R12-B (b) and R12-C (c). The mean error

with its confidence interval across all datasets is illustrated in (d) for each method, and is: for BLADE, $ε_z = 4.09 ± 0.85\%$; for BLADEu, $ε_z = 11.05 ± 6.61\%$; for DISP, $ε_z = 7.88 ± 3.74\%$; and, for RTRX, $ε_z = 6.76 ± 3.96\%$. First, we see that our scale correction effectively improves the depth estimation results, as expected from our depth scaling error analysis. With scale correction it outperforms the other methods, meaning that leveraging both defocus and correspondences provides accurate results. This behavior is also validated in simulation on R12-ES, where the mean error after correction is of the same order, i.e., $ε_z = 3.78\%$.

Secondly, the BLADE method presents the lowest standard deviation, a stable error across all distances, and errors of the same order for all configurations. The other methods vary significantly across the datasets, making our method the only one able to generalize to several configurations without losing precision. It thus means that leveraging both defocus and correspondences provides precise results.

Finally, for datasets R12-A, B, ES, the scale correction clearly improves the relative depth estimates. In R12-C, the results are similar with and without. The errors are nearly constant for the evaluated range when uncorrected as illustrated in Figure 7. The effect of scale correction is less visible as it can be assimilated to a bias correction.

## 8.3. Absolute depth evaluation on 3D scenes

We used the dataset R12-ELP20 to evaluate our depth estimation framework on absolute metric depth estimates.

**Central sub-aperture depth map.** To compare depth estimates, we generated for each scene the central sub-aperture depth map (CSAD) as follows:

1. From the raw depth map, we back-project each pixel having a virtual depth hypothesis into a 3D point in metric space;

2. We replace the plenoptic camera model by a pinhole model, where the sensor is now at a distance $F$ from the main lens, and we increase the pixel size by a factor $S$ (here, $S = 4$, the final resolution is thus $1020 × 767$ pixels);

3. We project each point of the point cloud with the new pinhole model, using a z-buffer like technique, and attributing the minimum depth value to the pixel (or the median value for noisy data);

4. And finally, we filter the resulting depth map image by applying a median filter and a morphological erosion to reduce noise.

For generating ground truth CSADs, we replace the first step by simply applying the extrinsic transformation $cT_l$ so that the 3D points are expressed in camera frame.

**Evaluated methods.** We evaluated our BLADE framework considering the following variations: 1) using relative blur information (B) or only disparity (D); 2) using the coarse (C) or the refined (R) estimation; and 3) using the scale-corrected (S) or scale-uncorrected (U) model. Note that we use the same intrinsic parameters for all evaluations. In the end, we presented the results for eight methods. For each method, we generated CSAD
<figure><img src="image_10.png" /><figcaption>Fig. 8: Relative depth error along the z-axis with respect to the ground truth displacement from the closest frame, for datasets R12-A (a), R12-B (b) and R12-C (c). The error $ε_z$ is expressed in percentage of the estimated distance, and truncated to 12 % to ease the readability and the comparison. The mean error with its confidence interval across all datasets for our method with (BLADE) and without scale correction (BLADEu), for the model of Noury et al. (2017) using only disparity (DISP), and for the proprietary software RxLive (RTRX) are reported in (d). Please refer to the color version for better visualization.</figcaption></figure>

for each of the five scenes of R12-ELP20, and compared them to the ground truths. It allows us to validate both the scaling model, and the improvement of accuracy due to the defocus information.

**Metrics.** To analyze the depth error, we computed a quality map as the absolute difference (AD) between the depth map and the ground truth. As the maps are sparse and not dense, we generated a mask corresponding to pixels in common where depth estimates are available. Errors are thus computed only for pixels in the mask. Finally, statistics over the errors are computed. We used the percentiles (at 25 and 75 %) and the median to describe the overall error of the depth map estimates.

**Results.** Statistics for all the variations of the BLADE framework for all the scenes are reported in Table 2. Bold font indicates the best results. The last column reports the mean of the median errors for all scenes. Snapshot of the colored point cloud, along with the ground truth CSAD, are reported for each scene in Figure 9. Depth map, mask and quality map are illustrated for the coarse and refined variations (i.e., B/C/S and B/R/S) of our framework. Results for the other variations can be found in supplementary material.

From the reported errors, the scenes can be divided into two groups: 1) the easy scenes (Calib, Plane-1 and Plane-2) containing mostly planar objects and presenting the lowest errors; and 2) the complex scenes (Figurines-1 and Figurines-2) containing more objects with less texture and more complex shapes, presenting a larger error. The errors distributions are similar for all the scenes, i.e., most of the errors are located on the objects' border.

First of all, the lower overall error is obtained for the variation leveraging blur in our coarse depth estimation framework (B/C/S). The mean median-error over all scenes is less than 19 mm, for distances ranging from 400 mm to 1500 mm. It corresponds to relative errors ranging from 1.27 % to 4.75 % of the

distance, which is coherent with the relative depth evaluation. For easy scenes, relative errors range from 0.96 % to 3.59 % of the distance. For complex scenes, relative errors range from 1.73 % to 6.49 % of the distance. As illustrated in the quality maps, most of the errors are located at object boundaries, whereas the errors are low everywhere else. This is due to our method not explicitly dealing with occlusion boundaries, leading to wrong estimates in those regions. Second, it is clear that the scaled variations outperform the unscaled ones, as expected from the previous experiments. Our depth scaling calibration efficiently corrects the depth estimates, and allows to generate metric depth map without scale errors. Third, integrating both correspondence and defocus cues shows lower median error and lower percentiles for all scenes, compared to only using disparity. It is due to compensating the mismatch of focus, which both helps the matching process and provides more cues about depth to be exploited. Finally, coarse estimation leads to denser depth maps as illustrated in Figure 9, and is able to extrapolate depth where there is enough texture within the micro-image using a locally planar approximation. Refined estimation captures depth information only on the textured areas. The maps are sparser but wrong estimates at object boundaries do not spread as much as using the locally planar approximation. Since proportionally there is more pixels corresponding to borders, the error is a bit higher for the refined version. With better strategy to deal with occlusion, results are expected to be similar for both. The choice of using which variation depends on the application behind: if we need more details, refined is preferred; if a denser map is required, the coarse is preferred.

Note that for the scene Figurines-2, the variation D/R/U has a smaller median error than the others. Recall that for R12-E, objects are reconstructed farther when unscaled (i.e., the scale error is negative, see Figure 7). As most of the errors appear on the objects' boundaries, objects in foreground are closer to the reference ones in background, and the difference of depth
<table><caption>Table 2: Statistics (percentiles and median) of the absolute difference (AD) error of central sub-aperture depth map for each variation of our BLADE framework (using relative blur information (B) or only disparity (D); using the coarse (C) or the refined (R) estimation; and using the scale-corrected (S) or scale-uncorrected (U) model), on the scenes of dataset R12-ELP20. All errors are expressed in mm. The last column indicates the mean of the median errors for all scenes.</caption><thead><tr><th rowspan="2">B/D</th><th rowspan="2">C/R</th><th rowspan="2">S/U</th><th colspan="3">Calib</th><th colspan="3">Plane-1</th><th colspan="3">Plane-2</th><th colspan="3">Figurines-1</th><th colspan="3">Figurines-2</th><th rowspan="2">Total</th></tr><tr><th>Q<sub>25</sub></th><th>med.</th><th>Q<sub>75</sub></th><th>Q<sub>25</sub></th><th>med.</th><th>Q<sub>75</sub></th><th>Q<sub>25</sub></th><th>med.</th><th>Q<sub>75</sub></th><th>Q<sub>25</sub></th><th>med.</th><th>Q<sub>75</sub></th><th>Q<sub>25</sub></th><th>med.</th><th>Q<sub>75</sub></th></tr></thead><tbody><tr><td>B</td><td>C</td><td>S</td><td><strong>7.661</strong></td><td><strong>17.761</strong></td><td><strong>39.146</strong></td><td><strong>6.498</strong></td><td><strong>13.940</strong></td><td><strong>26.402</strong></td><td><strong>5.482</strong></td><td><strong>11.370</strong></td><td><strong>24.468</strong></td><td><strong>11.720</strong></td><td><strong>24.867</strong></td><td><strong>47.270</strong></td><td><strong>12.310</strong></td><td>27.058</td><td>54.395</td><td><strong>18.999</strong></td></tr><tr><td>D</td><td>C</td><td>S</td><td>7.916</td><td>18.648</td><td>39.156</td><td>7.002</td><td>14.672</td><td>27.599</td><td>5.644</td><td>12.007</td><td>25.022</td><td>12.446</td><td>26.307</td><td>50.554</td><td>13.514</td><td>29.809</td><td>57.666</td><td>20.289</td></tr><tr><td>B</td><td>R</td><td>S</td><td>8.831</td><td>19.961</td><td>43.880</td><td>11.765</td><td>21.818</td><td>34.903</td><td>9.758</td><td>17.951</td><td>28.925</td><td>12.011</td><td>25.282</td><td>50.377</td><td>14.203</td><td>29.442</td><td>56.900</td><td>22.891</td></tr><tr><td>D</td><td>R</td><td>S</td><td>9.736</td><td>21.992</td><td>49.872</td><td>14.800</td><td>25.904</td><td>39.295</td><td>11.074</td><td>20.613</td><td>32.054</td><td>13.551</td><td>28.016</td><td>55.308</td><td>16.911</td><td>34.125</td><td>62.574</td><td>26.130</td></tr><tr><td>B</td><td>C</td><td>U</td><td>34.450</td><td>50.106</td><td>68.997</td><td>30.743</td><td>42.452</td><td>58.241</td><td>33.657</td><td>42.905</td><td>55.664</td><td>21.417</td><td>43.577</td><td>67.571</td><td>18.415</td><td>37.834</td><td>63.522</td><td>43.375</td></tr><tr><td>D</td><td>C</td><td>U</td><td>31.105</td><td>47.071</td><td>66.983</td><td>27.581</td><td>39.358</td><td>54.387</td><td>30.754</td><td>40.081</td><td>51.925</td><td>18.325</td><td>40.323</td><td>63.975</td><td>15.713</td><td>33.372</td><td>58.125</td><td>40.041</td></tr><tr><td>B</td><td>R</td><td>U</td><td>27.114</td><td>41.327</td><td>63.583</td><td>17.553</td><td>28.304</td><td>40.732</td><td>20.025</td><td>30.192</td><td>39.835</td><td>16.539</td><td>30.243</td><td>52.072</td><td>14.322</td><td>28.878</td><td>48.591</td><td>31.789</td></tr><tr><td>D</td><td>R</td><td>U</td><td>26.274</td><td>41.522</td><td>64.952</td><td>14.098</td><td>24.255</td><td>37.062</td><td>17.115</td><td>27.317</td><td>38.432</td><td>15.272</td><td>28.690</td><td>50.152</td><td>12.787</td><td><strong>26.733</strong></td><td><strong>46.932</strong></td><td>29.703</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 9: Snapshot view of the colored point cloud, along with the ground truth central sub-aperture depth map (CSAD) for each scene of the dataset R12-ELP20. CSAD, mask and quality map representing the absolute difference (AD) error are illustrated for the coarse and refined variations (i.e., B/C/S and B/R/S) of our framework. Results for the other variations can be found in supplementary materials. Please refer to the color version for better visualization.</figcaption></figure>

is then smaller. Furthermore, in the refined case, most of the estimations are done only on locally textured areas such as those boundaries. Combination of those two factors allows to explain why this variation has a smaller error.

## 9. Conclusion

With a plenoptic camera, depth estimation and 3D reconstruction can be performed directly from a single acquisition, with scale information. Inherently from its design, a multi-focus plenoptic camera captures both correspondence and defocus cues which are complementary for depth estimation.

In this paper, we presented a new metric depth estimation algorithm using only raw images from multi-focus plenoptic cameras. It is especially suited for the multi-focus configuration where several micro-lenses with different focal lengths are used. First, we completed our previous camera model by introducing the inverse projection model. We showed that depth recovered from virtual depth hypothesis suffers from a scale error. We included then an empirical depth scaling correction model as well as a methodology to calibrate it. Second, we introduced our blur aware depth estimation (BLADE) framework, improving disparity estimation for defocus stereo images via compensating
the mismatch of focus, i.e., integrating both correspondence and defocus cues. We formulated then a new residual error to leverage blur information for depth estimation which is used in two variations of our framework to recover depth either per micro-image or per pixel. Finally, our results showed that introducing defocus cue improves the depth estimation. We demonstrated the effectiveness of our depth scaling calibration on relative depth estimation setup and on real-world 3D complex scenes with ground truths acquired with a 3D lidar scanner. With our method, we obtained a median relative depth error ranging from 1.27% to 4.75% of the distance. In our experiments it corresponds to a median error of less than 19 mm, for distances ranging from 400 mm to 1500 mm.

**Discussions on limitations and improvements.** Regarding the generalization of our method to other plenoptic cameras, i.e., simple focused or unfocused plenoptic cameras, the same camera model can be exploited, though no defocus cues are included in the depth estimation process. Thanks to the scale correction model, metric precise and accurate depth should be expected. One limitation of our method is that computational cost is not addressed here. Our method is implemented purely on CPU with a brute force algorithm for finding the minimum of the cost function (about 10 to 30 seconds per frame). We can leverage neighborhood information and implement a belief propagation strategy to avoid having to compute an initial hypothesis for each micro-image. Combined with a GPU implementation, computation time can be significantly improved. Furthermore, as most of the errors are located at object boundaries, we can adapt several strategies: to explicitly manage occlusions; to check coherence between estimates ref-target and target-ref; to model uncertainty as the weight function in the cost function; and we can proceed to a robust filtering to eliminate outliers. Global refinement can also be considered as further steps to improve depth estimates. Future work will include the discussed improvements, as well as using the depth map for metrology and robotic applications.

## Acknowledgments

This work was supported by the AURA Region and the European Union (FEDER) through the MMII project of CPER 2015-2020 MMaSyF challenge. We would like to thank Laurent Malaterre, Trong-Lanh Rémi Nguyen, Ruddy Theodose and Etienne Cadet for their help during the acquisitions. We also thank the reviewers for their insightful comments.

## References

* Alain, M., Smolic, A., 2020. A Spatio-Angular Binary Descriptor for Fast Light Field Inter View Matching, in: Proceedings - International Conference on Image Processing (ICIP), pp. 2636-2640. doi:10.1109/ICIP40778.2020.9191118.

* Anisimov, Y., Wasenm, O., 2019. Rapid Light Field Depth Estimation with Semi-Global Matching. arXiv:1907.13449v1 arXiv:1907.13449v1.

* Bishop, T.E., Favaro, P., 2012. The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 972-986. doi:10.1109/TPAMI.2011.168.

* Bolles, R.C., Baker, H.H., Marimont, D.H., 1987. Epipolar-plane image analysis: An approach to determining structure from motion. International Journal of Computer Vision 1, 7-55. doi:10.1007/BF00128525, arXiv:1011.1669v3.

* Chen, C., Lin, H., Yu, Z., Kang, S.B., Yu, J., 2014. Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras, in: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1518-1525. doi:10.1109/CVPR.2014.197.

* Chen, C.H., Zhou, H., Ahonen, T., 2015. Blur-aware disparity estimation from defocus stereo images. Proceedings of the IEEE International Conference on Computer Vision 2015 Inter, 855-863. doi:10.1109/ICCV.2015.104.

* Cunha, F., Thomaz, L.A., Tavora, L.M.N., Assuncao, P.A.A., Fonseca-Pinto, R., Faria, S.M.M., 2020. Robust Depth Estimation From Multi-Focus Plenoptic Images, in: IEEE International Conference on Image Processing (ICIP), pp. 2626-2630. doi:10.1109/ICIP40778.2020.9190674.

* Custodio, J.P.B.L., 2014. Depth Estimation using Light-Field Cameras. Master thesis. Universidade de Coimbra.

* Dansereau, D.G., Girod, B., Wetzstein, G., 2019. LiFF: Light Field Features in Scale and Depth, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8034-8043. doi:10.1109/CVPR.2019.00823, arXiv:1901.03916.

* Ferreira, R., Goncalves, N., 2016. Fast and accurate micro lenses depth maps for multi-focus light field cameras. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9796 LNCS, 309-319. doi:10.1007/978-3-319-45886-1_25.

* Fleischmann, O., Koch, R., 2014. Lens-Based Depth Estimation for Multi-focus Plenoptic Cameras, in: Electronics and Power. volume 24, pp. 410-420. doi:10.1007/978-3-319-11752-2_33.

* Georgiev, T., Lumsdaine, A., 2012. The multifocus plenoptic camera, in: Digital Photography VIII, International Society for Optics and Photonics. SPIE. pp. 69-79. doi:10.1117/12.908667.

* Grossmann, P., 1987. Depth from focus. Pattern Recognition Letters 5, 63-69. doi:10.1016/0167-8655(87)90026-2.

* Hahne, C., Aggoun, A., Velisavljevic, V., Fiebig, S., Pesch, M., 2018. Baseline and Triangulation Geometry in a Standard Plenoptic Camera. International Journal of Computer Vision 126, 21-35. doi:10.1007/s11263-017-1036-4.

* Heber, S., Yu, W., Pock, T., 2016. U-shaped Networks for Shape from Light Field. Proceedings of the British Machine Vision Conference (BMVC) 1, 37.1-37.12. doi:10.5244/C.30.37.

* Heinze, C., Spyropoulos, S., Hussmann, S., Perwaß, C., 2016. Automated Robust Metric Calibration Algorithm for Multifocus Plenoptic Cameras. IEEE Transactions on Instrumentation and Measurement 65, 1197-1205.

* Held, R.T., Cooper, E.A., Banks, M.S., 2012. Blur and disparity are complementary cues to depth. Current Biology 22, 426-431. doi:10.1016/j.cub.2012.01.033.

* Hog, M., Sabater, N., Vandame, B., Drazic, V., 2017. An Image Rendering Pipeline for Focused Plenoptic Cameras. IEEE Transactions on Computational Imaging 3, 811-821. doi:10.1109/tci.2017.2710906.

* Huang, C.T., 2018. Empirical Bayesian Light-Field Stereo Matching by Robust Pseudo Random Field Modeling, 1-14doi:10.1109/TPAMI.2018.2809502.

* Jeon, H.G., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y.W., Kweon, I.S., 2015. Accurate depth map estimation from a lenslet light field camera, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1547-1555. doi:10.1109/CVPR.2015.7298762.

* Jeon, H.G., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y.W., Kweon, I.S., 2018. Depth from a Light Field Image with Learning-based Matching Costs. IEEE Transactions on Pattern Analysis and Machine Intelligence 8828, 1-1. doi:10.1109/TPAMI.2018.2794979.

* Jin, J., Hou, J., 2021. Occlusion-aware Unsupervised Learning of Depth from 4-D Light Fields. arXiv:2106.03043, 1-10arXiv:2106.03043.

* Johannsen, O., Honauer, K., Goldluecke, B., Alperovich, A., Battisti, F., Bok, Y., Brizzi, M., Carli, M., Choe, G., Diebold, M., Gutsche, M., Jeon, H.G., Kweon, I.S., Park, J., Park, J., Schilling, H., Sheng, H., Si, L., Strecke, M., Sulc, A., Tai, Y.W., Wang, Q., Wang, T.C., Wanner, S., Xiong, Z., Yu, J., Zhang, S., Zhu, H., 2017. A Taxonomy and Evaluation of Dense Light Field Depth Estimation Algorithms. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2017-July, 1795-1812. doi:10.1109/CVPRW.2017.226.

* Johannsen, O., Sulc, A., Goldluecke, B., 2016. What Sparse Light Field Coding Reveals about Scene Structure. IEEE Conference on Computer Vision and Pattern Recognition, 3262-3270doi:10.1109/CVPR.2016.355, arXiv:1011.1669v3.
Kiefer, J., 1953. Sequential Minimax Search for a Maximum. Proceedings of the American Mathematical Society 4, 502. doi:10.2307/2032161.

Kim, M.J., Oh, T.H., Kweon, I.S., 2014. Cost-aware depth map estimation for Lytro camera, in: IEEE International Conference on Image Processing (ICIP), pp. 36–40. doi:10.1109/ICIP.2014.7025006.

Kneip, L., Scaramuzza, D., Siegwart, R., 2011. A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2969–2976doi:10. 1109/CVPR.2011.5995464.

Konz, J., Zeller, N., Quint, F., Stilla, U., 2016. Depth Estimation from Micro Images of a Plenoptic Camera, in: BW-CAR Symposium on Information and Communication Systems (SInCom), pp. 17–23.

Labussière, M., Teulière, C., Bernardin, F., Ait-Aider, O., 2020. Blur Aware Calibration of Multi-Focus Plenoptic Camera, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2542–2551. doi:10. 1109/CVPR42600.2020.00262, arXiv:2004.07745.

Labussière, M., Teulière, C., Bernardin, F., Ait-Aider, O., 2022. Leveraging Blur Information for Plenoptic Camera Calibration. International Journal of Computer Vision, 1–23doi:10.1007/s11263-022-01582-z.

Lai, S.H., Fu, C.W., Chang, S., 1992. A Generalized Depth Estimation Algorithm with a Single Image 14, 405–411. doi:10.1109/34.126803.

Li, K., Zhang, J., Sun, R., Zhang, X., Gao, J., 2020. EPI-based Oriented Relation Networks for Light Field Depth Estimation. arXiv:2007.04538, 1-l1arXiv:2007.04538.

Liang, C.K., Ramamoorthi, R., 2015. A Light Transport Framework for Lenslet Light Field Cameras. ACM Transactions on Graphics 34, 1–19. doi:10. 1145/2665075, arXiv:arXiv:1204.6216v2.

Liu, X., Fu, D., Wu, C., Si, Z., 2019. The Depth Estimation Method Based on Double-Cues Fusion for Light Field Images, in: Proceedings of the 11th International Conference on Modelling, Identification and Control (ICMIC), pp. 719–726. doi:10.1007/978-981-15-0474-7_67.

Ma, H., Li, H., Qian, Z., Shi, S., Mu, T., 2018. VommaNet: an End-to-End Network for Disparity Estimation from Reflective and Texture-less Light Field Images. arXiv:1811.07124 arXiv:1811.07124.

Ng, R., Levoy, M., Duval, G., Horowitz, M., Hanrahan, P., 2005. Light Field Photography with a Hand-held Plenoptic Camera. Technical Report. Stanford University. doi:10.1.1.163.488, arXiv:0070242542.

Noury, C.A., 2019. Etalonnage de caméra pléoptique et estimation de profondeur à partir des données brutes. Ph.d thesis. Université Clermont Auvergne.

Noury, C.A., Teulière, C., Dhome, M., 2017. Light-Field Camera Calibration from Raw Images. International Conference on Digital Image Computing: Techniques and Applications (DICTA), 1–8doi:10.1109/DICTA. 2017.8227459.

Palmieri, L., Koch, R., 2017. Optimizing the Lens Selection Process for Multi- focus Plenoptic Cameras and Numerical Evaluation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2017- July, 1763–1774. doi:10.1109/CVPRW.2017.223.

Pentland, A.P., 1987. A New Sense for Depth of Field. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-9, 523–531. doi:10.1109/ TPAMI.1987.4767940.

Perez Nava, F., Luke, J.P., 2009. Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera, in: IEEE 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, pp. 1–4. doi:10.1109/3DTV.2009.5069675.

Perwass, C., Wietzke, L., 2012. Single Lens 3D-Camera with Extended Depth- of-Field, in: Human Vision and Electronic Imaging XVII, SPIE. p. 829108. doi:10.1117/12.909882.

Scharstein, D., Szeliski, R., 2002. A taxonomy and evaluation of dense two- frame stereo correspondence algorithms. International Journal of Computer Vision 47, 7–42. doi:10.1023/A:1014573219977.

Schechner, Y.Y., Kiryati, N., 2000. Depth from Defocus vs. stereo: How different really are they? International Journal of Computer Vision 39, 141–162. doi:10.1023/A:1008175127327.

Sheng, H., Zhao, P., Zhang, S., Zhang, J., Yang, D., 2018. Occlusion-aware depth estimation for light field using multi-orientation EPIs. Pattern Recognition 74, 587–599. doi:10.1016/j.patcog.2017.09.010.

Shin, C., Jeon, H.G., Yoon, Y., Kweon, I.S., Kim, S.J., 2018. EPINET: A Fully- Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images. IEEE/CVF Conference on Computer Vision and Pattern Recognition doi:10.1109/CVPR.2018.00499, arXiv:1804.02379.

Subbarao, M., 1989. Determining Distance from Defocused Images of Simple Objects. Technical Report.

Subbarao, M., Surya, G., 1994. Depth from defocus: A spatial domain approach. International Journal of Computer Vision 13, 271–294. doi:10. 1007/BF02028349.

Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R., 2013. Depth from combining defocus and correspondence using light-field cameras. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2, 673–680. doi:10.1109/ICCV.2013.89, arXiv:NIHMS150003.

Tao, M.W., Srinivasan, P.P., Malik, J., Rusinkiewicz, S., Ramamoorthi, R., 2015. Depth from shading, defocus, and correspondence using light-field angular coherence, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1940–1948. doi:10.1109/CVPR.2015.7298804.

Tosic, I., Berkner, K., 2014. Light Field Scale-Depth Space Transform for Dense Depth Estimation, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 441–448. doi:10.1109/CVPRW.2014.71.

de Villiers, J.P., Leuschner, F.W., Geldenhuys, R., 2008. Centi-pixel accurate real-time inverse distortion correction. Optomechatronic Technologies 7266, 726611. doi:10.1117/12.804771.

Wang, T.C., Efros, A.A., Ramamoorthi, R., 2015. Occlusion-aware depth estimation using light-field cameras. Proceedings of the IEEE International Conference on Computer Vision 2015 Inter, 3487–3495. doi:10.1109/ICCV. 2015.398.

Wanner, S., Fehr, J., Jähne, B., 2011. Generating EPI representations of 4D light fields with a single lens focused plenoptic camera. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6938 LNCS, 90–101. doi:10.1007/978-3-642-24028-7_9, arXiv:arXiv:1011.1669v3.

Wanner, S., Goldluecke, B., 2012. Globally consistent depth labeling of 4D light fields, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 41–48. doi:10.1109/CVPR.2012.6247656.

Williem, Park, I.K., 2018. Cost aggregation benchmark for light field depth estimation. Journal of Visual Communication and Image Representation 56, 38–51. doi:10.1016/j.jvcir.2018.08.015.

Wu, G., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., Chai, T., Liu, Y., 2017. Light Field Image Processing: An Overview. IEEE Journal on Selected Topics in Signal Processing 11, 926–954. doi:10.1109/JSTSP. 2017.2747126.

Xiao, Z., Zhang, M.E., Jin, H., Guillemot, C., 2021. A light field FDL-HSIFT feature in scale-disparity space, in: IEEE International Conference on Image Processing (ICIP).

Xu, S., 2016. 4D Light Field Reconstruction and Scene Depth Estimation from Plenoptic Camera Raw Images. Ph.d thesis. National University of Ireland, Galway.

Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.S., 2015. Learning a Deep Convolutional Network for Light-Field Image Super-Resolution, in: IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 57– 65. doi:10.1109/ICCVW.2015.17, arXiv:1506.02753.

Yu, Z., Guo, X., Ling, H., Lumsdaine, A., Yu, J., 2013. Line assisted light field triangulation and stereo matching, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 2792–2799. doi:10.1109/ICCV.2013. 347.

Zeller, N., Quint, F., Stilla, U., 2016. Depth estimation and camera calibration of a focused plenoptic camera for visual odometry. ISPRS Journal of Photogrammetry and Remote Sensing 118, 83–100. doi:10.1016/j. isprsjprs.2016.04.010.

Zhang, S., Sheng, H., Li, C., Zhang, J., Xiong, Z., 2016. Robust depth estimation for light field via spinning parallelogram operator. Computer Vision and Image Understanding 145, 148–159. doi:10.1016/j.cviu.2015.12.007.

Zhang, Y., Dai, W., Xu, M., Zou, J., Zhang, X., Xiong, H., 2019. Depth Estimation from Light Field Using Graph-Based Structure-Aware Analysis. IEEE Transactions on Circuits and Systems for Video Technology PP. doi:10. 1109/TCSVT.2019.2954948.
# Supplementary Material

As supplementary material, we first provide details about the projection model regarding the distances $d(k, l)$ (i.e., the distance between the micro-lens and the sensor) and $D(k, l)$ (i.e., the distance between the micro-lens and the main lens) for each micro-lens $(k, l)$. Second, we comment on the depth scaling phenomenon. Third, we quantify the approximation used in Eq. 18. For completeness, we then report the intrinsic and extrinsic parameters for the camera setup used in the evaluation. Finally, we give the additional results for corrected disparity analysis and for all variations of the method for the absolute depth evaluation on 3D scenes.

## Appendix A. Details on direct projection model

As noted in Labussière et al. (2022), technically we should consider specific distances $d(k, l)$ (i.e., the distance between the micro-lens and the sensor) and $D(k, l)$ (i.e., the distance between the micro-lens and the main lens) for each micro-lens $(k, l)$.

It corresponds to take the MLA tilt into consideration in the projection model. In this paper, we improve our model by taking into account the effect of the MLA tilt with respect to the sensor. It is reduced by using an orthogonal approximation and thus considering specific distances $d(k, l)$ (i.e., the distance between the micro-lens and the sensor) and $D(k, l)$ (i.e., the distance between the micro-lens and the main lens) for each micro-lens $(k, l)$. To ease the reading, we only used the notation $D$ and $d$, but the quantities can be replaced by their corresponding orthogonal approximation.

## Appendix B. Comments on depth scaling

At the end of the BLADE process, we have a virtual depth estimate associated to each pixel. To obtain a metric depth, we use the inverse projection model given in Eq. 5, where $\rho$ is computed from the virtual depth by Eq. 4, i.e.,

$$ \rho = m \cdot v^{-1} + q'_{i} $$

The algorithm effectively retrieves the virtual depth hypothesis corresponding to the observed images, i.e., the relation

$$ \hat{v} = \frac{B}{B - \Delta p} \qquad (B.1) $$

is verified, where $\Delta p$ is the Euclidean distance between two observations. It means that the correct disparity is correctly estimated from image space. However, when mapping from the virtual space to the object space with the inverse projection model, we observe that reconstructed objects are scaled up to a certain factor, in the z-dimension (e.g., points are projected farther) but also in the xy-dimensions (e.g., objects appear bigger). The scale factor grows approximately linearly as function of the distance with respect to the focus distance (see Fig. 7). This phenomenon appears both on real and simulated data.

It is due to the limitations of the thin-lens model. The proposed model describes efficiently the projective geometry for a

<figure><img src="image_12.png" /><figcaption>Fig. B.10: Illustration of the ray-tracing from a pixel containing a checkerboard corner with a real aperture. All elements are illustrated at the same scale (except the pixel size). Blue plane represents the MLA and the sensor planes. The gray plane is the main lens plane. The green circle models the real aperture. The yellow ray is the chief ray emanating from the considered pixel and going through the micro-lens center. Black and white rays are the rays from the cone of light that reach the pixel. As illustrated, the chief ray is correctly hitting the checkerboard corner in object space. However, since not all the rays from the cone of light go through the aperture, integration of the radiance from the passing ones induces a shift in the observed radiance in sensor space. Indeed, the theoretical projection of the corner in yellow corresponding to the yellow chief ray is not matching the observed corner position in cyan.</figcaption></figure>

point whose all rays inside the projection cone attain the retina. Indeed, when simulating the imaging process with a large aperture, the retrieved depths from the simulated images do not suffer from a scale error. All rays can go through, and the final radiance is the correct sum of all rays contained within the cone. With real aperture, some rays, including the chief ray, emanating from the pixel do not always go through the main lens and are blocked, as shown in Figure B.10. The observed radiance should be the integrate of all rays within the cone but only a portion of it can go through the aperture, meaning that the radiance associated to the projection is not the one that is effectively observed. This phenomenon must thus be taken into account.

## Appendix C. Quantification of the approximation in Eq. 18

Let $a = 2nd$ be the distance to the micro-lens $(i)$, with $n \in [1, 10]$, such as $v = a/d = 2n$. Let $\& = nB \sin(\alpha)$ be the z-shift between the micro-lenses $(i)$ and $(j)$ separated by a distance $nB$. The virtual depth $v'$ for the micro-lens $(j)$ is then given by $v' = (a + \&) / (d + \&)$.

First, as $B < d/2$, and with $\sin(\alpha) \approx \alpha$ since $\alpha$ is small, we can can expressed $v'$ as

$$ v' = \frac{a + \&}{d + \&} = \frac{a + \alpha n B}{d + \alpha n B} < \frac{2nd + \alpha \frac{n}{2} d}{d + \alpha \frac{n}{2} d} < \frac{2 + \frac{\alpha}{2}}{\frac{1}{n} + \frac{\alpha}{2}} \qquad (C.1) $$
Second, let measures the approximation error of $v'$ by $v$. The relative error is given by

$$ \epsilon = \left| 1 - \frac{v}{v'} \right| = \left| 1 - 2n \cdot \frac{1/n + \alpha/2}{2 + \alpha/2} \right| \qquad (C.2) \\ = \left| 1 - \frac{2 + n\alpha}{2 + \alpha/2} \right| = \left| \frac{\alpha/2 - n\alpha}{2 + \alpha/2} \right| $$

Finally, from calibration, $\alpha < 0.0005$ rad, the relative error is thus bounded such that $0.0125 \% < \epsilon_{\%} < 0.2375 \%$, which makes the approximation valid. □

## Appendix D. Intrinsic and extrinsic parameters

Intrinsic parameters for datasets R12-A, B, C, E are reported in Table F.3 for our camera model Labussière et al. (2020) (BAP), for the model of Noury et al. (2017) (NOUR), and for the proprietary RxLive software of Raytrix corresponding to the model of Heinze et al. (2016) (RTRX).

The extrinsic parameters correspond to the pose between the camera and the 3D lidar scanner. In our experiments, using a con- stellation of five points, the obtained optimized transformation is

$$ cT_l = \begin{bmatrix} -0.88925 & -0.070239 & 0.451992 & 177.278 \\ -0.456949 & 0.0917283 & -0.88475 & -394.36 \\ 0.020684 & -0.993304 & -0.11366 & -266.858 \\ 0 & 0 & 0 & 1 \end{bmatrix} $$

## Appendix E. Complementary results

### Appendix E.1. On corrected disparity

We evaluated the impact of the orthogonal approximation of the micro-lens baseline with respect to the micro-image baseline, i.e., using the corrected disparity (corresponding to Eq. (29)) instead of the commonly used disparity formulation (correspond- ing to Eq. (28)). Analysis is performed on dataset R12-A. With- out depth scaling correction, we have for the approximated disparity a mean relative error $\epsilon_z = 19.38 \%$, which is reduced to $\epsilon_z = 14.99 \%$ only by using the corrected disparity formula- tion. We thus used the disparity obtained from Eq. (29) for all evaluations.

### Appendix E.2. On depth evaluation on 3D scenes

Recall that we evaluated our BLADE framework considering the following variations: 1) using relative blur information (B) or only disparity (D); 2) using the coarse (C) or the refined (R) esti- mation; and 3) using the scale-corrected (S) or scale-uncorrected (U) model.

Snapshot of the colored point cloud, along with the ground truth CSAD are reported for each scene in Figure F.11 and Figure F.12. Depth map, mask and quality map are illustrated for all variations using the relative blur (B) in Figure F.11, and for all variations using only the disparity (D) in Figure F.12.

## Appendix F. Source code and datasets redistribution

The datasets and the source code are publicly available to the community. Datasets can be downloaded from https://github.com/comsee-research/plenoptic-datasets, and more details can be found on this page.

We also developed an open-source C++ library for plenoptic camera named libpleno which is available at https://github.com/comsee-research/libpleno.

Along with this library, we developed a set of tools to calibrate a multi-focus plenoptic camera, named COMPOTE (standing for Calibration Of Multi-focus PlenOpTic camEra) which is avail- able at https://github.com/comsee-research/compote.

Finally, we developed a set of tools to estimate depth from raw images obtained by a multi-focus plenoptic camera, named BLADE (for BLur Aware Depth Esti- mation with a plenoptic camera) which is available at https://github.com/comsee-research/blade.
<table><caption>Table F.3: Intrinsic parameters for datasets R12-A, B, C and R12-E obtained by our method Labussière et al. (2020) (BAP), by the method of Noury et al. (2017) (NOUR), and by RxLive software Heinze et al. (2016) (RTRX). Blur proportionality coefficient and scaling coefficients are also reported for our method.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">R12-A (h = 450 mm)</th><th colspan="3">R12-B (h = 1000 mm)</th><th colspan="3">R12-C (h = ∞)</th><th>R12-E (h = 2133 mm)</th></tr><tr><th>BAP</th><th>NOUR</th><th>RTRX</th><th>BAP</th><th>NOUR</th><th>RTRX</th><th>BAP</th><th>NOUR</th><th>RTRX</th><th>BAP</th></tr></thead><tbody><tr><td>F</td><td>[mm]</td><td>49.885</td><td>54.761</td><td>47.709</td><td>50.011</td><td>51.177</td><td>50.894</td><td>50.099</td><td>51.644</td><td>51.564</td><td>50.119</td></tr><tr><td>Q<sub>1</sub></td><td>[×10<sup>-5</sup>]</td><td>24.63</td><td>6.194</td><td>-</td><td>4.661</td><td>1.650</td><td>-</td><td>13.84</td><td>1.292</td><td>-</td><td>-6.823</td></tr><tr><td>-Q<sub>2</sub></td><td>[×10<sup>-6</sup>]</td><td>3.032</td><td>0.800</td><td>-</td><td>0.516</td><td>0.264</td><td>-</td><td>2.723</td><td>0.576</td><td>-</td><td>-0.408</td></tr><tr><td>Q<sub>3</sub></td><td>[×10<sup>-8</sup>]</td><td>1.095</td><td>0.252</td><td>-</td><td>0.156</td><td>0.078</td><td>-</td><td>1.260</td><td>0.185</td><td>-</td><td>-0.047</td></tr><tr><td>P<sub>1</sub></td><td>[×10<sup>-5</sup>]</td><td>-11.1</td><td>-18.1</td><td>-</td><td>12.84</td><td>11.27</td><td>-</td><td>2.51</td><td>12.13</td><td>-</td><td>20.749</td></tr><tr><td>-P<sub>2</sub></td><td>[×10<sup>-5</sup>]</td><td>3.599</td><td>5.186</td><td>-</td><td>24.33</td><td>23.16</td><td>-</td><td>-3.072</td><td>-0.027</td><td>-</td><td>11.128</td></tr><tr><td>-Q<sub>1</sub></td><td>[×10<sup>-5</sup>]</td><td>24.29</td><td>6.195</td><td>-</td><td>4.685</td><td>1.697</td><td>-</td><td>13.78</td><td>1.316</td><td>-</td><td>-6.853</td></tr><tr><td>Q<sub>2</sub></td><td>[×10<sup>-6</sup>]</td><td>2.971</td><td>0.814</td><td>-</td><td>0.528</td><td>0.279</td><td>-</td><td>2.705</td><td>0.589</td><td>-</td><td>-0.394</td></tr><tr><td>-Q<sub>3</sub></td><td>[×10<sup>-8</sup>]</td><td>1.066</td><td>0.258</td><td>-</td><td>0.160</td><td>0.082</td><td>-</td><td>1.246</td><td>0.186</td><td>-</td><td>-0.037</td></tr><tr><td>-P<sub>1</sub></td><td>[×10<sup>-5</sup>]</td><td>-10.89</td><td>-18.00</td><td>-</td><td>12.86</td><td>11.32</td><td>-</td><td>2.50</td><td>12.17</td><td>-</td><td>21.031</td></tr><tr><td>P<sub>2</sub></td><td>[×10<sup>-5</sup>]</td><td>3.540</td><td>5.218</td><td>-</td><td>24.47</td><td>23.38</td><td>-</td><td>-3.067</td><td>-0.053</td><td>-</td><td>11.325</td></tr><tr><td>D</td><td>[mm]</td><td>56.860</td><td>62.341</td><td>-</td><td>52.140</td><td>53.213</td><td>-</td><td>49.356</td><td>50.728</td><td>-</td><td>50.585</td></tr><tr><td>-t<sub>x</sub></td><td>[mm]</td><td>10.93</td><td>9.480</td><td>-</td><td>12.15</td><td>12.38</td><td>-</td><td>12.53</td><td>13.24</td><td>-</td><td>12.876</td></tr><tr><td>-t<sub>y</sub></td><td>[mm]</td><td>7.996</td><td>8.087</td><td>-</td><td>6.165</td><td>5.965</td><td>-</td><td>8.237</td><td>7.400</td><td>-</td><td>6.616</td></tr><tr><td>-θ<sub>x</sub></td><td>[μrad]</td><td>388.9</td><td>460.3</td><td>-</td><td>488.4</td><td>555.4</td><td>-</td><td>409.8</td><td>442.2</td><td>-</td><td>441.6</td></tr><tr><td>θ<sub>y</sub></td><td>[μrad]</td><td>271.4</td><td>363.4</td><td>-</td><td>286.5</td><td>330.1</td><td>-</td><td>306.1</td><td>333.4</td><td>-</td><td>289.2</td></tr><tr><td>θ<sub>z</sub></td><td>[μrad]</td><td>29.5</td><td>25.6</td><td>41.9</td><td>30.9</td><td>33.9</td><td>41.9</td><td>33.9</td><td>39.9</td><td>36.6</td><td>37.6</td></tr><tr><td>ΔC</td><td>[μm]</td><td>127.46</td><td>127.40</td><td>127.36</td><td>127.47</td><td>127.41</td><td>127.36</td><td>127.46</td><td>127.41</td><td>127.36</td><td>127.45</td></tr><tr><td>f<sup>(1)</sup></td><td>[μm]</td><td>582.67</td><td>-</td><td>-</td><td>566.39</td><td>-</td><td>-</td><td>580.80</td><td>-</td><td>-</td><td>601.58</td></tr><tr><td>f<sup>(2)</sup></td><td>[μm]</td><td>524.02</td><td>-</td><td>-</td><td>507.09</td><td>-</td><td>-</td><td>515.57</td><td>-</td><td>-</td><td>562.19</td></tr><tr><td>f<sup>(3)</sup></td><td>[μm]</td><td>560.57</td><td>-</td><td>-</td><td>542.47</td><td>-</td><td>-</td><td>552.84</td><td>-</td><td>-</td><td>583.54</td></tr><tr><td>u<sub>0</sub></td><td>[pix]</td><td>2078.3</td><td>2343.4</td><td>-</td><td>1855.8</td><td>1811.9</td><td>-</td><td>1786.6</td><td>1654.9</td><td>-</td><td>1722.5</td></tr><tr><td>v<sub>0</sub></td><td>[pix]</td><td>1591.0</td><td>1573.7</td><td>-</td><td>1926.2</td><td>1962.2</td><td>-</td><td>1547.1</td><td>1699.7</td><td>-</td><td>1843.6</td></tr><tr><td>d</td><td>[μm]</td><td>337.13</td><td>391.90</td><td>-</td><td>326.72</td><td>361.01</td><td>-</td><td>330.32</td><td>357.82</td><td>-</td><td>340.87</td></tr><tr><td>κ</td><td>-</td><td>0.813</td><td>-</td><td>-</td><td>0.776</td><td>-</td><td>-</td><td>0.740</td><td>-</td><td>-</td><td>1.020</td></tr><tr><td>γ<sub>2</sub></td><td>[×10<sup>-5</sup>]</td><td>79.709</td><td>-</td><td>-</td><td>23.017</td><td>-</td><td>-</td><td>2.736</td><td>-</td><td>-</td><td>4.617</td></tr><tr><td>γ<sub>1</sub></td><td>-</td><td>0.625</td><td>-</td><td>-</td><td>0.796</td><td>-</td><td>-</td><td>0.883</td><td>-</td><td>-</td><td>0.912</td></tr><tr><td>γ<sub>0</sub></td><td>-</td><td>31.512</td><td>-</td><td>-</td><td>10.913</td><td>-</td><td>-</td><td>10.910</td><td>-</td><td>-</td><td>-2.004</td></tr></tbody></table>
<figure><img src="image_13.png" /><figcaption>Fig. F.11: Snapshot view of the colored point cloud, along with the ground truth central sub-aperture depth map (CSAD) are reported for each scene of the dataset R12-ELP20. CSAD, mask and quality map representing the absolute difference (AD) error are illustrated for all variations using the relative blur (B) of our framework: top is the scaled (S) coarse (C) and refined variations (R), bottom is the unscaled (U) coarse (C) and refined variations (R). Please refer to the color version for better visualization.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Fig. F.12: Snapshot view of the colored point cloud, along with the ground truth CSAD are reported for each scene of the dataset R12-ELP20. CSAD, mask and quality map representing the absolute difference (AD) error are illustrated for all variations using only the disparity (D) of our framework: *top* is the scaled (S) coarse (C) and refined variations (R), *bottom* is the unscaled (U) coarse (C) and refined variations (R). Please refer to the color version for better visualization.</figcaption></figure>