# Leveraging Blur Information for Plenoptic Camera Calibration

Mathieu Labussière, Céline Teulière, Frédéric Bernardin, Omar Ait-Aider

► To cite this version:

Mathieu Labussière, Céline Teulière, Frédéric Bernardin, Omar Ait-Aider. Leveraging Blur Information for Plenoptic Camera Calibration. International Journal of Computer Vision, 2022, 10.1007/s11263-022-01582-z. hal-03659586

HAL Id: hal-03659586

https://hal.science/hal-03659586

Submitted on 5 May 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Leveraging Blur Information for Plenoptic Camera Calibration

Mathieu Labussière<fnref n="1" /> ID • Céline Teulière<fnref n="1" /> ID • Frédéric Bernardin<fnref n="2" /> ID • Omar Ait-Aider<fnref n="1" /> ID

Received: 17 February 2021 / Accepted: 25 January 2022
© The Author(s) 2022

## Abstract

This paper presents a novel calibration algorithm for plenoptic cameras, especially the multi-focus configuration, where several types of micro-lenses are used, using raw images only. Current calibration methods rely on simplified projection models, use features from reconstructed images, or require separated calibrations for each type of micro-lens. In the multi-focus configuration, the same part of a scene will demonstrate different amounts of blur according to the micro-lens focal length. Usually, only micro-images with the smallest amount of blur are used. In order to exploit all available data, we propose to explicitly model the defocus blur in a new camera model with the help of our newly introduced Blur Aware Plenoptic (BAP) feature. First, it is used in a pre-calibration step that retrieves initial camera parameters, and second, to express a new cost function to be minimized in our single optimization process. Third, it is exploited to calibrate the relative blur between micro-images. It links the geometric blur, i.e., the blur circle, to the physical blur, i.e., the point spread function. Finally, we use the resulting blur profile to characterize the camera's depth of field. Quantitative evaluations in controlled environment on real-world data demonstrate the effectiveness of our calibrations.

**Keywords** Plenoptic camera · Calibration · Multi-focus · Relative blur · Blur circle

## 1 Introduction

From *Lumigraph* (Lippmann 1911) to commercial *plenoptic cameras* (Ng et al. 2005; Perwaß et al. 2012), several designs have been proposed to capture information that cannot be captured by conventional cameras. Said cameras capture only one point of view of a scene, whereas a *plenoptic camera* is

Communicated by Zuzana Kukelova.

This work was supported by the AURA Region and the European Union (FEDER) through the MMII project of CPER 2015-2020 MMaSyF challenge.

<figure><img src="image_2.png" /></figure>

Mathieu Labussière
mathieu.labu@gmail.com

Céline Teulière
celine.teuliere@uca.fr

Frédéric Bernardin
frederic.bernardin@cerema.fr

Omar Ait-Aider
omar.ait-aider@uca.fr

a device that allows to retrieve spatial as well as angular information. A same point from a scene is projected into multiple observations on the sensor. For instance, this redundant information can be used for digitally refocusing and rendering (Bishop and Favaro 2012) or for depth estimation (Johannsen et al. 2017).

This paper focuses on plenoptic cameras based on a micro-lenses array (MLA) placed between a main lens and a sensor as illustrated in Fig. 3. The specific design of such a camera allows to multiplex both types of information onto the sensor in the form of a micro-images array (MIA), as shown in Fig. 1, but implies a trade-off between the angular and spatial resolutions (Georgiev et al. 2006; Levin et al. 2008; Georgiev and Lumsdaine 2009). It is balanced according to the (MLA) position with respect to the main lens focal plane and the sensor plane, corresponding to *unfocused* (Ng et al. 2005) or *focused* (Perwaß et al. 2012; Georgiev and Lumsdaine 2012) configurations.

To further extend the depth of field (DoF) of the plenoptic camera, a multi-focus configuration has been proposed by Perwaß et al. (2012); Georgiev and Lumsdaine (2012). In this setup, the MLA is composed of several micro-lenses with different focal lengths. The same part of a scene will be more or less focused according to the micro-lens' type.

<footnote n="1">¹ Clermont Auvergne INP, CNRS, Institut Pascal, Université Clermont Auvergne, 63000 Clermont-Ferrand, France</footnote>

<footnote n="2">² Cerema, Équipe-Projet STI, 10 rue Bernard Palissy, 63017 Clermont-Ferrand, France</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 1** The Raytrix R12 multi-focus plenoptic camera used in our experimental setup (a), along with a raw image of a checkerboard calibration target (b). The image is composed of several micro-images with different amounts of blur, arranged in a hexagonal grid. In each micro-image, our Blur Aware Plenoptic (BAP) feature is illustrated by its center and its blur circle (c).</figcaption></figure>

Usually, only micro-images with the smallest amount of blur are used. Alternatively, specific patterns are used to exploit the information (Palmieri and Koch 2017). If one were able to relate the camera parameters to the amount of blur in the image, all information could be used simultaneously, without distinction between types of micro-lenses. As a first step in that direction, we propose a calibration method that takes advantage of blur information.

Calibration is an initial step for applications using plenoptic imaging. Conventional cameras are usually modeled as pinhole or thin lens. Due to the complexity of plenoptic cameras' design, the developed models are generally high dimensional. Specific calibration methods have to be proposed to retrieve the intrinsic parameters of these models.

## 1.1 Related Work

**Unfocused plenoptic camera calibration:** In the unfocused configuration, the main lens is focused at the MLA plane and the sensor plane is placed at the MLA focal plane. The MLA is therefore focused at infinity, thus calling this configuration *unfocused*. The calibration of such plenoptic cameras (Ng et al. 2005) has been widely studied in the literature. Most approaches rely on a thin-lens model for the main lens and an array of pinholes for the micro-lenses. Dansereau et al. (2013) introduced a model to decode the pixels into rays, drawing inspiration from Grossberg et al. (2005), for the Lytro plenoptic camera (Ng et al. 2005). Their model is not directly associated with physical parameters and is based on corner detection in reconstructed sub-aperture images (SAIs). Zhou

et al. (2019) proposed a practical two-step calibration method for unfocused plenoptic cameras. Their model describes the camera physical parameters but still requires feature points extracted in reconstructed SAIs. Yunsu et al. (2014) formulated a geometric projection model to estimate intrinsic and extrinsic parameters by utilizing raw images directly to avoid errors from reconstruction steps. Their method includes analytical solution and non-linear optimization of the reprojection error of a novel line feature to overcome the difficulties in finding checkerboard corners. Shi et al. (2016) proposed a detailed model of a plenoptic camera in the context of particle image velocimetry (PIV). Based on linear optics, they derived a model based on ray-tracing: contrarily to previous methods, they modeled the main lens and each micro-lens as thin-lenses. Hahne et al. (2018) developed a ray model by ray-tracing from the sensor side to the object space. They consider only the chief ray, connecting micro-image centers (MICs) to the exit pupil center. O'Brien et al. (2018) introduced a projection model used for their calibration method suited both for unfocused and focused plenoptic cameras. They present a new feature called *plenoptic disc*, similar in nature to the circle of confusion (CoC) and defined by its center and its radius. Their feature parametrization is in 3D and is in one-to-one correspondence with point positions in the camera frame, as it is detected in reconstructed image. Zhao et al. (2020) recently presented a metric calibration method for unfocused plenoptic camera only also based on the *plenoptic disc* but directly from raw image.

In summary, most of the above methods require reconstructed images (SAIs) to extract features, and limit their model to the unfocused configuration, i.e., setting the sensor plane at the micro-lens focal plane. Therefore those models cannot be directly extended to the focused or multi-focus plenoptic camera.

**Focused plenoptic camera calibration:** With the arrival of commercial focused plenoptic cameras (Lumsdaine and Georgiev 2009; Perwaß et al. 2012), new calibration methods have been proposed. In this configuration, the micro-lenses focus on an intermediate image plane. Johannsen et al. (2013) formulated a general reprojection model in terms of the physical parameters of a Raytrix camera (Perwaß et al. 2012). They proposed a metric calibration and distortions correction using a grid of circular patterns. This work considered a relatively simple model of lens distortion and required careful initialization of the optimization to converge due to high sensitivity to local minima. Heinze et al. (2016) improved the previous model by considering more sophisticated models of the main lens distortions. They introduced new parameters including the tilt and shift for the main lens. They are able to distinguish each micro-lens type, calibrating then the distance between the MLA and the sensor for each one but in separated calibration processes. The projection model and the metric calibration procedure are incorporated in the

<figure><img src="image_4.png" /></figure>
RxLive software of Raytrix GmbH. Strobl and Linge-
nauber (2016) presented a step-wise calibration approach
to overcome the fragility of the initialization which hin-
ders the final optimization. They first determined main lens
parameters, then estimated MLA parameters. However, their
calibration framework relied on reconstructed total focus
images. Zeller et al. (2014) introduced two new methods
to calibrate a focused plenoptic camera and depth images
obtained from it. In further works (Zeller et al. 2016), they
improved the camera projection model by modeling the main
lens as a thin lens instead of a pinhole. The calibration process
uses the reconstructed total focus image and virtual depth
map to compute 3D observations.

All previous methods rely on reconstructed images (SAIs),
which can lead to the introduction of errors in the recon-
struction step as well as in the calibration process. Usually,
computation of reconstructed images requires camera param-
eters and/or depth information to avoid artifacts and recon-
struction error. To overcome this chicken and egg problem,
several calibration methods focus on using only raw plenop-
tic images. Zhang et al. (2016) proposed a calibration method
based directly on observations from raw images. They used
a parallel bi-planar checkerboard to have a depth-scale prior.
They considered a detailed model of the MLA geometry
that accounts for non-planarity of the array. Zhang et al.
(2018) presented a multi-projection-center model based on
the two planes parametrization (Levoy and Hanrahan 1996).
They derived a calibration algorithm based on this model
and projective transformation, suitable for both unfocused
and focused plenoptic cameras. Noury et al. (2017) pre-
sented a more complete geometrical model than the previous
works. This model relates 3D points to their correspond-
ing image projections, working directly with raw images.
They developed a new detector to find checkerboard corners
with sub-pixel accuracy in each micro-image. They intro-
duced a new cost function based on reprojection errors of
both checkerboard corners and micro-lens centers in raw
image space. This enforces projected micro-lens centers to
get closer to their corresponding MICs, and makes their
method robust to wrong parameters initialization especially
concerning those of the MLA. However, their method does
not consider different types of micro-lenses and forces them
to act as pinholes.

Several methods can account for the multi-focus setting.
Bok et al. (2017) extended their previous model (Yunsu et al.
2014) to work with the focused plenoptic camera. They did
not explicitly model the micro-lens focal lengths but intro-
duced two additional intrinsic parameters that account for
the MLA setting. Each setting – one for each type of micro-
lenses –, models a different distance between the MLA and
the sensor. Their method can retrieve different intrinsics by
running the optimization for each type separately. Nousias
et al. (2017) considered the geometric calibration of multi-

focus plenoptic cameras. Their method allows to identify the
micro-lens types and their spatial arrangement. It operates
on checkerboard corners retrieved by a custom micro-image
corner detector. Then, they applied their method on each
type of micro-lens independently to retrieve specific intrin-
sic and extrinsic parameters for each configuration. Latter
researches (Bok et al. 2017; Nousias et al. 2017; Noury et al.
2017) have achieved improved performance through automa-
tion and accurate identification of feature correspondences
in raw images. More recently, Wang et al. (2018) proposed
a geometric calibration method for focused plenoptic cam-
eras based on virtual image points, establishing the mapping
from object points behind the main lens and the MLA to
image points on the sensor. Their method can be extended
to calibrate multi-focus cameras by considering each type of
micro-lenses individually.

In conclusion, most of these methods rely on simplified
models for optic elements: the MLA misalignment is not
considered, and the micro-lenses are modeled as pinholes
thus not modeling their apertures. Some do not consider
distortions of the main lens or restrict themselves to the
focused case. Finally, few have considered the multi-focus
case (Heinze et al. 2016; Bok et al. 2017; Nousias et al. 2017;
Wang et al. 2018) but dealt with it in separate processes, lead-
ing to intrinsic and extrinsic parameters that vary depending
on the type of micro-lens.

## 1.2 Contributions

We present a new calibration method for plenoptic cameras.
To the best of our knowledge, it is the first to allow to calibrate
the multi-focus plenoptic camera within a single process tak-
ing into account all types of micro-lenses simultaneously. To
exploit all available information, we propose to explicitly
include the defocus blur in a new camera model. Thus, we
introduce a new Blur Aware Plenoptic (BAP) feature defined
in raw image space that enables us to handle the multi-focus
case. We present a new pre-calibration step using BAP fea-
tures from white images to provide a robust initial estimation
of camera parameters. We use our BAP features in a sin-
gle optimization process that retrieves intrinsic and extrinsic
parameters of a multi-focus plenoptic camera directly from
raw plenoptic images of a checkerboard target.

This paper extends our previous work (Labussière et al.
2020). In addition to our former contributions, we present
here an ablation study of the camera parameters and add fur-
ther comparisons with state-of-the-art calibration methods.
A new camera setup has also been tested to validate the gen-
eralization of our method, and a simulation setup is proposed
to evaluate our method on Lytro-like configuration. More-
over, we take advantage of our BAP features to develop a
new relative blur calibration process to link the geometric
blur to the physical blur, i.e., the circle of confusion (CoC)
<figure><img src="image_5.png" /><figcaption>**Fig. 2** Overview of our proposed method: first, the pre-calibration step retrieves initial camera parameters from white raw images at different apertures; then followed by the detection of BAP features that are used by the camera calibration process and calibration of the relative blur;</figcaption></figure>

finally, once the camera is calibrated, it can be used, as addressed here, for profiling the camera, i.e., to characterize the working range of the camera. Other applications can be considered, such as metric depth estimation.

<figure><img src="image_6.png" /><figcaption>**Fig. 3** Focused plenoptic camera model in Galilean configuration with the notations used in this paper. Object points are projected by the main lens behind the MLA into a virtual intermediate space, and then re-imaged by each micro-lens onto the sensor.</figcaption></figure>

to the point-spread function (PSF). This enables us to fully take advantage of blur in image space. Finally, we propose to use the blur to profile the plenoptic camera in terms of depth of field (DoF).

## 1.3 Paper Organization

An overview of our method is given in Fig. 2. The remainder of this paper is organized as follows. First, we present the camera model and how we model blur with our BAP feature in Sect. 2. Second, we explain in Sect. 3 how we leverage raw white images in the proposed pre-calibration step to initialize camera parameters. Then, we detail the feature detection in Sect. 4 and the calibration processes in Sect. 5, i.e., the camera calibration and the relative blur calibration. Our experimental setup is presented in Sect. 6. Finally, our results are given and discussed in Sect. 7. The notations used in this paper are shown in Fig. 3. Pixel counterparts of metric values are denoted in lower-case Greek letters. Bold font denotes vectors and matrices.

# 2 Camera and Blur Models

## 2.1 The (Multi-Focus) Plenoptic Camera

We consider the focused plenoptic camera, especially the multi-focus case as described by Georgiev and Lumsdaine (2012); Perwaß et al. (2012). The camera is composed of a main lens and photosensitive sensor with a micro-lenses array (MLA) in between, as illustrated in Fig. 3. The multi-focus configuration implies that the micro-lenses array consists of *I* different types of lenses. The setup corresponds to the multi-focus system described by Perwaß et al. (2012) with *I* = 3. Note that our model can be applied to the single-focus plenoptic camera as well, corresponding then to the case where *I* = 1. Finally, the unfocused configuration is a special case of our model where the micro-lens focal length is equal to the distance between the MLA and the sensor, i.e., *f* = *d*.
### 2.1.1 Main Lens

The main lens is modeled as a thin-lens and maps an object point to a virtual point in an intermediate space called the virtual space. An object at distance a is then projected at a distance b given the focal length F according to the thin-lens equation

$$ \frac{1}{F} = \frac{1}{a} + \frac{1}{b}. \qquad (1) $$

The main lens principal point is expressed as $[u_0 \ v_0]^T$ in image space. We model the main lens as parallel to the sensor plane. Deviations from this hypothesis will be compensated for by tangential distortion parameters. Furthermore, we define our camera reference frame as the main lens frame, with **O** being the origin, the z-axis coinciding with the optical axis and pointing outside the camera, and the y-axis pointing downwards. Distances are signed according to the following convention: F is positive when the lens is convergent; distances are positive when the point is real, and negative when virtual.

### 2.1.2 Distortions

We consider distortions of the main lens. Distortions represent deviations from the theoretical thin lens projection model. To correct those errors, we model the radial and tangential components of the lateral distortions using the model of Brown-Conrady (Brown 1966; Conrady 1919).

Depth distortions have also been studied by Heinze et al. (2016); Zeller et al. (2016), but Zeller et al. (2017); Noury (2019) both empirically observed that the effects of depth distortions, for large focal length and for large object distance, can be neglected compared to stochastic noise of the depth estimation process. Therefore, we do not include depth distortion in our model.

A distorted point $\mathbf{p} = [x \ y \ z]^T$ expressed in the main lens frame after projection (i.e., in the virtual intermediate space) is thus transformed into $\mathbf{p}_u = \phi(\mathbf{p}) = [x_u \ y_u \ z]^T$ and is computed as

$$ \left\{ \begin{array}{ll} x_u = x \left( 1 + Q_1 \mathbf{s}^2 + Q_2 \mathbf{s}^4 + Q_3 \mathbf{s}^6 \right) & \text{[radial]} \\ & + P_1 \left( \mathbf{s}^2 + 2x^2 \right) + 2P_2xy & \text{[tangential]} \\ y_u = y \left( 1 + Q_1 \mathbf{s}^2 + Q_2 \mathbf{s}^4 + Q_3 \mathbf{s}^6 \right) & \text{[radial]} \\ & + P_2 \left( \mathbf{s}^2 + 2y^2 \right) + 2P_1xy & \text{[tangential]} \end{array} \right. \qquad (2) $$

where $\mathbf{s}^2 = x^2 + y^2$. The three coefficients for the radial component are given by $\{Q_1, Q_2, Q_3\}$, and the two coefficients for the tangential by $\{P_1, P_2\}$.

### 2.1.3 Micro-Lenses Array

We also model the micro-lenses as thin-lenses allowing to take into account blur in the micro-image. The MLA consists then of I different lens types with focal lengths $f^{(i)}$ where $i \in [1..I]$ which are focused on I different planes.

We make the hypothesis that all micro-lenses lie on the same plane. The MLA is approximately centered around the optic axis. We define the farthest micro-lens along the $(-x)$-axis and the $(-y)$-axis as the origin of the MLA frame, i.e., the center of the upper-left micro-lens. The coordinates axes are orientated the same way as the ones of the main lens. The structural organization of the lenses can be an orthogonal or hexagonal arrangement. The MLA origin is at a distance D from the main lens and at a distance d from the sensor. Theoretically, the distances $d(k, l)$ and $D(k, l)$ are different for each micro-lens $(k, l)$ due to the MLA tilt. However, using the approximation of a constant distance d provides adequate results thanks to the low order of the MLA rotation angles' as shown in Sect. 7, while allowing simpler derivation of the equations.

Furthermore, a detected micro-image center (MIC) usually does not coincide with the optical center of the considered micro-lens. We take into account this deviation in opposition to orthographic projection of MICs which causes inaccuracy in decoded light field. Therefore, the principal point $\mathbf{c}_0^{k,l}$ of the micro-lens indexed by $(k, l)$ is given by

$$ \mathbf{c}_0^{k,l} = \begin{bmatrix} u_0^{k,l} \\ v_0^{k,l} \end{bmatrix} = \frac{d}{D+d} \left( \begin{bmatrix} u_0 \\ v_0 \end{bmatrix} - \mathbf{c}_{k,l} \right) + \mathbf{c}_{k,l}, \qquad (3) $$

where $\mathbf{c}_{k,l}$ is the center of the micro-image $(k, l)$ expressed in pixel, as illustrated in Fig. 3.

### 2.1.4 Micro-Images Array

Finally, each micro-lens produces a micro-image (MI) onto the sensor. The set of these micro-images has the same structural organization as the MLA. The data can therefore be interpreted as an array of micro-images, called by analogy the micro-images array (MIA). The MIA coordinates are expressed in image space. Let $\delta_i$ be the pixel distance between two arbitrary consecutive micro-images centers $\mathbf{c}_{k,l}$. With s the metric size of a pixel, let $\Delta_i = s\delta_i$ be its metric value, and $\Delta_\mu$ be the metric distance between the two corresponding micro-lens centers $\mathbf{C}_{k,l}$. From similar triangles, the ratio $\lambda$ between them is given by

$$ \lambda \triangleq \frac{D}{d+D} = \frac{\Delta_{\mu}}{\Delta_{i}} \iff \Delta_{\mu} = \lambda \Delta_{i} = \frac{D}{d+D} \cdot \Delta_{i}. \qquad (4) $$

We make the hypothesis that $\Delta_\mu$ is equal to the micro-lens aperture.
### 2.1.5 Camera Configuration

When the camera is in the unfocused configuration, the distance separating the sensor and the MLA is equal to the focal length of the micro-lenses, i.e., $d = f$. Dealing with the focused plenoptic camera, we usually consider two possible configurations as presented by Georgiev and Lumsdaine (2009): 1) Galilean, when objects are projected behind the image sensor; and 2) Keplerian, when objects are projected in front of the image sensor. When considering micro-lenses as thin-lenses, we have to take into account their focal lengths to configure the camera. In practice, considering an object projected at distance $b$ by the main lens, four cases are possible but only two are able to produce an exploitable image, i.e., with acceptable amount of blur, onto the sensor: $b < D$ and $f < d$ in Keplerian; and, $b > D$ and $f > d$ in Galilean. The condition $b > D$ can be achieved both when $F > D$ and $F < D$. The mode of operation is then constrained by the focal length of the micro-lenses, as suggested by Mignard-Debiseet al. (2017).

We introduce then the definition of the internal configuration according to the micro-lens focal length as

$$ \begin{cases} f < d \Rightarrow & \text{Keplerian internal configuration,} \\ f > d \Rightarrow & \text{Galilean internal configuration.} \end{cases} \qquad (5) $$

## 2.2 Modeling Blur Within the Plenoptic Camera

From optics geometry, the image of a point from a circular lens not focused on the sensor can be modeled by the circle of confusion (CoC). Using a camera with a circular aperture, the blurred image is also circular in shape and is called the *blur circle*. From similar triangles and from the thin-lens equation (eq. 1), the signed blur radius of the image of a point at a distance $a$ from the lens is expressed as

$$ \begin{cases} r = A \frac{d}{2} \left( \frac{1}{f} - \frac{1}{a} - \frac{1}{d} \right) & \text{[metric]} \\ \rho = r/s & \text{[pixel]} \end{cases} \qquad (6) $$

with $s$ being the size of a pixel, and $A$ the aperture of this lens. In continuous domain, the response of an imaging system to a not in-focus point, i.e., the blur, can be expressed by the point-spread function (PSF). Let $I(x, y)$ be the observed blurred image of an object at a constant distance. The image can be computed as the convolution of the PSF noted $h(x, y)$, with the in-focus image, $I^*(x, y)$, such as

$$ I(x, y) = h * I^*(x, y), \qquad (7) $$

where * denotes the convolution operator. If the lens aperture is circular and the level of blur low, the PSF $h(x, y)$ can be

efficiently modeled by a two-dimensional Gaussian given by

$$ h(x, y) = \frac{1}{2\pi\sigma^2} \exp\left(-\frac{x^2 + y^2}{2\sigma^2}\right), \qquad (8) $$

where the spread parameter $\sigma$ is proportional to the blur circle radius $\rho$. Therefore, we can write

$$ \sigma \propto \rho \Leftrightarrow \sigma = \kappa \cdot \rho \qquad (9) $$

where $\kappa$ is a camera constant that should be determined by calibration (Pentland 1987; Subbarao 1989). Note that the spatially-variant spread parameter $\sigma$ thus depends on the object distance $a$.

The blur radius $\rho$ appears at several levels within the camera projection: in the blur introduced by the thin-lens model of the micro-lenses and in the formation of the micro-images while taking a white image. Each micro-lens $(k, l)$ projects virtual points onto the sensor at a position $(u, v)$, with a blur radius $\rho$ depending on the distance to the point and the micro-lens type.

## 2.3 BAP Features and Projection Model

To leverage this blur information, we introduce a new Blur Aware Plenoptic (BAP) feature characterized by its center and its radius, noted $\mathbf{p} = [u \ v \ \rho \ 1]^\top$. The BAP feature are visualized in Fig. 1. Therefore, our complete plenoptic camera model allows us to link a scene point $\mathbf{p}_w = [x \ y \ z \ 1]^\top$ to our new BAP feature $\mathbf{p}$ in homogeneous coordinates through each micro-lens $(k, l)$ such as

$$ \begin{bmatrix} u \\ v \\ \rho \\ 1 \end{bmatrix} \propto P(i, k, l) \cdot T_{\mu}(k, l) \cdot \phi(K(F) \cdot T_c \cdot p_w), \qquad (10) $$

where $P(i, k, l)$ is the blur aware plenoptic projection matrix through the micro-lens $(k, l)$ of type $i$, and computed as

$$ \begin{aligned} P(i, k, l) &= \mathbf{P}(k, l) \cdot \mathbf{K}(f^{(i)}) \\ &= \begin{bmatrix} d/s & 0 & u_0^{k,l} & 0 \\ 0 & d/s & v_0^{k,l} & 0 \\ 0 & 0 & \frac{\Delta_{\mu}}{2s} & -\frac{\Delta_{\mu}}{2s} d \\ 0 & 0 & -1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & -1/f^{(i)} & 1 \end{bmatrix}. \end{aligned} \qquad (11) $$

$\mathbf{P}(k, l)$ is a matrix that projects the 3D virtual point onto the sensor and taking into account the blur radius. $\mathbf{K}(f)$ is the thin-lens projection matrix for the given focal length. $T_c$ is the pose of the main lens with respect to the world frame and $T_{\mu}(k, l)$ is the pose of the micro-lens $(k, l)$ expressed
in the camera frame. The function $\phi(\cdot)$ models the lateral distortions.

Finally, the projection model from eq. 10 consists of a set $\Xi$ of $(16 + I)$ intrinsic parameters to be optimized, including: the main lens focal length $F$, expressed in $K(F)$, and its five lateral distortion coefficients $Q_1, Q_2, Q_3, P_1$, and $P_2$, expressed in $\phi(\cdot)$; the sensor translations, encoded in $d$ and $(u_0, v_0)$ through eq. 3, from $P(k, l)$; the MLA pose, including its three rotations $(\theta_x, \theta_y, \theta_z)$ and three translations $(t_x, t_y, D)$, and the micro-lens pitch $\Delta_\mu$, expressed in $T_\mu(k, l)$; and, the $I$ micro-lens focal lengths $f^{(i)}$, in $K(f^{(i)})$.

## 2.4 Profiling the Depth of Field of the Plenoptic Camera

From calibrated camera parameters, we can compute the depth of field (DoF) of each micro-lens type and the *blur profile* – the blur radii as function of the object distance $–$, in order to profile the plenoptic camera. The analysis can be done with respect to the MLA pose, and then extended to object space by back-projection. A point at a distance $a$ from MLA is projected back into object space at a distance $a'$ according to the thin-lens equation through the main lens, such as

$$a' = \frac{(D - a) \cdot F}{(D - a) - F}. \qquad (12)$$

Let $r_0$ be the minimal acceptable radius of the CoC. The smallest diffraction-limited spot resolved by a lens in wave optics, i.e., the radius of the first null of the Airy disc, is $r^* = 1.22 \cdot \nu \cdot N^*$, where $\nu$ is the considered light wavelength, and $N^* = d/A$ is the working $f$-number of the lens. The minimal acceptable radius is the maximum between this limit and half the size of a pixel, such as $r_0 = \max(r^*, s/2)$. For a micro-lens of type (i), the focus plane distance is given by

$$a_0^{(i)} = \left( \frac{1}{f^{(i)}} - \frac{1}{d} \right)^{-1} = \frac{df^{(i)}}{d - f^{(i)}}. \qquad (13)$$

Let $A$ be the micro-lens aperture, we derive then the *far* $a_+$ and *near* $a_-$ focus planes distances:

$$\left\{ \begin{array}{l} a_{+}^{(i)} = \frac{dA \cdot a_{0}^{(i)}}{Af^{(i)} - 2r_{0}(a_{0}^{(i)} - f^{(i)})} \quad [\text{far}] \\ a_{-}^{(i)} = \frac{dA \cdot a_{0}^{(i)}}{Af^{(i)} + 2r_{0}(a_{0}^{(i)} - f^{(i)})} \quad [\text{near}]. \end{array} \right. \qquad (14)$$

The DoF of a micro-lens of type (i) is computed as the distance between the near and far focus planes, such as

$$\mathrm{DOF}^{(i)} = |a_{+}^{(i)}| - |a_{-}^{(i)}| = \frac{Af^{(i)} \cdot a_{0}^{(i)} \cdot 2r_{0}(a_{0}^{(i)} - f^{(i)})}{(Af^{(i)})^{2} - 4r_{0}^{2}(a_{0}^{(i)} - f^{(i)})^{2}}. \qquad (15)$$

Note that to fully exploit the combined extended DoF without gaps, the micro-lenses DoF should either just touch or slightly overlap (Perwaß et al. 2012). Finally, under this consideration, the total DoF of the plenoptic camera in MLA space is computed using the micro-lenses DoF as

$$\mathrm{DOF} = \max_{i} \left\{ |a_{+}^{(i)}| \right\} - \min_{i} \left\{ |a_{-}^{(i)}| \right\}. \qquad (16)$$

We can finally plot the *blur profile* of the camera, along with the focal planes and the total DoF as illustrated by the Fig. 10.

# 3 Pre-calibration Using Raw White Images

The goal of the pre-calibration step is to provide a strong initial estimate of the camera parameters. Inspired from *depth from defocus* theory (Subbarao and Surya 1994), we leverage blur information to estimate our blur radius by varying the main lens aperture and using the different micro-lenses focal lengths, in combination with parameters from the image space. This is achieved by using raw white images acquired with a light diffuser mounted on the main objective, and taken at different apertures. We then show how the blur radii are linked to camera parameters, thus enabling their initialization.

## 3.1 Micro-Images Array Calibration

First, the micro-images array (MIA) is calibrated using raw white images. We compute the micro-image centers $\{c_{k,l}\}$ by the intensity centroid method with sub-pixel accuracy (Thomason et al. 2014; Noury et al. 2017; Suligaet al. 2018). The distance between two micro-image centers $\delta_i$ is then computed as the optimized edge-length of a fitted 2D regular grid mesh. The optimization is conducted by non-linear minimization of the distances between the grid vertices and the corresponding detected MICs. The pixel translation offset in image coordinates, $(\tau_x, \tau_y)$, and the rotation around the $(-z)$-axis, $\vartheta_z$, are also determined during the optimization process.

## 3.2 Deriving the Micro-Image Radius

In white images taken with a light diffuser and a controlled aperture, each type of micro-lens produces a micro-image (MI) with a specific size and intensity. This provides a mean to distinguish between them (Fig. 5). The process of capturing a white image is equivalent for the micro-lenses to
<figure><img src="image_7.png" /><figcaption>Fig. 4 Formation of a micro-image with its radius R through a micro-lens while taking a white image using a light diffuser, at an aperture A, in Keplerian internal configuration. The point V is the vertex of the cone passing by the main lens and the considered micro-lens. V' is the image of V by the micro-lens and R is the radius of its blur circle.</figcaption></figure>

imaging a white uniform object of diameter A at a distance D. The imaging process is schematized in Fig. 4. Using optics geometry, the image of this object, i.e., the resulting MI, corresponds to the image of an imaginary point V constructed as the vertex of the cone passing through the main lens and the considered micro-lens. Let a be the signed distance of this point from the MLA plane, expressed from similar triangles and eq. 4 as

$$a = -D \frac{\Delta_{\mu}}{A - \Delta_{\mu}} = -D \left( A \left( \frac{d+D}{D} \cdot \frac{1}{\Delta_{i}} \right) - 1 \right)^{-1}, \quad (17)$$

with A being the main lens aperture. Note the minus sign is added because the vertex is always formed behind the MLA plane, and thus considered as a virtual object for the micro-lenses. Geometrically, the MI formed is the *blur circle* of this imaginary point V. Therefore, injecting the latter expression in eq. 6, the metric MI radius R is given by

$$\begin{aligned} R &= \frac{\Delta_{\mu}}{2} d \left( \frac{1}{f} - \frac{1}{a} - \frac{1}{d} \right) \\ &= \left( \frac{\Delta_{i} \cdot D}{d+D} \right) \cdot \frac{d}{2} \cdot \left( \frac{1}{f} + \left( A \left( \frac{d+D}{D} \cdot \frac{1}{\Delta_{i}} \right) - 1 \right) \frac{1}{D} - \frac{1}{d} \right) \\ &= A \cdot \frac{d}{2D} + \left( \frac{\Delta_{i} \cdot D}{d+D} \right) \cdot \frac{d}{2} \cdot \left( \frac{1}{f} - \frac{1}{D} - \frac{1}{d} \right). \end{aligned} \quad (18)$$

From the above equation, the MI radius R depends linearly on the aperture of the main lens. However, the main lens aperture cannot be measured directly whereas we have access to the f-number value. Recall that the f-number of an optical system is the ratio of the system's focal length F to the aperture, A, given by $N = F/A$. Finally, we can express the MI radius

for each micro-lens focal length type i as

$$R_i(N^{-1}) = m \cdot N^{-1} + q_i \quad (19)$$

with

$$m = \frac{dF}{2D} \quad \text{and} \quad q_i = \frac{1}{f^{(i)}} \cdot \left( \frac{\Delta_i \cdot D}{d+D} \right) \cdot \frac{d}{2} - \frac{\Delta_i}{2}. \quad (20)$$

We thus relate the MI radius to the plenoptic camera parameters. It is a function of fixed parameters (d, D, F), measured parameters ($\Delta_i = s \cdot \delta_i$) and variable parameters ($N$ and $f^{(i)}$ with $i \in [1..I]$).

Let $\Omega$ be the set of parameters {$m, q'_1, \dots, q'_I$}, where $q'_i$ is the value obtained by

$$q'_{i} = \frac{1}{f^{(i)}} \cdot \left( \frac{\Delta_{i} \cdot D}{d+D} \right) \cdot \frac{d}{2} = q_{i} + \frac{\Delta_{i}}{2}. \quad (21)$$

They are used to compute the radius part of the BAP feature and to initialize the camera parameters.

**Micro-image radii estimation:** From raw white images, we measure each MI radius $\rho = |R|/s$ in pixel based on image moments fitting. We use the second order central moments of the micro-image to construct a covariance matrix. The radius $\rho$ is proportional to the computed standard deviation $\sigma$. Recall that raw moments and centroid are given by

$$M_{ij} = \sum_{x,y} x^i y^j I(x,y) \quad \text{and} \quad \{\bar{x}, \bar{y}\} = \left\{ \frac{M_{10}}{M_{00}}, \frac{M_{01}}{M_{00}} \right\},$$

and the central moments by

$$\mu_{pq} = \sum_{x,y} (x - \bar{x})^p (y - \bar{y})^q I(x,y). \quad (22)$$

The covariance matrix is then computed as

$$\operatorname{cov}[I(x, y)] = \frac{1}{\mu_{00}} \begin{bmatrix} \mu_{20} & \mu_{11} \\ \mu_{11} & \mu_{02} \end{bmatrix} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} \\ \sigma_{yx} & \sigma_{yy} \end{bmatrix}. \quad (23)$$

We define $\sigma$ as the square root of the greatest eigenvalue of the covariance matrix, i.e.,

$$\sigma^2 = \frac{\sigma_{xx} + \sigma_{yy}}{2} + \frac{\sqrt{4\sigma_{xy}^2 + (\sigma_{xx} - \sigma_{yy})^2}}{2}. \quad (24)$$

The estimation is robust to noise, works under asymmetrical distribution and is easy to use, but requires a parameter $\alpha$ to convert the standard deviation $\sigma$ into a pixel radius $\rho = \alpha \cdot \sigma$. The parameter $\alpha$ is determined so that at least 98% of the distribution is taken into account. According to the standard normal distribution Z-score table, $\alpha$ is picked up in
<figure><img src="image_8.png" /><figcaption>Fig. 5 a Micro-image radii as function of the inverse f-number (in magenta), with their distributions represented by the violin-boxes, for our camera consisting of $I = 3$ different types. b Each type of micro-</figcaption></figure>

lens is identified by its color (type (1) in red, type (2) in green, and type (3) in blue) with its computed radius.

[2.33, 2.37]. In our experiments, we set $\alpha = 2.357$ as it best fits our measurements.

Recall that the pixel MI radius is given by $\rho = |R|/s$. The metric radius is either positive if formed after the rays inversion, as in Fig. 4, or negative if before, and thus depends on the internal configuration such as

$$R = \begin{cases} \rho \cdot s & \text{[Keplerian internal configuration],} \\ -\rho \cdot s & \text{[Galilean internal configuration].} \end{cases} \qquad (25)$$

*Coefficients estimation:* Given several raw white images taken at different apertures, we estimate the parameters $\Omega$, i.e., the coefficients of eq. 19, for each type of micro-image. Note that the standard full-stop f-number conventionally indicated on the lens differs from the real f-number. We use then the f-number calculated from the aperture value $AV$ by $N = \sqrt{2}^{AV}$. The coefficient $m$ is a function of fixed physical parameters independent of the micro-lens focal lengths and the main lens aperture. Therefore, we obtain a set of linear equations, sharing the same slope, but with different y-intercepts. With $X = [m \ q_1 \ \dots \ q_I]^{\top}$, the set of equations can be linearly rewritten as

$$AX = B, \text{ and then } X = (A^{\top}A)^{-1}A^{\top}B$$

where the matrix $A$, containing the f-numbers and a selector of the corresponding y-intercept coefficient, and the vector $B$, containing the radii measurements, are constructed by arranging the terms given the focal length at which they have been calculated. Finally, we compute $X$ with a least-square estimation. Figure 5 shows an example of radii distributions from our experiments computed from white images taken at several f-numbers, and the estimated linear functions. In practice, at least two aperture configurations are required. More can be used to improve the estimation but at the condi-

tion that radii measurement distributions are distinguishable from each others, with small overlap.

## 3.3 Camera Parameters Initialization

First, the pixel size $s$ is set according to the manufacturer values. The main lens focal length $F$ is also initialized from them. Given the parameters $\Omega$ and the focus distance $h$, the parameters $d$ and $D$ are initialized as

$$d \leftarrow \frac{2mH}{F + \xi \cdot 4m} \quad \text{and} \quad D \leftarrow H - \xi \cdot 2d, \qquad (26)$$

with $\xi = 1$ (resp., $\xi = -1$) in Galilean (resp., Keplerian) internal configuration, and where $H$ is given by Eq. (17) of (Perwaß et al. 2012),

$$H = \left| \frac{h}{2} \left( 1 - \sqrt{1 - 4 \frac{F}{h}} \right) \right|. \qquad (27)$$

For completeness, note that the *unfocused* configuration can be initialized with $d \leftarrow 2m$ and $D \leftarrow F$.

In a second step, all distortions coefficients are set to zero. The principal point is set as the center of the image. The sensor plane is thus set parallel to the main lens plane, with no rotation, at a distance $- (D + d)$. Seemingly, the MLA plane is initially set parallel to the main lens plane at a distance $-D$. From the pre-computed MIA parameters, the MLA translation takes into account the $(x, y)$-offsets $(-s\tau_x, -s\tau_y)$ and the rotation around the z-axis is initialized with $-\vartheta_z$. The micro-lenses pitch $\Delta_{\mu}$ is set according to eq. 4, where the ratio $\lambda$ is computed using eq. 26 such as

$$\lambda \leftarrow \frac{F}{F + 2m}. \qquad (28)$$
Finally, the initial micro-lenses' focal lengths are also computed from the parameters $\Omega$ as follows

$$f^{(i)} \leftarrow \frac{d}{2 \cdot q'_{i}} \cdot \Delta_{\mu}. \qquad (29)$$

Experiments will show that the initial model is close to the optimized model.

## 4 BAP Features Detection in Raw Images

At this point, the MIA is calibrated and micro-images centers are extracted. The raw images are devignetted by dividing them by a white raw image taken with the same aperture. We based our method on a checkerboard calibration pattern. The detection process is divided into two steps: 1) checkerboard images are processed to extract corners at position $(u, v)$; and 2) with the set of parameters $\Omega$ and the associated virtual depth estimate for each corner, the corresponding BAP feature is computed in image space.

### 4.1 Computing Blur Radius Through Micro-Lens

To respect the $f$-number matching principle (Perwaß et al. 2012), we configure the main lens $f$-number such that the micro-images fully tile the sensor without overlap. In this configuration the working $f$-number of the main imaging system and the micro-lens imaging system should match. We consider the general case of measuring an object $p$ at a distance $a$ from the main lens. First, $p$ is projected through the main lens according to the thin lens equation, $1/F = 1/a + 1/b$, resulting in a point $p'$ at a distance $b$ behind the main lens, i.e., at a distance $a' = D - b$ from the MLA. From eq. 6, the metric radius of the blur circle $r$ of a point $p'$ at distance $a'$ through a micro-lens of type (i) is expressed as

$$\begin{aligned} r &= \left( \frac{\Delta_i D}{d + D} \right) \cdot \frac{d}{2} \cdot \left( \frac{1}{f^{(i)}} - \frac{1}{a'} - \frac{1}{d} \right) \\ &= \underbrace{\frac{\Delta_i \cdot D}{d + D} \cdot \frac{d}{2} \cdot \frac{1}{f^{(i)}}}_{=q'_i[eq.21]} - \underbrace{\frac{\Delta_i \cdot D}{d + D} \cdot \frac{d}{2} \cdot \frac{1}{d}}_{=\Delta_{\mu}/2[eq.4]} - \underbrace{\frac{\Delta_i \cdot D}{d + D} \cdot \frac{d}{2} \cdot \frac{1}{a'}}_{=\Delta_{\mu}[eq.4]} \\ &= \left( -\Delta_{\mu} \cdot \frac{d}{2} \right) \cdot \frac{1}{a'} + \left( q'_i - \frac{\Delta_{\mu}}{2} \right). \end{aligned} \qquad (30)$$

In practice, $a'$ and $d$ cannot be measured in raw image space, but the virtual depth can, as it will be shown in the next subsection. Virtual depth refers to relative depth value obtained from disparity. It is defined as the ratio between the signed object distance $a'$ and the sensor distance $d$:

$$v = -\frac{a'}{d}. \qquad (31)$$

The sign convention is reversed for virtual depth computation. Distances are negative in front of the MLA plane. If we re-inject the virtual depth in eq. 30, taking caution of the sign, and using eq. 4, we can derive the radius of the *blur circle* of a point $p'$ at a distance $a'$ from the MLA by

$$r = \frac{\lambda \Delta_i}{2} \cdot v^{-1} + \left( q'_{i} - \frac{\lambda \Delta_i}{2} \right). \qquad (32)$$

This equation allows to express the pixel radius of the blur circle $\rho = r/s$ associated to each point having a virtual depth without explicitly evaluating the physical parameters $A, D, d, F$ and $f^{(i)}$ of the camera, directly in image space.

### 4.2 Features Extraction

First, we detect corners in raw images using the detector introduced by Noury et al. (2017) with sub-pixel accuracy in each micro-image. With a plenoptic camera, contrarily to a classic camera, a same point in object space is projected into multiple observations onto the sensor. The checkerboard is designed and positioned so that the sets of observations are sufficiently far from each others to be clustered. We use the DBSCAN algorithm (Ester et al. 1996) to identify the clusters. We then associate each point with its cluster of observations.

Secondly, once each cluster is identified, we compute the virtual depth $v$ from the disparity. Let $\Delta C_{1-2}$ be the distance between the centers of the micro-lenses $C_1$ and $C_2$, i.e., the baseline. Let $\Delta p = |p_1 - p_2|$ be the Euclidean distance between images of the same point in corresponding micro-images. The virtual depth $v$ is calculated with the intercept theorem:

$$v = \frac{\Delta C_{1-2}}{\Delta C_{1-2} - \Delta p} = \frac{\eta \cdot \Delta_{\mu}}{\eta \cdot \Delta_{\mu} - \Delta p} = \frac{\eta \cdot \lambda \Delta_{i}}{\eta \cdot \lambda \Delta_{i} - \Delta p}. \qquad (33)$$

If we consider two adjacent micro-lenses, the baseline $\Delta C_{1-2}$ is just the diameter of a micro-lens, i.e., $\Delta_{\mu} = \lambda \Delta_{i}$ and $\eta = 1$. For further apart micro-lenses the baseline is a multiple of that diameter, where $\eta$ is not necessarily an integer. To handle noise in corner detection, we use a median estimator to compute the virtual depth of the cluster, taking into account all combinations of point pairs in the disparity estimation.

Finally, we compute the BAP features from eq. 32, using the set of parameters $\Omega$ and the available virtual depth $v$. In each frame $n$, for each micro-image $(k, l)$ of type (i) containing a corner at position $(u, v)$ in the image, the feature $p_{k,l}^n$ is given by

$$p_{k,l}^n = [u \ v \ \rho \ 1]^T, \text{ with } \rho = r/s. \qquad (34)$$
In the end, our observations are composed of a set of micro-images centers {$c_{k,l}$} and a set of BAP features {$p_{k,l}^n$} allowing us to introduce two reprojection error functions corresponding to each set of features as explains in the next section.

## 5 Camera and Relative Blur Calibration

To retrieve the parameters of our camera model (eq. 10), we use a calibration process based on non-linear minimization of reprojection errors. The camera calibration process is divided into three phases: 1) the initial intrinsics are provided by the pre-calibration step; 2) the initial extrinsics are estimated from the raw checkerboard images; and 3) the parameters are refined with a non-linear optimization leveraging our new BAP features. In parallel, using our BAP features, the blur proportionality coefficient of eq. 9 is calibrated, by minimizing the relative blur in a new reprojection error with a non-linear optimization.

## 5.1 Camera Model Initialization

Iterative optimization of non-linear cost functions are sensitive to initial parameters setting. To ensure convergence and to avoid falling into local minima during the process, the parameters must be carefully initialized close to the solution. Our pre-calibration step provides a strong initial solution for the optimization. Intrinsic parameters are initialized as explained in Sect. 3.3 using only raw white images.

The camera poses {$T_c^n$}, i.e., the extrinsic parameters, are initialized using the same method as by (Noury et al. 2017). For each cluster of observations, the barycenter is computed, as illustrated by Fig. 6. Those barycenters can been seen as the projections of the checkerboard corners through the main lens using a standard pinhole model. For each frame, the pose is then estimated using the PnP algorithm (Kneip et al. 2011), like in classic pinhole imaging system. To associate 3D-2D correspondences, we reproject checkerboard corners based on the estimated pose in image space and link them to their nearest cluster of observations.

<figure><img src="image_9.png" /><figcaption>Fig. 6 Checkerboard raw image with: a clusters of observations; b their barycenter used as approximation for extrinsics initialization.</figcaption></figure>

## 5.2 Optimizing the Camera Parameters

By introducing blur in our model, we can optimize all parameters within one single optimization process. We propose a new cost function $\Theta$ taking into account the blur information of our new BAP feature. The cost is composed of two main terms both expressing errors in the image space: 1) the blur aware plenoptic reprojection error and 2) the main lens center reprojection error.

In the first term, for each frame $n$, each checkerboard corner $p_w^n$ is reprojected into the image space through each micro-lens ($k, l$) of type (i) according to the projection model of eq. 10 and compared to its observations $p_{k,l}^n$. In the second term, the main lens center $O$ is reprojected according to a pinhole model in the image space through each micro-lens ($k, l$) and compared to its detected micro-image center $c_{k,l}$. Let $S = \{\Xi, \{T_c^n\}\}$ be the set of intrinsic $\Xi$ and extrinsic $\{T_c^n\}$ parameters to be optimized. The cost function $\Theta(S)$ is expressed as

$$ \Theta(S) = \sum \| p_{k,l}^n - \Pi_{k,l}(p_w^n) \|^2 + \sum \| c_{k,l} - \Pi_{k,l}(O) \|^2. \quad (35) $$

The optimization is conducted using the Levenberg-Marquardt algorithm.

## 5.3 Relative Blur Calibration Using BAP Features

Relative blur estimation has been studied by Ens and Lawrence (1993); Mannan and Langer (2016). Up to our knowledge, it has never been studied in context of plenoptic camera. As a new contribution, we leverage the relative blur between different micro-images and our BAP features to calibrate the blur proportionality coefficient $\kappa$ of eq. 9.

**Relative blur model:** A point imaged by two different micro-lenses of type (i) and (j) will have different blur amount, i.e., the resulting images will have different spread parameters for the PSF model, such as

$$ \begin{cases} I_{(i)}(x, y) = h_{(i)} * I^*(x, y) \\ I_{(j)}(x, y) = h_{(j)} * I^*(x, y), \end{cases} \quad (36) $$

where $I^*(x, y)$ is the latent in-focus image. We approximate the PSF with a 2D Gaussian as in eq. 8, where the diameter of the blur kernel $h_{(i)}$ is $\sigma_{(i)}$. To compare two views with different amount of blur, we use the relative blur model in spatial domain (Pentland 1987; Subbarao 1988; Subbarao and Surya 1994; Ens and Lawrence 1993). As stated by Mannan and Langer (2016), the Gaussian relative blur approximation works well mainly for small relative blurs (up to $\rho \approx 5$ pixels) and when the aperture has a simple shape, which is the case with the plenoptic camera. We then use the equally-
defocused representation by applying additional blur to the relatively in-focus micro-image, hence,

$$ \begin{cases} I_{(i)}(x, y) \approx h_r * I_{(j)}(x, y) & \text{if } \sigma_{(i)} \ge \sigma_{(j)} \\ h_r * I_{(i)}(x, y) \approx I_{(j)}(x, y) & \text{if } \sigma_{(i)} \le \sigma_{(j)}. \end{cases} \quad (37) $$

Note that $h_r$ is the relative blur kernel applied to either one of the views such that both views are equally-defocused. The diameter of the relative blur kernel $h_r$ is approximated as

$$ \sigma_r(i, j) \approx \sqrt{|\sigma_{(i)}^2 - \sigma_{(j)}^2|}. \quad (38) $$

This approximation is exact when the PSF is a Gaussian. Since the radius of the relative blur kernel $\sigma_r$ cannot indicate whether the (i) or the (j) view is more in-focus than the other, we define the relative blur similarly to Chen et al. (2015), as

$$ \Delta\sigma^2(i, j) \triangleq \sigma_{(i)}^2 - \sigma_{(j)}^2, \quad (39) $$

where $\Delta\sigma^2(i, j) > 0$ indicates that a pixel in the (j)-micro-image is more in-focus than its corresponding pixel in the (i)-micro-image. Symmetrically, $\Delta\sigma^2(i, j) < 0$ indicates that the (i)-micro-image is more in-focus. In a similar fashion, we define the relative blur radius as

$$ \rho_r(i, j) \approx \sqrt{|\Delta\rho^2(i, j)|} = \sqrt{|\rho_{(i)}^2 - \rho_{(j)}^2|} \quad (40) $$

with $\sigma_r = \kappa \cdot \rho_r$, and where $\rho_{(i)}$, $\rho_{(j)}$ are the blur radii of the BAP features through a micro-lens of type (i) and (j).

**Blur proportionality coefficient calibration:** To calibrate $\kappa$, we use our BAP features and the relative blur model applied on micro-images of different types. BAP features $\{p_i\}$ from a same cluster C represent the same point in object space $p_w$. We extract two windows W around the BAP features $p_i$, $p_j \in C(p_w)$ of different types, and express them using the equally-defocused representation (eq. 37). As the relative blur radius does not exceed 2.5pix, windows W of size 9 × 9 are extracted at (u, v) with sub-pixel precision, and represent therefore the same part of the scene in both micro-images. Additional blur is applied using a Gaussian kernel of spread parameter $\sigma_r$. The spread parameter is computed from the $\rho$ part of the BAP features and the parameter $\kappa$ to be optimized, with initial value $\kappa = 1$. Let $\Theta(\kappa)$ be the cost function to be minimized. It is expressed as

$$ \Theta(\kappa) = \sum_{n} \sum_{p_i^n, p_j^n \in C(p_w^n)} \|W(p_j^n) - h_r * W(p_i^n)\|_2^2, \quad (41) $$

given $|\rho_{(i)}| < |\rho_{(j)}|$ and where $h_r$ is the PSF with spread parameter $\sigma_r = \kappa \cdot \sqrt{|\rho_{(i)}^2 - \rho_{(j)}^2|}$. The optimization is conducted using the Levenberg-Marquardt algorithm.

<table><caption>**Table 1** Summary of R12-A, B, C, D, and UPC-S datasets contents</caption><thead><tr><th rowspan="2">h</th><th colspan="2">Target</th><th colspan="2">Calib. dist.</th><th colspan="3">Eval. dist.</th></tr><tr><th>Size</th><th>Scale</th><th>Min</th><th>Max</th><th>Min</th><th>Max</th><th>Step</th></tr></thead><tbody><tr><td>A</td><td>450</td><td>9 × 5</td><td>10</td><td>175</td><td>400</td><td>265</td><td>385</td><td>10</td></tr><tr><td>B</td><td>1000</td><td>8 × 5</td><td>20</td><td>400</td><td>775</td><td>450</td><td>900</td><td>50</td></tr><tr><td>C</td><td>∞</td><td>6 × 4</td><td>30</td><td>500</td><td>2500</td><td>400</td><td>1250</td><td>50</td></tr><tr><td>D</td><td>1500</td><td>5 × 3</td><td>20</td><td>850</td><td>1300</td><td>750</td><td>1200</td><td>50</td></tr><tr><td>S</td><td>hyperf.</td><td>9 × 6</td><td>26.25</td><td>250</td><td>800</td><td>200</td><td>500</td><td>50</td></tr></tbody></table>

All distances are given in mm. Scale refers to checkerboard square size. Evaluation distances refer to the linear motion table setup

# 6 Experimental Setup

To validate our camera model, we evaluate our method on real-world data obtained with a multi-focus plenoptic camera in a controlled environment. Our experimental setup is illustrated in Fig. 1. The camera is mounted on a linear motion table with micro-metric precision. The target plane is orthogonal to the translation axis, and the camera optical axis is aligned with this axis. The approximate absolute distances at which the images have been taken with the corresponding step lengths are reported in Table 1.

## 6.1 Hardware Environment

For our experiments we used a Raytrix R12 color 3D-light-field-camera, with a MLA of F/2.4 aperture. The camera is in Galilean internal configuration. We used two different mounted lens, a Nikon AF Nikkor F/1.8D with a 50 mm focal length for comparison with state-of-the-art, and a Nikon AF DC-Nikkor F/2D with a 135 mm focal length to validate the generalization of our model. The MLA organization is hexagonal row-aligned, and composed of 176 × 152 (width × height) micro-lenses with $I = 3$ different types. The sensor is a Basler beA4000-62KC with a pixel size of $s = 0.0055$ mm. The raw image resolution is 4080 × 3068 pixel. We calibrate our camera for four focus distance configurations, with $h \in \{450, 1000, \infty\}$ mm for the 50 mm lens, and with $h = 1500$ mm for the 135 mm lens. Note that when changing the focus setting, the main lens moves with respect to the block MLA-sensor.

## 6.2 Software Environment

All images have been acquired using the MultiCamStudio free software (v6.15.1.3573) of the Euresys company. We set the shutter speed to 5 ms. While taking white images for the pre-calibration step, we set the gain to its maximum value. For Raytrix data, we use their proprietary software RxLive (v4.0.50.2) to calibrate the camera, and compute the depth maps used in the evaluation. Our source code has
been made publicly available: https://github.com/comsee-research/libpleno, and https://github.com/comsee-research/compote.

## 6.3 Datasets

We build four datasets with different focus distance *h*: for the 50 mm lens, R12-A for *h* = 450 mm, R12-B for *h* = 1000 mm, and R12-C for *h* = ∞; for the 135 mm lens, R12-D for *h* = 1500 mm. Each dataset is composed of:

- white raw plenoptic images acquired at different apertures (*N* ∈ {4, 5.66, 8, 11.31, 16}) using a light diffuser mounted on the main objective for pre-calibration,

- free-hand calibration target images acquired at various poses (in distance and orientation), separated into two subsets, one for the calibration process (16 images) and the other for reprojection error evaluation (15 images),

- a white raw plenoptic image acquired in the same luminosity condition and with the same aperture as in the calibration targets acquisition for devignetting,

- and, calibration targets acquired with a controlled translation motion for quantitative evaluation, along with the depth maps computed by the RxLive software.

Examples of calibration targets acquired for the R12-B dataset are given in Fig. 7 along with their 3D poses. A summary for each dataset is given in Table 1, indicating checkerboard information and the distances at which the targets have been acquired for calibration and for the controlled evaluation. Our datasets have been made publicly available,

and can be downloaded from our public repository at https://github.com/comsee-research/plenoptic-datasets.

## 6.4 Simulation Environment

In order to validate our model on Lytro-like plenoptic camera configuration, i.e., unfocused plenoptic camera (UPC), we propose to evaluate our model in a simulation environment. We built our own simulator based on raytracing to generate images. Similar to the real-world dataset, we generated a dataset, named UPC-S, composed of several white images taken at different apertures (with $N \in \{2, 4, 5.6\}$), various checkerboard poses for calibration and validation, and for evaluation, checkerboard images with known translation along the z-axis. Details are also given in Table 1. We used the Lytro Illum intrinsic parameters reported in Table 4 of Bok et al. (2017) as baseline for the simulation. They have been converted into our parameters and reported in Table 3. The MLA organization is hexagonal row-aligned, and composed of 541 × 434 (width × height) micro-lenses of the same type ($I = 1$). The raw image resolution is 7728 × 5368 pixel, with a pixel size of $s = 0.0014$ mm and with micro-image of radius 7.172 pixel.

## 7 Results and Discussions

Our evaluation process follows the steps given in the overview (Fig. 2). First, we present the pre-calibration results, where white raw plenoptic images are used for computing micro-image centers, and for estimating initial camera

<figure><img src="image_10.png" /><figcaption>**Fig. 7** Example of calibration targets acquired for distances between 775 and 400 mm from the checkerboard used in the dataset R12-B, and their respective poses in 3D.</figcaption></figure>
<table><caption>**Table 2** Set of parameters $\Omega$ (in [$\mu$m]) computed during the pre-calibration step for each dataset, along with the calibrated relative blur proportionality coefficient.</caption><thead><tr><th></th><th>R12-A</th><th>R12-B</th><th>R12-C</th><th>R12-D</th></tr></thead><tbody><tr><td>Δ<sub>i</sub></td><td>128.222</td><td>128.293</td><td>128.333</td><td>127.851</td></tr><tr><td>λ</td><td>0.99441</td><td>0.99358</td><td>0.99380</td><td>0.99746</td></tr><tr><td>m</td><td>-140.596</td><td>-159.562</td><td>-155.975</td><td>-171.288</td></tr><tr><td>q'<sub>1</sub></td><td>35.135</td><td>36.489</td><td>35.443</td><td>38.599</td></tr><tr><td>q'<sub>2</sub></td><td>40.268</td><td>42.075</td><td>41.278</td><td>43.129</td></tr><tr><td>q'<sub>3</sub></td><td>36.822</td><td>38.807</td><td>37.858</td><td>40.788</td></tr><tr><td>κ</td><td>0.6988</td><td>0.6989</td><td>0.6531</td><td>0.8824</td></tr></tbody></table>

parameters. Second, from the set of devignetted calibration target images, BAP features are extracted, and camera intrinsic and extrinsic parameters are then computed using our non-linear optimization process. In parallel, the same BAP features are also used to calibrate the relative blur proportionality coefficient. Third, we evaluate our model quantitatively, firstly, using the reprojection error as a metric, and secondly, using the relative translation error in a controlled environment. Then, we propose an ablation study of the camera parameters. Finally, we illustrate how to characterize the plenoptic camera extended DoF using the blur profile.

## 7.1 Pre-calibration

To estimate the parameters $\Omega$, we set $\alpha = 2.357$, and since the camera is in Galilean internal configuration, we use $R = -\rho \cdot s$, following eq. 25. Fig. 5 shows the micro-image radii as function of the inverse f-number with the estimated lines for dataset R12-B. Their distributions are represented by the violin-boxes. For $N = 5.66$, we can see that radii distributions overlap, and that radii values are slightly overestimated as they do not fit exactly the borders of the micro-images. In practice, we only use white images that present distinguishable radii distributions in the estimation process, usually corresponding to small apertures. In case of R12-B, only white images at $N = 11.31$ and $N = 8$ are used. The corresponding coefficients for all datasets are summarized in Table 2. As expected, the parameter $m$ is different for each dataset, since $D$ and $\Delta_i$ vary with the focus distance $h$, whereas the $q'_i$ values are close for all datasets, even for different camera setup (R12-D).

## 7.2 Free-Hand Camera Calibration

**Comparison with state-of-the-art:** Since our model is close to the one of Noury et al. (2017), we compare our intrinsic with the ones obtained under their pinhole assumption using only corner reprojection error and with the same initial

parameters. In addition, we evaluate against the method of Nousias et al. (2017), which provides a set of intrinsics and extrinsics for each micro-lens type. The equivalence of our parameters and their parameters is given by

$$ 
\begin{gathered}
F = \frac{(f_x + f_y)}{2} \cdot s, \quad D = -F \cdot \left(\frac{K_1}{K_2} \cdot F + 1\right)^{-1}, \\ 
d^{(i)} = D - \frac{K_2 D}{D + K_2}, \quad u_0 = c_x \quad \text{and} \quad v_0 = c_y,
\end{gathered} 
\quad (42) 
$$

where $K_1$ and $K_2$ are the two additional intrinsic parameters that account for the MLA setting in their model. The equivalence also stands for the parameters of Bok et al. (2017). The provided detector from Nousias et al. (2017) was not able to detect corner observations on our datasets. Therefore, we used the same observations for our method (noted BAP in Table 3), Noury et al. (2017) method (NOUR), and Nousias et al. (2017) method for each type (NOUS1, NOUS2, and NOUS3), which allowed us to focus the comparison on the camera model only. Finally, we provide the calibration parameters obtained from the RxLive software (RTRX) corresponding to the model of Heinze et al. (2016), and compare our depth measurements to their depth maps.

**Initialization:** We initialize $\lambda$ from eq. 28. Its value for each dataset is reported in Table 2. The difference between the initial value of $\lambda$ and its value computed from optimized camera parameter is less than 0.024%, which validates the use of the initial value from eq. 28 when computing our BAP features. The initial camera parameters reported in Table 3 are computed using the methodology presented in Sect. 3.3. They are used for the BAP and NOUR methods. The camera internal configuration is set to Galilean. When $h$ decreases, $D$ increases. Yet when the main lens focus distance is at infinity, the main lens should focus on the plane $v = 2$, which implies that $D$ tends to $F - 2d$ as lower bound, as $H$ tends to $F$. In most cases (here, for R12-A, B, D), we will still have $F < D$, which usually can describe the camera in Keplerian configuration. In Keplerian internal configuration, the condition $F < D$ stands regardless of the focus distance, as $D$ lower bound is $F + 2d$.

When using the linear initialization from NOUS, the initial parameters of some configurations corresponded to impossible physical setup or were too far from the solution, hindering the convergence of the optimization. Therefore, in order to continue comparison, we manually set the initial parameters close enough to a solution. In contrast, we can see that the optimized parameters for BAP and NOUR are close to initial values, which shows that our pre-calibration step provides a strong initial solution for the optimization process.

**Intrinsic camera parameters:** Optimized intrinsic parameters are reported for each dataset and for all the evaluated
<table><caption>**Table 3** Initial intrinsic parameters for each dataset along with the optimized parameters obtained by our method (BAP) and with the methods of Noury et al. (2017) (NOUR), of Nousias et al. (2017) for each micro-lens type (NOUS1, NOUS2, NOUS3) and the parameters obtained from RxLive software (RTRX)</caption><thead><tr><th rowspan="2"></th><th colspan="5">R12-A (F = 50mm, h = 450mm)</th><th colspan="6">R12-B (F = 50mm, h = 1000mm)</th></tr><tr><th>Init.</th><th>BAP</th><th>NOUR</th><th>NOUS1</th><th>NOUS2</th><th>NOUS3</th><th>RTRX</th><th>Init.</th><th>BAP</th><th>NOUR</th><th>NOUS1</th><th>NOUS2</th><th>NOUS3</th><th>RTRX</th></tr></thead><tbody><tr><td>F [mm]</td><td>50</td><td>49.714</td><td>54.888</td><td>61.305</td><td>62.476</td><td>63.328</td><td>47.709</td><td>50</td><td>50.047</td><td>51.262</td><td>53.913</td><td>52.988</td><td>52.977</td><td>50.894</td></tr><tr><td>Q<sub>1</sub> [×10<sup>-5</sup>]</td><td>0</td><td>24.66</td><td>6.099</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>2.900</td><td>0.023</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>−Q<sub>2</sub> [×10<sup>-6</sup>]</td><td>0</td><td>2.998</td><td>0.925</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>0.300</td><td>0.093</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Q<sub>3</sub> [×10<sup>-8</sup>]</td><td>0</td><td>1.063</td><td>0.303</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>0.064</td><td>0.007</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>P<sub>1</sub> [×10<sup>-5</sup>]</td><td>0</td><td>−14.6</td><td>−15.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>14.13</td><td>12.14</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>−P<sub>2</sub> [×10<sup>-5</sup>]</td><td>0</td><td>6.340</td><td>5.020</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>21.54</td><td>18.16</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>D [mm]</td><td>56.658</td><td>56.701</td><td>62.425</td><td>71.131</td><td>72.541</td><td>73.530</td><td>—</td><td>52.113</td><td>52.125</td><td>53.296</td><td>56.062</td><td>55.128</td><td>55.124</td><td>—</td></tr><tr><td>−t<sub>x</sub> [mm]</td><td>11.29</td><td>10.97</td><td>9.771</td><td>—</td><td>—</td><td>—</td><td>—</td><td>11.30</td><td>12.44</td><td>12.67</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>−t<sub>y</sub> [mm]</td><td>8.411</td><td>7.887</td><td>8.334</td><td>—</td><td>—</td><td>—</td><td>—</td><td>8.416</td><td>5.988</td><td>6.114</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>−θ<sub>x</sub> [μrad]</td><td>0</td><td>843.1</td><td>468.6</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>607.2</td><td>576.2</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>θ<sub>y</sub> [μrad]</td><td>0</td><td>637.1</td><td>321.8</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>514.5</td><td>350.4</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>θ<sub>z</sub> [μrad]</td><td>0.6</td><td>31.5</td><td>25.3</td><td>—</td><td>—</td><td>—</td><td>—</td><td>41.9</td><td>17.0</td><td>46.0</td><td>35.3</td><td>—</td><td>—</td><td>41.9</td></tr><tr><td>Δμ [μm]</td><td>127.51</td><td>127.46</td><td>127.38</td><td>—</td><td>—</td><td>—</td><td>—</td><td>127.36</td><td>127.47</td><td>127.45</td><td>127.40</td><td>—</td><td>—</td><td>127.36</td></tr><tr><td>f<sup>(1)</sup> [μm]</td><td>578.15</td><td>578.18</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>581.10</td><td>580.49</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>f<sup>(2)</sup> [μm]</td><td>504.46</td><td>505.42</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>503.96</td><td>504.31</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>f<sup>(3)</sup> [μm]</td><td>551.67</td><td>552.08</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>546.39</td><td>546.36</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>u<sub>0</sub> [pix]</td><td>2039</td><td>2070.9</td><td>2289.8</td><td>1984.9</td><td>2034.5</td><td>1973.7</td><td>—</td><td>2039</td><td>1958.3</td><td>1934.9</td><td>2074.7</td><td>2094.7</td><td>1837.0</td><td>—</td></tr><tr><td>v<sub>0</sub> [pix]</td><td>1533</td><td>1610.9</td><td>1528.2</td><td>1482.1</td><td>1481.0</td><td>1495.2</td><td>—</td><td>1533</td><td>1802.9</td><td>1759.3</td><td>1640.2</td><td>1649.1</td><td>1620.4</td><td>—</td></tr><tr><td>d [μm]</td><td>318.63</td><td>324.77</td><td>402.32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>336.84</td><td>336.38</td><td>363.17</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>d<sup>(1)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>585.16</td><td>—</td><td>—</td><td>—</td><td>407.81</td><td>—</td><td>—</td><td>—</td><td>447.81</td><td>—</td><td>407.81</td></tr><tr><td>d<sup>(2)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>—</td><td>527.59</td><td>—</td><td>—</td><td>406.007</td><td>—</td><td>—</td><td>—</td><td>—</td><td>401.93</td><td>—</td><td>406.007</td></tr><tr><td>d<sup>(3)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>561.93</td><td>—</td><td>406.90</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>414.32</td><td>406.90</td></tr></tbody></table>
<table><caption>Table 3 continued</caption><thead><tr><th rowspan="2"></th><th colspan="10">R12-C (F = 50mm, h = ∞)</th><th colspan="3">R12-D</th><th colspan="2">UPC-S</th></tr><tr><th>Init.</th><th>BAP</th><th>NOUR</th><th>NOUS1</th><th>NOUS2</th><th>NOUS3</th><th>RTRX</th><th>Init.</th><th>BAP</th><th>Ref.</th><th>Init.</th><th>BAP</th></tr></thead><tbody><tr><td>F [mm]</td><td>50</td><td>50.013</td><td>53.322</td><td>51.113</td><td>49.919</td><td>50.812</td><td>51.564</td><td>135</td><td>136.105</td><td>9.9845</td><td>10</td><td>10.230</td></tr><tr><td>Q<sub>1</sub> [×10<sup>-5</sup>]</td><td>0</td><td>18.61</td><td>1.393</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>35.974</td><td>0</td><td>0</td><td>-47.913</td></tr><tr><td>-Q<sub>2</sub> [×10<sup>-6</sup>]</td><td>0</td><td>2.646</td><td>0.382</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>8.083</td><td>0</td><td>0</td><td>-3.882</td></tr><tr><td>Q<sub>3</sub> [×10<sup>-8</sup>]</td><td>0</td><td>1.038</td><td>0.104</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>4.82</td><td>0</td><td>0</td><td>-5.821</td></tr><tr><td>P<sub>1</sub> [×10<sup>-5</sup>]</td><td>0</td><td>19.11</td><td>27.72</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>-4.31</td><td>0</td><td>0</td><td>-3.577</td></tr><tr><td>-P<sub>2</sub> [×10<sup>-5</sup>]</td><td>0</td><td>7.311</td><td>4.461</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>-3.76</td><td>0</td><td>0</td><td>0.117</td></tr><tr><td>D [mm]</td><td>49.384</td><td>49.362</td><td>52.379</td><td>50.331</td><td>49.067</td><td>49.882</td><td>—</td><td>149.24</td><td>149.10</td><td>9.8479</td><td>10</td><td>10.005</td></tr><tr><td>-t<sub>x</sub> [mm]</td><td>11.26</td><td>13.12</td><td>14.16</td><td>—</td><td>—</td><td>—</td><td>—</td><td>11.30</td><td>11.21</td><td>5.459</td><td>5.405</td><td>0</td></tr><tr><td>-t<sub>y</sub> [mm]</td><td>8.418</td><td>7.446</td><td>6.231</td><td>—</td><td>—</td><td>—</td><td>—</td><td>8.387</td><td>8.351</td><td>3.718</td><td>3.748</td><td>0</td></tr><tr><td>-θ<sub>x</sub> [μrad]</td><td>0</td><td>490.9</td><td>487.7</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>371.1</td><td>0</td><td>0</td><td>-0.529</td></tr><tr><td>θ<sub>y</sub> [μrad]</td><td>0</td><td>388.9</td><td>366.1</td><td>—</td><td>—</td><td>—</td><td>—</td><td>0</td><td>287.0</td><td>0</td><td>0</td><td>0.944</td></tr><tr><td>θ<sub>z</sub> [μrad]</td><td>28.0</td><td>41.1</td><td>49.1</td><td>—</td><td>—</td><td>—</td><td>36.6</td><td>5.8</td><td>35.4</td><td>0</td><td>0</td><td>-0.033</td></tr><tr><td>Δμ [μm]</td><td>127.54</td><td>127.48</td><td>127.42</td><td>—</td><td>—</td><td>—</td><td>127.36</td><td>127.53</td><td>127.51</td><td>20</td><td>19.987</td><td>19.987</td></tr><tr><td>f<sup>(1)</sup> [μm]</td><td>554.35</td><td>569.88</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>625.63</td><td>636.06</td><td>40.0873</td><td>47.753</td><td>47.747</td></tr><tr><td>f<sup>(2)</sup> [μm]</td><td>475.98</td><td>491.71</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>559.91</td><td>572.52</td><td>—</td><td>—</td><td>—</td></tr><tr><td>f<sup>(3)</sup> [μm]</td><td>518.98</td><td>535.28</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>592.05</td><td>604.23</td><td>—</td><td>—</td><td>—</td></tr><tr><td>u<sub>0</sub> [pix]</td><td>2039</td><td>1692.1</td><td>2131.6</td><td>1966.3</td><td>1913.8</td><td>2052.5</td><td>—</td><td>2039</td><td>2028.7</td><td>3842.8</td><td>3863</td><td>3861.7</td></tr><tr><td>v<sub>0</sub> [pix]</td><td>1533</td><td>1677.8</td><td>1445.9</td><td>1484.6</td><td>1487.2</td><td>1492.7</td><td>—</td><td>1533</td><td>1526.7</td><td>2719.5</td><td>2683</td><td>2715.5</td></tr><tr><td>d [μm]</td><td>307.93</td><td>319.53</td><td>367.40</td><td>—</td><td>—</td><td>—</td><td>—</td><td>378.72</td><td>382.30</td><td>40.087</td><td>47.279</td><td>47.323</td></tr><tr><td>d<sup>(1)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>357.80</td><td>—</td><td>—</td><td>—</td><td>407.81</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>d<sup>(2)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>—</td><td>349.99</td><td>—</td><td>—</td><td>406.00</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>d<sup>(3)</sup> [μm]</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>353.26</td><td>—</td><td>406.90</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>

Reference and initial intrinsic parameters for the simulated Lytro dataset UPC-S along with the optimized parameters obtained by our method (BAP)

<figure><img src="image_11.png" /></figure>
methods in Table 3. First, BAP, NOUR and NOUS all verify the condition $F \approx D + 2d$ when the focus is set at infinity (R12-C). Second, the focal lengths obtained from NOUR, NOUS and RTRX change significantly given the focus distance, and the ones obtained from NOUS even vary according to the micro-lens types. In contrast, only BAP shows stable parameters across all three R12-A, B, C datasets. Shared parameters across datasets (i.e., the focal lengths and the distance between the MLA and the sensor) are close enough to indicate that our model successfully generalizes to different focus configurations. Furthermore, the parameters obtained by our method with an other main lens, i.e., R12-D, are coherent with the previously obtained parameters, stressing out that our model can be applied to a different camera setting. Finally, our method is the only one providing the micro-lenses focal lengths in a single unified model. The other methods calibrate either several MLA-sensor distances (RTRX), or several models, one for each type (NOUS).

Note that distortion coefficients and MLA rotations are close to zero. The influence of these parameters will be analyzed in the proposed ablation study of the camera model in Sect. 7.4.

*On simulated data:* First, pre-calibration has been performed using the white raw images. The resulting parameters $\Omega$ are coherent with the simulation parameters. With parameters $m = -23.639\mu$m and $q = -0.146\mu$m, we have $d \approx f$, which describes the *unfocused* configuration. Reference and initial intrinsic parameters are reported in Table 3, along with the optimized parameters. Second, calibration has been performed. The obtained intrinsic parameters are close enough to the references parameters, indicating that our method is able to generalize to the *unfocused* plenoptic camera.

For completeness, we also quantitatively evaluated the optimized parameters, by estimating the relative displacement between checkerboard with known motion along the z-axis. It results a translation error $\epsilon_z = 1.64\%$, which validates the model.

## 7.3 Quantitative Evaluations of the Camera Model

*Reprojection error:* In the absence of ground truth, we first evaluated the intrinsic parameters by estimating the reprojection error using the previously computed intrinsics. We consider only free-hand calibration target images which are not used in the calibration process. We use the RMSE as a metric to evaluate the reprojection error on the corner part of the features, for each dataset. For the BAP method, the corner reprojection part is reported in Table 4, as well as the radius reprojection part within parentheses. Regarding the NOUS methods, the original error is expressed using the mean reprojection error (MRE). We converted the final error to the RMSE metric for comparison. Note that the latter method operates separately on each type of micro-lens, meaning that

<table><caption>**Table 4** Corner reprojection error for each evaluation dataset (i.e., free-hand calibration target images not part of the calibration dataset) using the RMSE metric</caption><thead><tr><th></th><th>BAP</th><th>NOUR</th><th>NOUS1</th><th>NOUS2</th><th>NOUS3</th></tr></thead><tbody><tr><th>R12-A</th><td>0.856 (0.083)</td><td>0.713</td><td>0.773</td><td><strong>0.667</strong></td><td>0.958</td></tr><tr><th>R12-B</th><td>0.674 (0.183)</td><td>0.618</td><td>0.538</td><td><strong>0.519</strong></td><td>0.593</td></tr><tr><th>R12-C</th><td>0.738(0.041)</td><td>0.713</td><td>1.287</td><td>0.681</td><td><strong>0.411</strong></td></tr></tbody></table>

Bold values indicate the lowest errors

For the BAP method, reprojection error of the radius part is indicated within parentheses

the number of features is not the same as with NOUR and BAP. First, the reprojection error is less than 1 pixel for all methods, for each dataset, demonstrating that the computed intrinsics lead to an accurate reprojection model and can be generalized to images which are not from the calibration set. Second, even though the NOUS method provides the lowest RMSE, it shows a significant discrepancy according to the considered type. The error obtained by our method is slightly higher than the error from NOUR, but this can be explained by the fact that our optimization does not aim at minimizing only the corner reprojection error, but the radius reprojection as well. Note that the positional error $\epsilon_{u,v}$ predominates in the total cost by two orders of magnitude compared to the blur radius error $\epsilon_\rho$, but the latter still helps to constrain our model as shown by the relatively close intrinsics between the datasets.

*Controlled environment poses evaluation:* With our experimental setup, we acquired several images with known relative translation between each frame. We compare the estimated displacements along the z-axis from the extrinsic parameters to the ground truth. The extrinsics are computed with the models estimated from the free-hand calibration. In the case of the RTRX method, we use the filtered depth maps obtained with the proprietary software RxLive to estimate the displacements.

The translation errors along the z-axis with respect to the ground truth displacement from the closest frame are reported in Fig. 8 for datasets R12-A (a), R12-B (b) and R12-C (c). The relative error $\epsilon_z$ for a known displacement $\delta_z$ is computed as the mean absolute relative difference between the estimated displacement $\hat{\delta}_z$ and the ground truth, for each pair of frames $(T_i, T_j)$ separated by a distance $\delta_z$, i.e.,

$$ \epsilon_z(\delta_z) = \eta^{-1} \sum_{(T_i, T_j)|z_i - z_j = \delta_z} \left| \delta_z - \hat{\delta}_z \right| / \delta_z, \quad (43) $$

where $\hat{\delta}_z = \hat{z}_i - \hat{z}_j$, and $\eta$ is a normalization constant corresponding to the number of frames pair.
Translation error w.r.t. the ground truth displacement

<figure><img src="image_12.png" /><figcaption>**Fig. 8** Translation error along the z-axis with respect to the ground truth displacement from the closest frame, for datasets R12-A (**a**), R12-B (**b**) and R12-C (**c**). The error $ε_z$ is expressed in percentage of the estimated distances, and truncated to 7% to ease the readability and the comparison. The mean error with its confidence interval across all datasets for</figcaption></figure>

our method (BAP), (Noury et al. 2017) method (NOUR), (Nousias et al. 2017) method for each type (NOUS1, NOUS2, NOUS3), and for the proprietary software RxLive (RTRX) are reported in (d). Please refer to the color version for better visualization.

The mean error with its standard deviation across all datasets for BAP, NOUR, NOUS, and RTRX are reported in (d).

Firstly, the mean error across R12-A, B, C datasets are of the same order for the evaluated methods around 3%: for BAP, $ε_z = 2.92 ± 0.73\%$; for NOUR, $ε_z = 3.50 ± 3.08\%$; for NOUS1, $ε_z = 1.68 ± 1.53\%$; for NOUS2, $ε_z = 3.40 ± 2.19\%$; for NOUS3, $ε_z = 3.30 ± 3.35\%$; and, for RTRX, $ε_z = 4.96 ± 4.44\%$. This is also the case for the dataset R12-D where our model has a mean translation error of $ε_z = 3.37\%$. Note that all evaluated methods outperform RTRX as the depth maps computation might not be as precise as the optimization of extrinsic parameters. Our method ranks second in terms of relative mean error. Even though lowest error is obtained by the method NOUS for type (1), it presents a large standard deviation and the errors for the other two types are significantly higher. In real application context, there is no way to know in advance which type will produce the smallest error. Nousias et al. (2017) suggest that when extrinsics are sufficiently close, we can use representative extrinsics that are calculated by averaging the extrinsics from the individual types. Our results do not match this observation as the estimated extrinsics are significantly different for each type. As shown, only the first type gives satisfactory results

whereas the other two present a larger error with a significant standard deviation. Averaging the extrinsics from all types will therefore minimize the difference between poses but will not provide the best possible estimation.

Secondly, the standard deviation can be seen as an indicator of the estimation precision across the datasets, and thus indicates whether the model can generalize to several configurations or not. Our model presents the lowest standard deviation as illustrated in Fig. 8 (d). This indicates a low discrepancy between datasets and thus that the model is precise and consistent for all configurations.

Thirdly, we analyze the behavior of each method for each dataset across different distances. None of the methods suffered from a constant bias, as we do not observe a decreasing relative error as the distance increases. BAP and NOUR present a stable relative error for all distances, i.e., with approximately 0.3 % of standard deviation. This indicates that the estimation suffered only from a scale error. One could thus re-scale the poses to provide a precise and accurate estimation. We cannot draw any conclusion for the other methods since the variations do not follow any obvious pattern.

Finally, our model differs from the model of Noury et al. (2017) by modeling the micro-lens focal lengths. Compar-

<figure><img src="image_13.png" /></figure>
<table><caption>**Table 5** Ablation study of some camera parameters. For each dataset, the reprojection error $ε_{all}$, computed using the RMSE along with the relative translation error $ε_z$, expressed in %, are reported</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Tilt</th><th rowspan="2">Pitch</th><th rowspan="2">Dist</th><th colspan="2">R12-A</th><th colspan="2">R12-B</th><th colspan="2">R12-C</th></tr><tr><th>ε<sub>all</sub></th><th>ε<sub>z</sub></th><th>ε<sub>all</sub></th><th>ε<sub>z</sub></th><th>ε<sub>all</sub></th><th>ε<sub>z</sub></th></tr></thead><tbody><tr><td>1</td><td>✓</td><td>✓</td><td>✓</td><td><strong>0.860</strong></td><td><strong>3.23</strong></td><td><strong>0.698</strong></td><td><strong>3.15</strong></td><td>0.739</td><td>2.31</td></tr><tr><td>2</td><td>✓</td><td>✓</td><td>×</td><td>0.866</td><td>3.34</td><td>0.700</td><td>3.20</td><td><strong>0.737</strong></td><td>3.18</td></tr><tr><td>3</td><td>✓</td><td>×</td><td>✓</td><td>0.884</td><td>3.88</td><td>0.755</td><td>3.21</td><td>0.770</td><td><strong>2.00</strong></td></tr><tr><td>4</td><td>✓</td><td>×</td><td>×</td><td>0.891</td><td>3.98</td><td>0.752</td><td>3.24</td><td>0.773</td><td>3.13</td></tr><tr><td>5</td><td>×</td><td>✓</td><td>✓</td><td>0.865</td><td>3.48</td><td>0.784</td><td><strong>3.15</strong></td><td>0.760</td><td>2.89</td></tr><tr><td>6</td><td>×</td><td>✓</td><td>×</td><td>0.864</td><td>3.58</td><td>0.716</td><td>3.16</td><td>0.749</td><td>3.04</td></tr><tr><td>7</td><td>×</td><td>×</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>8</td><td>×</td><td>×</td><td>×</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

Bold values indicate the lowest errors

The symbol ✓ (resp., ×) indicates if we keep (resp., remove) the considered parameters

ing those two models, the mean error as well as the standard deviation is smaller with our method. The inclusion of the micro-lens focal lengths in the camera model improves the estimation precision and accuracy, and enables to generalize to several configurations. Dealing with different intrinsics which produce different extrinsics is not satisfactory when using the multi-focus plenoptic camera. In contrast, our model is able to manage all micro-lens types simultaneously, and proves to be stable across various configurations and working distances.

## 7.4 Ablation Study of Camera Parameters

To evaluate the influence of each parameter of the camera model, we present an ablation study of some of them. We focus the analysis on distortion coefficients ($Q_1, Q_2, Q_3, P_1$, and $P_2$), on some degrees of freedom of the MLA, especially its tilt with respect to the sensor ($θ_x, θ_y$), and the pitch between micro-lenses ($Δ_μ$). All combinations of the parameters have been tested, resulting in eight configurations. For each configuration and on each dataset of R12-A, B, C: first, we calibrate the camera intrinsic parameters; second, we evaluate the model using the RMSE of the reprojection error; and finally, we quantitatively estimate the relative translation error on the evaluation dataset. Each configuration has been initialized with the same intrinsic parameters, and used the same observations for all processes. Results are reported in Table 5. The first column is the configuration number. The Tilt column indicates if we keep (✓) or remove (×</) the parameters $θ_x$ and $θ_y$. The Pitch column stands for the parameter $Δ_μ$, and the column Dist for the distortion parameters $Q_1, Q_2, Q_3, P_1$, and $P_2$. The reprojection error $ε_{all}$ is given by its RMSE, and the relative translation error $ε_z$ is expressed in percent with respect to the ground truth displacement.

The configuration 1 is our reference, corresponding to the complete model. The optimized parameters are close to the

ones from Table 3, i.e., with less than 1% of variation, for all converging configurations and for all datasets.

First, the distortions do not impact the reprojection error of the model. Considering the pairs of configurations (1, 2), (3, 4), and (5, 6), the errors are similar with or without distortions, indicating that our camera does not suffer from lateral distortions. This is due to the relatively large main lens focal length. Nevertheless, distortions may have a role to play in case of shorter focal length.

Second, removing the rotations of the MLA does not improve nor worsen the reprojection error and the pose estimation. When keeping the tilt but freezing the pitch, the model is able to converge. The tilt, in combination with other factors (such as a slight decrease of the main lens focal length), compensates for the error introduced by the approximate value of the pitch. In contrast, configurations 7 and 8 do not converge to a solution, showing that when removing both the tilt and the pitch of the MLA, the model is not constrained enough, and the reprojection error cannot be minimized, resulting in a failure.

Finally, when freezing the pitch to its initial value, the positional part of the reprojection error increases. It is especially the case for dataset R12-A, where the reported errors in Table 5 are the highest of all configurations. This confirms our previous observation that the deviation of the micro-image centers and their optical centers does not satisfy an orthographic projection between the MIA and the MLA. The pitch should be taken into account, on one hand to improve the precision of the model, and on the other hand not to hinder the optimization process.

## 7.5 Relative Blur Calibration

We calibrate the blur proportionality coefficient $κ$ for the three datasets using our BAP features. Figure 9 presents two windows extracted around BAP features of different types from the same cluster, showing different amount of blur. The target image to be equally-defocused according to our model
<figure><img src="image_14.png" /><figcaption>**Fig. 9** a Reference image with highest amount of blur. b Target image to be equally-defocused. c Target image with additional blur. d Estimated PSF.</figcaption></figure>

is shown before, (b), and after, (c), blur addition. The estimated PSF of the relative blur is given in (d).

The optimized blur proportionality coefficients $κ$ are reported in Table 3. Theoretically, the parameter should be the same for all three datasets. Empirically this observation is validated for R12-A and R12-B. Estimated $κ$ for R12-C is lower. This is because the micro-lenses focal lengths in R12-C are slightly shorter than in R12-A and R12-B. Analytically, this difference generates a higher amount of relative blur, and thus a shorter estimate of $κ$ to match the observed blur in image space. In other words, $κ$ compensates for the slight differences in $f^{(i)}$ estimates. Therefore, $κ$ should be calibrated for each dataset.

## 7.6 Profiling the Plenoptic Camera

Using the parameters from our calibration process, we plot the *blur profile* of the camera, i.e., the evolution of the blur radius with respect to depth for each micro-lens type along with its corresponding DoF. Figure 10 shows the blur profiles obtained for our three focus distance configurations, with their DoFs expressed in mm. The blur radius is expressed in pixel and is given for each type, in *red* for type (1), in *green* for type (2) and in *blue* for type (3). Distances are given in object space in mm with their corresponding virtual depth on a secondary x-axis, spanning from $v = 1$ to 15, except for the configuration $h = \infty$ where we cropped just after the farthest focal plane. In MLA space, the profiles have the same behavior for all focus distances, as it only depends on the MLA parameters.

First, the horizontal dashed line represents the radius of the minimal acceptable e circle of confusion $r_0$. In our case, at a wavelength of 750nm, the radius of the smallest diffraction-limited spot is $r^* = 2.4\mu$m which is less than half the pixel size. We then choose $r_0 = s/2$. Despite not illustrated in the figure, the blur radius grows exponentially when getting closer to the plane $v = 0$. Once this limit is exceeded, the blur decreases and converges to a constant value of approximately 6 pixel. This happens for more distant objects when points are projected in front of MLA implying a negative virtual depth. This is the case for $h = 450$ and $h = 1000$ mm, but not for $h = \infty$, as the points were never projected closer

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /><figcaption>**Fig. 10** Blur profile, including each micro-lens type, in object space, at different focus distances: a $h = 450$ mm; b $h = 1000$ mm; and c $h = \infty$ mm. Focal planes and depth of fields are illustrated for each type. The blur radius is expressed in pixel as function of the object distance to the camera in mm. Corresponding virtual depth is reported on the secondary x-axis.</figcaption></figure>

than $v = 2$. In the working distance range, the blur does not exceed 5 pixel and grows when points are closer to the camera.

Secondly, we can use the DoF to select the range of working distances where the blur is not noticeable. The DoF increases in object space as the focus distance increases. As reported on the figures: for R12-A, the DoF is of 14.44 mm; for R12-B of 120 mm; and finally, for R12-C, the total DoF is of 223 m. In MLA space the total DoF is constant and spans from $v = 2.15$ to 3.45. As expected, the DoF overlap. In particular, the DoF of the type (3) micro-lens is entirely included in the other two, whereas the DoF of the
type (1) and (2) just touch. Within the total DoF, a point can
then be seen focused in two micro-images of different types
simultaneously, which eases the matching problem between
views.

Finally, we can easily identify the distance limits at which
the point will not be in the DoF anymore nor be projected
on multiple micro-images, i.e., corresponding to virtual dis-
tances |v| < 2. At these distances, disparity cannot be
computed in image space, and no depth estimation can be
performed. Such estimation can also be hindered by the res-
olution in virtual space compared to the resolution in object
space as disparity is inversely proportional to virtual depth.
For instance, for close objects, points will be projected on
more micro-images but with a low disparity. So the profiles
can be used to efficiently characterize the range of distances
according to the desired application. Furthermore, once the
MLA parameters are available, we can simulate an approx-
imate blur profile for the desired focus distance *h* with the
desired main lens focal length *F* by updating the value of *D*
using eq. 26 and eq. 27.

**8 Conclusion**

To calibrate a plenoptic camera, state-of-the-art methods rely
on simplifying hypotheses, on reconstructed data or require
separate calibration processes to take into account the multi-
focus configuration. Taking advantage of blur information
we propose: 1) a more complete plenoptic camera model
with the introduction of a new BAP feature that explicitly
models the defocus blur; this new feature is exploited in
our calibration process based on non-linear optimization of
reprojection errors; 2) a new relative blur calibration to fill the
gap between the physical and geometric blur, which enables
us to fully exploit blur in image space; and 3) a way to profile
the plenoptic camera and its extended depth of field (DoF).

Our camera model is applicable to the multi-focus plenop-
tic camera (both in Galilean and Keplerian configuration), as
well as to the single-focus and unfocused plenoptic camera.
In case of the Raytrix multi-focus camera, our ablation
study shows that main lens distortions and MLA tilt can
be omitted without hindering the calibration process nor
the pose estimation. The study also indicates that explic-
itly including the pitch of the micro-lenses in the model
improves the results. In addition, our calibration methods are
validated by quantitative evaluations in controlled environ-
ment on real-world data. Our method provides strong initial
intrinsics during the pre-calibration step, and coherent opti-
mized camera parameters for all evaluated configurations. It
shows a low and stable relative translation error across all the
datasets.

In the future, we plan to use blur information in comple-
ment to disparity to improve metric depth estimation.

**Acknowledgements** We thank Charles-Antoine Noury and Adrien Coly for their insightful discussions and their help during the acquisitions.

**Data Availability Statement** The code source has been made open-source and publicly available in the following repositories: https://github.com/comsee-research/libpleno, and https://github.com/comsee-research/compote.

**Code Availability** The datasets analyzed for this study can be found in the following repository: https://github.com/comsee-research/plenoptic-datasets. All data supporting the results of this paper will be provided upon request.

**Declarations**

**Conflict of interest** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

**References**

Bishop, Tom E., & Favaro, Paolo. (2012). The light field camera: Extended depth of field, aliasing, and superresolution. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 34(5), 972–986.

Bok, Y., Jeon, H.-G., & Kweon, I. S., (2014). Computer vision—ECCV 2014. Springer International Publishing. *Geometric Calibration of Micro-Lens-Based Light-Field Cameras Using Line Features*, pp. 47–61.

Bok, Y., Jeon, H.-G., & Kweon, I. S. (2017). Geometric calibration of micro-lens-based light field cameras using line features 39. *IEEE Transactions on Pattern Analysis and Machine Intelligence* (2), pp. 287–300.

Brown, D. (1966). Decentering distortion of lenses—The prism effect encountered in metric cameras can be overcome through analytic calibration. *Photometric Engineering* 32(3), 444–462.

Chen, C. H., Zhou, H., & Ahonen, T. (2015). Blur-aware disparity estimation from defocus stereo images. In: *2015 Inter Proceedings of the IEEE International Conference on Computer Vision*, pp. 855–863.

Conrady, A. (1919). Decentered lens-systems. *Monthly Notices of the Royal Astronomical Soceity*, 79, 384–390.

Dansereau, D. G., Pizarro, O., & Williams, S. B. (2013). Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*. pp. 1027–1034.

Ens, J., & Lawrence, P. (1993). An investigation of methods for determining depth from focus. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 15(2), 97–108.
Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. KDD.

Georgiev, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S. K., & Intwala, C. (2006). Spatio-angular resolution tradeoff in integral photography. *Rendering Techniques* 21, 263–272.

Georgiev, T., & Lumsdaine, A. (2009). Depth of field in plenoptic cameras eurographics (1), 5–8.

Georgiev, T., & Lumsdaine, A. (2009). CTuB3 resolution in plenoptic cameras frontiers in optics 2009/laser science XXV/fall 2009 OSA optics & photonics technical digest. CTuB3.

Georgiev, T., & Lumsdaine, A. (2012). The multifocus plenoptic camera. In: *Digital Photography VIII. International Society for Optics Photonics SPIE*, pp. 69–79.

Grossberg, M. D., & Nayar, S. K. (2005). The raxel imaging model and ray-based calibration. In: *61 International Journal of Computer Vision* (2), 119–137.

Hahne, C., Aggoun, A., Velisavljevic, V., Fiebig, S., & Pesch, M. (2018). Baseline and triangulation geometry in a standard plenoptic camera. In: *126 International Journal of Computer Vision* (1), 21–35.

Heinze, C., Spyropoulos, S., Hussmann, S., & Perwaß, C. (2016). Automated Robust metric calibration algorithm for multifocus plenoptic cameras. *65 IEEE Transactions on Instrumentation and Measurement*, (5) 1197–1205

Johannsen, O., Heinze, C., Goldluecke, B., Perwaß, C. (2013). On the calibration of focused plenoptic cameras. *8200 LNCS Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, pp. 302–317.

Johannsen, O., Honauer, K., Goldluecke, B., Alperovich, A., Federica, B., Yunsu, B., Brizzi, M. et al. (2017). A Taxonomy and evaluation of dense light field depth estimation algorithms. In: *2017-IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops*, pp. 1795–1812.

Kneip, L., Scaramuzza, D., & Siegwart, R. (2011). A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation. In: *Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, pp. 2969–2976.

Labussière, M., Teulière, C., Bernardin, F., Ait-Aider, O. (2020). IEEE 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). *Blur Aware Calibration of Multi-Focus Plenoptic Camera*, pp. 2542–2551.

Levin, A., Freeman, W. T., Durand, F. (2008). Understanding camera trade-offs through a Bayesian analysis of light field projections. *5305 LNCS Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, (PART 4), pp. 88–101.

Levoy, M., Hanrahan, P. (1996). *ACM Press Proceedings of the 23rd annual conference on Computer graphics and interactive techniques—SIGGRAPH '96*. Light field rendering, New York, New York, USA, pp. 31–42.

Lippmann, G. (1911). *Integral photography*. Academy of the Sciences

Lumsdaine, Andrew., Georgiev, Todor. (2009). The focused plenoptic camera. In: *IEEE International Conference on Computational Photography (ICCP)*, pp. 1–8.

Mannan, F., Langer, M. S. (2016). Blur calibration for depth from defocus. *13th Conference on Computer and Robot Vision (CRV)*, pp. 281–288.

Mannan, F., Langer, M. S. (2016). What is a good model for depth from defocus? *13th Conference on Computer and Robot Vision (CRV)*, pp. 273–280.

Mignard-Debise, L., Restrepo, J., Ihrke, I. (2017). A unifying first-order model for light-field cameras: the equivalent camera array. *3 IEEE Transactions on Computational Imaging, 4*, 798–810

Ng, R., Levoy, M., Duval, G., Horowitz, M., Hanrahan, P. (2005). Light field photography with a hand-held plenoptic camera. Stanford University, pp. 1–11.

Noury, C.-A. (2019). Etalonnage de caméra plénoptique et estimation de profondeur à partir des données brutes. Université Clermont Auvergne,

Noury, C. A., Teulière, C., Dhome, M. (2017). Light-field camera calibration from raw images DICTA. In: *2017—International Conference on Digital Image Computing: Techniques and Applications*, pp. 1–8.

Nousias, S., Chadebecq, F., Pichat, J., Keane, P., Ourselin, S., Bergeles, C. (2017). Corner-based geometric calibration of multi-focus plenoptic cameras. *Proceedings of the IEEE International Conference on Computer Vision*, pp. 957–965.

O'Brien, S., Trumpf, J., Ila, V., Mahony, R. (2018). Calibrating light-field cameras using plenoptic disc features. In: *IEEE 2018 International Conference on 3D Vision (3DV)*, pp. 286–294.

Palmieri, L., & Koch, R. (2017). Optimizing the lens selection process for multi-focus plenoptic cameras and numerical evaluation. In: *2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops*, pp. 1763–1774.

Pentland, Alex P. (1987). A new sense for depth of field PAMI-9. *IEEE Transactions on Pattern Analysis and Machine Intelligence* 4, pp. 523–531

Perwaß, C., Wietzke, L. (2012). SPIE human vision and electronic imaging XVII. Single Lens 3D-Camera with Extended Depth-of-Field 49(431), 829108.

Shi, S., Wang, J., Ding, J., Zhao, Z., New, T. H. (2016). Elsevier 70–88 parametric study on light field volumetric particle image velocimetry. *49 Flow Measurement and Instrumentation*.

Strobl, K. H., Lingenauber, M. (2016). Elsevier Inc. Stepwise calibration of focused plenoptic cameras. *145 Computer Vision and Image Understanding*, pp. 140–147.

Subbarao, M. (1988). Parallel depth recovery by changing. *Camera Parameters*, pp. 149–155.

Subbarao, M. (1989). Determining distance from defocused. *Images of Simple Objects*, pp. 11794–12350. 89-07-20.

Subbarao, M., Surya, G. (1994). Depth from defocus: a spatial domain approach. In: *13 International Journal of Computer Vision, 3*, 271–294.

Suliga, P., Wrona, T. (2018). IEEE microlens array calibration method for a light field camera. In: *Proceedings of the 19th International Carpathian Control Conference (ICCC)*, pp. 19–22.

Thomason, C. M., Thurow, B. S., Fahringer, T. W. (2014). Calibration of a microlens array for a plenoptic camera. In: *52nd Aerospace Sciences Meeting*, pp. 1–18.

Wang, Y., Qiu, J., Liu, C., He, D., Kang, X., Li, J., Shi, L. (2018). Virtual image points based geometrical parameters' calibration for focused light field camera. *6 IEEE Access*, (c), 71317–71326.

Zeller, N., Quint, F., Stilla, U. (2014). Calibration and accuracy analysis of a focused plenoptic camera. *II-3 ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences*, pp. 205–212.

Zeller, N., Quint, F, Sutterlin, M., Stilla, U. (2016). Investigating mathematical models for focused plenoptic cameras. *IEEE 2016 12th IEEE International Symposium on Electronics and Telecommunications (ISETC)*, pp. 301–304.

Zeller, N., Noury, C. A., Quint, F., Teulière, C., Stilla, U., Dhome, M. (2016). Metric calibration of a focused plenoptic camera based on a 3D calibration target III-3 ISPRS annals of photogrammetry. *Remote Sensing and Spatial Information Sciences*, pp. 449–456.

Zeller, N., Quint, F., Stilla, U. (2017). From the calibration of a light-field camera to direct plenoptic odometry. In: *11 IEEE Journal of Selected Topics in Signal Processing*, 7, 1004–1019.

Zhang, C., Ji, Z., Wang, Q. (2016). Unconstrained two-parallel-plane model for focused plenoptic. *Cameras Calibration*, pp. 1–20.
Zhang, Q., Zhang, C., Ling, J., Wang, Q., Yu, J. (2018). A generic multi-projection-center model and calibration method for light field cameras. *IEEE Transactions on Pattern Analysis and Machine Intelligence*

Zhao, Y., Li, H., Mei, D., Shi, S. (2020). Metric calibration of unfocused plenoptic cameras for three-dimensional shape measurement. *59 Optical Engineering*, 1(07).

Zhou, P., Cai, W., Yu, Y., Zhang, Y., Zhou, G. (2019). Elsevier Ltd 190–196 a two-step calibration method of lenslet-based light field cameras. *115 Optics and Lasers in Engineering*.

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Terms and Conditions

Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).

Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.

These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription (to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will apply.

We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as detailed in the Privacy Policy.

While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may not:

1. use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access control;

2. use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is otherwise unlawful;

3. falsely or misleadingly imply or suggest endorsement, approval, sponsorship, or association unless explicitly agreed to by Springer Nature in writing;

4. use bots or other automated methods to access the content or redirect messages

5. override any security feature or exclusionary protocol; or

6. share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal content.

In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue, royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any other, institutional repository.

These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.

To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied with respect to the Springer Nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law, including merchantability or fitness for any particular purpose.

Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed from third parties.

If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not expressly permitted by these Terms, please contact Springer Nature at

onlineservice@springernature.com