<figure><img src="image_1.png" /></figure>

HAL
open science

# Camera Models and Fundamental Concepts Used in Geometric Computer Vision

Peter Sturm, Srikumar Ramalingam, Jean-Philippe Tardif, Simone Gasparini,

Joao Barreto

► To cite this version:

Peter Sturm, Srikumar Ramalingam, Jean-Philippe Tardif, Simone Gasparini, Joao Barreto. Camera Models and Fundamental Concepts Used in Geometric Computer Vision. Foundations and Trends in Computer Graphics and Vision, 2011, 6 (1-2), pp.1-183. 10.1561/0600000023. inria-00590269

HAL Id: inria-00590269

https://inria.hal.science/inria-00590269

Submitted on 3 May 2011

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Camera Models and Fundamental Concepts Used in Geometric Computer Vision

By Peter Sturm, Srikumar Ramalingam, Jean-Philippe Tardif, Simone Gasparini and João Barreto

## Contents

1 Introduction and Background Material 3

1.1 Introduction 3

1.2 Background Material 6

2 Technologies 8

2.1 Moving Cameras or Optical Elements 8

2.2 Fisheyes 14

2.3 Catadioptric Systems 15

2.4 Stereo and Multi-camera Systems 31

2.5 Others 33

3 Camera Models 36

3.1 Global Camera Models 41

3.2 Local Camera Models 66

3.3 Discrete Camera Models 72

3.4 Models for the Distribution of Camera Rays 75

3.5 Overview of Some Models 82

3.6 So Many Models ... 84
<table>
  <tbody>
    <tr>
      <td><b>4 Epipolar and Multi-view Geometry</b></td>
      <td><b>90</b></td>
    </tr>
    <tr>
      <td>4.1 The Calibrated Case</td>
      <td>91</td>
    </tr>
    <tr>
      <td>4.2 The Uncalibrated Case</td>
      <td>92</td>
    </tr>
    <tr>
      <td>4.3 Images of Lines and the Link between Plumb-line Calibration and Self-calibration of Non-perspective Cameras</td>
      <td>100</td>
    </tr>
    <tr>
      <td><b>5 Calibration Approaches</b></td>
      <td><b>103</b></td>
    </tr>
    <tr>
      <td>5.1 Calibration Using Calibration Grids</td>
      <td>103</td>
    </tr>
    <tr>
      <td>5.2 Using Images of Individual Geometric Primitives</td>
      <td>110</td>
    </tr>
    <tr>
      <td>5.3 Self-calibration</td>
      <td>114</td>
    </tr>
    <tr>
      <td>5.4 Special Approaches Dedicated to Catadioptric Systems</td>
      <td>124</td>
    </tr>
    <tr>
      <td><b>6 Structure-from-Motion</b></td>
      <td><b>127</b></td>
    </tr>
    <tr>
      <td>6.1 Pose Estimation</td>
      <td>128</td>
    </tr>
    <tr>
      <td>6.2 Motion Estimation</td>
      <td>130</td>
    </tr>
    <tr>
      <td>6.3 Triangulation</td>
      <td>133</td>
    </tr>
    <tr>
      <td>6.4 Bundle Adjustment</td>
      <td>134</td>
    </tr>
    <tr>
      <td>6.5 Three-Dimensional Scene Modeling</td>
      <td>136</td>
    </tr>
    <tr>
      <td>6.6 Distortion Correction and Rectification</td>
      <td>137</td>
    </tr>
    <tr>
      <td><b>7 Concluding Remarks</b></td>
      <td><b>143</b></td>
    </tr>
    <tr>
      <td><b>Acknowledgements</b></td>
      <td><b>145</b></td>
    </tr>
    <tr>
      <td><b>References</b></td>
      <td><b>146</b></td>
    </tr>
  </tbody>
</table>
# Camera Models and Fundamental Concepts Used in Geometric Computer Vision

Peter Sturm<fnref n="1" />, Srikumar Ramalingam<fnref n="2" />, Jean-Philippe Tardif<fnref n="3" />, Simone Gasparini<fnref n="4" />, and João Barreto<fnref n="5" />

## Abstract

This survey is mainly motivated by the increased availability and use of panoramic image acquisition devices, in computer vision and various of its applications. Different technologies and different computational models thereof exist and algorithms and theoretical studies for geometric computer vision (“structure-from-motion”) are often re-developed without highlighting common underlying principles. One of the goals of this survey is to give an overview of image acquisition methods used in computer vision and especially, of the vast number of camera models that have been proposed and investigated over the years,

<footnote n="1">¹ INRIA Grenoble — Rhône-Alpes and Laboratoire Jean Kuntzmann, Grenoble, Montbonnot, France, Peter.Sturm@inrialpes.fr</footnote>

<footnote n="2">² MERL, Cambridge, MA, USA, ramalingam@merl.com</footnote>

<footnote n="3">³ NREC — Carnegie Mellon University, Pittsburgh, PA, USA, tardifj@gmail.com</footnote>

<footnote n="4">⁴ INRIA Grenoble — Rhône-Alpes and Laboratoire Jean Kuntzmann, Grenoble, Montbonnot, France, Simone.Gasparini@inrialpes.fr</footnote>

<footnote n="5">⁵ Coimbra University, Coimbra, Portugal, jpbar@deec.uc.pt</footnote>
where we try to point out similarities between different models. Results
on epipolar and multi-view geometry for different camera models are
reviewed as well as various calibration and self-calibration approaches,
with an emphasis on non-perspective cameras. We finally describe what
we consider are fundamental building blocks for geometric computer
vision or structure-from-motion: epipolar geometry, pose and motion
estimation, 3D scene modeling, and bundle adjustment. The main goal
here is to highlight the main principles of these, which are independent
of specific camera models.
1.1 Introduction

Many different image acquisition technologies have been investigated
in computer vision and other areas, many of them aiming at providing
a wide field of view. The main technologies consist of catadioptric and
fisheye cameras as well as acquisition systems with moving parts, e.g.,
moving cameras or optical elements. In this monograph, we try to give
an overview of the vast literature on these technologies and on com-
putational models for cameras. Whenever possible, we try to point out
links between different models. Simply put, a computational model for
a camera, at least for its geometric part, tells how to project 3D entities
(points, lines, etc.) onto the image, and vice versa, how to back-project
from the image to 3D. Camera models may be classified according to
different criteria, for example the assumption or not of a single view-
point or their algebraic nature and complexity. Also, recently several
approaches for calibrating and using "non-parametric" camera mod-
els have been proposed by various researchers, as opposed to classical,
parametric models.

In this survey, we propose a different nomenclature as our main
criterion for grouping camera models. The main reason is that even
so-called non-parametrics models do have parameters, e.g., the coordinates of camera rays. We thus prefer to speak of three categories: (i) A **global camera model** is defined by a set of parameters such that changing the value of any parameter affects the projection function all across the field of view. This is the case for example with the classical pinhole model and with most models proposed for fish-eye or catadioptric cameras. (ii) A **local camera model** is defined by a set of parameters, each of which influences the projection function only over a subset of the field of view. A hypothetical example, just for illustration, would be a model that is “piecewise-pinhole”, defined over a tessellation of the image area or the field of view. Other examples are described in this monograph. (iii) A **discrete camera model** has sets of parameters for individual image points or pixels. To work with such a model, one usually needs some interpolation scheme since such parameter sets can only be considered for finitely many image points. Strictly speaking, discrete models plus an interpolation scheme are thus not different from the above local camera models, since model parameters effectively influence the projection function over regions as opposed to individual points. We nevertheless preserve the distinction between discrete and local models, since in the case of discrete models, the considered regions are extremely small and since the underlying philosophies are somewhat different for the two classes of models.

These three types of models are illustrated in Figure 1.1, where the camera is shown as a black box. As discussed in more detail later in the monograph, we mainly use back-projection to model cameras, i.e., the mapping from image points to camera rays. Figure 1.1 illustrates back-projection for global, discrete and local camera models.

After describing camera models, we review central concepts of geometric computer vision, including camera calibration, epipolar and multi-view geometry, and structure-from-motion tasks, such as pose and motion estimation. These concepts are exhaustively described for perspective cameras in recent textbooks [137, 213, 328, 336, 513]; our emphasis will thus be on non-perspective cameras. We try to describe the various different approaches that have been developed for camera calibration, including calibration using grids or from images of higher level primitives, like lines and spheres, and self-calibration. Throughout
<figure><img src="image_2.png" /><figcaption>Fig. 1.1 Types of camera models. *Left*: For global models, the camera ray associated with an image point **q** is determined by the position of **q** and a set of global camera parameters contained in a vector **c**. *Middle*: For discrete models, different image regions are endowed with different parameter sets. *Right*: For discrete models, the camera rays are directly given for sampled image points, e.g., by a look-up table containing Plücker coordinates, here the Plücker coordinates **L**<sub>q</sub> of the ray associated with image point **q**.</figcaption></figure>

this monograph, we aim at describing concepts and ideas rather than
all details, which may be found in the original references.

The monograph is structured as follows. In the following section,
we give some background material that aims at making the math-
ematical treatment presented in this monograph, self-contained. In
Section 2, we review image acquisition technologies, with an emphasis
on omnidirectional systems. Section 3 gives a survey of computational
camera models in the computer vision and photogrammetry literature,
again emphasizing omnidirectional cameras. Results on epipolar and
multi-view geometry for non-perspective cameras are summarized
in Section 4. Calibration approaches are explained in Section 5, followed by an overview of some fundamental modules for structure-from-motion in Section 6. The monograph ends with conclusions, in Section 7.

## 1.2 Background Material

Given the large scope of this monograph, we rather propose summaries of concepts and results than detailed descriptions, which would require an entire book. This allows us to keep the mathematical level at a minimum. In the following, we explain the few notations we use in this monograph. We assume that the reader is familiar with basic notions of projective geometry, such as homogeneous coordinates, homographies, etc. and of multi-view geometry for perspective cameras, such as the fundamental and essential matrices and projection matrices. Good overviews of these concepts are given in [137, 213, 328, 336, 513].

**Fontes.** We denote scalars by italics, e.g., $s$, vectors by bold characters, e.g., $t$ and matrices in sans serif, e.g., $\mathbf{A}$. Unless otherwise stated, we use homogeneous coordinates for points and other geometric entities. Equality between vectors and matrices, up to a scalar factor, is denoted by $\sim$. The cross-product of two 3-vectors $\mathbf{a}$ and $\mathbf{b}$ is written as $\mathbf{a} \times \mathbf{b}$.

**Plücker coordinates for 3D lines.** Three-dimensional lines are represented either by two distinct 3D points, or by 6-vectors of so-called Plücker coordinates. We use the following convention. Let $\mathbf{A}$ and $\mathbf{B}$ be two 3D points, in homogeneous coordinates. The Plücker coordinates of the line spanned by them, are then given as:

$$ \begin{pmatrix} B_4 \bar{\mathbf{A}} - A_4 \bar{\mathbf{B}} \\ \bar{\mathbf{A}} \times \bar{\mathbf{B}} \end{pmatrix}, \qquad (1.1) $$

where $\bar{\mathbf{A}}$ is the 3-vector consisting of the first three coordinates of $\mathbf{A}$ and likewise for $\bar{\mathbf{B}}$.

The action of displacements on Plücker coordinates is as follows. Let $\mathbf{t}$ and $\mathbf{R}$ be a translation vector and rotation matrix that map points according to:

$$ \mathbf{Q} \mapsto \begin{pmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^{\top} & 1 \end{pmatrix} \mathbf{Q}. $$
Plücker coordinates are then mapped according to:

$$ L \mapsto \begin{pmatrix} R & 0 \\ -[t] \times R & R \end{pmatrix} L, \qquad (1.2) $$

where 0 is the 3 × 3 matrix composed of zeroes.

Two lines cut one another exactly if

$$ L_2^T \begin{pmatrix} 0_{3 \times 3} & l_{3 \times 3} \\ l_{3 \times 3} & 0_{3 \times 3} \end{pmatrix} L_1 = 0. \qquad (1.3) $$

**Lifted coordinates.** It is common practice to linearize polynomial expressions by applying Veronese embeddings. We use the informal term “lifting” for this, for its shortness. Concretely, we apply lifting to coordinate vectors of points. We will call “n-order lifting” of a vector **a**, the vector $\mathcal{L}^n(\mathbf{a})$ containing all n-degree monomials of the coefficients of **a**. For example, second and third order liftings for homogeneous coordinates of 2D points, are as follows:

$$ \mathcal{L}^2(\mathbf{q}) \sim \begin{pmatrix} q_1^2 \\ q_1 q_2 \\ q_2^2 \\ q_1 q_3 \\ q_2 q_3 \\ q_3^2 \end{pmatrix} \qquad \mathcal{L}^3(\mathbf{q}) \sim \begin{pmatrix} q_1^3 \\ q_1^2 q_2 \\ q_1 q_2^2 \\ q_2^2 q_3 \\ q_2 q_3^2 \\ q_3^3 \end{pmatrix}. \qquad (1.4) $$

Such lifting operations are useful to describe several camera models. Some camera models use “compacted” versions of lifted image point coordinates, for example:

$$ \begin{pmatrix} q_1^2 + q_2^2 \\ q_1 q_3 \\ q_2 q_3 \\ q_3^2 \end{pmatrix}. $$

We will denote these as $\tilde{\mathcal{L}}^2(\mathbf{q})$, and use the same notation for other lifting orders.
We briefly describe various image acquisition technologies, with an emphasis on omnidirectional ones. This section aims at describing the most commonly used technologies in computer vision and related areas, without any claim of being exhaustive. More information, including historical overviews, can be found in the following references, which include textbooks, articles, and webpages [50, 51, 54, 71, 103, 220, 227, 294, 326, 335, 528, 541].

## 2.1 Moving Cameras or Optical Elements

### 2.1.1 Slit Imaging

Slit imaging has been one of the first techniques to acquire panoramic images. Various prototypes existed already in the nineteenth century, usually based on a moving slit-shaped aperture. Historical overviews are given in the references in the first paragraph of this section. In the following, we only review some more recent, digital slit imaging systems, mainly those developed for robotic and computer vision; similar systems were also developed for photogrammetric applications [326].
<figure><img src="image_3.png" /><figcaption>Fig. 2.1 Examples of generation of central slit images. Top: “Standard” slit imaging principle and an early realization of it, the cylindrograph of Moëssard [351]. Bottom: Slit imaging with a tilted 1D sensor, the so-called “cloud camera” and an image acquired with it (see text).</figcaption></figure>

Most of these systems either use a 2D camera or a 1D camera (also called linear camera or pushbroom camera) which “scans” a scene while moving, generating a panoramic image (cf. Figure 2.1). In the 2D camera case, only one or several columns or rows of pixels are usually kept per acquired image, and stitched together to form a panoramic image. Note that pushbroom images are highly related to the so-called epipolar plane images, see for example [56].

Sarachik, Ishiguro et al., and Petty et al. acquired panoramas from a rotating perspective camera by glueing together pixel columns from each image, and used them for map building or 3D measurement [247, 400, 434]. Barth and Barrows developed a panoramic image acquisition
system based on a fast panning linear camera, thus significantly
decreasing acquisition times [38]. A similar system was developed by
Godber et al. [178]. Benosman et al. developed a panoramic stereo
sensor based on the same principle, consisting of two linear cameras
mounted vertically on top of each other [52]; they rotate together
about their baseline and panoramic images are generated by stacking
the acquired 1D images together. Issues of calibration, matching, and
3D reconstruction are addressed in [52]. Klette et al. reviewed several
design principles and applications of such devices [281].

Usually, the rotation axis is parallel to the “slits” of pixels used
to form the panoramic image. A historic example of rotating a tilted
camera, is a design described by Fassig [136], called “cloud camera”: it
acquires a hemispheric field of view by revolving a tilted camera with
a wedge-shaped opening around an axis (cf. Figure 2.1).

There are many more such systems; an exhaustive list is out
of scope. The above systems are all designed to deliver **central
panoramic images**: panoramic referring to omnidirectionality in one
orientation, central referring to images having a single effective point
of view. This is achieved by rotating the 1D or 2D camera about an
axis containing its optical center.

In the following, we review several **non-central slit imaging
systems**.

One-dimensional cameras, or pushbroom cameras, are routinely
used in satellite imaging, since they provide a cheap way of obtaining
high-resolution images and since they are well adapted to the way satel-
lites scan planets. Pushbroom panoramas are non-central since each col-
umn is acquired at a different position of the satellite. The special case
of linear pushbroom panoramas, where the camera motion is assumed
to be a straight line (cf. Figure 2.2(a)), was extensively modeled by
Gupta and Hartley [197], see Section 3.1.4. They also proposed a sen-
sor model and a calibration method for the case of pushbroom cameras
on an orbiting satellite, e.g., moving along an elliptical trajectory [198].

Concerning close-range applications, Zheng and Tsuji seem to be
among the first researchers to have introduced and used non-central
panoramic images [570, 571, 572], sometimes also called non-central
mosaics, motion panoramas, or omnivergent images. Like in other
<figure><img src="image_4.png" /><figcaption>(a) Linear pushbroom acquisition.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Circular non-central panorama.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c) Dual slit principle to generate stereo panoramas using one rotating 2D camera.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d) Cyclograph generation with inward looking camera.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(e) Example of a cyclograph, courtesy of Steve Seitz [444].</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(f) Example of a panorama obtained with the approach of [4], see text. Courtesy of Aseem Agarwala.</figcaption></figure>

<caption>Fig. 2.2 Several ways of generating non-central slit or slit-like images.</caption>

systems, they proceeded by acquiring images using a moving camera, through a vertical slit, and stacking the acquired slit images one next to the other. If the camera is rotating about the vertical axis through its optical center, then the acquired image is a cylindrical mosaic, or panorama. If the camera is rotating about some other vertical axis, then we obtain non-central mosaics (cf. Figure 2.2(b)). Zheng and Tsuji have also proposed a generalization of this principle, for a camera moving on any smooth path. In the case of a straight line, we of course find the
above linear pushbroom panoramas. Zheng and Tusji used such motion
panoramas for route following of robots. Besides explaining the gener-
ation of panoramas, they also analyzed the apparent distortions in the
obtained images and other issues. They also used dual-slit panoramas
(two panoramas acquired using the same camera and two vertical slits)
for stereo computations (cf. Figure 2.2(c)), an idea that was later gener-
alized by Peleg et al. [394, 395, 397] and Li et al.[314], Seitz et al. [444].
Here, 3D points can be reconstructed from point matches between the
two non-central images by triangulation.

Ishiguro et al. used such panoramic views for map generation [246, 247]. They also used the dual-slit panoramas explained above for stereo computations, as well as stereo from panoramas acquired at different positions.

McMillan and Bishop used panoramic images for image-based ren-
dering, inspired by the plenoptic function concept [338].

Krishnan and Ahuja studied how to obtain the sharpest panoramic
images from a panning camera [288]. They showed that when using a
regular camera, whose CCD is perpendicular to the optical axis, the
camera should be rotated about an off-center point on the optical axis,
together with acquiring images with a varying focus setting. This effec-
tively yields non-central panoramas. Krishnan and Ahuja also proposed
another sensor design, where the CCD plane is not perpendicular to the
optical axis, and used it for panoramic image acquisition. The advan-
tage of such a sensor is that the depth of field volume is skewed and
so, while panning with a *fixed* focus setting, the union of the images'
depth of field volumes, is larger than for a fronto-parallel CCD, hence
avoiding to vary the focus setting while panning.

Usually, cameras are facing outward when acquiring slit images.
Inward facing cameras were also considered, as early as in 1895 [105],
leading to the principle of peripheral photography or images called
cyclographs. A sample cyclograph obtained using this approach is
shown in Figure 2.2(e). Let us mention another acquisition principle
that does not produce slit panoramas but is considering the acquisi-
tion of images taken around an object: Jones et al. proposed a setup to
acquire images as if taken all around an object, with a static camera and
object [258], whereas the usual approach is to use multiple cameras or a
turntable. To do so, they placed a cylindrical mirror around the object and used a rotating planar mirror to select the effective viewpoint.

Peleg et al. showed how to acquire non-central panoramas without having to move the camera, by using a mirror of a particular shape or a lens with a particular profile [396, 395]. When combining a static central camera with the appropriate mirror or lens, the camera rays of the compound system are distributed the same way as in circular non-central panoramas, i.e., they are incident with a circle containing the camera's optical center (cf. Figure 2.2(b)).

Seitz et al. studied these ideas with the goal of obtaining the most accurate 3D reconstruction from possibly non-central stereo pairs [444]. Their problem definition is that only a 2D set of lines of sight may be acquired as an image, from viewpoints constrained within a sphere. In order to maximize the reconstruction accuracy of points outside the sphere, each point should be viewed along two lines of sight at least, with maximum vergence angle. The result is that the lines of sight to be stored are tangents of the sphere. How to best choose the appropriate tangents is described in [444], resulting in so-called omnivergent images. Interestingly, omnivergent stereo pairs, although being pairs of non-central images, do have a standard epipolar geometry, with horizontal epipolar lines, thus allow to directly apply standard stereo algorithms (more on this in Section 3.4.2). However, acquiring such spherical omnivergent images is not simple, although Nayar and Karmarkar proposed systems for this task, based on catadioptric or fisheye cameras [371]. An approximate omnivergent stereo pair can be acquired via the dual-slit principle introduced by Zheng et al., see above.

Agarwala et al. have extended the principle of using slits of images to compose panoramic images, toward the use of image regions, chosen in order to better conform to the scene's shape [4]. They apply this to generate multi-viewpoint panoramas of urban scenes, such as of the facades of buildings along streets (see Figure 2.2(f) for an example).

Ichimura and Nayar studied the problem of motion and structure recovery from freely moving 1D sensors and rigid rigs of two or three 1D sensors, as well as from special motions; their study subsumes several previously studied cases such as linear pushbroom panoramas [243].
**2.1.2 Classical Mosaics**

By “classical” mosaics, we refer to classical in terms of the com-
puter vision community, i.e., mosaics generated by stitching together
2D images, but noting that slit imaging to generate mosaics has been
done before, at least with analog cameras. A tutorial on image align-
ment and stitching has been published in the same journal as this
survey, by Szeliski [487]. Due to this good reference and the fact, that
classical mosaic generation is widely known, we do not describe this
any further and simply give a few additional references.

An early and often overlooked work on digital mosaics is by Yelick and Lippman, who showed how to combine images obtained by a rotating camera to generate a mosaic [322, 547]. In his bachelor thesis [547], Yelick also discussed other omnidirectional image acquisition techniques, such as the fisheye lens and rotating slit cameras. Later works on digital mosaics include those by Teodosio and Mills [496], Teodosio and Bender [495] and Chen [92], to name but a few among the many existing ones.

**2.1.3 Other Technologies**

Murray analyzed the setup by Ishiguro et al. and others (see Sec-
tion 2.1.1) and proposed an alternative solution where instead of rotat-
ing the camera about some axis, the camera looks at a planar mirror
which rotates about an axis [362]. Such a system was already proposed
for aerial imaging by Bouwers and van der Sande [58]. However, the
issue of how to compensate for the camera displacement during rota-
tions of the mirror, due to the airplane's motion, was not fully discussed.
Other systems that use rotating mirrors or prisms are referenced by
Yagi [541], see also Section 2.3.4.

**2.2 Fisheyes**

The concept of fisheye view and lens dates back to more than a cen-
tury [48, 226, 350, 536]. Fisheye lenses or converters can achieve larger
than hemispheric fields of view but are usually still relatively costly.
A technical report suggesting inexpensive simple solutions to build
fisheye lenses was provided by Dietz [115]. Some more references on fisheyes are given in Section 3.1.7.

## 2.3 Catadioptric Systems

Using external mirrors together with cameras allows for a broad range of design possibilities, which is one of the reasons of the large number of catadioptric devices proposed alone in the computer vision community. Camera design may follow different goals, foremost in our context being a wide field of view, others being for example the compactness of a sensor, a single effective viewpoint, image quality, focusing properties, or a desired projection function. In the following, we exclusively describe geometric properties, in terms of projection function (focusing etc. of course also depend on mirror geometry).

We may distinguish five types of catadioptric systems: (i) single-mirror central systems, having a single effective viewpoint, (ii) central systems using multiple mirrors, (iii) non-central systems, (iv) single-lens stereo systems, and (v) programmable devices.

The fourth category, single-lens stereo systems, is of course a subset of the third category, non-central systems, but is singled out here since the goal is to obtain a sufficiently non-central system in order to enable accurate 3D modeling whereas for other systems, “non-centrality” is usually not a design goal but an artifact or a consequence of other design goals.

### 2.3.1 Single-Mirror Central Catadioptric Systems

Baker and Nayar derived all single-mirror central catadioptric systems [22] (Bruckstein and Richardson obtained some of these results independently [66]). Essentially the same results were also known outside the scope of catadioptric vision, e.g., were reported in 1637 by Descartes [112] and later by Feynman et al. [142] and Drucker and Locke [121]; further, it is likely that they were already known in antiquity to Greek geometers. Single-mirror central catadioptric systems can only be constructed using mirrors whose surface is obtained by revolving a conic about a symmetry axis of the conic. In addition, the camera looking at the mirror must be central and its optical center
<figure><img src="image_10.png" /><figcaption>Fig. 2.3 Illustration of (compositions of) single-mirror central catadioptric systems. Effective viewpoints are shown in green and the position of the true camera, in blue. The latter is only shown for the hyper-catadioptric case, for the para-catadioptric one, the camera is telecentric, thus has an optical center at infinity.</figcaption></figure>

has to coincide with one of the conic's foci, otherwise the whole system is non-central.<fnref n="1" /> The other focus is then the effective viewpoint of the catadioptric system: any (back-projection) line going out from the camera center goes, after being reflected in the mirror, through that second focus (cf. Figure 2.3).

The following special cases exist: hyperboloidal (cf. Figure 2.3(a)), paraboloidal (cf. Figure 2.3(b)), ellipsoidal, cone-shaped or planar mirrors. As for paraboloidal mirrors, one of the two real focus points is a point at infinity. In order to obtain a wide field of view, the only option is to “position” the camera at that point, which can be achieved using a telecentric lens. As for cone-shaped mirrors, the camera’s optical center has to be located at the cone’s tip; hence the only part of the mirror the camera sees corresponds to the rays that graze the mirror surface. Cone-shaped mirrors are thus theoretically excluded from the set of

<footnote n="1">¹It may be possible to achieve a central catadioptric system using a *non-central* camera looking at a mirror which is not necessarily conic-based; all that matters is that the rays that are back-projected from the non-central camera converge to a single point after reflection in the mirror. For any non-central camera and a desired effective viewpoint, one may be able to design an appropriate mirror.</footnote>
useful central catadioptric mirrors, although it has been shown that,
with a more general modeling of optics than applied in [22], practical
central viewing can still be achieved, see further below in this section.
Ellipsoidal mirrors do not allow to increase the field of view and are
thus not directly appropriate to build omnidirectional cameras; how-
ever, they are useful in designing so-called folded catadioptric systems,
consisting of two or more mirrors (see Section 2.3.2). As for spherical
mirrors, a special case of ellipsoidal ones, both real foci lie at the sphere
center; a camera positioned there only sees the reflection of itself, which
makes this case impractical. Overall, the only two systems deemed gen-
erally practical are the para-catadioptric and hyper-catadioptric ones,
based on paraboloidal and hyperboloidal mirrors.

Central hyper-catadioptric systems seem to have been used first, cf. the patent by Rees in 1970 [424] and first applications in robotics and computer vision by Yagi and his co-workers [540, 546]. They highlighted the single viewpoint property if the camera is positioned at one of the mirror's foci and also discussed optical properties of the system such as blur. They also combined cameras of different types, such as in the MISS system which is composed of a cone-based catadioptric camera (non-central) and a standard stereo system [544]. This allows to combine mutual advantages of the sensors, such as good localization using the catadioptric camera and larger resolution with the stereo system. A full trinocular analysis of line images was also proposed.

Nayar introduced central para-catadioptric systems, consisting of
an orthographic camera and a parabolic mirror, positioned such that
the viewing direction of the camera is parallel to the mirror axis [367,
368]. He also showed that two para-catadioptric sensors with a field of
view of 180° each, can be put back-to-back to achieve a full spherical
field of view while still preserving a single effective optical center (cf.
Figure 2.3(c)). This is possible since a 180° field of view is achieved with
a para-catadioptric system if the mirror extends till the cross section
containing its finite focus. Since the finite focus is exactly the effective
optical center, putting two such mirrors back-to-back makes their finite
foci coincide and thus also the two effective optical centers.

Such systems are rather widely used nowadays and commercialized
by several companies, see e.g., [103].
As mentioned above, cone-shaped mirrors were predicted as being impractical to achieve central catadioptric systems. This issue was reconsidered by Lin and Bajcsy [319, 320]. Their starting point was that the geometric analysis of Baker and Nayar considered that each 3D point is imaged along a single light path, whereas due to the finite aperture of real lenses, light rays emitted by a point within a finite volume, hit the image area. Reciprocally, a camera whose main lens is located at the tip of a cone (the actual tip being cut off) actually sees the mirror surface and not only rays grazing it. Lin and Bajcsy showed that it is possible to obtain sharp single-viewpoint catadioptric images in this case.

### 2.3.2 Central Catadioptric Cameras with Multiple Mirrors — Folded Catadioptric Cameras

Central catadioptric cameras can also be achieved when using more than one mirror. This allows more compact sensor designs and gives additional degrees of freedom to improve optical properties. Such designs are also termed *folded catadioptric cameras*; an excellent reference is [372], where Nayar and Peri presented several designs, design issues, and references to other works in this area.

Roughly speaking, when combining conic-shaped mirrors, and positioning them such that foci of successive mirrors in a sequence of reflections, coincide, then central image acquisition is possible: by placing a camera at the left-over focus of the “last” mirror, the compound system has a single effective viewpoint at the left-over focus of the “first” mirror (cf. Figure 2.4(a)).

Previously, Yagi and Yachida proposed such a system, consisting of two paraboloidal mirrors [542]. Nagahara et al. proposed a design for catadioptric optics for head mounted devices (HMDs), consisting of three mirrors, a planar, a hyperboloidal, and an ellipsoidal one, arranged such as to provide a single effective viewpoint, while achieving a wide field of view and avoiding occlusions of the field of view by the mirrors themselves [363]. Takeya et al. proposed another similar design [488]. Kim and Cho addressed the problem of calibrating a system composed of multiple successive mirrors, using a learning-based
<figure><img src="image_11.png" /><figcaption>(a) A folded central catadioptric system based on two hyperboloidal mirrors. The camera is now located inside one of the mirrors, at the blue position. The effective viewpoint of the compound systems is shown in green.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) A camera looking at a planar mirror is identical to a virtual camera behind the mirror, looking “the other way round”. Shown is the field of view of the true and the virtual camera; the green point is the effective viewpoint.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) Assembling camera-mirror pairs such that effective viewpoints coincide.</figcaption></figure>

<caption>Fig. 2.4 Illustration of multi-mirror central catadioptric systems.</caption>

approach [280]. Nagahara et al. showed how to achieve uniform angular resolution for a folded catadioptric camera with two mirrors and a single effective viewpoint [364].

Another way of achieving central projection with multiple mirrors is what is often called the “Nalwa pyramid”, although an earlier patent on essentially the same system is due to Iwerks [250]. Iwerks, Nalwa and independently, Kawanishi et al., proposed to use several regular cameras and as many planar mirrors [250, 271, 365]. A camera looking at a planar mirror produces the same image (up to side-swapping)
as a camera located behind the mirror, at the position obtained by
reflecting the original center in the mirror, and “inversely” oriented
(cf. Figure 2.4(b)). The reflected optical center position is the effective
viewpoint here. The main idea consists in arranging camera-mirror
pairs such that the effective viewpoints coincide. This is the easiest done
in a pyramidal layout (cf. Figure 2.4(c)), but others are imaginable.
Nalwa’s original design consists of four camera-mirror pairs, Kawanishi
et al. used six. Main advantages of such a system are omnidirectional
viewing (the different camera images can be stitched together), a single
effective viewpoint, and high resolution (as compared to mono-camera
omnidirectional systems).

Gao et al. propose a similar design that has a hemispherical field of
view [155]. Majumder et al. and Hua et al. placed two mirror-camera
pyramids back-to-back such that the effective viewpoints of the two
pyramids coincide, thus also enhancing the vertical field of view of the
compound sensor [234, 235, 329].

Greguss developed the so-called panoramic annular lens (PAL)
which combines reflective and refractive elements in a compact layout
and achieves panoramic viewing with a full spherical field of view in
horizontal direction and a vertical field of view of about 40° [188, 425].
Greguss' design has been used and/or improved for example by Pow-
ell [407] and Zhu et al. [574].

Yin and Boult built a sensor consisting of a tree of three different-
sized coaxial paraboloidal mirrors and an orthographic camera looking
at them [549]. Their motivation was not to achieve a single-lens stereo
system, rather to obtain an image pyramid by optical means: the three
mirrors are chosen such that they lead to omnidirectional images where
each one doubles the resolution with respect to the previous one. Ide-
ally, the mirrors' foci should coincide in order for the three pyramid
images to correspond to the same viewpoint; this is not exactly the
case in the proposed system but it was shown that the misalignment is
negligible if the scene is sufficiently far away.

2.3.3 Non-central Catadioptric Cameras

**Spherical mirrors.** Hong et al. used a non-central catadioptric
system with a spherical mirror for robot homing, a visual servoing
task [231]. They actually did not use the entire field of view that the
camera offers, but just a region around the horizon plane; the hori-
zon plane, i.e., the plane swept out by horizontal back-projection rays,
corresponds to a circle in the image plane. The robot on which the
camera is mounted is supposed to move on a horizontal ground plane.
Hence, points in the horizon plane are always imaged on the above
circle. In order to perform homing, the authors thus exploit a narrow
region around that circle, perform image analysis and matching in it
and use the result for a 2D visual servoing (translation and rotation
in the ground plane). While a spherical mirror always leads to a non-
central catadioptric system (unless the camera is placed at the sphere
center), considering only the image portion corresponding to the hori-
zon plane, actually corresponds to using an omnidirectional 1D camera
with a single effective viewpoint.

**Cone-shaped mirrors.** Yagi and Kawato used a catadioptric system with a cone-shaped mirror (called COPIS) for robot motion estimation and map building [542]. The system was non-central since the camera was located at a certain distance from the tip of the cone. Yagi and Kawato described the forward projection model as well as 3D point triangulation and epipolar geometry for translational motion. They as well as other researchers have used cone-shaped mirrors mounted vertically on robots since vertical lines, which are omnipresent in man-made scenes, are imaged as (long) radial lines and are thus easy to extract, e.g., [61, 83, 321, 393, 456]. Mouaddib and his co-workers used this idea for pose estimation, motion estimation, map building, etc. with the sensor SYCLOP they developed [393]. They proposed a calibration method, where the sensor is immersed in a hollow cube with a calibration checkerboard painted on the inside of each face [61, 83]. They then use both, extracted points and vertical lines of the calibration object, for calibration. A method for pose estimation (or, absolute localization) is presented in [82].

**Purpose-made mirrors optimizing resolution or satisfying other design goals.** A drawback of “standard” catadioptric sensors is a significant variation of effective resolution, or spatial/angular resolution, across the image area. Several works aimed at removing
or alleviating this, see the nice overview by Hicks and Perline [225]. Different works aimed at achieving different properties for sensor resolution. Uniform angular resolution is achieved by what is usually called equidistant or equiangular projection: the angle spanned by the back-projection ray of an image point and the optical axis, is proportional to the distance between the image point and the principal point. This is approximately the case for most fisheye lenses. Ollis et al. showed how to compute mirror surfaces that provide an equiangular catadioptric sensor [386]. Previously, Chahl and Srinivasan obtained a similar result, a mirror shape where the angle between the optical axis and a ray back-projected from the camera, is proportional to the same angle after reflecting the ray in the mirror [84]. Conroy and Moore achieved a sensor with solid angle pixel density invariance, i.e., where the surface area of a circular image portion is proportional to the solid angle of the corresponding field of view [99]. A similar work is due to Hicks and Perline [225]. Another related work by Gächter et al. achieved a similar goal [153], but for the case where the camera looking at the mirror has a log-polar sensor arrangement, such as cameras studied by Tistarelli and Sandini [505] and Pardo et al. [392].

All these works lead to non-central cameras. Nagahara et al. proposed a catadioptric system that is central and has uniform angular resolution [364]; this was possible by using two appropriately shaped and placed curved mirrors, whereas the above works used a single mirror each.

Hicks and Bajcsy as well as Kweon et al. showed how to compute mirror shapes such that a particular scene plane that is “fronto-parallel” to the mirror is imaged without distortion, while still providing a very wide field of view [222, 223, 297]. This was done for both, perspective and orthographic cameras looking at the mirror. Srinivasan contributed a similar result, i.e., a mirror that gives a wide field of view while directly providing a rectified image: in [457], he showed how to compute the shape of a mirror that directly provides cylindrical panoramas, i.e., rectangular images where the two coordinates relate to azimuth respective elevation angles of points relative to the camera’s optical axis. He found that there is no smooth mirror shape achieving this property, but that this can be approximated
by a piecewise planar mirror with a layout similar to a Fresnel lens
array.

Kondo et al. proposed an anisotropic mirror shape, i.e., that is not a
surface of revolution, with the aim of obtaining panoramic vision while
allocating higher spatial resolution to a preferred azimuth range, e.g.,
corresponding to the driving direction of a robot [284, 286].

Nayar and Karmarkar showed how to acquire 360 × 360 mosaics
by stitching together image slices acquired by a rotating *slice camera*
[371]. The slice camera is designed such as to have a 360° field of view
in one direction, while being orthographic in the other direction. This
is achieved by a specially designed mirror; in case the camera looking at
the mirror is orthographic itself, the mirror is a cone. Nayar and Kar-
markar's design extends slit imaging, cf. Section 2.1.1, to the acquisition
of a full spherical field of view, by being based on an omnidirectional
slit camera that rotates.

Peleg et al. showed how to acquire circular non-central mosaics using
a mirror of a special shape [395, 396], see Section 2.1.1.

A few more general approaches exist for mirror design, as follows.
Gaspar et al. proposed a general approach allowing to derive several
of the above mirror designs in a unified framework [156, 157]. Hicks as
well as Menegatti formulated and provided a solution to the so-called
prescribed projection problem [221, 341]: the input is a desired mapping
between an object surface and the image plane of a camera in a given
position. The goal is to compute a mirror and its location, that together
realize this mapping, i.e., the image taken by the resulting catadioptric
system is as specified. In many cases, there is no exact solution to
the problem, but approximate solutions can be found. Swaminathan
et al. addressed the same problem as Hicks in [486] and proposed a
solution that minimizes reprojection errors in the image plane, i.e.,
where the desired and the actual scene-to-image mapping (the “inverse”
of the prescribed projection problem) give image points as close to one
another as possible. Kondo et al. proposed another approach for the
same problem, akin to photometric stereo [285]. This approach allows
to conceive discontinuous mirror shapes.

All the above sensors are necessarily non-central (with the excep-
tion of [364]), although for many applications, one may model them
sufficiently well using a central approximation, see also the discussion
in Section 3.6.

Krishnan and Nayar recently proposed a catadioptric camera whose main optics is a fisheye, a camera they call cata-fisheye [289]. The motivation for this design is to achieve a spherical field of view in azimuth while not necessarily having a hemispherical one in elevation. Indeed, in many applications, such as videoconferencing and vehicle-mounted computer vision, a vertical field of view of a few dozens of degrees is often sufficient. Krishnan and Nayar proposed to achieve this by putting a convex mirror in front of a fisheye. The produced image consists of two annular regions: the outer one corresponds to the direct view at the scene through the fisheye, whereas the inner one shows the reflection in the mirror. Here, the outer image region would usually correspond to the part of the vertical field of view that is above the camera, while the inner region shows a part that is below. One advantage of this system is thus that the full image resolution is dedicated to the desired vertical field of view (as opposed to “wasting” resolution on a full hemispherical fisheye field of view if parts thereof are not relevant for a given application). Other advantages are a good image quality since a mirror of low curvature may be used, and a rather compact design. It is furthermore easy to regulate the desired vertical field of view by displacing the mirror or employing mirrors of different shapes. Strictly speaking, the system is non-central in general, but the working range in which the parallax is negligible, is explained in [289].

2.3.4 Single-Lens Stereo Systems

As mentioned at the beginning of Section 2.3, we consider here cata-
dioptic systems that are intentionally non-central in order to enable
stereovision, whereas the systems in the previous section are non-
central “by accident” or due to conforming to design goals prohibiting a
single effective viewpoint. Most catadioptric single-lens stereo systems
are achieved by setting one or more planar or curved mirrors in front
of the camera [73, 74, 99, 143, 183, 203, 255, 333, 366, 386, 455] or in
addition by keeping a direct view on the scene for a part of the camera’s
field of view [15, 49, 261, 295, 343, 422, 569].
Advantages of mirror-based stereo systems over multi-camera setups are that no camera synchronization and radiometric alignment of camera are required and that only one set of intrinsic parameters has to be calibrated, as opposed to calibrating each camera in a multi-camera setup.

Mouaddib et al. proposed a set of performance criteria to compare different single-lens catadioptric stereo systems [354, 355].

In the following, we describe some systems, first some using planar mirrors.

**Planar mirrors.** One of the earliest references suggesting a design for a mirror-based single-lens stereo system is probably a paper of 1899 by Finsterwalder [144], a seminal paper which summarized various results on camera calibration, pose and motion estimation, epipolar geometry, and even projective reconstruction from uncalibrated images. On pages 20–22 of this paper (written in German), Finsterwalder proposed to use a setup of three mutually perpendicular planar mirrors to photograph small objects put between the camera and the mirrors. He suggested that this gives up to eight perspective images (direct view up to multiple reflections) which can be used for stereo reconstruction. We do not know if this system has ever been built.

The idea of using a single planar mirror to acquire stereo images for 3D modeling, a direct and a reflected view of an object, has been considered by various researchers in the early twentieth century, see for example [39, 117, 409, 562, 563]. Some of these works were motivated by the modeling of coastal landscapes from images taken aboard a ship, the mirror surface being formed by a sea or lake. Note also that systems composed of planar mirrors were proposed for stereo *viewing* (as opposed to stereo imaging) as early as in 1838, by Wheatstone [526, 527] (see also [229]).

Kaneko and Honda showed that when acquiring an image of an object consisting of a part with a direct view of the object and another with its reflection in a planar mirror, this indeed corresponds to a stereo configuration and allows to reconstruct the object in 3D [261] (see also a similar approach by Zhang and Tsui [569]). This approach was extended by Mitsumoto et al. whose approach relaxed the requirement
of knowing the mirror’s position and who also proposed to use an addi-
tional planar mirror, to better recover occluded object parts [343].
Arnspang et al. also proposed to put planar mirrors in front of the
camera [15, 422]. One of the setups they suggested consisted of two
mirrors facing each other and which are parallel to the camera’s opti-
cal axis (cf. Figure 2.5(a) and (b)). Here, a scene point may be seen
multiple times: in the direct camera view, reflected in the mirrors or
even after a double reflection, once in each mirror. Arnspang et al. for-
mulated the triangulation problem for their setup and also proposed to
extend the setup by arranging more than two mirrors in a cylindrical
ring in front of the camera (cf. Figure 2.5(c)).

A similar idea was suggested by Han and Perlin who proposed a
single-lens stereo system akin to kaleidoscopes [203]. The camera looks
through a tapered tube whose inside is made of planar mirror facets.
Each planar mirror, together with the actual camera, corresponds to a
virtual camera. Hence, each scene point can be seen multiple times in
what is effectively a multi-view stereo system. Furthermore, multiple
reflections of light rays may happen inside the tube, thus multiplying
the number of virtual cameras and viewpoints, much like what one can
observe in a kaleidoscope. Han and Perlin’s idea was to use this acquisi-
tion system for the acquisition of the Bidirectional Texture Reflectance
of objects, based on the large number of viewpoints contained in a sin-
gle snapshot. Kuthirummal and Nayar proposed a similar system, see
further below.

Goshtasby and Gruver used two planar mirrors and no direct view of
the scene [183]. Cafforio and Rocca, Inaba et al. as well as Mathieu and
Devernay used four planar mirrors, where a pair each gives one virtual
camera, via the successive reflections in two mirrors (see Figure 2.6);
the system is thus basically equivalent to one with two planar mirrors,
but allows for an easier setup by more easily avoiding self-reflections of
the camera in the mirrors [74, 244, 333]. Gluckman and Nayar analyzed
in detail the relative orientation and epipolar geometry of such systems
as well as their self-calibration [174, 175].

In [177], Gluckman and Nayar studied the question of how to obtain
an optically rectified stereo pair using a single-lens catadioptric system
with planar mirrors. They found that an odd number of mirrors is
<figure><img src="image_14.png" /></figure>

(a) The camera (blue position) has a direct view of the scene as well as views reflected in two planar mirrors. Green dots show the position of virtual cameras corresponding to reflected views, purple dots that associated with double reflections. In the example, the scene point is seen four times in total.

<figure><img src="image_15.png" /></figure>

(b) Same setting as in figure (a). The numbers indicate how often regions of the scene are seen in the image, in direct, reflected or doubly reflected views.

<figure><img src="image_16.png" /></figure>

(c) Now the camera is bordered by four planar mirrors. The effective viewpoints of the system are shown using the same color code as in (a).

<figure><img src="image_17.png" /></figure>

(d) Camera looking through a cylinder whose inside is specular. The effective viewpoints are now located on circles. Each viewpoint only “sees” within the plane spanned by it and the axis of the cylinder.

<figure><img src="image_18.png" /></figure>

(e) When using a cone instead of a cylinder, the effective viewpoints are no longer coplanar, shown in 2D here.

<figure><img src="image_19.png" /></figure>

(f) Image obtained with (e), courtesy of Sujit Kuthirummal [295]. The inner part shows the direct view of the scene, the outer part the reflection in the cone.

<caption>Fig. 2.5 Illustration of some single-lens catadioptric stereo systems.</caption>

required and for the cases of one and three mirrors, derived the constraints on mirror placement that ensure rectified images. They also showed how to optimize the mirror placement in order to minimize the overall size of the entire sensor.

Avni et al. built a system composed of two cameras and two planar mirrors: the cameras as well as an object to be modeled in 3D are
<figure><img src="image_20.png" /><figcaption>Fig. 2.6 Left: Single-lens stereo system using planar mirrors. Here, four mirrors are used for practical reasons, leading to two effective viewpoints, shown in green. Right: A practical realization of the system, image courtesy of Frédéric Devernay [333].</figcaption></figure>

positioned above the mirrors such that each camera sees the object's reflections in both mirrors [20]. Hence, a total of four images of the object are obtained, allowing a multi-view 3D modeling.

**Non-planar mirrors.** Nayar used two specular spheres in the field of view of a camera to obtain stereo information [366]. Southwell et al. presented a design of a single-lens catadioptric stereo sensor with curved mirrors where one convex mirror is fixed on top of a second, larger one [143, 455]. Hence, points in the common field of view of the two individual catadioptric images are seen twice and can be reconstructed. Similar designs were proposed by Cabral et al. [73] and Jang et al. [255]. Jang et al. especially discussed how to maximize the effective stereo baseline.

Nene and Nayar described several single-lens catadioptric stereo configurations where both (virtual) cameras in a stereo system are central [374]. The first uses, like in other systems, two planar mirrors; Nene and Nayar described the epipolar geometry of this setup. The other proposed configurations use mirrors of revolution of conic-shape placed such that the camera is at a focus point of each of the mirrors, see Figure 2.7(a). Hence, each mirror corresponds to a central catadioptric system and by using two or more mirrors arranged this way, a stereo system is obtained where each (virtual) camera is central. Nene and Nayar described the cases of pairs of ellipsoidal, pairs of hyperboloidal and pairs of paraboloidal mirrors and derived the epipolar geometry for
<figure><img src="image_21.png" /><figcaption>Fig. 2.7 Other single-lens catadioptric stereo systems. *Top*: System using two hyperbolic mirrors with a coinciding focus point, at which the camera is located. This is a single-lens stereo system where each view is central. *Bottom*: Single-lens stereo system by Sagawa et al. composed of paraboloidal mirrors and sample acquired image; the camera looks at the mirrors from below. Images courtesy of Ryusuke Sagawa.</figcaption></figure>

each of those. Other cases are straightforward to imagine, e.g., combing ellipsoidal/hyperboloidal/planar mirrors in the appropriate manner.

Murphy presented a panoramic imaging system for planetary rovers where a single camera looks at a convex mirror with a hole in the middle, through which the camera sees through a fisheye lens [361]. Hence, a single image contains two panoramas, one for the lower part (the catadioptric view) and one for the upper one (fisheye). The system is similar to that of Krishnan and Nayar (cf. Section 2.3.3), although its aim is to obtain single-lens stereo while that of Krishnan and Nayar's design is omnidirectional viewing with a desired distribution of spatial resolution across the image. Benosman et al. proposed a sensor where a camera has both, a direct view of the scene and a view on a hyperbolic mirror [49]. The motivation is to have a high resolution in a dominant direction via the direct perspective image, e.g., the direction ahead of a robot, together with a lower-resolution panoramic view
of the surroundings. Yi and Ahuja proposed another single-lens stereo
system, consisting of a hyperboloidal mirror and a concave lens [548].
The camera has a direct view of the mirror and an indirect one of it,
through the concave lens, thus effectively producing a stereo pair.

Sagawa et al. built a single-lens catadioptric stereo system consisting of one camera looking at seven spherical or paraboloidal mirrors [432], see Figure 2.7(b) and (c). Although in principle such a system can be used for multi-baseline stereo, the prototype shown in [432] is of small dimension and thus has a small baseline. Its intended application is the detection of close-by objects, for which accurate 3D reconstruction is not necessary. The calibration of the sensor is discussed in [282], see also Section 5.3.2.

Lanman et al. built a similar catadioptric acquisition system con-
sisting of a single high-resolution camera looking at a set of spherical
mirrors arranged on a plate [298]. Their system is larger than Sagawa’s
since one intended application is 3D modeling. The system is obviously
non-central since already a single spherical mirror leads to a non-central
image. While a single spherical mirror is only “slightly non-central”
(more on this in Section 3.6), the setup by Lanman et al. effectively gen-
erates a large “baseline”, allowing for single image 3D reconstruction.
A practical method for calibrating the system, including the proper-
ties of second surface mirrors, is proposed in [298], and the multi-mirror
view geometry is analyzed, i.e., conditions that hold for points observed
in different mirrors, to be images of the same scene point.

Kuthirummal and Nayar proposed a system similar to the one by
Arnspang et al. and Han and Perlin (see above), where the piecewise
planar tube is replaced by a cone or cylinder-shaped one [295], see
Figure 2.5(d)–(f). They demonstrated the use of their system for single-
image 3D scene and reflectance recovery.

Orghidan et al. designed a structured-light type depth sensor composed of a camera looking at two hyperboloidal mirrors and a laser emitter [388]. They showed how to model and calibrate this system and use it for 3D modeling.

**Rotating mirrors.** Other catadioptric single-lens stereo systems have been proposed, using mirrors that rotate between different image
acquisitions in order to produce stereo images, e.g., by Teoh and
Zhang [497], Nishimoto and Shirai [376], Murray [362], and Gao and
Ahuja [154].

**2.3.5 Programmable Systems**

Hicks and Nayar et al. proposed methods and system designs allowing to modify the shape of the mirror in a catadioptric system in order to adapt the sensor to a change in the scene or to acquire an image in a desired way [224, 296, 370].

Nayar et al. proposed to use a programmable array of micro-mirrors to achieve programmable, or purposive, imaging [370, 369]. They demonstrated the use of digital micro-mirror devices (DMDs), routinely used in projectors, for tasks such as high dynamic range image acquisition, optical appearance matching, or generally speaking, the change of imaging geometry. In a similar work, Hicks et al. also suggested to use micro-mirrors to control the way an image is acquired, e.g., by actively controlling the spatial resolution across an image [224].

The following two ideas are not strictly speaking programmable
systems. Kuthirummal and Nayar studied the possibility of using a
flexible mirror sheet as reflector in a catadioptric system [296]. They
proposed to recover the current shape of the mirror from its outline in
the image and additional assumptions; once the shape is determined,
the catadioptric system is effectively calibrated. The system is not pro-
grammable in the same sense as those by Hicks and Nayar et al. but is
described in this section since it operates by changing the mirror shape
during image acquisition.

Fergus et al. proposed the concept of random lens imaging [140]. One practical instance of this concept is a camera looking at a collection of randomly positioned small mirrors or refractive elements. A calibration method is proposed and potential applications of the general concept for tasks, such as super-resolution and depth sensing, are described.

**2.4 Stereo and Multi-camera Systems**

Using two or more cameras to achieve omnidirectional viewing is a well-
known principle and will not be covered in great detail here, besides a
few historical references and references to stereo systems based on omni-
directional cameras. A nice overview of omnidirectional stereo systems
is due to Zhu [573].

Lin and Bajcsy proposed a catadioptric stereo system consisting
of two cameras looking at one cone-shaped mirror from different dis-
tances [321]. The cameras are aligned with the cone's axis of revolution;
to be precise, one camera lies on the axis, the other one looks at the
mirror through a beam splitter such that it virtually comes to lie on the
mirror axis. Hence, epipolar planes contain the axis, which simplifies
the epipolar geometry and 3D reconstruction.

Spacek proposed to use two cone-camera pairs, one on top of the
other and all cameras and mirrors axis-aligned [456]. He derives the
equations for stereo computations and highlights that cone-based cata-
dioptic cameras, although being non-central, can be used for stereo
and may be advantageous over the more common central catadioptric
systems due to their higher spatial resolution (lower vertical field of
view). A similar system, consisting of two hyper-catadioptric cameras,
was used in [57, 413].

Other stereo systems combine omnidirectional and traditional cam-
eras, to combine their respective advantages, see e.g., [3, 79, 118, 268].

Multi-camera systems composed of several *perspective* cameras have
been built at least as early as in 1884, initially mainly if not exclu-
sively for aerial imaging. The earliest work known to us (no effort was
made for an exhaustive bibliography research) is that of Triboulet,
who, as reported in [504] experimented from 1884 on with a multi-
camera system consisting of seven cameras attached to a balloon: one
camera looked downward and six cameras were equally distributed
around the balloon's circumference (the system thus resembles the
popular Ladybug sensor by Point Grey, http://www.ptgrey.com).
Similar other multi-camera systems from the early twentieth century
include those by Scheimpflug [438] and Thiele [499], cf. Figure 2.8.
Several other systems were developed throughout the first half of the
twentieth century, see for example [87, 403, 423, 502].

Other systems used multiple lenses in the same camera body. Gasser
as well as Aschenbrenner invented systems where multiple cameras
share the same focal plane and film; due to a suitable arrangement
<figure><img src="image_22.png" /><figcaption>Fig. 2.8 Three multi-camera systems from the first decade of the twentieth century. *Top*: two systems developed by Thiele [499]. The first one, called auto-panoramograph, was attached at the bottom of a balloon. The second system, the stereo-panoramograph, is equivalent to an omnidirectional stereo rig with a baseline of 2 m. *Bottom*: A system developed by Scheimpflug [438], consisting of seven oblique cameras and a central one. Shown are eight original images acquired with the system and a mosaic composed from them using a special projection equipment called photo-perspectograph.</figcaption></figure>

of prisms and mirrors, the different cameras directly generate a com-
posite wide-angle image on the film [18, 160]. These systems were,
like most of the above, intended for aerial imagery. Multi-lens cameras
were used at least as early as in 1882: Londe used multi-lens cameras
to study epileptic seizures, by capturing a succession of time-delayed
images [71], similar to the famous motion series by Muybridge according
to an acquisition principle he developed in 1872 [71].

2.5 Others

2.5.1 Plenoptic Camera

Adelson and Wang proposed the so-called plenoptic camera, a design
that essentially gives multiple pinhole images while using a single main
aperture and a single sensor array [2]. When placing a lenticular array
in front of a camera's sensor plane, each lenslet together with a subset
of the sensor array's pixels forms a tiny pinhole camera. That pinhole
camera effectively only captures rays entering the camera's main aper-
ture within a subregion of the aperture, as opposed to regular cameras,
where each pixel integrates light rays from all over the aperture. Con-
ceptually, a plenoptic camera thus corresponds to a multi-camera sys-
tem consisting of cameras arranged on a planar grid. Acquired images
can be used to perform stereo computations or various computational
photography tasks. We will not consider plenoptic cameras further in
this article since they are not usually used for panoramic imaging,
although in principle nothing would prevent to use them with fish-
eye lenses (besides the drop of spatial resolution for the already low-
resolution individual pinhole sub-cameras).

More information on plenoptic cameras and similar designs
can for example be found on http://en.wikipedia.org/wiki/
Integral_photography and in [429, 162] The concept of plenoptic
camera is highly related to that of integral imaging [323] and paral-
lax stereogram [248] or panoramagram [249].

**2.5.2 Biprism**

A single-lens stereo system using a biprism rather than a mirror, that
directly produces rectified images, has been developed by Lee et al.
[303].

**2.5.3 Spherical Lens and Spherical Image Area**

Krishnan and Nayar proposed an omnidirectional sensor consisting of
a transparent sphere, around which sensing elements are uniformly
arranged on a spherical surface, with free space between neighboring
elements [290]. Hence, individual sensing elements can see through the
transparent sphere and other sensing elements, to perceive the scene.
The design goals of the system are an omnidirectional field of view with
a single center of projection and uniform spatial resolution.

**2.5.4 Krill-Eye**

Hiura et al. proposed an optical arrangement for wide-angle imaging
system, termed krill-eye, that is inspired by compound animal eyes [228]
The system consists of gradient refractive index (GRIN) lenses aligned
on a spherical surface. Hiura et al. proposed a theoretical study on
image quality and focusing properties and built a proof-of-concept
prototype.

2.5.5 Rolling Shutter Cameras

Meingast et al. and Ait-Aider et al. studied the case of rolling shutter CMOS cameras, where the image area is not exposed simultaneously, but in a rolling fashion across rows of pixels [10, 171, 340]. The consequence is that when taking a picture of a moving object, it will appear distorted. Meingast et al. developed projection models for the general and several special cases of object motion and showed how optical flow can be modeled [340]. They also addressed the calibration of the sensor. Ait-Aider et al. used the rolling shutter's apparent drawback to develop an approach that computes both, the pose of a known object and its instantaneous velocity, from a single image thereof [10]. There is a conceptual link to pushbroom cameras, as follows, highlighted in a similar fashion in [171, 340]. An alternative way of looking at the rolling shutter phenomenon is to consider a static object and a pushbroom camera moving around it with a velocity that combines the inverse velocity of the object and a motion that compensates the difference in pose associated with different rows of pixels in the camera.

The effect of rolling shutters was also taken into account by Wilburn
et al. who built a multi-camera array with CMOS cameras [529, 530],
with several applications such as synthetic aperture imaging or allowing
to acquire videos with increased frame rate, dynamic range, resolution,
etc. A method for calibrating such a camera array was proposed by
Vaish et al. [516].
In the following, we review a certain number of camera models that were proposed in the literature. Some of them are based on physical models, others are of a more algebraic nature.

These models can be described and sorted according to various criteria. A first characteristic concerns the spatial distribution of the camera rays along which a camera samples light in the environment. Most models have a single optical center through which all camera rays pass. We also speak of **central camera models**. For these, the back-projection function (see below) delivers the direction of the camera ray. **Non-central camera models** do not possess a single optical center. In that case, the back-projection operation has to deliver not only the direction but also the position of a camera ray, e.g., some finite point on the ray. We will also use Plücker coordinates to represent camera rays. Special cases of non-central cameras are **oblique camera**, where no two camera rays meet [390] and **axial cameras** where there exists a line that cuts all camera rays. A special case of this are **x-slit** or **two-slit models** where there exist two lines that cut all camera rays. This is for example the case for linear pushbroom cameras [197].

A second property of camera models and one we would like to stress particularly, concerns how global or local a model is. The following
definitions have already been introduced in Section 1.1, cf. also Fig-
ure 1.1. Most models have a small set of parameters and in addition,
each parameter influences the projection function all across the field of
view. We call these, **global models**, since they hold for the entire field
of view/image area. Second, there exist several more **local models**,
e.g., models with different parameter sets for different portions of the
image. These models have usually more parameters than global ones,
but have a higher descriptive power. Finally, at the extreme, we have
**discrete models**, where the projection or back-projection function is
merely sampled at different points, e.g., sampling the back-projection
function at every pixel. These models have many parameters, one set of
them per sampled location. They are sometimes called non-parametric
models, but we do not find this entirely appropriate, since they do have
parameters; hence the proposed name of discrete models. Let us note
that the transition from global to local to discrete models is not dis-
continuous: some local models have the same parametric form as global
ones, and to use discrete models, one usually needs some interpolation
scheme, for example to be able to back-project any image point, not
only sampled ones. In that respect, discrete models plus, such as inter-
polation scheme, are actually local models in our language.

Many of the global and discrete models described in the following
are well known. This is less true for the local models, although they
may represent a good compromise between tractability (number
of parameters, stability of calibration) and generality. We would
like to point out the work of Martins et al. [331], which is rather
often cited but often described only partially or not appropriately.
They proposed three versions of the so-called two-plane model, see
Sections 3.1.3 and 3.2.1 for more details. These foreshadowed several
important contributions by others, mostly achieved independently.
First, it contains one of the first proposals for a local camera model
(the spline-based version of the developed two-plane model). Second,
it sparked works on discrete camera models, where camera rays are
calibrated individually from images of two or more planar calibration
grids [85, 189], an approach rediscovered by [191, 477]. Third, the
linear and quadratic versions of the two-plane model, when written
in terms of back-projection matrices, are nothing else than particular
instances of rational polynomial models; they are thus closely related
to the division and rational models explained by Fitzgibbon [145]
and Claus and Fitzgibbon [98] or the general linear cameras of Yu
and McMillan [559], as well as others, which found a lot of interest
recently. Martins et al. may also be among the first to explicitly use
lifted coordinates (cf. Section 1.2) in camera models.

Finally, a third main property of camera models we consider is that
some models have a direct expression for forward projection, others for
back-projections, some work easily both ways. This is important since
forward and back-projections are convenient for different tasks: forward
projection for distortion correction of images and bundle adjustment,
back-projection for minimal methods for various structure-from-motion
tasks, such as pose and motion estimation.

A few more general notes and explanations of notations follow.

**Back-projection versus forward projection.** One defines the other but one or the other may be more difficult to formulate algebraically. For example, the classical radial distortion model is written using polynomials; even with only one distortion coefficient, the polynomials are cubic, making it cumbersome to write the inverse model. This model is traditionally used for back-projection although some researchers used the same expression for forward projection. Other polynomial models are also used for either direction.

Generally speaking though, back-projection is often easier to for-
mulate. This is especially true for catadioptric systems, where back-
projection comes down to a deterministic “closed-form” ray-tracing
whereas forward projection entails a search for which light ray(s) emit-
ted by a 3D point gets reflected by the mirror(s) into the camera.
Also, when using rational polynomial functions in point coordinates
to express camera models, this is straightforward for back-projection;
using such a model for forward projection, results in general in curved
“camera rays”, i.e., the set of 3D points mapped onto an image point,
form a curve, not a straight line (cf. Section 3.1.8 on the cubic camera
model by Hartley and Saxena [210]).

For these and other reasons, we will emphasize back-projection
formulations in the following. Another reason for this is that back-
projection is less often used in the literature although it enables to
clearly formulate several structure-from-motion tasks and epipolar geometry.

Back-projection will be formulated in two different ways, either by giving expressions allowing to compute two points on a camera ray, or by directly giving the Plücker coordinates of the ray. As for the first case, one of the two points on the ray will be its point at infinity and the other, a finite point, denoted by

$$ \mathbf{B}^i \text{ and } \mathbf{B}^f, $$

respectively. The point at infinity will be given as a 3-vector, the finite point as a 4-vector, both in homogeneous coordinates. Note that in this survey we do not consider the possibility of camera rays that lie completely on the plane at infinity; such a case is of small practical interest and if necessary, all formulas are easy to adapt. Plücker coordinates of camera rays will be written as

$$ \mathbf{B}^l. $$

They can of course be easily computed from the two points $\mathbf{B}^i$ and $\mathbf{B}^f$, cf. Section 1.2.

The reason for using both expressions for back-projection is that the two-point expression is better suited to formulate pose estimation, whereas the Plücker-based expression allows to formulate motion estimation and epipolar geometry in a straightforward manner. When we deal with central camera models, we will only give the point at infinity $\mathbf{B}^i$ of camera rays, i.e., their direction, always assuming that cameras are in canonical position, i.e., with the optical center at the origin.

Whenever possible, we try to write the back-projection operation using back-projection matrices, operating on (lifted) image point coordinates. We will use full back-projection matrices

$$ \mathbf{B}^l $$

of size $6 \times n$, that map image points to camera rays in Plücker coordinates, and partial back-projection matrices

$$ \mathbf{B}^i $$

of size $3 \times n$, that map image points to camera ray directions. The values of $n$ for different models depend on which liftings of image coordinates are required.
Finally, let us note that the back-projection for a true camera usu-
ally gives rise to a half-line in 3D; many camera models do not take
this fully into account and only model back-projection via infinite lines.
We do the same in this monograph and, for ease of language, use the
term ray or camera ray to denote a line of sight of a camera.

**Radial distortion.** In this section, models are presented in differ-
ent forms, depending if they are backward or forward ones, or only
model radial distortion or more general distortions. Radial distor-
tion models are given in different forms; they are defined by a 1D
(un)distortion profile or (un)distortion function, mapping radial dis-
tances in the distorted image (distances of image point to distortion
center) to either radial distances in the undistorted image or the inci-
dence angle between camera rays and the optical axis.

We use the following notations. The radial distance is denoted
by $r_d$ (in distorted images) and $r_u$ (in undistorted images). The angle
between a camera ray and the optical axis is denoted by $\theta$. Radial
distortion models are usually given as a mapping between any of these
three entities. For simplicity, we always assume in the following that
the radial distortion center is equal to the principal point and that
it is known and located at the origin. When considering a distortion
center different from the origin, the given back-projection equations
can be modified in a straightforward manner, by preceding them with
the translation bringing the distortion center to the origin. When
considering a distortion center different from the principal point, the
modifications of back-projection equations are equally straightforward.
Whereas in the following, we suppose that the perspective back-
projection consists merely in “undoing” the focal length, considering
a distortion center that is different from the principal point, requires
a full perspective back-projection.

**Extrinsic parameters.** Camera pose, or extrinsic parameters, is straightforward to take into account. As for back-projection, translations and rotations can be applied as follows (cf. Section 1.2):

$$
\mathbf{B}^f \mapsto \begin{pmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^\top & 1 \end{pmatrix} \mathbf{B}^f
$$
$$ 
\begin{aligned}
\mathbf{B}^i & \mapsto \mathbf{R}\mathbf{B}^i \\
\mathbf{B}^l & \mapsto \begin{pmatrix} \mathbf{R} & 0 \\ -[\mathbf{t}]_\times \mathbf{R} & \mathbf{R} \end{pmatrix} \mathbf{B}^l.
\end{aligned}
$$

# 3.1 Global Camera Models

## 3.1.1 Classical Models

By “classical models”, we mean those used most often in applications and academic research, i.e., the pinhole model, affine camera models, and pinhole models enhanced with classical terms for radial and tangential distortions.

**Pinhole model.** The pinhole model, or perspective projection, assumes that all camera rays pass through a single point, the optical center and that there is a linear relationship between image point position and the direction of the associated camera ray. That relationship can be expressed via a so-called calibration matrix which depends on up to five intrinsic parameters:

$$ K = \begin{pmatrix} f_u & s & x_0 \\ 0 & f_v & y_0 \\ 0 & 0 & 1 \end{pmatrix}, $$

where $f_u$ respectively $f_v$ is the focal length measured in pixel dimensions, horizontally respectively vertically, $s$ is a so-called skew term which may for example model non-rectangular pixels or synchronization errors in the image read-out, and $(x_0, y_0)$ are the coordinates of the principal point, the orthogonal projection of the optical center onto the image plane. The linear relation between image points and ray directions can be written as:

$$ q \sim K\mathbf{B}^i. $$

The model is equally easy to use for forward and back-projections. In what follows, we will assume square pixels, i.e., $s = 0$ and $f_u = f_v = f$:

$$ K = \begin{pmatrix} f & 0 & x_0 \\ 0 & f & y_0 \\ 0 & 0 & 1 \end{pmatrix}. \qquad (3.1) $$
**Affine models.** We also briefly mention affine camera models, where the optical center is a point at infinity. Various submodels thereof exist, often called orthographic, weak-perspective, and para-perspective models, going from the most specific to the most general affine model [13, 232]. These models can give good approximations of pin-hole cameras in the case of very large focal lengths or if the scene is shallow in the direction of the camera’s optical axis as well as far from the camera. Back-projection is slightly different from the pinhole model in that the direction is the same for all camera rays (the direction corresponding to the optical center) and the essential back-projection operation is thus not the computation of the ray direction, but of a finite point on the ray.

**Classical polynomial distortion models.** The pinhole model has been enhanced by adding various terms for radial, tangential, and other distortions. The most widely established models are described for example in [63, 64, 149, 152, 451]. The usual model for combining radial and decentering distortion (one type of tangential distortion) is:

$$ 
\begin{align} 
\bar{x}_d &= x_d - x_0 \nonumber \\ 
\bar{y}_d &= y_d - y_0 \nonumber \\ 
r_d^2 &= \bar{x}_d^2 + \bar{y}_d^2 \tag{3.2} \\ 
x_u &= x_d + p_1(r_d^2 + 2\bar{x}_d^2) + 2p_2\bar{x}_d\bar{y}_d + \bar{x}_d \sum_{i=1}^{n} k_i r_d^{2i} \nonumber \\ 
y_u &= y_d + 2p_1\bar{x}_d\bar{y}_d + p_2(r_d^2 + 2\bar{y}_d^2) + \bar{y}_d \sum_{i=1}^{n} k_i r_d^{2i}, \nonumber 
\end{align} 
$$

where $n$ is the number of radial distortion coefficients used. The distortion corrected image points correspond to a pinhole model, i.e., are related to camera ray directions by:

$$ \mathbf{B}^i \sim \mathcal{K}^{-1} \begin{pmatrix} x_u \\ y_u \\ 1 \end{pmatrix}. $$

In the above model, it is assumed that the radial distortion center coincides with the principal point. Other models exist that consider them as distinct and/or include additional distortion terms, see e.g., [149, 215].
If we neglect tangential distortion and suppose that image coordinates are centered in the distortion center and that the latter is equal to the principal point, then we can deduce, from Equation (3.2), the radial (un)distortion function:

$$r_u = r_d \left( 1 + \sum_{i=1}^{n} k_i r_d^{2i} \right). \qquad (3.3)$$

This relates the distance of a point from the distortion center, before and after distortion. From this, it is easy to extract the relationship between the radial distance in the distorted image and the incidence angle of the camera ray with the optical axis:

$$\theta = \operatorname{atan}\frac{r_u}{f} = \operatorname{atan}\frac{r_d(1 + \sum_{i=1}^{n} k_i r_d^{2i})}{f}, \qquad (3.4)$$

which depends on the focal length of the camera.

Let us now consider a radial distortion with a single coefficient $k_1$. We can write down the camera ray direction in the following form:

$$\mathbf{B}^i \sim \mathbf{K}^{-1} \begin{pmatrix} x_d + k_1 \bar{x}_d r_d^2 \\ y_d + k_1 \bar{y}_d r_d^2 \\ 1 \end{pmatrix} \sim \begin{pmatrix} (x_d - x_0)(1 + k_1 r_d^2) \\ (y_d - y_0)(1 + k_1 r_d^2) \\ f \end{pmatrix} \\ \sim \mathbf{B}_{3 \times 10}^i \mathcal{L}^3(\mathbf{q}_d),$$

where the lifting operation $\mathcal{L}^3(\cdot)$ is given in Equation (1.4) and the back-projection matrix is:

$$\mathbf{B}^i \sim \begin{pmatrix} k_1 & 0 & k_1 & 0 & -3k_1x_0 & -2k_1y_0 & -k_1x_0 & 1 + k_1a & 2k_1x_0y_0 & -x_0(1 + k_1c) \\ 0 & k_1 & 0 & k_1 & -k_1y_0 & -2k_1x_0 & -3k_1y_0 & 2k_1x_0y_0 & 1 + k_1b & -y_0(1 + k_1c) \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & f \end{pmatrix},$$

with $a = 3x_0^2 + y_0^2, b = x_0^2 + 3y_0^2$ and $c = x_0^2 + y_0^2$.

Since the first and third columns are equal, as well as the second and fourth ones, we may use a more compact back-projection matrix of size 3 × 8:

$$\mathbf{B}_c^i \sim \begin{pmatrix} k_1 & 0 & -3k_1x_0 & -2k_1y_0 & -k_1x_0 & 1 + k_1a & 2k_1x_0y_0 & -x_0(1 + k_1c) \\ 0 & k_1 & -k_1y_0 & -2k_1x_0 & -3k_1y_0 & 2k_1x_0y_0 & 1 + k_1b & -y_0(1 + k_1c) \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & f \end{pmatrix},$$
which operates on lifted image coordinate vectors of the form:

$$ \begin{pmatrix} q_1^3 + q_1 q_2^2 \\ q_1^2 q_2 + q_2^3 \\ q_1^2 q_3 \\ q_1 q_2 q_3 \\ q_2^2 q_3 \\ q_1 q_3^2 \\ q_2 q_3^2 \\ q_3^3 \end{pmatrix}. $$

If we assume a known principal point (equal to the distortion center here) and translate image points such that the principal point lies at the origin, then the back-projection equations reduce further to:

$$ B_c^i \sim \begin{pmatrix} k_1 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & k_1 & 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & f \end{pmatrix}, $$

which can be compacted similarly to the above:

$$ B_{cc}^i \sim \begin{pmatrix} k_1 & 0 & 1 & 0 & 0 \\ 0 & k_1 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & f \end{pmatrix} \quad \text{operating on} \quad \begin{pmatrix} q_1^3 + q_1 q_2^2 \\ q_1^2 q_2 + q_2^3 \\ q_1 q_3^2 \\ q_2 q_3^2 \\ q_3^3 \end{pmatrix}. $$

Back-projection matrices for combined radial and tangential distortion models are equally straightforward to derive. Note that this back-projection formulation allows to easily write down equations for the epipolar geometry, cf. Section 4.2.1.

**Inversion of models.** Devernay and Faugeras discussed the analytical inversion of the classical radial distortion model with one coefficient, using Cardan's method. This analysis was adapted by Ma et al. to other variants of polynomial radial distortion models, especially rational polynomials [327], cf. also Section 3.1.8. Heikkilä showed how to formulate an approximate inverse model for radial and tangential distortions [215]. A more detailed and accurate inverse model was proposed by Mallon and Whelan [330].
**Other models and works.** The standard radial and tangential distortion model is a backward one, going from distorted to undistorted coordinates. Often, the same model is also employed for forward projection, see e.g., [41, 216, 525, 521], in which case the model coefficients' values will differ of course.

Many researchers have used other polynomial models for distortion/undistortion, e.g., the so-called polynomial fisheye transform (PFET) by Basu and Licardie [41], which is nothing else than a general polynomial relating $r_d$ and $r_u$, including even terms. Such models were for example applied for endoscope calibration, see [17, 204, 454, 565]. Other calibration approaches using such models are [8, 42, 134, 148, 215, 216, 274, 302, 330, 381, 436, 437, 514, 515, 521, 525, 534]. These approaches often differ with respect to the choice of calibration object, potentially carrying out controlled motions to position it in front of the camera, and with respect to the strategy for estimating the unknowns (e.g., estimating some parameters first while keeping others at initial values).

Kölbl proposed to use a trigonometric series to model radial distortion, in order to better condition the normal equations occurring in distortion estimation [283]. Several other distortion models are described in [275, 276].

Shih et al. established an approach for evaluating the errors made when neglecting lens distortion during calibration [449]. This is a function of the actual amount of true distortion, the number and positions/distribution of calibration points, the amount of noise and possibly other factors.

**Very wide fields of view.** It is well known that the above classical radial distortion model is not appropriate for cameras with a very wide field of view. To see this, refer to Equation (3.4): in order to achieve a hemispheric field of view ($\theta = 90^\circ$), one would need $r_d$ and thus the extent of the image area, to be infinitely large. Many alternative models exist, described in subsequent sections.

### 3.1.2 Extensions for Modeling Zooming and Focusing

In [64], Brown explained that lens distortion varies with a camera's focus setting but that knowledge of the distortion for two focal settings
allows to interpolate the distortion for any other setting. Also, radial distortion depends on the distance of the object to the camera. These effects are rigorously proven but seem to be used relatively seldomly in practice.

There exist various approaches for modeling zoom lenses and the influence of zooming on the intrinsic parameters, as well as for modeling other intricate phenomena related to camera calibration, see e.g., [11, 16, 53, 70, 96, 100, 102, 128, 150, 313, 469, 531, 533, 532].

### 3.1.3 Two-Plane Model

Chen et al. introduced implicit camera models in computer vision [89, 90]. Their two-plane model is mostly known through the subsequent paper by Martins et al. [331] and follow-up papers by Wei and Ma [522, 524] and others. As explained in the introduction of this section, the two-plane model is rather rich in ideas, which is why it will be covered in some detail here.

The model is defined as follows [331]. Consider two images of a planar calibration grid, put in known positions and assume that we can extract matches between the image and grid in both images. Since the grid positions are known, points on the grid actually define a 3D calibration object, which may be used as input to classical calibration approaches. The motivations for the two-plane model are to not rely on pre-defined camera models for the calibration and to handle non-central cameras. This may be achieved as follows. If we have dense matches, we may compute, for each image point for which we have two matches, the associated camera ray, simply by computing the line spanned by the two matched grid points. This idea was fully introduced later, by Gremban et al. [189] and Grossberg and Nayar [191]. Martins et al. did not assume dense matches; hence, in order to compute camera rays, one must have recourse to some interpolation scheme. Basically, for an image point we want to back-project, we need to determine the corresponding points on the two grids. To do so, Martins et al. proposed three interpolation schemes to compute corresponding points on calibration grids, from extracted matches: two global schemes (linear
<figure><img src="image_23.png" /><figcaption>Fig. 3.1 Illustration of the two-plane model. A₁ and A₂ represent the interpolation transformations for the two grids, affine respectively quadratic transformations for the linear respectively quadratic versions of the model.</figcaption></figure>

and quadratic) and a local one (spline-based). The global schemes are illustrated in Figure 3.1.

**First version — “Linear interpolation”**. The mapping between the image plane and each planar calibration grid was supposed to be an affine transformation:

$$ \begin{pmatrix} X_i \\ Y_i \\ Z_i \end{pmatrix} = A_i \mathbf{q}, $$
where $i = 1, 2$ is the index of the grid and

$$A_i = \begin{pmatrix} A_{i,11} & A_{i,12} & A_{i,13} \\ A_{i,11} & A_{i,12} & A_{i,13} \\ 0 & 0 & Z_i \end{pmatrix},$$

and it is assumed that $q_3 = 1$. Here, Martins et al. assumed that the two calibration grids are parallel to the X, Y plane, at known depths $Z_i$. They stated that other cases can also be handled similarly, but did not develop this further.

A more proper formulation using homogeneous coordinates also for 3D points, would separate the pose of the grid, parameterized by a rotation and translation, from the interpolation function “inside” the plane, which can be an affine transformation (as in Martins et al.’s work) or a projective one:

$$Q_i \sim \underbrace{\begin{pmatrix} R_i & \mathbf{t}_i \\ \mathbf{0}_3^\top & 1 \end{pmatrix}}_{\text{pose matrix}} \underbrace{\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}}_{M} A_i \mathbf{q} \sim \begin{pmatrix} \bar{R}_i & \mathbf{t}_i \\ \mathbf{0}_2^\top & 1 \end{pmatrix}_{4 \times 3} A_i \mathbf{q},$$

where $\bar{R}_i$ consists of the first two columns of $R_i$ and $M$ maps 2D coordinates on the grid plane, to 3D coordinates. The computation of the interpolation mappings $A_i$ (i.e., the **calibration**) would be carried out between image points and 2D points on calibration grids, instead of 3D points as in the original formulation above.

The back-projection ray of an image point $\mathbf{q}$ can be computed from the two grid points $Q_1$ and $Q_2$ according to Equation (1.1) in Section 1.2. Let $P_i = (\bar{R}_i \mathbf{t}_i)$. Then:

$$\mathbf{B}^l \sim \begin{pmatrix} (A_2\mathbf{q})_3 P_1 A_1\mathbf{q} - (A_1\mathbf{q})_3 P_2 A_2\mathbf{q} \\ (P_1 A_1 \mathbf{q}) \times (P_2 A_2 \mathbf{q}) \end{pmatrix}.$$

This expression is quadratic in the coordinates of the image point $\mathbf{q}$. It is thus clear that back-projection can be written via a $6 \times 6$ back-projection matrix:

$$\mathbf{B}^l \sim \mathbf{B}_{6 \times 6}^l \mathcal{L}^2(\mathbf{q}) \qquad (3.5)$$

operating on lifted image point coordinates $\mathcal{L}^2(\mathbf{q})$ (cf. Equation (1.4)).
In the special case of calibration planes parallel to the XY-plane and of affine transformations $A_i$, we get:

$$ B^l = \begin{pmatrix} 0 & 0 & 0 & A_{1,11} - A_{2,11} & A_{1,12} - A_{2,12} & A_{1,13} - A_{2,13} \\ 0 & 0 & 0 & A_{1,21} - A_{2,21} & A_{1,22} - A_{2,22} & A_{1,23} - A_{2,23} \\ 0 & 0 & 0 & 0 & 0 & Z_1 - Z_2 \\ 0 & 0 & 0 & Z_2 A_{1,21} - Z_1 A_{2,21} & Z_2 A_{1,22} - Z_1 A_{2,22} & Z_2 A_{1,23} - Z_1 A_{2,23} \\ 0 & 0 & 0 & Z_1 A_{2,11} - Z_2 A_{1,11} & Z_1 A_{2,12} - Z_2 A_{1,12} & Z_1 A_{2,13} - Z_2 A_{1,13} \\ b_1 & b_2 & b_3 & b_4 & b_5 & b_6 \end{pmatrix}, \quad (3.6) $$

where the last row is given by the vector

$$ \mathbf{b} = \begin{pmatrix} A_{1,11}A_{2,21} - A_{1,21}A_{2,11} \\ A_{1,11}A_{2,22} + A_{1,12}A_{2,21} - A_{1,21}A_{2,12} - A_{1,22}A_{2,11} \\ A_{1,12}A_{2,22} - A_{1,22}A_{2,12} \\ A_{1,11}A_{2,23} + A_{1,13}A_{2,21} - A_{1,21}A_{2,13} - A_{1,23}A_{2,11} \\ A_{1,12}A_{2,23} + A_{1,13}A_{2,22} - A_{1,22}A_{2,13} - A_{1,23}A_{2,12} \\ A_{1,13}A_{2,23} - A_{1,23}A_{2,13} \end{pmatrix}. $$

If instead of affine transformations, projective ones are adopted for $A_1$ and $A_2$, then the back-projection matrix $B^l$ has no zero coefficients anymore.

**Second version — “Quadratic interpolation”**. Here, the interpolation function between the image plane and each of the calibration grids, is assumed to be quadratic, and modeled by:

$$ \begin{pmatrix} X_i \\ Y_i \\ Z_i \end{pmatrix} = A_i \mathcal{L}^2(\mathbf{q}), $$

with 3 × 6 matrices $A_i$. Obviously, camera rays can be expressed via a back-projection matrix operating on degree-4 liftings of the image coordinates:

$$ \mathbf{B}^l \sim \mathbf{B}_{6 \times 15}^l \mathcal{L}^4(\mathbf{q}). $$

**Third version — “Linear spline interpolation”**. This is a local camera model according to our definition and will thus be explained in Section 3.2.1.
**Observations.** In [331] it is noted that “The basic difference between this model and the pinhole model is that all lines of sight are not forced to go through the same point”. A second major difference is that this model allows to handle non-perspective distortions that may in principle be quite general.

The linear two-plane model is actually identical to the general linear camera (GLC) model of Yu and McMillan [559], cf. Section 3.4.4, although it is formulated differently. Also, whereas the two-plane model was proposed in order to interpolate calibration data, the GLC was introduced in order to generate novel spatial distributions of camera rays, with applications for example in image synthesis.

The linear and quadratic two-plane models are particular instances of rational polynomial camera models, which were later introduced in full generality, cf. Section 3.1.8. Also, the two-plane models may be among the first to explicitly use lifted image coordinates in their formulation.

Finally, as mentioned above, they inspired works on discrete camera models; with the possibility of getting dense matches, it is indeed natural to calibrate camera rays directly for sampled image positions instead of computing interpolation transformations, as proposed by Gremban et al. [189], Grossberg and Nayar [191], and others, see Section 3.3.

**Extensions.** Izaguirre et al. showed how to incorporate the camera motion in the back-projection matrix [251]. Since we have explicitly written the back-projection, this can also be done easily by combining back-projection (e.g., Equation (3.5) for the linear two-plane model) with extrinsic parameters, as explained in Equation (1.2).

The two-plane model is formulated as a back-projection; Gremban et al. proposed an approach to deduce the forward projection operation [189]. In addition, they already posed the basis for a fully ray-based camera calibration, which is treated in Section 3.3.

The two-plane model was further generalized by Wei and Ma in several articles, as follows. In [522] necessary and sufficient constraints were derived for the back-projection matrix to correspond to a perspective back-projection, i.e., when no non-perspective distortion is present. In [523, 524] the authors also modeled the forward projection, using
the same algebraic model but with independent parameters and esti-
mation processes. There, they also investigated the epipolar geometry
associated with their camera model. The findings of [524] do not seem to
be fully general since both for the forward projection and the epipolar
geometry, the existence of an optical center of the cameras seems to be
assumed, whereas the considered camera model in general corresponds
to non-central cameras.

It would be worthwhile to establish the conditions under which two
cameras following the two-plane model, form a stereo pair, along the
lines of Seitz and Kim's study [445], see Section 3.4.2. Further, it would
be interesting to establish necessary and sufficient constraints on the
back-projection matrices under which the camera is central.

Champleboux et al. extended the two-plane model in two respects
[85]. First, by allowing the use of more than two images of calibration
planes. Second, by modeling the function that maps image points to
lines of sight, using bicubic B-splines. They applied their approach to
the calibration of cameras but also of radiographic and range sensors.

Fiala and Basu used a 1D version of the two-plane model to calibrate
the radial distortion profile of catadioptric cameras [143].

Lavest, Delherm et al. used the two-plane model to calibrate zoom
cameras for different zoom settings and to obtain 3D reconstructions
from images taken with different zoom factors but without moving the
camera [109, 300, 301].

**3.1.4 Models for Slit Cameras**

Consider slit image acquisition proceeding by moving a 1D perspective
camera with a constant translational velocity while acquiring images,
which are stacked together to form a 2D image, i.e., a linear pushb-
room panorama (cf. Section 2.1.1 and [197]). It is obviously an axial
camera since all viewpoints lie on a straight line due to the camera's
translational motion. In addition, all camera rays, for all acquisitions,
cut the plane at infinity in the same line — the intersection of the
1D camera's view plane with the plane at infinity. Linear pushbroom
cameras are thus a special case of two-slit camera, with one slit being
a line at infinity [389].
Gupta and Hartley proposed models for the linear pushbroom panorama and its multi-view geometry [197]. They showed that forward projection can be modeled by a 3 × 4 projection matrix, which relates 3D point coordinates with lifted coordinates of the corresponding image point. They also showed expressions for back-projection. Let us re-derive them here. Let the 1D camera be modeled by the following back-projection operations, where **q** are the homogeneous coordinates of image points:

$$ \mathbf{B}^f = \begin{pmatrix} \mathbf{t} \\ 1 \end{pmatrix} + \frac{q_2}{q_3} \begin{pmatrix} \mathbf{v} \\ 0 \end{pmatrix} \sim q_3 \begin{pmatrix} \mathbf{t} \\ 1 \end{pmatrix} + q_2 \begin{pmatrix} \mathbf{v} \\ 0 \end{pmatrix} $$

$$ \mathbf{B}^i = (\mathbf{a} \quad \mathbf{0} \quad \mathbf{b}) \mathbf{q} = q_1 \mathbf{a} + q_3 \mathbf{b}. $$

Here, the second image coordinate is assumed to correspond to the acquisition time and **v** gives the translational velocity of the camera; **t** is the optical center position at the first acquisition and vectors **a** and **b** depend on the 1D camera's intrinsic parameters and orientation. The latter are constant, thus the point at infinity of a back-projected ray does not depend on the second image coordinate (which explains the column of zeroes in the second above equation).

The Plücker coordinates of camera rays may now be expressed as follows, based on the above two equations defining $\mathbf{B}^f$ and $\mathbf{B}^i$ (cf. Equation (1.2)):

$$ \mathbf{B}^l = \begin{pmatrix} -q_3(q_1\mathbf{a} + q_3\mathbf{b}) \\ (q_3\mathbf{t} + q_2\mathbf{v}) \times (q_1\mathbf{a} + q_3\mathbf{b}) \end{pmatrix} = \underbrace{\begin{pmatrix} \mathbf{0} & -\mathbf{a} & \mathbf{0} & -\mathbf{b} \\ \mathbf{v} \times \mathbf{a} & \mathbf{t} \times \mathbf{a} & \mathbf{v} \times \mathbf{b} & \mathbf{t} \times \mathbf{b} \end{pmatrix}}_{\mathbf{B}_{6 \times 4}^l} \begin{pmatrix} q_1 q_2 \\ q_1 q_3 \\ q_2 q_3 \\ q_3^2 \end{pmatrix} $$

The geometry of more general two-slit cameras was studied by Pajdla and Zomet et al. [389, 576]. Among other issues, Zomet et al. derived the forward projection equations and have shown that 3D lines are imaged as conics. Feldman et al. showed that the back-projection for the same two-slit cameras can be expressed via a 6 × 6 back-projection matrix, operating on second order lifted image coordinates [139]. As will
be seen later (Sections 4.2.1 and 4.3), once it is established that back-projection can be written as a linear mapping operating on order-2 lifted image coordinates, it follows automatically that 3D lines are imaged as conics and that there exists a fundamental matrix of size 6 × 6.

Models for other slit image systems are in principle straightforward to establish. Consider the case of a 1D camera rotated about some axis (not necessarily containing the 1D camera’s optical center). Let the axis be the Z-axis and the optical center at the first acquisition ($q_2 = 0$) be at position $\mathbf{t}$. Further, let the intrinsic parameters and orientation of the 1D camera be subsumed by vectors $\mathbf{a}$ and $\mathbf{b}$, as above for the linear pushbroom case. Then, full back-projection operations can be written as:

$$ \mathbf{B}^f = \begin{pmatrix} \cos(kq_2/q_3) & -\sin(kq_2/q_3) & 0 & 0 \\ \sin(kq_2/q_3) & \cos(kq_2/q_3) & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} \mathbf{t} \\ 1 \end{pmatrix} $$

$$ \mathbf{B}^i = \begin{pmatrix} \cos(kq_2/q_3) & -\sin(kq_2/q_3) & 0 \\ \sin(kq_2/q_3) & \cos(kq_2/q_3) & 0 \\ 0 & 0 & 1 \end{pmatrix} (q_1\mathbf{a} + q_3\mathbf{b}). $$

Here, $k$ expresses the rotational velocity and $q_2/q_3$ the time. Due to the trigonometric expressions in the rotation matrix, it is not possible to write the back-projection in polynomial form though and for example to establish fundamental matrices.

### 3.1.5 Models for Central Catadioptric Cameras

Geyer and Daniilidis proposed a unifying model for single-mirror central catadioptric systems (cf. Section 2.3.1) [164, 165]. It is embodied by a two-step projection procedure, defined by a unit sphere and a pin-hole camera, as follows (cf. Figure 3.2). First, a 3D point is mapped onto the sphere, by intersecting the sphere with the line spanned by the point and the sphere’s center. Second, that intersection point is projected into a (virtual) pinhole camera. Geyer and Daniilidis showed that this model subsumes all single-mirror central catadioptric systems
<figure><img src="image_24.png" /><figcaption>Fig. 3.2 Unified central catadioptric model of Geyer and Daniilidis. Left: Projection from 3D to 2D. The line joining the 3D point and the center of the sphere, cuts the sphere. The intersection points are then projected by a virtual pinhole camera. Right: Back-projection. An image point is first back-projected relative to the virtual pinhole camera. The intersections of the camera ray with the sphere are determined. The final camera rays are the lines spanned by the intersections points and the center of the sphere. For clarity, the second final camera ray is shown as a dashed line.</figcaption></figure>

(see Section 2.3.1 for a list of these); the center of the sphere is the effective viewpoint of the modeled system. Note that the first step, projection onto the sphere, has two mathematical solutions; in practice, one usually knows which one to pick but when formulating multi-view geometry for instance, one may have to carry along both solutions. Also note that the virtual pinhole camera of the model is not equivalent to the camera looking at the actual mirror in the true catadioptric system; it is just an algebraic model.

Back-projection with this model can be formulated as follows. Without loss of generality, let us assume that the sphere is of radius 1 and centered in the origin. Let the projection matrix of the virtual pinhole camera be given as:

$$M_{3 \times 3}^{-1} (l_{3 \times 3} - \mathbf{t}).$$

Then, an image point is back-projected, relative to the virtual pinhole, as:

$$\mathbf{B}_v^f \sim \begin{pmatrix} \mathbf{t} \\ 1 \end{pmatrix}$$

$$\mathbf{B}_v^i \sim M\mathbf{q}.$$
The back-projection ray intersects the sphere in two points, written in homogeneous coordinates as:

$$ \left( \frac{(Mq)^T(Mq)t - (t^T Mq)Mq \pm \sqrt{D}Mq}{(Mq)^T(Mq)} \right), $$

with $D = (t^T Mq)^2 - (t^T t - 1)(Mq)(Mq)^T$. The final back-projection directions are then given by the first three coordinates of these two points (since the sphere center is the origin):

$$ B^i \sim (Mq)^T(Mq)t - (t^T Mq)Mq \pm \sqrt{D}Mq. $$

Due to the square root in this expression, it is in general not possible to directly obtain polynomial expressions.<fnref n="1" /> For the special case where the virtual pinhole lies on the sphere though, i.e., for $t^T t = 1$, the two directions are:

$$ B_1^i \sim t $$

$$ B_2^i \sim (Mq)^T(Mq)t - 2(t^T Mq)Mq. $$

This special case corresponds to para-catadioptric cameras. The first direction is independent of the image point $q$ and can thus be discarded; the second one can be written via a 3 × 6 back-projection matrix $B$ which depends on $M$ and $t$:

$$ B_2^i \sim B_{3 \times 6}^i \mathcal{L}^2(q). $$

Let us finally consider the usual special case where the principal point is supposed to be known and to be the origin and the optical axis of the camera to be aligned with the mirror axis. Further, without loss of generality, we assume that the camera’s optical center lies on the Z-axis. Then, $M \sim \text{diag}(1, 1, k)$ and $t^T = (0, 0, -1)$ and the back-projection becomes:

$$ B_2^i \sim \begin{pmatrix} 0 & 0 & 0 & -2k & 0 & 0 \\ 0 & 0 & 0 & 0 & -2k & 0 \\ 1 & 0 & 1 & 0 & 0 & -k^2 \end{pmatrix} \mathcal{L}^2(q). $$

<footnote n="1">¹However, Sturm and Barreto showed that polynomial back-projection expressions can be obtained by using degree-4 liftings of image coordinates [473].</footnote>
The first and third columns are identical and the second one only con-
tains zeroes. We can thus write the back-projection in a more compact
form as:

$$
\mathbf{B}_2^i \sim \begin{pmatrix} 0 & 2k & 0 & 0 \\ 0 & 0 & 2k & 0 \\ -1 & 0 & 0 & k^2 \end{pmatrix} \begin{pmatrix} q_1^2 + q_2^2 \\ q_1 q_3 \\ q_2 q_3 \\ q_3^2 \end{pmatrix}.
$$

Geyer and Daniilidis analyzed various properties of catadioptric sys-
tems using their model, among which the projection of points and lines,
the concepts of projective duality, epipolar geometry, etc. Barreto and
Araújo proposed a modified version of this model and used it to study
various geometric properties, especially those related to line images and
their application for calibration [25, 28, 29]. Mei and Rives proposed
to extend the model by adding classical terms for radial and tangential
distortion [339].

A similar model was previously proposed by Smith et al. [453]. Their
two-step model is defined by an arbitrary quadric surface, an effective
viewpoint and a direction for an orthographic projection. A 3D point is
first mapped onto the quadric surface by intersecting the line spanned
by the point and the effective viewpoint, with the quadric. The inter-
section point is then orthographically projected to give the final image
point. In practice, Smith et al. assumed that the quadric’s highest point,
in the direction of the orthogonal projection, lies above the effective
viewpoint; the orthographic projection of the effective viewpoint itself
is also considered as a distortion center. The final model used by Smith
et al. has four parameters and it can model non-radially symmetric
systems. It subsumes central para-catadioptric cameras as well as the
division model (cf. Section 3.1.8) but not hyper-catadioptric cameras.

Ying and Hu proposed another extension by replacing the sphere
with a general quadric of revolution [550]. One of the quadric’s foci is
chosen as effective viewpoint. The projection of a 3D point is modeled
in two steps analogous to Geyer and Daniilidis’ model: the line spanned
by the 3D point and the effective viewpoint is cut by the quadric. The
two intersection points are then projected into a virtual pinhole camera
whose optical center is located on the mirror’s revolution axis. This
gives the two theoretical images of a 3D point. Back-projection follows
the inverse procedure. Ying and Hu showed that this model, besides
central catadioptric cameras, also subsumes some fisheye models, such
as stereographic projection (cf. Section 3.1.7) and the division model.
A similar extension was proposed by Barreto, who showed that when
replacing the sphere with a paraboloid, one may represent the division
model [26, 27].

Bayro-Corrochano and López-Franco rewrote the model of Geyer
and Daniilidis in terms of conformal geometry and gave the correspond-
ing formulas for forward projection of points and lines and the back-
projection of image points [44]. Tolvanen et al. expressed the model of
Geyer and Daniilidis in terms of Clifford algebra [507]. Perwass and
Sommer proposed a model similar to Geyer and Daniilidis', expressed
in geometrical algebra terms; it subsumes the para-catadioptric and
division models [399].

**3.1.6 Non-central Catadioptric Cameras**

Consider any non-central catadioptric system. Even if everything is
known about the system, i.e., the shape of the mirror, the calibration
of the camera looking at it as well as the relative camera-mirror pose,
forward projection is a difficult problem. As for back-projection, it suf-
fices to first back-project an image point relative to the camera, to find
the (first) intersection point of the back-projected ray with the mir-
ror and to reflect the ray according to Snell's law. For some setups,
back-projection may be written analytically, e.g., for spherical or cone-
shaped mirrors.

Forward projection is much harder since it is not immediately given
at which point on the mirror the reflection happens. Again, for some
cases, analytical models can be formulated but in general, there is no
closed-form solution for this problem, so researchers have proposed to
solve it by non-linear optimization. One possibility is for example to
optimize the desired image point coordinates, where the cost function
is the distance between the back-projected and reflected ray, and the
3D point whose projection is sought. Another possibility is to optimize
the reflection point on the mirror, using for example a similar cost
function: the distance between the camera's optical center and the line joining that point and the 3D point, after reflecting that line in the mirror.

The special case of quadric-shaped mirrors was investigated by Gonçalves and Araújo who showed how to formulate the problem via three polynomials and to reduce it to an optimization in a single variable [179]. Vandeportaele also characterized the problem using polynomials and studied the number and nature of roots for the general as well as special cases of quadric-shaped mirrors [517]. Agrawal et al. contributed similar results for different types of quadric-shaped mirrors as well as for the case of cameras looking through refractive spheres [5].

### 3.1.7 Fisheye Models

**Classical models.** Several models were suggested for fisheye lenses (or rather, the opposite: fisheye lenses are constructed in order to satisfy such models) [48, 146, 226, 350, 536] — stereographic, equiangular,<fnref n="2" /> sine-law, and equi-solid angle projection:

$$r_d = k \tan(\theta/2)$$

$$r_d = k\theta$$

$$r_d = k \sin \theta$$

$$r_d = k \sin(\theta/2).$$

Their back-projection equations are given in Table 3.1 (where the coefficient $k$ is replaced by $f$ to homogenize notations). We observe that stereographic back-projection can be written using polynomials; it can be written via a 3 × 4 back-projection matrix operating on lifted image coordinates (here, we suppose that the principal point is known and located at the origin) and that it is (not surprisingly) equivalent to the back-projection of para-catadioptric cameras. The other three models do not have back-projection expressions in terms of polynomials.

In [146], Fleck compared these models with respect to several criteria, among which the maximum theoretical field of view and light

<footnote n="2">²This is more often called equidistant model, but in the context of back-projection, we prefer to use the word equiangular.</footnote>
fall-off. The stereographic model is preferred by Fleck, due to representing a good compromise between field of view, light fall-off, and a good approximate local preservation of Euclidean shapes. It seems though that most fisheyes are designed to approach the equiangular model, which is thus the model used most often for fisheye camera calibration.

Hall et al. for example used an equiangular model for a fisheye lens-based imaging system used for robot guidance [80, 382, 383].

An experimental evaluation of some of the above fisheye models for a particular camera was performed by Schneider et al. [439].

As a sidenote, let us remark that interestingly, some of the first scientific applications of fisheye lenses concerned the area of forest management (see http://en.wikipedia.org/wiki/Hemispherical_photography), where they were already used in the 1950s. Researchers in agricultural and forestry departments came up with calibration methods, independently from photogrammetrists. For example, Clark and Follin used the equiangular model for fisheyes [94]. Their application was forest management and panoramic images were used for assessing light interception and leaf areas in forests. Therefore, they were interested in images with equi-area properties, i.e., where the area of an image region is proportional to the solid angle of the corresponding field of view. In [94], they assessed how well different fisheye lenses approach this property.

**Other models.** Various other models were proposed for fisheye lenses, or lenses with a wide field of view in general, some of them aiming at subsuming several of the above classical models.

Herbert proposed the following alternative model for fisheye cameras [219]:

$$r_d = \theta + \sum_{i=1}^{5} a_i \sin \frac{i \pi \theta}{\theta_{\max}},$$

where $\theta_{\max}$ is the camera's field of view (assumed known).

Various fisheye lens designs and some performance characteristics were presented by Kumler and Bauer [294]. They used the following model:

$$r_d = k \sin(\beta\theta),$$
which encompasses the sine-law and equi-solid angle models shown above. Also, if $\beta$ is very small, this model approaches the equiangular one.

A similar general model was proposed by Gennery [161]:

$$r_d = \frac{\sin(\beta\theta)}{k \cos(\max(0, \beta\theta))}.$$

For $\beta = 1$ this gives the pinhole model, for $\beta = 0.5$ stereographic projection, $\beta = -0.5$ gives equi-solid angle projection, and $\beta = -1$ the sine-law projection. As $\beta$ approaches zero, this model approaches the equiangular one. Gennery suggested to couple this model (which subsumes ideal fisheye models) with a classical polynomial distortion model. Gennery's full model includes also the position of the effective optical center along the optical axis, as a function of $\theta$ or $r_d$; his full model is thus a non-central one (an axial model to be precise) and resembles in that respect that of Tardif et al. [491], see Section 3.3.2. Besides this effect, Gennery also included the orientation of the optical axis relative to the image plane, in order to model non-perpendicularity.

Yet another similar model was suggested by Bakstein and Pajdla [23], who used a combination of the stereographic and equi-solid angle models, depending on four parameters:

$$r_d = a \tan \frac{\theta}{b} + c \sin \frac{\theta}{d}.$$

Basu and Licardie proposed a distortion function called fisheye transform (FET) [41], inspired by the work of Schwartz on the geometry of the human visual system [441]:

$$r_d = s \log(1 + \lambda r_u)$$

$$r_u = \frac{e^{r_d/s} - 1}{\lambda}.$$

Although it is called fisheye transform, this model does not handle fields of view of 180° or higher.

Devernay and Faugeras proposed the FOV model (field-of-view model) [113]:

$$r_u = \frac{\tan(r_d \omega)}{2 \tan \frac{\omega}{2}}.$$
According to their description, the model is equivalent to the equiangular model. However, this seems to be the case only for particular values of the parameter $\omega$ (cf. Figure 3.4(c) in Section 3.5). If we include the focal length of the perspective projection in the model, we get the following expression for the back-projection angle $\theta$:

$$ \theta = \operatorname{atan}\frac{\tan(r_d\omega)}{2f\tan\frac{\omega}{2}} $$

For

$$ f = \frac{1}{2\tan\frac{1}{2\omega}} $$

the FOV model becomes equivalent to the equiangular model:

$$ \theta = r_d\omega. $$

Like Gennery (see above), Devernay and Faugeras suggested to complement their model (or other fisheye models) by additional terms, e.g., the classical polynomial radial distortion model, to calibrate cameras.

### 3.1.8 Polynomial and Rational Polynomial Models

Most of the models in the previous section were based on trigonometric functions, whereas polynomial expressions were used in the classical distortion models of Section 3.1.1. The two-plane models of Section 3.1.3 were also embodied by polynomials. In this section, we review several other polynomial-based models.

A first possibility is to model radial distortion using polynomials, but instead of relating the distorted and undistorted radial distances as in Equation (3.3), one considers the relation between distorted radial distances and incidence angles between camera rays and the optical axis:

$$ r_d = \sum_{i=1}^{n} c_i \theta^i. \qquad (3.7) $$

This model was for example used by Herbert [219], Xiong and Turkowksi [539], Kannala and Brandt [266], Kannada et al. [267], Ying et al. [553], and Stehle et al. [459]. Kannala and Brandt further added
non-polynomial terms to account for tangential distortions [266]. They also showed an approximation of the inverse (backward) model. Note that the model in Equation (3.7) subsumes the equiangular model, which corresponds to using only the first coefficient $c_1$.

The reciprocal is of course also possible, i.e., models of the form:

$$ \theta = \sum_{i=1}^{n} c_i r_d^i . $$

This was used for example by Inoue et al. where a 12-th degree polynomial was found to be suitable to calibrate a fisheye converter [245].

Beauchemin et al. proposed a polynomial model, but where polynomials relate polar image point coordinates before and after distortion, instead of Cartesian ones [46, 47]. Their model consists of two polynomials, one for the radial distortion function (distance from distortion center), one for the radial angle, capturing tangential distortions.

Hellmeier as well as Remy et al. used splines to model the 1D distortion function [218, 427].

**Rational polynomials for radial distortion.** Finally, several researchers have proposed models based on rational polynomials, as follows.

Fitzgibbon and independently, Bräuer-Burchardt and Voss, proposed the so-called division model, which has one distortion coefficient $c$ [145, 62]. Before this, that model was already used by Lenz and Tsai [305, 306] (see also the textbook [253, 254] by Jähne); however, they used it as forward model, i.e., in the below equations, one would swap the forward and back-projection equations. The division model as defined by Fitzgibbon and Bräuer-Burchardt and Voss, is given as:

$$ r_u = \frac{r_d}{1 - cr_d^2} \\ r_d = \frac{\sqrt{1 + 4cr_u^2} - 1}{2cr_u} . $$

A nice feature is that the model has a closed-form inverse.

The division model can be generalized by using general rational polynomials to relate $r_u$ and $r_d$, as proposed by Ma et al. [327]. Rational
polynomials were already used before the introduction of the division model, e.g., by Hellmeier [218].

Mičušik and Pajdla [346] and Mičušik [344] used a similar model for the relation between image distance $r_d$ and the angle between optical axis and camera rays, instead of the relation between image distances in distorted and undistorted images, as above. Their model has two parameters and is of the form:

$$ \theta = \frac{ar_d}{1 + br_d^2}. $$

It is in some sense a division model for the equiangular fisheye model, which it extends. Like for the equiangular model, expressing back-projection directions involves trigonometric functions; hence, Mičušik and Pajdla proposed to linearize their back-projection model using a Taylor development in order to perform an initial estimation of the model parameters during (self-)calibration.

**Rational polynomials for full back-projection models.** Instead of using rational functions to model the 1D distortion function, as above, one may use them for the actual back-projection mapping, as foreshadowed by the two-plane methods of Chen and Martins et al. [89, 90, 331] (see Section 3.1.3) and fully introduced by Claus and Fitzgibbon [98]. In general, one may write general mappings between distorted and undistorted image points:

$$ \mathcal{L}^a(\mathbf{q}_u) \sim \mathbf{M} \mathcal{L}^b(\mathbf{q}_d), $$

where the matrix $\mathbf{M}$ would contain the coefficients of homogeneous polynomials relating the lifted coordinates in the distorted and undistorted image planes (for full generality, we assume that different liftings may be possible, i.e., $a \neq b$). We speak of *rational* polynomial models here since “undoing” homogeneous point coordinates involves divisions of coordinates.

It is more common though to use this kind of model directly for the back-projection mapping of distorted image points to back-projection directions or rays:

$$ \mathbf{B}^i \sim \mathbf{B}^i \mathcal{L}^b(\mathbf{q}_d), \qquad (3.8) $$

$$ \mathbf{B}^l \sim \mathbf{B}^l \mathcal{L}^b(\mathbf{q}_d). \qquad (3.9) $$
In practice, Claus and Fitzgibbon used a bi-quadratic such model for central cameras, i.e., $b = 2$ and $B^i$ of size $3 \times 6$ [98]. It is easy to see that this encompasses the division model as well as the para-catadioptric and stereographic models. Note that in order to represent the classical radial distortion model, higher order liftings are required (cubic in the case of a single distortion coefficient, cf. Section 3.1.1).

Let us also note that the two-plane models of Chen and Martins et al. (see Section 3.1.3) are instances of rational polynomial back-projection models.

It may be worthwhile to study which instances of the models (Equations (3.8) and (3.9)) correspond for example to radially symmetric, central, perspective, or otherwise special camera models.

**Rational polynomial forward models.** Rational polynomial models are usually either applied for the back-projection function as proposed by Claus and Fitzgibbon [98], or for the 1D radial undis- tortion function as proposed by Ma et al. [327]. Hartley and Saxena also proposed a rational camera model, but for the forward projection function [210]: the image point coordinates are expressed via ratio- nal polynomial functions of the 3D point coordinates (Hartley and Saxena used cubic polynomials). In the general case, this corresponds to curved camera rays, i.e., the set of 3D points that get projected onto the same image point, form a curve. Hartley and Saxena showed that this model is a good fit for SAR cameras. It also subsumes “linear” cameras such as perspective and linear pushbroom ones, although fit- ting it to data obtained from these may lead to overfitting due to a strong overparameterization.

Essentially the same approach seems to be in routine use to model satellite imagery, cf. [147]. In Section 11.3 of that reference, the gen- eral term of “replacement sensor models” is employed to denote any camera model that replaces a rigorous, physics-based, model by empir- ical expressions. The most used ones are rational polynomial functions, much like in the above works.

Hall et al. proposed the RTcam (rational tensor camera model) [200]. It essentially proceeds by first transforming 3D points via bi- quadratic rational functions, followed by a perspective projection to a
2D image plane (the full RTcam model is defined for arbitrary dimensions of scene and image). Together, the two imaging steps still correspond to bi-quadratic rational functions. Similar to the GLC model (see Section 3.4.4), the development of the RTcam is mainly motivated by applications in stylized, or non-photorealistic, rendering.

### 3.1.9 One-Dimensional Radial Models

Thirthala and Pollefeys formally introduced so-called 1D radial camera models [500, 501]. They hinge on the existence of an optical axis and a center of distortion. Planes containing the optical axis are called radial planes and lines going through the distortion center, radial lines. The only main assumption about the projection function is that all 3D points in a radial plane are imaged to points on a same radial line. This effectively allows completely arbitrary radial distortions, and especially, does not require any radial symmetry of the distortion (the distortion "function" may be different for each radial plane-line pair) and it allows for non-central projections.

To make the model tractable, a second assumption is used though, namely that the projection can be modeled by a 2 × 4 projection matrix, mapping 3D points onto radial lines, represented by two homogeneous coordinates. In other words, it is assumed that there exists a projective relationship between the pencil of radial planes in 3D and that of radial lines in the image. Thirthala and Pollefeys showed that quadrifocal tensors can be formulated based on these projection matrices, and used for self-calibrating cameras. In general, no matching constraints exist for less than four images. However, for the case of pure rotational camera motion, a trifocal tensor exists and can also be used for self-calibration.

One-dimensional radial models have actually been used somewhat implicitly before, especially embodied within the radial alignment constraint (RAC) of Tsai [515], used also in other works [468, 106]. Tsai proposed a camera calibration approach for cameras with radial distortion, where one of the initial steps concerns the estimation of the calibration grid's pose relative to the camera. At this stage, it is assumed that the camera follows a 1D radial model and that the
distortion center is known. Compared to the general model of Thirthala and Pollefeys, Tsai further assumed that angles between radial lines and associated radial planes are identical (e.g., he had to know the aspect ratio). Using these assumptions, the radial plane associated with each extracted calibration point can be computed. One may then estimate the pose of the calibration grid using the constraint that with the correct pose, each point on the grid, must lie in the radial plane of the matched image point. It is clear that the pose cannot be fully computed: any translation along the optical axis will still satisfy the above constraint. Hence, 5 degrees of freedom are computed, from a minimum of 5 point matches.

A 1D radial model assumption was also used in the calibration approach of Hartley and Kang [209]. In the initial step, they used this to compute the distortion center from the image of a calibration grid; see more on their approach in Section 5.1.2.

### 3.1.10 Neural Networks

The projection or back-projection function has also been modeled and computed using neural networks, see for example [9, 7, 535, 577]. This is an interesting approach since it is inherently flexible and can be applied to calibrate various kinds of cameras with various types of distortion.

## 3.2 Local Camera Models

### 3.2.1 Two-Plane Model

The third version of the two-plane model proposed by Chen and Martins et al. [89, 90, 331] (cf. Section 3.1.3) is a local camera model. Here, for each of the two calibration images, the extracted matches define triangulations of the image and calibration grids respectively, cf. Figure 3.3. Back-projection is now modeled separately for each pair of matching triangles, by an affine transformation as in the first version of the two-plane model, computed from the matches associated with the three triangle vertices. To back-project an image point, the following is done. For each calibration image, the triangle into which the image point falls is determined. A point on the calibration grid is computed
<figure><img src="image_25.png" /><figcaption>Fig. 3.3 Illustration of a local two-plane model (cf. text). For the second image, the trian-
gulation is shown only partially.</figcaption></figure>

via the affine transformation associated with that triangle. This is done
separately for both calibration images/grids. The back-projected cam-
era ray is then computed by spanning the line between the points on
the two calibration grids.

**3.2.2 Warping-Based Models**

Very similar to the two-plane approach is the following idea. Whereas
the two-plane model interpolates 3D points on calibration grids, the
following methods perform an interpolation in the image domain. They
can all be applied with a single image of a grid, as for the correction of
non-perspective distortions.
The general idea is as follows: consider one calibration plane and an image thereof, as well as matches between the plane and the image. Consider any warping scheme that maps the acquired image into the “ideal” image of the calibration plane (let’s say, a texture map of the plane). Since the ideal image is distortion-free (it is a perspective image), this warping thus removes non-perspective distortions. If we determine the parameters of this warping using the acquired image, we may then apply it to any other image and thus remove its non-perspective distortions. This gives images which are distortion-corrected, i.e., images as acquired by some perspective camera. However, we don’t yet know the associated perspective camera. Still, when acquiring and then warping one or more images, the resulting images correspond to images acquired by the same perspective camera. This enables the application of any algorithm valid for perspective cameras, to these images. For example, if images of a calibration grid are acquired (be it planar or three-dimensional), any classical perspective calibration algorithm may be applied to calibrate the virtual camera. The same holds true for self-calibration. The complete calibration of the true camera is then the combination of the warp, with the computed perspective (self-)calibration.

Several methods following this general idea were proposed; they mostly differ in the choice of interpolation scheme which is required to warp entire images (the role played by the piecewise affine warp in the two-plane method). Without going into details, we cite the approaches of Green et al. [186], Goshtasby [182], Yokobori et al. [557], Brand et al. [59, 60], and Jackowski et al. [252]. Peuchot and Saint-André [401] and Bogner [54] also mentioned the feasibility of this type of approach. The method of Jackowski et al. corrects for color non-linearities in addition to geometrical distortions [252].

Sagawa et al. used the same idea and combined it with a structured light-type approach in order to get dense matches between the image and the calibration grid (a flat screen) [433], an approach used also for example by Grossberg and Nayar, Tardif et al. and Dunne et al. (see Section 3.3). From dense matches, the distortion-corrected image can be constructed pixel-by-pixel instead of requiring piecewise warps and an interpolation scheme.
It seems somewhat surprising that apparently, none of the above works went through the last obvious step to obtain a complete camera calibration. Namely, given these distortion-corrected, hence “virtual” perspective images, one may apply any calibration approach for pinhole cameras to calibrate the perspective part of the projection in addition to the non-perspective one. Such approaches were proposed previously, but only in combination with a plumb-line type distortion correction based on classical polynomial distortion models, see e.g., the work of Grosky and Tamburino [190]. Dunne et al. seem to be the first to combine the above generic distortion correction with a subsequent pinhole calibration [125]. They also used a structured-light type approach to get dense, per-pixel, matches between calibration grids and images; due to the per-pixel nature of the approach, we consider it in the discrete class of models and describe it in the next section.

**Comment.** All approaches in this section are valid for central cameras only since they are based on the idea of (bijective) warps of distorted images into perspective ones. The two-plane approach of Section 3.2.1 is in some sense a non-central counterpart.

### 3.2.3 Other Local Models

Qiu and Ma proposed an approach that is similar in spirit to the warp-based ones described above [410]. They used as input an image of a 3D calibration object, as opposed to a planar grid as in most other similar approaches. From 2D-to-3D matches, they then seemed to compute a best-fit perspective projection matrix for that image and projected the 3D calibration points using it. The result will constitute the ideal image onto which the distorted input image will be warped, for distortion correction. Input to the next step of calibration are the images of calibration points in the distorted image and in the ideal image, respectively. The goal is then to compute distortion functions $\Delta x$ and $\Delta y$ such that:

$$ \begin{pmatrix} x_d \\ y_d \end{pmatrix} = \begin{pmatrix} x_u + \Delta x(x_u, y_u) \\ y_u + \Delta y(x_u, y_u) \end{pmatrix}. $$
In [410], the distortion functions were estimated at a regular lattice of image pixels using non-parametric regression, and extended to the whole image by linear or nearest-neighbor interpolation.

As mentioned above, Qiu and Ma used a 3D calibration object and fitted a perspective projection matrix to the 2D-to-3D matches. The approach could easily be adopted to using a 2D calibration grid: then one may directly use the model of the grid as ideal image, such as in most other similar approaches or, one may compute the best-fit homography (instead of projection matrix) between the 2D-to-2D matches and use it to project the calibration points into an ideal image that is “closer” to the actual input image. The rest of the approach would be strictly identical.

Ragot proposed a similar idea, where 3D-to-2D matches obtained from an image of a calibration grid are interpolated to determine the forward respectively back-projection of generic 3D respectively image points [412]. The interpolation is done within triangular tessellations of 3D respectively image points.

Munjy proposed another very similar approach, based on the finite element method [359, 360]. He considered a regular triangular or rectangular tessellation of the image plane and modeled the distortion function with the help of one focal length per vertex (giving the angle between the line of sight associated with the vertex, and the optical axis). Focal lengths of all other image points can then be computed via bilinear interpolation of the focal lengths of the vertices of the triangle/rectangle in which image points lie. The parameters of the model (principal point, focal lengths of vertices) are computed by bundle adjustment (can be done when using a calibration grid but also in full self-calibration mode where 3D point coordinates are also optimized, using multiple images).

Lichti and Chapman extended this approach by coupling it with the classical radial and tangential distortion terms [315]. Back-projection is thus carried out by correcting for these classical distortions and then back-projecting the resulting points using the local focal length computed from the finite-element model. Lichti and Chapman also discussed correlations among the model's parameters and how to acquire images in order to minimize them. In [316], they further extended the
approach by including continuity constraints in order to overcome discontinuities of the distortion function due to the bilinear interpolation carried out over neighboring rectangles.

Schreier et al. [440] and Cornille [101] used a similar model, where B-splines are used to map positions in distorted images to those in undistorted ones. They used that approach to calibrate optical and scanning electron microscopes.

Stevenson and Fleck computed a 2D distortion function that is piecewise over a triangular tessellation of the image plane [463]. The basic idea is to apply an affine warp to each individual triangle such that the image becomes distortion-free. A notable difference with most other similar works is that the distortion-free image is generated with respect to stereographic projection instead of perspective, i.e., distortion-free does not mean as usual that straight lines remain straight (as in perspective projection) but rather that any sphere is imaged as a circle (a property of stereographic projection). Consequently, the input are images of spherical objects. In a first step, ellipses are fitted to their outlines in the images (which is an approximation, precluding the use of too large spheres). Then, the Delaunay triangulation of the centers of all ellipses is computed. Let us now consider one triangle and the associated three ellipses, centered in the triangle's vertices. We may compute an affine transformation that maps the three ellipses as closely as possible into circles. This transformation is naturally only defined up to a scale change ("circle-ness" does not depend on scale). The goal is then to find a new triangulation that is consistent with the original one, *modulo* these local affine transformations. This new triangulation would be nothing else than the stereographic image of the set of sphere centers. This description does not exactly correspond to the approach taken by Stevenson and Fleck, but transcribes its basic idea. Once the new triangulation computed, the input image can be warped into a stereographic image by piecewise warping it from the original into the new triangulation. The result is an uncalibrated stereographic image, i.e., whose "focal length" (overall scale) is arbitrary and whose distortion center is unknown.

The approach is somewhat similar in spirit to the warping-based ones of Section 3.2.2, in that it results in a piecewise warp of an image,
allowing to undistort it. Differences however are the use of stereographic
projection as reference instead of perspective and that the warp is
not computed on the basis of an ideal input image (the model of a
2D calibration grid), but using images of 3D primitives (here, spheres).

3.3 Discrete Camera Models

3.3.1 Ray-Based Models

As explained in the introduction to this section, most discrete camera
models can be seen as the limiting case of the above local models,
where interpolation is required in the immediate vicinity of pixels
instead of in larger image regions. The main difference between
the following discrete approaches and the local ones is the explicit
motivation to calibrate individual camera rays, as opposed to forward
or back-projection mappings.

The idea of calibrating individual camera rays for discrete image
locations was formulated by several researchers. Among the first were
probably Gremban et al. who proposed extensions of the two-plane
method in [189] and discussed the possibility of a ray-based calibra-
tion. However, they concluded that “A lookup table of calibration data
for each pixel would be prohibitively expensive”. This has to be under-
stood in the historical context; such a lookup table requires four coeffi-
cients per pixel, i.e., is of a size which does not pose memory problems
anymore nowadays. As alternative solution, Gremban et al. effectively
proposed global and local camera models, based on global and local
interpolation schemes to compute camera rays for any pixel: global
schemes are based on a function that is valid for the whole image plane
whereas local schemes look up the closest image points from calibration
data (in the case of back-projection) or the closest lines of sight (in the
case of forward projection) and interpolate accordingly.

Using structured light-type acquisition setups, such interpolations
can be avoided altogether, by matching image points directly to calibra-
tion points. This has been used by various researchers, e.g., Grossberg
and Nayar [191], Dunne et al. [123, 124, 125], Sagawa et al. [433] (cf.
Section 3.2.2), and Tardif et al. [491].
Southwell et al. proposed an idea for a structured-light type
ray-based camera calibration approach [455]. The idea was to insert
the camera in a cylinder painted on the inside in smoothly varying
color shades such that each “point” on the cylinder has a unique color.
Then, for each camera pixel, the scene point it sees can in theory be
uniquely determined. Southwell et al. proposed to calibrate a camera
by generating a look-up table where for each pixel, the direction of its
camera ray is stored, after computing it from the matched point on
the cylinder. They did not use this approach in practice though, due
to practical and signal processing problems.

Grossberg and Nayar proposed a generic imaging model consisting
of geometric as well as radiometric and optical parts [191]. The geomet-
ric part is identical in spirit to the two-plane model and its successors.
However, Grossberg and Nayar calibrated pixels individually, without
any interpolation, using structured light-type approaches allowing to
densely match images and calibration objets (e.g., flat screens). Fur-
ther, instead of computing lines of sight for individual pixels, they actu-
ally computed half-lines. This was mainly achieved by computing and
using the caustic of the imaging system which is used as the hull of
the imaging system's model, from which camera rays emanate in an
outward direction. Besides the geometric part of the imaging model, it
also contains radiometric and optical aspects: for example, each pixel is
associated with individual radiometric response and point spread func-
tions. Overall, a line of sight (or rather, a half-line), together with these
non-geometric properties, make what Grossberg and Nayar termed a
*raxel*, a sort of tiny camera associated with each individual pixel of the
imaging system.

Such a ray-based imaging model has been used recently by several
researchers to devise calibration and structure-from-motion algorithms,
for example by Ramalingam et al. [414, 415, 418, 477], Dunne et al.
[123, 124, 125], and Gonçalves and Araújo [180]. Ray-based calibration
approaches are described in Section 5.1.2.

Debaecker et al. proposed to model the field of view of each pixel
by a cone instead of a (half-)line in order to model the fact that pixels
gather a volume of light rays [108]. This may be seen as an alternative
to using a point spread function like Grossberg and Nayar.
**Comments.** In the above works, it is usually proposed to compute one camera ray per image pixel. Note however that this choice, while being intuitive, is not the only one for a discrete imaging model. Indeed, one may compute rays for any other discrete sample of image locations. With a regular super-pixel-based sampling for example, one ray for every *n*-th pixel would be computed; to compute rays for every pixel, an interpolation scheme would then be required, i.e., we are looking at local camera models, see the previous section. Even with a per-pixel sampling, interpolation is required to perform back-projection for every image *point*. Finally, nothing prevents in principle to apply a sub-pixel sampling for ray-based calibration. The appropriate choice of sampling density is a model selection question and depends on several factors, see the general discussion in Section 3.6.

### 3.3.2 Discrete Sampling of the Distortion Curve

Instead of computing camera rays at a discrete set of image locations, which is nothing else than gathering samples of a generic back-projection mapping, one may also gather such samples for any variant of the 1D (un) distortion function: $r_d(\theta), \theta(r_d), r_d(r_u)$, or $r_u(r_d)$.

Most calibration approaches based on this idea then fit some global distortion model to these samples. Examples are some collimator-based calibration approaches (cf. Section 5.1.4) or an early approach for plane-based calibration by Hallert (cf. Section 5.1.2) [201].

More recent such approaches are the ones by Stevenson and Fleck [462] (see Section 5.3.2), Hartley and Kang [209] and Tardif et al. [491] (see Section 5.1.2), as well as Ying and Hu [552] (see Section 5.1.1). In addition to modeling the usual 1D distortion function, Tardif et al. and Ying and Hu also considered a non-central radial distortion model: to each radial distance $r_d$, is associated an angle $\theta$ as well as an optical center on the optical axis. This is thus a radially symmetric axial model, similar to the one proposed by Gennery (cf. Section 3.1.7) [161].

### 3.3.3 "Isolated" Camera Rays

Grossmann et al. have considered what might be the most discrete camera model, consisting of a sparse set of pixels and associated camera
rays, with no requirement that rays have close-by neighboring rays [192, 194]. Their proposed self-calibration approach (see Section 5.3.3) only makes the assumption that the camera is central.

## 3.4 Models for the Distribution of Camera Rays

What a camera sees of a scene depends on the light rays it captures, not on its intrinsic geometry, i.e., which pixel exactly catches the photons of which light ray. In that respect, we may split the camera's intrinsic parameters into two sets: one that explains which light rays the camera captures (these will then be considered as camera rays) and one that explains the mapping between camera rays and image points. To make the distinction clearer, let us consider any two different central cameras, e.g., a perspective camera and a central catadioptric one. Being central, both cameras catch exactly the same set of light rays: all light rays that contain the optical center (here, we neglect of course the finite aperture of our cameras and their different fields of view). What differs is the mapping from image locations to rays and consequences thereof (e.g., non-perspective distortions and the spatial/angular resolution profile). On top of this come the extrinsic parameters, telling the global orientation and position of the camera, or in other words, of its set of camera rays.

In this section, we summarize several works which are primarily concerned with the first part of the “intrinsic geometry” of cameras, namely a description of which rays a camera captures: the locus, or spatial distribution, of camera rays. Several taxonomies and computational models have been proposed in the last years [390, 405, 445, 472, 559]. The main motivation of these works is to reason on the spatial distribution of camera rays; the mapping between rays and image points is of secondary importance although some of these works also propose such a mapping in order to obtain a full camera model.

### 3.4.1 From Central to Oblique Cameras

Cameras may be classified according to the existence of a geometric entity that is incident with all camera rays [390, 405, 472]. For a **central** camera, there exists a 3D point, the optical center, that is incident
with all rays. Any other camera is **non-central**. Among non-central
cameras, several subgroups exist, as follows.

In an **axial** camera, all rays cut some real 3D line. A special case is the **two-slit** camera, where there exist two real lines that cut all rays (for example, linear pushbroom panoramas, cf. Section 2.1.1). Cameras falling outside any of these categories are usually termed **fully non-central** or the like. A special case here are **oblique** cameras, for which any two camera rays are either identical or skew [390]. Oblique cameras are the most non-central cameras possible. A special case are **linear oblique** cameras, see Section 3.4.3, for which all rays are cut by an imaginary 3D line [389] (in fact, by two conjugated imaginary lines [405]).

Other cases are cameras, whose rays are incident with a circle, as is
the case with circular non-central mosaics (cf. Section 2.1.1). Another
example is catadioptric camera consisting of a cone-shaped mirror and
a camera located on the mirror axis. All reflected rays are incident with
a circle centered in the mirror axis. However, such systems are also of
the axial type, since all reflected rays also intersect the mirror axis.

One may define many other categories of cameras, by choosing other
geometric primitives to which all rays are incident.

**3.4.2 Ray Distributions Allowing for Standard**
**Stereo Geometry**

Pajdla [390] and Seitz and Kim [445] were interested in studying which
sets of camera rays admit stereo imaging conditions. What is meant by
this? To explain this, we look ahead to Section 4 on epipolar geome-
try. The basic question of epipolar geometry and stereovision is, given
a point in one image, which are the potentially matching points in
the other image. The answer to this is: all points whose camera rays
intersect the ray of the first point. These define an epipolar curve in
the second image. Epipolar curves in the first image are defined anal-
ogously. Essentially, two cameras are in standard stereo geometry if
epipolar curves in the two images are in a one-to-one relationship: the
camera ray of any point on an epipolar curve in one image cuts all
camera rays of points on the related epipolar curve in the other image
(for a more precise definition, see [445]). An auxiliary requirement is that the set of epipolar curves in each image is one dimensional.

Not surprisingly perhaps, the set of rays of two cameras in standard stereo geometry, lie on a set of doubly ruled quadrics<fnref n="3" /> [390, 445]. The two individual cameras then correspond to the two rulings of these quadrics. Note that this comprises several types of non-central cameras, such as circular non-central mosaics and others [445] as well as linear oblique cameras [390].

This result concerns the locus of camera rays and is independent of the mapping from rays to image points. What it implies is that for any two cameras in standard stereo geometry, the acquired images may be warped into images where epipolar curves are horizontal lines and matching epipolar lines have the same vertical coordinate. In other words, it is possible to rectify images and then apply any standard stereo algorithm.

### 3.4.3 Linear Oblique Cameras

Pajdla studied the case of linear oblique cameras, already mentioned above [390]. A defining characteristic of these is the existence of a linear transformation that maps each 3D point to the unique camera ray containing that point. Pajdla showed that all linear oblique cameras are projectively equivalent to the one defined as follows: the camera ray containing a 3D point $(X, Y, Z, W)^T$ is spanned by that point and the point with coordinates $(Y, -X, W, -Z)^T$.

This defines the set of camera rays in a linear oblique camera. In order to get a full camera model, including the mapping between rays and image points, Pajdla proposed to use a plane as reference. Let $\mathbf{q}$ be the homogeneous 2D coordinates of a point in the reference plane (the image plane if one likes). Each point inside the plane is mapped to a 3D point by some $4 \times 3$ pose matrix $M$:

$$ \mathbf{Q} \sim \begin{pmatrix} X \\ Y \\ Z \\ W \end{pmatrix} \sim \mathbf{Mq}. $$

<footnote n="3">³For central cameras, this is the pencil of epipolar planes.</footnote>
The camera ray is then spanned by **Q** and the second 3D point, defined above. It is easy to prove that the camera ray can be written as:

$$ \mathbf{B}^l \sim \mathbf{B}^l \mathcal{L}^2(\mathbf{q}), $$

with $\mathbf{B}^l$ a 6 × 6 back-projection matrix depending on $\mathcal{M}$, operating on lifted image point coordinates. The back-projection matrix is always of rank at most 4.

For example, if we choose the following pose matrix:

$$ \mathcal{M} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}, $$

which corresponds to choosing the $XY$-plane as reference plane, then one obtains the following back-projection matrix:

$$ \mathbf{B}^l \sim \begin{pmatrix} 0 & 0 & 0 & 0 & -1 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & -1 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 & 0 & 0 \\ -1 & 0 & -1 & 0 & 0 & 0 \end{pmatrix}. \qquad (3.10) $$

We may see that, similarly to what was shown for the classical radial distortion model (cf. Section 3.1.1) or para-catadioptric cameras (cf. Section 3.1.5), the back-projection matrix may be compacted. The second column only contains zeroes and the first and third ones are identical. Hence, back-projection for linear oblique cameras may be written as:

$$ \mathbf{B}^l \sim \begin{pmatrix} 0 & 0 & -1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & -1 \\ 0 & 0 & 1 & 0 \\ 0 & -1 & 0 & 0 \\ -1 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} q_1^2 + q_2^2 \\ q_1 q_3 \\ q_2 q_3 \\ q_3^2 \end{pmatrix}. $$
**Observation.** Linear oblique cameras are actually subsumed by the linear two-plane model of Chen and Martins et al. (cf. Section 3.1.3): it is easy to prove that the back-projection matrix of Equation (3.10) can be obtained from the one of the two-plane model (cf. Equation (3.6)) by an appropriate choice of affine transformations $A_1$ and $A_2$.

### 3.4.4 General Linear Camera Model (GLC)

Yu and McMillan proposed the so-called General Linear Camera model (GLC) [559, 558], which parameterizes the set of camera rays by affine transformations of three basis rays. Concretely, they adopted a two-plane parameterization of rays (see also a paper by Gu et al. [196] on this parameterization) as follows. The three basis rays are given by points:

$$A_i = \begin{pmatrix} u_i \\ v_i \\ 0 \\ 1 \end{pmatrix} \qquad B_i = \begin{pmatrix} s_i \\ t_i \\ 1 \\ 1 \end{pmatrix},$$

where we assume that the two planes are $Z = 0$ and $Z = 1$. All other rays are then given by two parameters $\alpha$ and $\beta$, defining points in the two planes via affine transformations of the basis points parameterized by two scalars $\alpha$ and $\beta$:

$$A = \begin{pmatrix} \alpha u_1 + \beta u_2 + (1 - \alpha - \beta) u_3 \\ \alpha v_1 + \beta v_2 + (1 - \alpha - \beta) v_3 \\ 0 \\ 1 \end{pmatrix} \qquad B = \begin{pmatrix} \alpha s_1 + \beta s_2 + (1 - \alpha - \beta) s_3 \\ \alpha t_1 + \beta t_2 + (1 - \alpha - \beta) t_3 \\ 1 \\ 1 \end{pmatrix}.$$

Yu and McMillan showed that this model allows to generate the set of camera rays of several imaging geometries, such as pinhole and orthographic cameras but also non-central cameras such as some two-slit ones. It also allows to generate less classical imaging geometries, called by Yu and McMillan the pencil camera, the twisted orthographic camera, and the bilinear camera.

Like for linear oblique cameras, the initial main motivation of the GLC model is to propose a model for the locus of camera rays, not necessarily a full camera model. For linear oblique cameras, a way to
get such a full model, mapping image points to rays, was shown above. As for the GLC, a natural choice is to consider the two parameters $\alpha$ and $\beta$ as 2D image coordinates. Before going further, let us replace the image coordinates by homogeneous ones such that:

$$ \mathbf{q} \sim \begin{pmatrix} \alpha \\ \beta \\ 1 \end{pmatrix}. $$

Then:

$$ \mathbf{A} \sim \begin{pmatrix} (u_1 - u_3)q_1 + (u_2 - u_3)q_2 + u_3q_3 \\ (v_1 - v_3)q_1 + (v_2 - v_3)q_2 + v_3q_3 \\ 0 \\ q_3 \end{pmatrix} \quad \mathbf{B} \sim \begin{pmatrix} (s_1 - s_3)q_1 + (s_2 - s_3)q_2 + s_3q_3 \\ (t_1 - t_3)q_1 + (t_2 - t_3)q_2 + t_3q_3 \\ q_3 \\ q_3 \end{pmatrix}. $$

Finally, the camera ray, spanned by $\mathbf{A}$ and $\mathbf{B}$, can be written via the following 6 × 6 back-projection matrix, operating on lifted image coordinates $\mathcal{L}^2(\mathbf{q})$:

$$ \mathbf{B}^l = \begin{pmatrix} 0 & 0 & 0 & u_1 - u_3 - s_1 + s_3 & u_2 - u_3 - s_2 + s_3 & u_3 - s_3 \\ 0 & 0 & 0 & v_1 - v_3 - t_1 + t_3 & v_2 - v_3 - t_2 + t_3 & v_3 - t_3 \\ 0 & 0 & 0 & 0 & 0 & -1 \\ 0 & 0 & 0 & v_1 - v_3 & v_2 - v_3 & v_3 \\ 0 & 0 & 0 & u_3 - u_1 & u_3 - u_2 & -u_3 \\ b_1 & b_2 & b_3 & b_4 & b_5 & b_6 \end{pmatrix}. \quad (3.11) $$

For the sake of conciseness, the coefficients $b_i$ are not shown here; they can be easily computed if necessary.

**Observations.** We see that the rays in a GLC can be modeled by a quadratic rational camera model, cf. Section 3.1.8. Furthermore, by comparing the back-projection matrices of the GLC and the linear version of the two-plane model (Equation (3.6) in Section 3.1.3), we observe that they have exactly the same form. It can indeed be proven that the two models are equivalent. This is maybe not surprising since both models are based on affine transformations in two reference planes, although they are parameterized in different ways. Whereas the motivation for developing the two-plane model of Chen and Martins et al.
was to interpolate between calibration data, Yu and McMillan were
aiming at developing novel camera geometries. Compared to works on
the two-plane model, Yu and McMillan carried out an in-depth study
of the modeled camera geometries.

**Other considerations on GLCs.** In [560], Yu and McMillan gave a projection equation for a special case of GLCs (concretely, a transformation that maps each 3D point to the ray it lies on in the GLC). They also showed how to relate two GLCs which differ in the choice of image plane. Whereas for perspective cameras, corresponding points in two different image planes (but with the same optical center) are related by a homography, this relationship is a quartic rational function for the considered special case of GLCs. In the general case of GLCs, this relation will be even more complicated. However, if parallel reference planes are used, then changing the distance between them does not matter: for any GLC with a particular distance between the planes, there is an equivalent GLC defined with planes separated by another distance.

The GLC model is defined by three basis rays. An extension to
multiple basis rays was proposed and analyzed for up to six rays, by
Popescu et al. [406].

**3.4.5 Linear Families of Lines**

Ponce proposed to model the set of camera rays by two-parameter lin-
ear families of straight lines, either degenerate reguli (rank-3 families)
and non-degenerate linear congruences (rank-4 families) [405]. This
framework subsumes the GLCs of Yu and McMillan [558, 559] and
the linear oblique cameras of Pajdla [390]. Ponce gives formulas for
projection, back-projection, and epipolar and multi-view geometry, for-
mulated such that the two parameters of the considered linear families
of camera rays are considered as image point coordinates. The back-
projection matrix is of size 4 × 6 for two-slit cameras and linear oblique
ones, and 5 × 6 for the “pencil cameras” of Yu and McMillan [559].
All models subsumed by Ponce’s formulation can be expressed by
6 × 6 back-projection matrices, leading to 6 × 6 fundamental matrices
(cf. Section 4). Batog et al. generalized the theoretical framework
of [405] by characterizing all admissible maps that allow to model linear
cameras, i.e., cameras whose rays form a linear family of lines [43].

**3.4.6 Caustics**

Roughly speaking, the caustic of an imaging system is the surface
enveloping the set of camera rays, i.e., it is tangent to all rays. Caus-
tics are an excellent tool to analyze properties of imaging devices, as
nicely advocated by Swaminathan et al. [484]. A caustic provides one
possible description of a viewpoint locus of a camera, i.e., a point set
such that any camera ray goes through at least one of the points in
the set (ideally through exactly one). For a central camera, the caustic
degenerates into a point and for axial ones, into a line segment. The
caustic allows to study “how central” a camera is, i.e., how much its
extent deviates from a single point. Swaminathan et al. showed how
to compute the extent and shape of the caustic for axial catadiop-
tric systems based on conic-shaped mirrors and such that the camera’s
optical center lies on the mirror axis [484]. They showed how to use
this analysis to determine a best approximation of a single viewpoint,
useful for example for approximate perspective view generation. They
also analyzed how resolution varies across the image area, and recipro-
cally, proposed an approach to compute a mirror profile that satisfies
a desired resolution profile. Finally, the parametric form of the caustic
for a class of catadioptric sensors is used in a calibration approach,
described in Section 5.3.2.

Usually, to each image point corresponds exactly one point on the
caustic. This corresponds to the “viewpoint” associated with that image
point. To compute the entire camera ray, one still needs its direction.
For axial systems, it can be easily computed since the camera ray must
pass through the axis and also be tangent to the caustic. For fully non-
central systems, the full determination of camera rays requires a second
map (of directions), besides a map from image points to the caustic.

**3.5 Overview of Some Models**

In Table 3.1, some of the models seen in this section are summarized.
Note that in the table, we do not always use the same notation for these
<table><caption>Table 3.1. Back-projection and forward projection equations for some camera models.</caption><thead><tr><th>Name of model</th><th>B<sup>1</sup></th><th>θ(r<sub>d</sub>)</th><th>r<sub>u</sub>(r<sub>d</sub>)</th><th>P<sup>1</sup></th><th>r<sub>d</sub>(θ)</th><th>r<sub>d</sub>(r<sub>u</sub>)</th></tr></thead><tbody><tr><td>Perspective</td><td>(<table><tr><td>1</td><td>0</td><td>0</td></tr><tr><td>0</td><td>1</td><td>0</td></tr><tr><td>0</td><td>0</td><td>f</td></tr></table>)<table><tr><td>k</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>0</td><td>k</td><td>0</td><td>1</td><td>0</td></tr><tr><td>0</td><td>0</td><td>0</td><td>f</td><td></td></tr></table></td><td>atan <sup>r<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>r<sub>d</sub></td><td>(<table><tr><td>f</td><td>0</td><td>0</td></tr><tr><td>0</td><td>f</td><td>0</td></tr><tr><td>0</td><td>0</td><td>1</td></tr></table>)</td><td>f tan θ</td><td>r<sub>u</sub></td></tr><tr><td>Classical radial dist.</td><td>(<table><tr><td>k</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>0</td><td>k</td><td>0</td><td>1</td><td>0</td></tr><tr><td>0</td><td>0</td><td>0</td><td>f</td><td></td></tr></table>)<table><tr><td>q<sub>1</sub><sup>3</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td></tr><tr><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td></tr><tr><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td></tr></table></td><td>atan <sup>r<sub>d</sub>(1+kr<sub>d</sub><sup>2</sup>)</sup>&frasl;<sub>f</sub></td><td>r<sub>d</sub>(1 + kr<sub>d</sub><sup>2</sup>)</td><td></td><td>Requires solution of cubic equation</td><td></td></tr><tr><td>Para-catadioptric</td><td>(<table><tr><td>0</td><td>2k</td><td>0</td><td>0</td></tr><tr><td>0</td><td>0</td><td>2k</td><td>0</td></tr><tr><td>-1</td><td>0</td><td>0</td><td>k<sup>2</sup></td></tr></table>)<table><tr><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td></tr><tr><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td></tr><tr><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td></tr></table></td><td>atan <sup>2kr<sub>d</sub></sup>&frasl;<sub>k<sup>2</sup> - r<sub>d</sub><sup>2</sup></sub></td><td>N/A</td><td></td><td>Requires solution of quadratic equation</td><td></td></tr><tr><td>Division [62, 145]</td><td>(<table><tr><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>k f</td><td>0</td><td>0</td><td>f</td></tr></table>)<table><tr><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td></tr><tr><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td></tr><tr><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td></tr></table></td><td>atan <sup>r<sub>d</sub></sup>&frasl;<sub>f(1+kr<sub>d</sub><sup>2</sup>)</sub></td><td><sup>r<sub>d</sub></sup>&frasl;<sub>1+kr<sub>d</sub><sup>2</sup></sub></td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>Mičušik, Pajdla [344]</td><td>N/A</td><td>atan <sup>2kr<sub>d</sub></sup>&frasl;<sub>k<sup>2</sup> - r<sub>d</sub><sup>2</sup></sub></td><td>N/A</td><td></td><td>N/A</td><td>N/A</td></tr><tr><td>Stereographic</td><td>(<table><tr><td>0</td><td>2f</td><td>0</td><td>0</td></tr><tr><td>0</td><td>0</td><td>2f</td><td>0</td></tr><tr><td>-1</td><td>0</td><td>0</td><td>f<sup>2</sup></td></tr></table>)<table><tr><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td><td>q<sub>1</sub><sup>2</sup></td></tr><tr><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td><td>q<sub>2</sub><sup>2</sup></td></tr><tr><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td><td>q<sub>3</sub><sup>2</sup></td></tr></table></td><td>atan <sup>2kr<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>N/A</td><td></td><td>N/A</td><td>N/A</td></tr><tr><td>Equidistant/-angular</td><td>N/A</td><td><sup>r<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>N/A</td><td>N/A</td><td>fθ</td><td>N/A</td></tr><tr><td>Sine-law</td><td>N/A</td><td>asin <sup>r<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>N/A</td><td>N/A</td><td>f sin θ</td><td>N/A</td></tr><tr><td>Equi-solid angle</td><td>N/A</td><td>2asin <sup>r<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>N/A</td><td>N/A</td><td>f sin(θ/2)</td><td>N/A</td></tr><tr><td>Fisheye transform [41]</td><td>N/A</td><td>atan <sup>cr<sub>d</sub></sup>&frasl;<sub>f</sub></td><td>N/A</td><td>N/A</td><td>k log(1 + f tan θ)</td><td>N/A</td></tr><tr><td>Herbert [219]</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>θ + Σ<sup>5</sup><sub>i=1</sub> a<sub>i</sub> sin <sup>iπθ</sup>&frasl;<sub>g<sub>max</sub></sub></td><td>N/A</td></tr><tr><td>Kumler and Bauer [294]</td><td>N/A</td><td>asin <sup>r<sub>d</sub></sup>&frasl;<sub>k</sub></td><td>N/A</td><td>N/A</td><td>f sin(kθ)</td><td>N/A</td></tr><tr><td>FOV model [113]</td><td>N/A</td><td>atan <sup>tan(kr<sub>d</sub>)</sup>&frasl;<sub>2f tan <sup>k</sup>&frasl;<sub>2</sub></sub></td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>Bakstein and Pajdla [23]</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>k<sub>1</sub> tan <sup>θ</sup>&frasl;<sub>k<sub>2</sub> sin(θ/2)</sub> + k<sub>3</sub> sin <sup>θ</sup>&frasl;<sub>k<sub>4</sub></sub></td><td>N/A</td></tr><tr><td>Gennery [161]</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td><sup>k cos(max(θ,3θ))</sup>&frasl;<sub>k cos(max(θ,3θ))</sub></td><td>N/A</td></tr></tbody></table>
models as in the previous sections; rather, we try to homogenize notations by naming coefficients of camera models *k* or *k*₁, *k*₂, .... Further, parameters akin to the focal length are denoted by *f*.

The shown models are all radial distortion models; their equations are given under the assumption that the distortion center is identical to the principal point and located at the origin. How to relax these assumptions is outlined in the introduction to this section.

The table contains six different ways of modeling the imaging transformation. Cells annotated by “N/A” correspond to cases where there is no closed-form expression or, if there is no expression of back-projection in terms of polynomials (second column).

Sample graphs of the function $\theta(r_d)$ for several camera models are given in Figure 3.4.

## 3.6 So Many Models ...

This section reviewed some camera models existing in the computer vision literature. Although it is certainly not exhaustive, the list of models is rather long. One may ask, which model one should choose or recommend. For example, recurring questions are, should one use a non-central camera model for a slightly misaligned catadioptric system or stick with a central model, or should one model fisheye lenses with a central or non-central model.

The general answer to these questions is: it depends...

The choice of camera model depends of course on how appropriate a model is in general for the type of camera used. For example, one would obviously never use a pinhole model for fisheye cameras. More interesting parts of the question are: how to determine the appropriate order of a polynomial model, how to select among altogether different types of models, how to decide if it is worth using a non-central model rather than a central one, etc.

The choice of camera model or of the number of coefficients of a model may depend on several factors:

* The amount of noise in the calibration data.

* The amount of calibration data: how many images are used and how many calibration points (or other primitives used for calibration)?
<figure><img src="image_26.png" /><figcaption>Fig. 3.4 Graphs of the function $\theta(r_d)$ for some camera models. In the first two rows, distortion model coefficients $k$ are chosen such that for $r_d = 1$, we have $\theta = 90^\circ$.</figcaption></figure>

* The quality of the spatial distribution of calibration data: how well do the data cover the field of view (both across the field of view and in depth), how good is the spatial distribution of the set of camera positions, how different are the camera orientations, etc.?

* The “true” camera model: is it central or not, how pronounced are non-perspective distortions, etc.?
The choice depends on the combination of these and possibly other factors. For example, Zhang showed that the self-calibration of the classical radial distortion model, even with a single coefficient, from matches between images, is only stable if the actual distortion is pronounced **and** if extracted image points are accurate [566].

Let us consider extreme cases. Without noise in the data, the true model can be estimated without error (assuming we are not in a degenerate situation for the position of the calibration data, such as the twisted cubic for pinhole camera calibration [67]). Even with noise, the availability of infinitely many calibration data may allow a perfect calibration of the true camera model.

In practice, there is noise and the number of data is finite and we must consider the interplay of the above factors. To stress that the choice of camera model depends on all of these factors, we may note that even for one and the same camera, different models may be chosen for different scenarios. For example, fisheye lenses are probably not perfectly central, i.e., not all camera rays go through a same 3D point. However, the viewpoint locus is at most as large as the aperture of the lens, i.e., a few centimeters, and in practice, much smaller. Hence, if the fisheye lens is used for an application where the scene of interest is meters away from the camera, then an approximation by a central imaging model should be sufficient (unless we use many images and extremely accurate image points for calibration). However, if the scene in some hypothetical application is much less than a meter away, a non-central model may make sense.

So far, the discussion was rather qualitative. To actually implement a method for choosing among camera models requires quantitative goodness measures.

Clearly, our question is one of model selection, for which a large body of literature exists. The main principle of model selection is to find a good compromise between the goodness of fit to the data and the complexity of a model; roughly speaking, a good model should explain the data well but not have too many parameters. The reason for this is that in general, a model with more parameters has a better chance to
<figure><img src="image_27.png" /><figcaption>Fig. 3.5 Illustration of overfitting. Seven points are sampled from a quadratic polynomial (red curve) and perturbed by random noise. Then, a quadratic and a degree-6 polynomial are fitted to the points (green and orange curve respectively). The degree-6 polynomial fits the data perfectly as expected but is an extreme overfit as can be seen from the two random trials above, whereas the degree-2 polynomial fits stably and accurately.</figcaption></figure>

well fit the data.<fnref n="4" /> If more parameters than necessary are allocated then this constitutes an overparameterization and it may lead to overfitting: in addition to explaining the phenomenon underlying the observed data (here, an imaging process) the model parameters will tend to explain the noise in the data.<fnref n="5" /> If we tip the balance among the above factors, e.g., too few or too badly distributed data compared to the amount of noise, then the model will no longer be a trustworthy explanation of the phenomenon. The usual sample illustration of overfitting concerns the problem of fitting a polynomial to data points, cf. Figure 3.5.

The problem of overfitting in camera calibration is real but can be rather safely reduced by following a few guidelines. If an application allows it, one should always acquire more than the minimum required

<footnote n="4">⁴The fit gets always better if one extends the number of parameters of the *same* model, e.g., by using additional coefficients for modeling radial distortion or a finer tessellation for local camera models. As for *different* models, this needs not always be the case.</footnote>

<footnote n="5">⁵Another manifestation of this are correlations between the estimated parameters.</footnote>
amount of data (in terms of number of images, number of calibration
points, etc.). The image area should be completely covered by calibra-
tion points: distortions are usually strongest, i.e., most observable at
the image border and even in the absence of distortions, a better cov-
erage of the image area gives a better leverage on calibration and other
structure-from-motion tasks (e.g., motion estimation gets stabler the
wider the field of view that is effectively used). By design, calibration
grids usually have well-distributed calibration points. One of the most
important aspects is that images should be taken from as different as
possible viewpoints and, crucially, with as different camera orientations
as possible. The latter aspect includes rotations about the optical axis
(required if an aspect ratio is to be computed) but more importantly,
one should acquire images with different directions of the optical axis
(or, orientations of the image plane). Naturally, one should also spend
an effort to get the most accurate image point extraction.

These are just a few general guidelines. More intricate ones which
may be important for high accuracy calibration are that cameras should
best be calibrated in the same conditions as in the final application,
e.g., same temperature, atmospheric conditions, even in the same ori-
entation (tilting a camera may already affect its intrinsic parameters),
etc. More on these issues can be found in photogrammetric literature,
e.g., [336, 451].

Coming back to the model selection problem: model selection meth-
ods for camera calibration were proposed for example by El-Melegy
and Farag [127], Ma et al. [327], and Orekhov et al. [387]. See also
the good discussion by Derrien and Konolidge in [111], who also evalu-
ated the geometric error committed when approximating a non-central
camera by a central model. An alternative to usual model selection
methods is to evaluate the quality of different camera models (and of
the estimated coefficients thereof) using other data than that used for
calibration. One possibility is cross-validation, i.e., evaluate calibration
results using calibration images not used for calibration. Another one is
to judge the quality of calibration within an actual intended application
of the calibration: calibration is rarely a goal in its own but is carried
out in order to realize some application, such as 3D modeling or ego-
motion estimation. If ground truth data for these or other applications
are available, then it is obviously a good idea to measure the quality
of a calibration result via the quality of the subsequent result in the
considered application.

A final comment is that while it is not possible to recommend this or
that camera model in general, it seems a good idea to enhance “ideal”
models for cameras with large fields of view (e.g., the equiangular one),
with classical radial or tangential models, as proposed for example by
Devernay and Faugeras [113], Kannala and Brandt [266], Gennery [161],
and Mei and Rives [339].
Epipolar geometry is central to stereovision. On the one hand, knowing it enables to strongly constrain the problem of matching two images. On the other hand, it can be estimated from matches and then allows for motion estimation, self-calibration, and triangulation of 3D points or other geometric primitives. Epipolar geometry has been well studied for perspective cameras, since the nineteenth century (see, e.g., the excellent paper [144] by Finsterwalder on geometric foundations of photogrammetry, as well as modern treatments [137, 213]).

As for non-perspective cameras, much has been done in the last 20 years and this section aims at giving a concise overview. It will be seen in the following that epipolar geometry is a very powerful tool for the self-calibration of non-perspective distortions; this is based on a strong link to the plumb-line calibration approach (cf. Section 5.2.1), explained in Section 4.3.

We start the section by considering the case of calibrated cameras, where the epipolar geometry can be represented by an essential matrix. Except the question if cameras are central, non-central, axial, etc., the essential matrix is independent of the camera type. In Section 4.2, we consider the epipolar geometry of uncalibrated cameras
and particularly, fundamental matrices for non-perspective cameras. Finally, since epipolar curves are images of straight lines (camera rays of the other camera), we consider the computation of line images in Section 4.3 and, most importantly, the above mentioned strong links between self-calibration and plumb-line calibration of non-perspective cameras.

## 4.1 The Calibrated Case

The epipolar geometry for a pair of calibrated *central* cameras can be represented by the essential matrix E [324] and the associated epipolar constraint on back-projected image points:

$$ (\mathbf{B}_2^i)^T ([\mathbf{t}] \times \mathbf{R}) \mathbf{B}_1^i = 0, \qquad (4.1) $$

where $\mathbf{t}$ and $\mathbf{R}$ are the relative translation and rotation between the two views. Note that since we directly work with the directions of camera rays, the above formulation is independent of the type of central camera and also works when considering two cameras of different types.

The essential matrix for *non-central* cameras was introduced by Pless [402]. For non-central cameras, we must consider the complete camera rays instead of only their directions as in Equation (4.1). Let $\mathbf{L}_1$ and $\mathbf{L}_2$ be the camera rays associated with matching image points in a pair of views. They must intersect one another (in the 3D point whose images are the two matching image points). Putting together Equations (1.2) and (1.3) of Section 1.2, this leads to the essential matrix and epipolar constraint for non-central calibrated cameras:

$$ \mathbf{L}_2^T \begin{pmatrix} -[\mathbf{t}] \times \mathbf{R} & \mathbf{R} \\ \mathbf{R} & 0 \end{pmatrix} \mathbf{L}_1 = 0. \qquad (4.2) $$

The essential matrix acts on Plücker coordinates of camera rays and is thus of size $6 \times 6$.

In [402], Pless also gave the epipolar constraint for continuous camera motion. We refer to Section 6.2 for notes on the estimation of the central and non-central essential matrices and subsequent camera motion estimation.
Sturm specialized Pless' epipolar constraint to calibrated axial, two-slit, and central cameras, for all cases with finite/infinite slits respectively optical centers [472]. For axial and two-slit cameras, the essential matrices are of size 5 × 5 and 4 × 4, respectively. It was also shown how to establish trifocal matching tensors for these camera models.

Gasparini and Sturm showed how to derive matching tensors for line images as opposed to images of points, for central, two-slit, axial, and fully non-central calibrated cameras [158].

## 4.2 The Uncalibrated Case

### 4.2.1 Generalities

The epipolar geometry of uncalibrated perspective cameras can be represented by the fundamental matrix [137, 213]. How about other central cameras or the case of non-central ones? The answer is immediate if back-projection can be represented as a matrix-vector product (or, by polynomials or rational polynomials, cf. Section 3.1.8).

**Central cameras.** Let $B^i$ be the back-projection matrix of a central camera, of size $3 \times n$ and operating on lifted image coordinates $\bar{\mathcal{L}}^a(\mathbf{q})$, where $a$ and $n$ depend on the camera model (e.g., $n = 5$ and $a = 3$ for the one-coefficient classical radial distortion model, cf. Section 3.1.1). By plugging $B_1^i \sim B^i \bar{\mathcal{L}}^a(\mathbf{q}_1)$ and $B_2^i \sim B^i \bar{\mathcal{L}}^a(\mathbf{q}_2)$ into the epipolar constraint (4.1), we directly get the fundamental matrix:

$$ (\bar{\mathcal{L}}^a(\mathbf{q}_2))^T \underbrace{(B^i)^T [\mathbf{t}] \times R B^i}_{\mathbf{F}} \bar{\mathcal{L}}^a(\mathbf{q}_1) = 0. $$

It is of size $n \times n$.

It is of course possible to derive the fundamental matrix for two cameras of different types with back-projection matrices of different sizes $3 \times m$ respectively $3 \times n$:

$$ (\bar{\mathcal{L}}^b(\mathbf{q}_2))^T \underbrace{(B_2^i)^T [\mathbf{t}] \times R B_1^i}_{\mathbf{F}} \bar{\mathcal{L}}^a(\mathbf{q}_1) = 0. \quad (4.3) $$

The fundamental matrix will accordingly be of size $m \times n$.
Due to the presence of the rank-2 skew-symmetric matrix $[t]_×$, the fundamental matrix is always of rank smaller or equal than 2, in general of rank 2.

**Non-central cameras.** The situation for non-central cameras is analogous: it suffices to plug back-projection matrices $B^l$ of size $6 \times n$ into the non-central epipolar constraint (4.2). A difference from the central case is that the rank of the non-central fundamental matrix depends on the rank of the back-projection matrices as well as on the relative position of the two cameras. For example, the back-projection matrices of two-slit cameras are of rank 4; the rank of the two-slit fundamental matrix is 4 if the two views are in general position, and lower if the two views share one slit or if their slits are incident (cf. [139] and Section 4.2.2).

**Epipolar curves.** For perspective cameras, $Fq_1$ represents the epipolar line in the second view, associated with a point $q_1$ in the first view. In our general setting, epipolar lines are replaced by epipolar curves. It is easy to see that the order of epipolar curves is identical to the order of image coordinate lifting (a and b respectively in Equation (4.3)). For example, for liftings of order 2,3 respectively 4, epipolar curves are conics, cubic respectively quartic curves. This observation tells us immediately that for all cameras that can be modeled by back-projection matrices operating on second order lifted image coordinates, epipolar curves are conics, e.g.,: linear pushbroom cameras and other types of two slits (cf. Section 3.1.4), linear oblique cameras (cf. Section 3.4.3), para-catadioptric cameras (cf. Section 2.3.1), the division model (cf. Section 3.1.8), all GLCs (cf. Section 3.4.4), etc. Also, epipolar curves for the one-coefficient classical radial distortion model (cf. Section 3.1.1), are obviously cubic (cf. [566]).

As described in Section 3.4.2, non-central cameras may or may not be in standard stereo configuration: they are, if epipolar curves in the two images are in a one-to-one correspondence. This is for example the case for two-slit cameras with relative positions as mentioned above (e.g., they share a slit), which corresponds to a fundamental matrix of rank 2. Otherwise, one can still compute epipolar curves in both
images, but they now form 2D families in each image and are no longer
in one-to-one correspondence. An algebraic consequence is an increase
in the rank of the fundamental matrix, for the above example of two-slit
cameras, an increase from 2 to 3 or 4.

Let us note that Rademacher and Bishop already described how to
understand epipolar curves for arbitrary non-central images [411]. They
also suggested the concept of an internal epipolar geometry, i.e., an
epipolar geometry of a non-central camera with itself: when considering
any ray of the camera, all other rays that cut it, are associated with an
epipolar curve. For oblique cameras, no internal epipolar curves exist
by definition.

**Epipoles.** For central cameras, one may define epipoles analogously to the pinhole case: the image of the optical center of one camera is the epipole in the other camera’s image plane. Further, all epipolar curves go through the epipole. In general, there may be more than one epipole, for example for central catadioptric cameras (see an example in Figure 4.1) for which each 3D point is mathematically imaged in two points (cf. Section 3.1.5). In that case, all epipolar curves go through all epipoles. Another example is the classical radial distortion model;

<figure><img src="image_28.png" /><figcaption>Fig. 4.1 Epipolar geometry for a perspective and para-catadioptric image pair. Cyan circles indicate three point matches and associated epipolar lines/curves are shown in red. Epipoles are shown in white; there exist two of them in the para-catadioptric image.</figcaption></figure>
in the case of one distortion coefficient, there are three epipoles: either three real points or one real and two conjugate imaginary points.

For non-central cameras, the notion of optical center and thus of epipole does not exist anymore. However, in the special cases of axial and two-slit cameras, a weaker notion can be defined as follows: the image of the camera axis (or axes in the case of a two-slit camera) in the other camera gives a curve that is incident to all epipolar curves.

**The non-polynomial case.** The results in this section so far only apply to cameras whose back-projection can be modeled by matrices. In other cases, e.g., for most of the fisheye models of Section 3.1.7 (cf. also Table 3.1), fundamental matrices do not exist. The epipolar geometry is still defined of course, it just cannot be expressed in the usual matrix-vector style. The epipolar geometry has been studied for some of these cases, see the following section.

### 4.2.2 Existing Works

In the following, a few existing works on the epipolar geometry of non-perspective cameras are reviewed.

**Classical polynomial distortion model.** Zhang, in [566], studied the epipolar geometry of two radially distorted images and showed that epipolar curves are cubic if only one radial distortion coefficient is considered. He proposed a non-linear optimization method for estimating the distortion coefficients of the two images and the perspective fundamental matrix.

**Central catadioptric cameras.** Svoboda et al. studied the epipolar geometry for central catadioptric cameras [482, 479, 481]. They showed that epipolar curves are conics and gave formulas for computing these conics, from known camera calibration and motion. They also showed the existence of two epipoles per image, in general, which are incident with each epipolar conic.

Geyer and Daniilidis showed that the epipolar geometry of para-catadioptric cameras can be represented via a fundamental matrix [166]. It acts on lifted coordinates of image points and is of
size 4 × 4 if one assumes that images have square pixels (in the gen-
eral case, it is of size 6 × 6 [471]). It can thus be estimated linearly
from 15 or more point matches. Geyer and Stewénius proposed a min-
imal 9-point algorithm to compute the fundamental matrix between
two para-catadioptric cameras, based on constructing a Gröbner basis
of the polynomial equation system given by the point matches [172].

Geyer and Daniilidis [168] and independently, Sturm [471], showed
how to systematically derive multi-view relationships for para-
catadioptric cameras. Their approaches use a direct extension of the
framework used for perspective images by Faugeras and Mourrain [138]
and Triggs [511]. They allow to re-derive the above para-catadioptric
fundamental matrix and in addition, trifocal or quadrifocal tensors.
Sturm showed that using the same framework, one may obtain multi-
view relations for any combination of para-catadioptric, perspective,
or orthographic images [471]. It is clear from Section 4.2.1 that this
easily generalizes to all types of camera models represented by back-
projection matrices.

Other central catadioptric cameras than para-catadioptric ones,
i.e., those based on hyperboloidal or ellipsoidal mirrors, are less
nicely modeled. The main reason for this is that mathematically, each
image point is back-projected to two camera rays (cf. Section 3.1.5).
Although only one of them is physically plausible, it is not possible
to dissociate the two rays while expressing them using polynomials
in the image point coordinates. Sturm and Barreto showed that it
is possible to represent the two rays by a single geometric object (a
line complex) which can be represented using polynomials [473]. This
enabled the definition of back-projection and fundamental matrices,
the latter being of size 15 × 15.

**Non-central catadioptric cameras.** Cone-shaped mirrors usually lead to non-central catadioptric cameras (cf. Section 2.3.1). The epipolar geometry for such cameras, when restricted to planar motions, was derived by Yagi and Kawato [542]. As for two cameras mounted one on top of the other, with aligned mirror axes, the epipolar geometry is simple [456]: the mirror axes pierce the two image planes in the epipoles and the radial lines going through the epipoles, are epipolar lines. One
may see that this situation is always true for pairs of radially symmetric axial cameras with aligned camera axes.

Würz-Wessel proposed a numerical approach for computing epipolar curves in images of free-form mirror surfaces [538].

**Central and non-central panoramas.** Ishiguro et al. showed that in central panoramic images (see Section 2.1.1), 3D lines are imaged as sinusoidal curves [246, 247]. Consequently, epipolar curves are sinusoidal too, a result also derived by McMillan and Bishop [338] and Bunschoten and Kröse [68, 69]. Torii et al. studied the multi-view geometry of central panoramas [509].

Huang et al. studied the epipolar geometry for non-central panoramic images generated by a perspective 1D sensor rotated about an axis parallel to the sensor line but not necessarily passing through the 1D camera's optical center [236, 237, 238]. Further, the 1D sensor may be oriented such that the sensor line, the optical center, and the rotation axis are not coplanar. Huang et al. derived the epipolar geometry for such images as well as several special cases [236, 238] and proposed motion estimation methods [237]. Šivic and Pajdla proposed an in-depth geometric study of non-central panoramic images and their stereo configurations [450].

Menem and Pajdla established the epipolar geometry between a non-central panoramic image and a pinhole image [342].

**Polynomial and rational polynomial models.** Claus and Fitzgibbon uncovered the fundamental matrix for the bi-quadratic rational function model (see Section 3.1.8) [98]. It is a 6 × 6 matrix that maps lifted image points to the six coefficients (five plus a scale factor) of the epipolar conics in the other image. If one restricts the model to the one-parameter division model, then image lines are circles and epipolar geometry may be written via a 4 × 4 fundamental matrix [31, 98].

Barreto and Daniilidis showed that this is the case even if the two cameras have different distortion coefficients [31]. They also showed how to compute the fundamental matrix and extract the distortion coefficients from it. In [35], they showed that the combination of a para-catadioptric camera and one following the division model is also subject to a fundamental matrix.