# Euclidean Bundle Adjustment Independent on Camera Intrinsic Parameters

Ezio Malis, Adrien Bartoli

► To cite this version:

Ezio Malis, Adrien Bartoli. Euclidean Bundle Adjustment Independent on Camera Intrinsic Parameters. RR-4377, INRIA. 2001. inria-00072211

HAL Id: inria-00072211

https://inria.hal.science/inria-00072211

Submitted on 23 May 2006

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE

# Euclidean Bundle Adjustment Independent
on Camera Intrinsic Parameters

Ezio Malis — Adrien Bartoli

N° 4377

Décembre 2001

THÈMES 4 et 3

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

INRIA
SOPHIA ANTIPOLIS

# Euclidean Bundle Adjustment Independent on Camera Intrinsic Parameters

Ezio Malis *, Adrien Bartoli †

Thèmes 4 et 3 — Simulation et optimisation
de systèmes complexes — Interaction homme-machine,
images, données, connaissances
Projets Icare et Movi

Rapport de recherche n° 4377 — Décembre 2001 — 27 pages

**Abstract:** This research report presents a bundle adjustment technique which is independent on camera intrinsic parameters. Standard bundle adjustment techniques consider camera intrinsic parameters as unknowns in the optimization process. The scheme proposed in this report differs from previous standard techniques since unknown camera intrinsic parameters are computed as a function of the 3D structure and the camera pose. Considering less unknowns in the optimization process produces a faster algorithm which is more adapted to real-time applications. Computationally expensive metric reconstruction, using for example several zooming cameras, considerably benefits from an intrinsics-free bundle adjustment. Indeed, experimental results, obtained using both simulated data and real images, show that the algorithm we propose not only considerably reduces the computational cost but also improves the accuracy of the metric reconstruction.

**Key-words:** structure from motion, Euclidean reconstruction, bundle adjustment, projective invariance.

<footnote>* Ezio Malis is with the ICARE Project at INRIA Sophia-Antipolis</footnote>

<footnote>† Adrien Bartoli is with the MOVI Project at INRIA Rhône-Alpes</footnote>
Ajustement euclidien de faisceaux indépendant
des paramètres intrinsèques des caméras

**Résumé** : Ce rapport de recherche présente une technique d'ajustement de faisceaux in-
dépendante des paramètres intrinsèques des caméras. Les techniques standards d'ajustement
de faisceaux considèrent ces paramètres comme des inconnues dans le processus d'optimisation.
Le principe proposé dans ce rapport est différent. En effet, les paramètres intrinsèques des
caméras sont calculés à partir de la structure 3D et la position des caméras. Un nombre
inférieur d'inconnues est alors impliqué dans le processus d'estimation, lequel converge alors
plus rapidement. Les reconstructions métriques coûteuses, par exemple à partir de plusieurs
caméras à zoom variable, tirent un bénéfice non négligeable de cet ajustement de faisceaux
indépendant des données de calibrage. En effet, les résultats expérimentaux sur données
simulées et réelles montrent qu'une réduction notable du temps de calcul ainsi qu'une légère
amélioration de la précision de la reconstruction métrique sont ls obtenues.

**Mots-clés** : invariance projective, ajustement de faisceaux, reconstruction métrique.
# Contents

<table><tr><td>1</td><td>Introduction</td><td>4</td></tr><tr><td>2</td><td>Theoretical background</td><td>5</td></tr><tr><td>2.1</td><td>Perspective projection</td><td>5</td></tr><tr><td>2.2</td><td>Camera model</td><td>6</td></tr><tr><td>3</td><td>Projective spaces invariant to camera parameters</td><td>6</td></tr><tr><td>3.1</td><td>Invariance by projective transformation</td><td>7</td></tr><tr><td>3.2</td><td>Computing camera parameters "a posteriori"</td><td>9</td></tr><tr><td>4</td><td>Metric reconstruction from several images</td><td>13</td></tr><tr><td>4.1</td><td>Computational cost of the non-linear optimization</td><td>14</td></tr><tr><td>4.2</td><td>Initialization of the non-linear optimization</td><td>14</td></tr><tr><td>4.3</td><td>Imposing geometric constraints</td><td>15</td></tr><tr><td>4.4</td><td>Expected improvements over standard bundle adjustment</td><td>15</td></tr><tr><td>4.4.1</td><td>Fixed intrinsic parameters</td><td>15</td></tr><tr><td>4.4.2</td><td>Varying focal length and principal point.</td><td>16</td></tr><tr><td>4.4.3</td><td>Time-varying structure and motion tracked by several zooming cameras.</td><td>16</td></tr><tr><td>5</td><td>Experimental Results</td><td>16</td></tr><tr><td>5.1</td><td>Simulated Data</td><td>16</td></tr><tr><td>5.1.1</td><td>Fixed intrinsic parameters</td><td>17</td></tr><tr><td>5.1.2</td><td>Varying focal length and principal point.</td><td>19</td></tr><tr><td>5.1.3</td><td>Time-varying structure and motion tracked by several zooming cameras.</td><td>22</td></tr><tr><td>5.2</td><td>Real Images</td><td>24</td></tr><tr><td>5.2.1</td><td>Fixed intrinsic parameters</td><td>24</td></tr><tr><td>5.2.2</td><td>Varying focal length and principal point.</td><td>25</td></tr><tr><td>6</td><td>Conclusion</td><td>26</td></tr></table>
# 1 Introduction

Bundle adjustment is the final refinement step of a reconstruction process. Standard metric reconstruction methods start with a projective reconstruction of the scene obtained from several images [4] [6] [2]. The projective reconstruction is then updated to an Euclidean reconstruction after camera self-calibration [11] [8] [12]. Finally, the visual reconstruction is refined using bundle adjustment in the Euclidean space [10]. Standard bundle adjustment [13] [7] is based on the joint estimation of the 3D structure, of the camera extrinsic parameters (i.e. the rotation and the translation of the cameras with respect to the absolute frame) and of the camera intrinsic parameters (i.e. the focal length, the principal point, the skew and the aspect ratio). Therefore, bundle adjustment is a high dimensional optimization problem which involves a large number of unknowns. The objective is achieved by minimizing a cost function which contains the error between the features measured from the images and the features predicted from a model. The cost function is generally non-linear and it can eventually be subject to non-linear constraints. Obviously, the speed of an optimization process is directly related to the number of unknowns and to the form of the cost function. Many applications of metric reconstruction, and in particular robotic vision applications, need a computational speed close to real-time. Thus, it is extremely important to have a fast algorithm despite the large number of images used for the reconstruction. The primary objective of this research report is to investigate a new fast bundle adjustment method to be used in real-time applications. As an example of robotic application needing a huge amount of computation, consider the measurement and the tracking over time of the 3D model of a moving non-rigid object using several zooming cameras. In order to accomplish such difficult applications, the bundle adjustment that we propose is based on parameterizing the reconstruction problem by using the 3D structure and the camera extrinsic parameters but without considering the camera intrinsic parameters despite the fact that they are unknowns. The key idea is to partially work in a projective space which is invariant to camera intrinsic parameters. A general approach to invariance theory for systematically extracting invariants is given by Lie group theory [14]. Obviously, there are various ways in which invariance to camera parameters can be obtained. For example, simple invariants to focal length and principal point have been proposed in [15] for robotics applications. Contrarily to previous work, in this report we do not limit the invariance to focal length and principal point but we exploit invariance to all the camera intrinsic parameters, following an idea proposed in [9], where invariance is obtained by a change of projective coordinates. Thus, the number of unknowns in the bundle adjustment optimization problem can be considerably reduced. Even if the computational cost in terms of CPU time is the key issue of this report, a second objective of our work is to have a fast convergence of the optimization process without losing the accuracy of the solution. Obviously, the accuracy of 3D measurements increases with the number of cameras and/or the number of images. If the number of cameras increases, so does the number of unknown camera parameters in a standard bundle adjustment. On the contrary, the bundle adjustment proposed in this report does not add any supplementary unknown camera parameter in that case. Consequently, we eliminate the possibility to find, in the non-linear optimization, local minima which may be due to correlation between
intrinsic and extrinsic parameters of the camera. Experimental results, obtained using simulated data and real images, show that our algorithm not only considerably reduces the computational cost but also improves the accuracy of the reconstruction.

# 2 Theoretical background

In this section, we give the necessary theoretical background and we introduce the mathematical notations used in this report.

## 2.1 Perspective projection

Let $\mathcal{F}_0$ be the absolute frame attached to a rigid object. Let the 3D point $\mathcal{C}$ be the origin of frame $\mathcal{F}$ and the center of projection (see Figure 1). Let the plane of projection $\pi$ be parallel to the plane $(\vec{x}, \vec{y})$. A 3D point with homogeneous coordinates $\mathbf{X}_i = (X_i, Y_i, Z_i, 1)$ (with respect to frame $\mathcal{F}_0$) is projected to the point $\mathbf{m}_i = (x_i, y_i, 1)$ on $\pi$:

$$ \zeta_i \mathbf{m}_i = [ \mathbf{R} \quad \mathbf{t} ] \mathbf{X}_i, \qquad (1) $$

where $\zeta_i$ is a positive scalar factor, $\mathbf{R}$ and $\mathbf{t}$ are respectively the rotation and the translation between frame $\mathcal{F}_0$ and $\mathcal{F}$. Let $\mathbf{r} = \theta \mathbf{u}$ be the (3×1) vector containing the axis of rotation $\mathbf{u}$ and the angle of rotation $\theta$ ($0 \le \theta < 2\pi$). If $[\mathbf{r}]_\times$ is the skew symmetric matrix associated to vector $\mathbf{r}$, then $\mathbf{R} = \exp([\mathbf{r}]_\times)$. Let $\xi = (\mathbf{t}, \mathbf{r})$ be the (6×1) vector containing global coordinates of an open subset $\mathcal{S} \subset \mathbb{R}^3 \times \mathcal{SO}(3)$. The vector $\mathbf{m}_i(\xi, \mathbf{X}_i)$ depends on the position $\xi$ of the camera and on the 3D coordinates $\mathbf{X}_i$ of the point.

<figure><img src="image_4.png" /><figcaption>Figure 1: Perspective projection of 3D points in the frame $\mathcal{F}$.</figcaption></figure>
## 2.2 Camera model

Pinhole cameras perform a perspective projection of a 3D point. However, the point $m_i$ is not directly measured by the cameras. The information given by a pinhole camera is an image point $p_i$. The point $p_i = (u_i, v_i, 1)$ observed in the image $\mathcal{I}$, taken at the position $\mathcal{F}$, depends on the camera internal parameters:

$$ \mathbf{p}_i = \mathbf{K} \mathbf{m}_i, \qquad \mathbf{p}_i \in \mathcal{I}(\xi, \mathbf{K}), \qquad (2) $$

where:

$$ \mathbf{K} = \begin{bmatrix} f_m k_u & -f_m k_u \cot(\phi) & u_0 \\ 0 & f_m k_v / \sin(\phi) & v_0 \\ 0 & 0 & 1 \end{bmatrix} \qquad (3) $$

$u_0$ and $v_0$ are the coordinates of the principal point (pixels), $f_m$ is the metric focal length (meters), $k_u$ and $k_v$ are the magnifications respectively in the $\vec{u}$ and $\vec{v}$ directions (pixels/meters), and $\phi$ is the angle between these axes. The image $\mathcal{I}(\xi, \mathbf{K})$ depends on the intrinsic parameters of the camera. For the sake of simplicity, we will also use the following equivalent notation of the camera parameters:

$$ \mathbf{K} = \begin{bmatrix} f_p & s & u_0 \\ 0 & r f_p & v_0 \\ 0 & 0 & 1 \end{bmatrix} \qquad (4) $$

where $f_p = f_m k_u$ is the focal length measured in pixel, $s = -f_m k_u \cot(\phi)$ is the skew and $r = k_v / (k_u \sin(\phi))$ is the aspect ratio.

## 3 Projective spaces invariant to camera parameters

Invariance to camera intrinsic parameters can be obtained if there exist a non-singular $(3 \times 3)$ matrix $\mathbf{Q}$ measured from image features, such that:

$$ \mathbf{Q} = \mathbf{K} \mathbf{M}, $$

where $\mathbf{M}$ is a matrix only depending on the 3D coordinates of the object with respect to frame $\mathcal{F}$. The matrix $\mathbf{Q}$ can be used to define a $\mathcal{P}^2$ projective space $\mathcal{Q}$. The transformed point $\mathbf{q}_i \in \mathcal{Q}$ is:

$$ \mathbf{q}_i = \mathbf{Q}^{-1} \mathbf{p}_i, \qquad \mathbf{q}_i \in \mathcal{Q}(\xi). \qquad (5) $$

From now on we will refer to the transformed space $\mathcal{Q}(\xi)$ as an “invariant” space where the invariance is related to camera intrinsic parameters. It is easy to show that the transformed image point $\mathbf{q}_i$ does not depend on the intrinsic parameters of the camera. Indeed, from equation (3) we obtain:

$$ \mathbf{q}_i = \mathbf{Q}^{-1} \mathbf{p}_i = \mathbf{M}^{-1} \mathbf{K}^{-1} \mathbf{K} \mathbf{m}_i = \mathbf{M}^{-1} \mathbf{m}_i. $$
Therefore, $q_i$ only depends on the position of the camera with respect to the observed object and on its 3D structure. Matrix $Q$ can be built by arbitrarily choosing three non collinear image points as proposed in [9]. However, the transformation proposed in [9] depends on the choice of the three points and the transformed coordinates $q_i$ are also invariant to a rotation $r_z$ around the $\vec{z}$ axis of frame $\mathcal{F}$. In order to avoid the selection of the points, we propose a different method to compute the projective transformation $Q$. This method uses all available features in the image.

## 3.1 Invariance by projective transformation

Suppose that $n$ points, not all collinear, are available in image $\mathcal{I}$. Let $S_p$ be the following symmetric (3×3) matrix:

$$S_p = \frac{1}{n} \sum_{i=1}^{n} \mathbf{p}_i \mathbf{p}_i^T = \frac{1}{n} \begin{bmatrix} \sum_{i=1}^{n} u_i^2 & \sum_{i=1}^{n} u_i v_i & \sum_{i=1}^{n} u_i \\ \sum_{i=1}^{n} u_i v_i & \sum_{i=1}^{n} v_i^2 & \sum_{i=1}^{n} v_i \\ \sum_{i=1}^{n} u_i & \sum_{i=1}^{n} v_i & \sum_{i=1}^{n} 1 \end{bmatrix} .$$

This matrix, computed using all observed points, depends on camera intrinsic parameters and it has a simple geometric meaning. Indeed, if:

$$S_p = \begin{bmatrix} \sigma_{11} & \sigma_{12} & \sigma_{13} \\ \sigma_{12} & \sigma_{22} & \sigma_{23} \\ \sigma_{13} & \sigma_{23} & 1 \end{bmatrix},$$

then $\sigma_{13}$ and $\sigma_{23}$ are the coordinates of the centroid of the $n$ points, while $\sigma_{11}$, $\sigma_{12}$ and $\sigma_{22}$ are the moments of inertia of the points about the origin axes. Similarly, let $S_m$ be the following positive symmetric (3×3) matrix:

$$S_m = \frac{1}{n} \sum_{i=1}^{n} \mathbf{m}_i \mathbf{m}_i^T = \frac{1}{n} \begin{bmatrix} \sum_{i=1}^{n} x_i^2 & \sum_{i=1}^{n} x_i y_i & \sum_{i=1}^{n} x_i \\ \sum_{i=1}^{n} x_i y_i & \sum_{i=1}^{n} y_i^2 & \sum_{i=1}^{n} y_i \\ \sum_{i=1}^{n} x_i & \sum_{i=1}^{n} y_i & \sum_{i=1}^{n} 1 \end{bmatrix} .$$

This matrix only depends on the structure of the object and on its position with respect to frame $\mathcal{F}$. Since $\mathbf{p}_i = \mathbf{K} \mathbf{m}_i$, $S_p$ can be written as a function of $S_m$ and of the camera intrinsic parameters $\mathbf{K}$:

$$S_p = \frac{1}{n} \sum_{i=1}^{n} \mathbf{p}_i \mathbf{p}_i^T = \frac{1}{n} \sum_{i=1}^{n} \mathbf{K} \mathbf{m}_i \mathbf{m}_i^T \mathbf{K}^T = \mathbf{K} \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{m}_i \mathbf{m}_i^T \right) \mathbf{K}^T = \mathbf{K} S_m \mathbf{K}^T . \quad (6)$$
It is possible to decompose $S_p$ and $S_m$ using a Cholesky decomposition as:

$$S_m = T_m T_m^T \qquad (7)$$

$$S_p = T_p T_p^T \qquad (8)$$

where $T_m$ and $T_p$ are upper triangular matrices. Thus, from equations (6), (7) and (8) we obtain:

$$T_p = K T_m \qquad (9)$$

Note that $T_m$ does not depend on $K$, thus choosing $Q = T_p$ the transformed space defined by equation (5) is independent on camera intrinsic parameters:

$$q = T_p^{-1} p = T_m^{-1} K^{-1} p = T_m^{-1} K^{-1} K m = T_m^{-1} m$$

Note that $T_p$ is easily obtained from the Cholesky decomposition of matrix $S_p$:

$$Q = T_p = \begin{bmatrix} \sqrt{\sigma_{11} - \sigma_{13}^2 - \frac{(\sigma_{12} - \sigma_{13}\sigma_{23})^2}{\sigma_{22} - \sigma_{23}^2}} & \sigma_{13} \\ 0 & \sqrt{\frac{(\sigma_{12} - \sigma_{13}\sigma_{23})^2}{\sigma_{22} - \sigma_{23}^2} - \sigma_{23}} & \sigma_{23} \\ 0 & 0 & 1 \end{bmatrix}$$

The computation of $Q$ from image points is extremely stable since it is based on averaging measurements. This is proven by simulation results obtained with $n = 100$ points randomly distributed in a sphere with $\rho = 10$ cm radius. The image coordinates of each point are corrupted with an additive Gaussian noise with zero mean and increasing standard deviation. For each value of $\sigma$, we compute the mean and the standard deviation, over $N = 10000$ samples, of the relative errors on the estimated matrix $\tilde{Q}$. Let $\tilde{q}_{ij}$ and $q_{ij}$ be respectively the $ij$ entries of estimated matrix $\tilde{Q}$ and of the true matrix $Q$. Since $\tilde{q}_{21} = q_{21} = 0$, $\tilde{q}_{31} = q_{31} = 0$, $\tilde{q}_{32} = q_{32} = 0$ and $\tilde{q}_{33} = q_{33} = 1$, we compute the following five relative errors:

$$\frac{\tilde{q}_{ij} - q_{ij}}{q_{ij}} \quad (i, j) \in \{(1, 1), (1, 2), (1, 3), (2, 2), (2, 3)\}$$

The results plotted in Figure 2 show that the average relative error on $\tilde{Q} - Q$ remains small despite $\sigma$ increases. Obviously the standard deviation of the error increases but it is small enough to consider $\tilde{Q}$ as a very good approximation of $Q$. This is very important since matrix $\tilde{Q}$ will be used in the cost function minimized in the bundle adjustment. We will see in the next subsection that having a good approximation of the true matrix $Q$ allows us to correctly estimate “a posteriori” the camera intrinsic parameters.
<figure><img src="image_5.png" /><figcaption>Figure 2: Relative error on the entries of the matrix Q estimated with an increasing standard deviation of noise on pixel coordinates. The mean and the standard deviation of the error are sufficiently small to consider $\tilde{Q}$ as a good approximation of the true matrix Q.</figcaption></figure>

## 3.2 Computing camera parameters “a posteriori”

From equation (9), the camera parameters can be computed “a posteriori” as a function of the known matrix $T_p$ and of the unknown matrix $T_m$:

$$K = T_p T_m^{-1} \qquad (10)$$

Since the matrix $T_p$ is measured from image points corrupted by noise, only an estimation $\tilde{T}_p$ can be used to compute an estimated matrix $\hat{K}$:

$$\hat{K} = \tilde{T}_p \hat{T}_m^{-1} \qquad (11)$$

where $\hat{T}_m$ is an estimation of the unknown matrix $T_m$. During the bundle adjustment, we will change the estimation $\hat{T}_m$ trying to find the true matrix. Suppose that the matrix $\hat{T}_m$ is perfectly estimated (i.e. we have $\hat{T}_m = T_m$). In this case, we can show that the error $\tilde{K} - K$ can be neglected. As a consequence, using $\tilde{K}$ instead of $K$ in the bundle adjustment has a little effect on the optimization process preserving the optimality of the solution. In order to have a ground truth, simulations have been carried out using a pinhole camera with $f = 1000$ pixels, $r = 1$, $\phi = \pi/2$ and $u_0 = v_0 = 250$ pixels. For each simulation, the results are evaluated by computing the following relative errors:
* the relative error on the focal length: $\frac{\hat{f}_p - f_p}{f_p}$

* the relative error on the aspect ratio: $\frac{\hat{r} - r}{r}$

* the relative error on the skew: $\frac{\hat{\phi} - \phi}{\phi}$

* the relative errors on the principal point: $\frac{\hat{u}_0 - u_0}{u_0}$ and $\frac{\hat{v}_0 - v_0}{v_0}$

In these simulations, the distribution of the 3D points is the same one used in the previous subsection. In the first simulation, the image coordinates of each point are corrupted with an additive Gaussian noise with zero mean and increasing standard deviation ($0 \le \sigma \le 3$ pixels). For each value of $\sigma$, we compute the mean and the standard deviation, over $N = 10000$ samples, of the relative errors. The results given in Figure 3(a) show that the error on the camera intrinsic parameters is considerably smaller than the error usually obtained with standard self-calibration methods (see the results in Section 5).

Figure 4(b) shows the results obtained with a fixed standard deviation ($\sigma = 1$) of the Gaussian noise. The number of points increases ($10 \le n \le 100$) at each iteration. For each value of $n$, we compute the mean and the standard deviation, over $N = 10000$ samples, of the relative errors. This means that the error is very small and practically constant while the standard deviation slowly decreases as the number of points increases. This result confirms that the error on the intrinsic camera parameters is negligible for any number of points.
<figure><img src="image_6.png" /><figcaption>Figure 3: Results with an increasing standard deviation of noise on pixel coordinates (with $n = 100$ points). The mean error slightly increases for all the camera intrinsic parameters. The standard deviation increases faster but it remains small.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>(a) Relative error on the focal length *f*</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Relative error on the principal point (*u*₀, *v*₀)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) Relative error on the angle *φ*</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(d) Relative error on the ratio *r*</figcaption></figure>

Figure 4: Simulation results with an increasing number of points and with fixed standard deviation σ = 1. The mean error is almost constant for all the camera intrinsic parameters. The standard deviation of the error decreases when the number of points used in the computation increases.
# 4 Metric reconstruction from several images

Suppose to have $m$ different images (obtained from several zooming cameras) of the same $n$ points. Bundle adjustment is achieved by minimizing the following cost function:

$$C = \sum_{i=1}^{n} \sum_{j=1}^{m} \| \hat{\mathbf{p}}_{ij} - \tilde{\mathbf{p}}_{ij} \|^2,$$

where $\tilde{\mathbf{p}}_{ij}$ is the known vector containing the pixel coordinates of point $i$ measured in the image $j$ and $\hat{\mathbf{p}}_{ij}$ is the model estimated from unknown parameters. Standard bundle adjustment uses the following model:

$$\hat{\mathbf{p}}_{ij}(\mathbf{K}_j, \xi_j, \mathcal{X}_i) = \mathbf{K}_j \hat{\mathbf{m}}_{ij}(\xi_j, \mathcal{X}_i)$$

thus, for each image the unknowns are the camera parameters $\mathbf{K}_j$ (i.e. 5m unknowns for all images) and the camera pose $\xi_j$ (i.e. 6m unknowns for all images), and for each point the unknowns are the 3D coordinates $\mathcal{X}_i$ (i.e. 3n unknowns for all points). The total number of unknowns for a standard bundle adjustment is thus $s = 3n + 11m$.

In this report we propose to use a different model of $\mathbf{p}_{ij}$ in the bundle adjustment:

$$\hat{\mathbf{p}}_{ij}(\xi_j, \mathcal{X}) = \tilde{\mathbf{T}}_{pj} \hat{\mathbf{T}}_{mj}^{-1}(\xi_j, \mathcal{X}) \hat{\mathbf{m}}_{ij}(\xi_j, \mathcal{X}_i) = \tilde{\mathbf{Q}}_j \hat{\mathbf{q}}_{ij}(\xi_j, \mathcal{X})$$

where $\tilde{\mathbf{Q}}_j$ is measured from image features and $\hat{\mathbf{q}}_{ij}(\xi_j, \mathcal{X})$ depends on the camera pose $\xi_j$, on the 3D coordinates of all the points $\mathcal{X} = (\mathcal{X}_1, \mathcal{X}_2, ..., \mathcal{X}_n)$ but not on the camera intrinsic parameters. Consequently, the bundle adjustment we propose is intrinsics-free. The total number of unknowns of the intrinsics-free bundle adjustment is $p = 3n + 6m$. When compared to the standard bundle adjustment, the relative difference of unknowns is $(s-p)/s = 5m/(3n+11m)$. When $m \gg n$ (for example when tracking a given object in a long sequence of images taken with several zooming cameras) we can obtain a reduction up to $(s-p)/s \approx 0.45$ (i.e. 45%) of the number of unknowns. Therefore, as it will be proven by the experimental results presented in Section 5, the computational cost of the optimization process can be considerably reduced.

Let $\mathbf{x} = (\xi_1, \xi_2, ..., \xi_m, \mathcal{X}_1, \mathcal{X}_2, ..., \mathcal{X}_n)$ be the $(p \times 1)$ vector containing all the unknowns. The intrinsics-free bundle adjustment consists in solving the following optimization problem:

$$\min_{\mathbf{x}} C(\mathbf{x}) = \min_{\mathbf{x}} \sum_{i=1}^{n} \sum_{j=1}^{m} \| \tilde{\mathbf{Q}}_j \hat{\mathbf{q}}_{ij}(\xi_j, \mathcal{X}) - \tilde{\mathbf{p}}_{ij} \|^2 . \qquad (12)$$

The cost function is thus defined as the sum of the residual errors of the reprojections in each image. If the error on the measured features $\tilde{\mathbf{p}}_{ij}$ has a Gaussian distribution and $\tilde{\mathbf{Q}}_j \approx \mathbf{Q}_j$ (see the experiment in Section 3) then our bundle adjustment is close to be a maximum likelihood estimator.

By setting $\tilde{\mathbf{q}}_{ij} = \tilde{\mathbf{Q}}_j^{-1} \tilde{\mathbf{p}}_{ij}$, the intrinsics-free bundle adjustment could also be defined as the minimization of a cost function completely defined in the invariant space $\mathcal{Q}$
$\min_{\mathbf{x}} \sum_{i=1}^{n} \sum_{j=1}^{m} \|\hat{\mathbf{q}}_{ij}(\xi_j, \mathbf{X}) - \tilde{\mathbf{q}}_{ij}\|^2$. However, in the report we will use the cost function defined in (12) in order to compare the results with respect to the standard bundle adjustment which cost function is defined in the image space. Finally, note that, without loss of generality, we can set the absolute frame $\mathcal{F}_0$ to one of camera frame $\mathcal{F}_k$. Thus, the number of unknowns is reduced to $s = 3n + 11m - 6$ for the standard bundle adjustment and to $p = 3n + 6m - 6$ for the intrinsics-free bundle adjustment.

## 4.1 Computational cost of the non-linear optimization

The optimization problem (12) is efficiently solved using iterative second order methods. They are generally fast since they provide quadratic convergence to the solution if the starting point of the minimization $\mathbf{x}_0$ is in a neighborhood of the solution. Second order methods are based on the Taylor expansion of the cost function at each iteration $k$ of the optimization process:

$$C(\mathbf{x}_{k+1}) = C(\mathbf{x}_k) + (\mathbf{x}_{k+1} - \mathbf{x}_k)^T \frac{\partial C(\mathbf{x}_k)}{\partial \mathbf{x}} + \frac{1}{2} (\mathbf{x}_{k+1} - \mathbf{x}_k)^T \frac{\partial^2 C(\mathbf{x}_k)}{\partial \mathbf{x}^2} (\mathbf{x}_{k+1} - \mathbf{x}_k)^T + O^3(\mathbf{x}_{k+1} - \mathbf{x}_k)$$

where $\frac{\partial C(\mathbf{x}_k)}{\partial \mathbf{x}}$ is the $(p \times 1)$ gradient vector, $\frac{\partial^2 C(\mathbf{x}_k)}{\partial \mathbf{x}^2}$ is the $(p \times p)$ Hessian matrix and $O^3(\mathbf{x}_{k+1} - \mathbf{x}_k)$ is the residual error. Starting from this expansion, several methods have been proposed to find and update $\mathbf{x}_{k+1}$ of the unknowns. For example, a standard Newton method compute the update $\mathbf{x}_{k+1}$ of the unknowns by setting $O^3(\mathbf{x}_{k+1} - \mathbf{x}_k) = 0$:

$$\mathbf{x}_{k+1} = \mathbf{x}_k + \left( \frac{\partial^2 C(\mathbf{x}_k)}{\partial \mathbf{x}^2} \right)^{-1} \frac{\partial C(\mathbf{x}_k)}{\partial \mathbf{x}}$$

At each iteration, one needs to compute or approximate the gradient vector and the Hessian matrix. It is clear that the computational cost of the non-linear minimization is directly related to the number of unknowns $p$. The lower is $p$ the faster is the computation. In this report, we use the non-linear second order optimizer Levenberg-Marquardt [3] with numerical differentiation (i.e. the gradient vector and the Hessian matrix are approximated). Note that, even if they are not currently used in our experimental results, the speed of any bundle adjustment algorithm can be increased by using optimization techniques which optimally exploit sparseness and data structure [13].

## 4.2 Initialization of the non-linear optimization

Non-linear optimization requires an initial value $\mathbf{x}_0$ of the unknowns. In our experiments, we first obtain a projective reconstruction and then use a rough self-calibration to upgrade the reconstruction to metric. The projective reconstruction is obtained by computing the fundamental matrix using [4] between the two farthest images, usually the first one and the last one, and triangulating point features in projective space using [5]. We then register each other image in turn by using 3D to 2D point correspondences. Finally, we refine the estimated structure and motion with a bundle adjustment in projective space [1]. The
self-calibration is obtained from the projective reconstruction by using the linear method proposed in [8] and inspired by [12]. We assume that each camera has zero skew, unity aspect ratio and its principal point lying at the image center. These assumptions allow to obtain a linear solution for the (eventually varying) focal length through the recovery of the absolute quadric.

## 4.3 Imposing geometric constraints

For most of pinhole cameras it is a reasonable approximation to suppose a zero skew (i.e. $\phi = \pi/2$) and unit aspect ratio (i.e. $r = 1$). Thus, there are some geometric constraints on camera internal parameters that can be imposed during the optimization. Let $\tau_{ij}^p$ and $\tau_{ij}^m$ be the $ij$ entries of matrices $T_p$ and $T_m(x)$ respectively. It is easy to obtain from equation (11) the equivalence between the following constraints:

$$ \bullet \text{ zero skew: } \phi = \frac{\pi}{2} \Leftrightarrow \tau_{12}^p \tau_{11}^m(x) = \tau_{11}^p \tau_{12}^m(x) $$

$$ \bullet \text{ unit aspect ratio: } r = \frac{k_u}{k_v} = 1 \Leftrightarrow \tau_{11}^p \tau_{22}^m(x) = \tau_{22}^p \tau_{11}^m(x) $$

Imposing constraints on intrinsic parameters is thus equivalent to impose constraints on the unknowns $x$ of the intrinsics-free bundle adjustment. Constraints on the intrinsics are modeled by using either Lagrange multipliers or heavily weighted artificial measurements added to the cost function. In the former case, each constraint is modeled by one additional parameter and in the latter case, not any parameter are necessary. We do not have observed significant differences during our experiments, in terms of accuracy, between these two means of modeling constraints.

## 4.4 Expected improvements over standard bundle adjustment

As mentioned before, the exact count of the unknowns depends on the considered setup (number of cameras, structure of the object,...). We present different setups used for metric reconstruction, supposing that singular configurations have been avoided and that a minimal number of data is available to achieve the reconstruction. For each case, we compare the expected performances of the intrinsics-free bundle adjustment with respect to the standard one. These considerations are confirmed by the experimental results presented in Section 5.

### 4.4.1 Fixed intrinsic parameters

In the first setup, we consider that all intrinsic parameters are fixed and unknown (all images are provided by the same camera). A minimum number of 3 images are necessary to self-calibrate. The unknowns of the standard bundle adjustment are $s = 3n + 6m + 5$ while the unknowns of the intrinsics-free bundle adjustment are $p = 3n + 6m$. Since $p \approx s$, the improvements of the new method on the computational cost are expected not to be impressive. However, it is an interesting setup since it is very common and it can be used
to prove that the accuracy of the solution found by the intrinsics-free bundle adjustment is
as good as for standard bundle adjustment.

4.4.2 Varying focal length and principal point.

The second setup assumes that cameras have zero skew and unity aspect ratio. This case is often encountered in practice since these hypotheses are satisfied by most modern cameras. It corresponds to either a unique zooming camera in motion observing a static scene (note that not only the focal length but also the principal point position may change when zooming and/or focusing) or a set of cameras taking synchronized pictures of the same, eventually dynamic, scene. A minimum number of 4 images are then needed to self-calibrate. The unknowns of the standard bundle adjustment are $s = 3n + 6m + 3m$ while the unknowns of the intrinsics-free bundle adjustment are $p = 3n + 6m$. In this case, when $m \gg n$ we can obtain a reduction up to $(s-p)/s \approx 0.33$ (i.e. 33%) of the number of unknowns. As a consequence, the computational cost is expected to be reduced.

4.4.3 Time-varying structure and motion tracked by several zooming cameras.

The third setup is the most general one. The structure, the camera intrinsic and extrinsic
parameters are continuously changing. It is an interesting setup for modern robotic vision
since it allows the achievement of new applications in visual servoing, object tracking or
computer-aided surgery. The speed of the reconstruction algorithm is extremely important
in order to perform the tracking. With this setup, the number of unknowns is the same
as for the previous setup at each iteration. Thus, the intrinsics-free bundle adjustment is
expected to work better than the standard bundle adjustment, especially when the number
of cameras involved in the reconstruction is large.

5 Experimental Results

In this section, we validate our algorithm using both simulated data and real images and
compare it to standard bundle adjustment methods.

**5.1 Simulated Data**

To test whether the theory presented in the report is a reasonable approximation we consider
the metric reconstruction of $n = 25$ points lying inside a cube of 1 meter side. Several
images (minimum is $m = 3$), taken with one or more cameras (the focal length and the
principal point may also be changed) depending on the setup, are used to reconstruct the
3D coordinates of the points. Images are taken around the cube from different positions
between 5 and 9 meters away from its center. The image coordinates of the points are
corrupted by a centered Gaussian noise with varying standard deviation. The algorithms
are repeated in order to compute the mean and the standard deviation over 1000 trials.
Both standard and intrinsics-free bundle adjustment are ran on the same computer. We measure the CPU time and the number N of iterations needed to the convergence of each algorithm to assess the corresponding computational cost. The accuracy of the solution is measured by evaluating the error on the metric reconstruction. Obviously the reconstruction is made up to a scalar factor, thus we use the ground truth just to fix the scale and make it possible to compare the results in a metric space. In order to have an error independent on the frame in which the reconstruction is done, we consider only differences between the estimated distance $||\hat{\mathcal{X}}_i - \hat{\mathcal{X}}_j||$ and the true distance $||\mathcal{X}_i - \mathcal{X}_j||$ between two points in the 3D space. Thus, the error $\epsilon_{\mathcal{X}}$ on the structure is obtained as:

$$ \epsilon_{\mathcal{X}} = \sum_{i,j}^{n} (||\hat{\mathcal{X}}_i - \hat{\mathcal{X}}_j|| - ||\mathcal{X}_i - \mathcal{X}_j||)^2 $$

In order to verify that the camera parameters obtained “a posteriori” by our algorithm are a good approximation of the true parameters, we measure the relative error $\epsilon_f$ on the recovered focal length as:

$$ \epsilon_f = \frac{\hat{f}_p - f_p}{f_p} $$

Measuring the error on the focal length is justified by the fact that this is usually the only parameter obtained by self-calibration and that may lie quite far from the optimal solution.

### 5.1.1 Fixed intrinsic parameters

In the first set of experiments, we consider a single moving camera with fixed intrinsics parameters. The focal length of the camera is $f = 1000$ pixels, the principal point is at the image center, the skew is zero and the aspect ratio is one. Figure 5 presents the comparison between the properties of convergence and computational cost of the algorithms. As expected, the CPU time until convergence is slightly higher for the standard bundle adjustment algorithm especially when the standard deviation $\sigma$ of the noise increases (see Figure 5(a)). This result is confirmed by figure 5(b) which shows that the convergence to the minimum of the cost function requires less iterations for the intrinsics-free method. It is worth noting that the algorithms converge almost to the same minimum of the cost function. This has been the case in most of our experiments.
<figure><img src="image_11.png" /><figcaption>Figure 5: Comparison of the computational cost between the standard (green line) and the intrinsics-free (red line) bundle adjustment.</figcaption></figure>

Figure 6 plots the error on the recovered structure. Even if the difference between the two algorithms is not impressive, the intrinsics-free method performs better when the noise $\sigma$ is high (Figure 6(a)) and when only few $m$ images are used (Figure 6(b)). As mentioned in Section 3, the camera intrinsic parameters are not directly considered as unknown in our algorithm but they can be estimated “a posteriori”.

<figure><img src="image_12.png" /><figcaption>Figure 6: Comparison of the accuracy of the 3D reconstruction between the standard (green line) and of the intrinsics-free (red line) bundle adjustment.</figcaption></figure>
Since the 3D structure estimated with the intrinsics-free bundle adjustment is more accurate, it is not surprising, as shown in Figure 7, that the relative error $\epsilon_f$ on the focal length is also smaller. Figure 7(a) shows the results obtained when the standard deviation of the noise in the image $\sigma$ varies from 0 to 3 pixels. When the level of added noise is lower than 1 pixel, the results are undistinguishable. Beyond a certain level of added image noise the intrinsics-free bundle adjustment performs better. Figure 7(b) shows the results obtained when varying the number of images used in the reconstruction from 3 to 10. We observe that for a small number of images, in this case less than 5, the intrinsics-free method performs better than the standard one. When the number of images increases beyond 6 the results are undistinguishable

<figure><img src="image_13.png" /><figcaption>Figure 7: Comparison of the relative error on the recovered focal length between the standard (green line) and of the intrinsics-free (red line) bundle adjustment.</figcaption></figure>

### 5.1.2 Varying focal length and principal point.

In this set of experiments we consider a moving and zooming camera. The focal length $f_p$ changes between 800 and 1300 pixels while moving the camera around the 3D structure. We suppose that $s = 0$, $r = 1$ and that the principal point is at the image center. In this setup, the intrinsics-free bundle adjustment has less unknowns than the standard method. Thus, the CPU time and the number of iterations needed to converge are smaller, as it is shown in Figure 8.
<figure><img src="image_14.png" /><figcaption>Figure 8: Comparison of the computational cost between the standard (green line) and the intrinsics-free (red line) bundle adjustment.</figcaption></figure>

Again, the estimation processes is more stable. Thus, the reconstruction provided by the
intrinsics-free bundle adjustment is more accurate. Figure 9 confirms the results obtained
in the previous simulation. With a small number *m* of images and for higher noise σ the
intrinsics-free bundle adjustment performs better. Otherwise, the results are undistinguishable.

<figure><img src="image_15.png" /><figcaption>Figure 9: Comparison of the accuracy of the 3D reconstruction between the standard (green line) and of the intrinsics-free (red line) bundle adjustment.</figcaption></figure>
In this experiment, multiple different focal lengths are recovered. Thus, we use the average error value over all images to compare the results:

$$ \epsilon_f = \sum_{j=1}^{m} \frac{\hat{f}_{pj} - f_{pj}}{f_{pj}} $$

Figure 10(a) plots the relative error on the focal length when varying the level of noise on the image point coordinates, while the principal point is left constant at the center of the image. We observe in Figure 10(a) that, as in the previous case, the two methods perform equally well up to a certain level of noise ($\sigma = 1$ pixel), above which the intrinsics-free method begins to perform better than the standard one.

<figure><img src="image_16.png" /><figcaption>Figure 10: Comparison of the relative error on the recovered focal lengths between the standard (green line) and of the intrinsics-free (red line) bundle adjustment.</figcaption></figure>

Previous results are obtained supposing that the principal point is at the image center where effectively it was. On the other hand, what happen if we suppose that the principal point is at the image center but in reality it is not ? To answer this question, the principal point position is corrupted from the center of the camera with an additive centered Gaussian variable with varying standard deviation. Figure 11 shows that if the principal point is not at the image center the performance of both algorithm is reduced: the CPU time, the error on the recovered structure and focal length increase.
<figure><img src="image_17.png" /><figcaption>Figure 11: CPU time versus the perturbation of the principal point position.</figcaption></figure>

5.1.3 Time-varying structure and motion tracked by several zooming cameras.

This experiment consists in tracking time-varying structure and motion using successive bundle adjustments. At each iteration, structure and motion are estimated using bundle adjustment. The estimate of iteration *k* is the starting point of iteration *k* + 1. We simulate a moving object (eventually non-rigid) observed by a set of zooming cameras. At each iteration the object is roughly maintained at the center of the images using zooming and by changing camera orientations. Zooming affects camera focal length and principal point. Therefore, both camera intrinsic and extrinsic parameters might change. When we deform the structure through time, the deformation consists in offsetting each point from its position by a random value drawn from a normal distribution and with a variance of 10% of the object size. Figure 12 shows the needed CPU time when the object considered is either rigid (a) or non rigid (b). Object motion is a weighted combination between a circular motion which induces great changes in camera orientation and small changes in their intrinsic parameters, and a translation away from the cameras inducing the opposite effect. Figure 12(a) shows that when the change in camera orientation is dominant the behavior of the two bundle adjustments are very close, the intrinsics-free being slightly better. On the other hand, the more the object moves away from the cameras, the more their intrinsics vary which increases the difference between the two algorithms. The intrinsics-free method needs less CPU time to convergence than the standard bundle adjustment. Figure 12(b) shows that when the previous scenario is combined to an object deformation, the difference between the algorithms is slightly reduced but remains significant when the change in the intrinsics is dominant.
<figure><img src="image_18.png" /><figcaption>Figure 12: CPU time to convergence versus relative variation *k* on camera parameters. When *k* = 0 % only camera extrinsic parameters change, while when *k* = 100 % only camera intrinsic parameters change.</figcaption></figure>

The reduction of the number of the unknowns has a beneficial effect on the accuracy
of the reconstruction, as it is shown by the results plotted in Figure 13. We observe in
Figure 13(a) that when the structure is rigid and camera intrinsic parameters change, the
intrinsic-free approach performs much better than the standard bundle adjustment. When
the structure is non-rigid (see Figure 13(b)), the difference between the two algorithm is
reduced but still the intrinsics-free bundle adjustment is more accurate.
<figure><img src="image_19.png" /><figcaption>Figure 13: Error on recovered structure as a function of the relative variation *k* on camera parameters. When *k* = 0 % only camera extrinsic parameters change, while when *k* = 100 % only camera intrinsic parameters change.</figcaption></figure>

## 5.2 Real Images

In this section, we validate our algorithm using real images of a calibration grid. Image point positions, calibration and pose of the camera can be estimated with a very high accuracy. This serves as ground truth.

### 5.2.1 Fixed intrinsic parameters

In the first experiment, we use 3 images of the calibration grid shown in Figure 14 observed by the same camera.

<figure><img src="image_20.png" /><figcaption>Figure 14: Three images of a calibration grid used in the experiment.</figcaption></figure>
We run the bundle adjustments from an initial solution obtained by the linear method while enforcing the constraint that the intrinsics are identical for all the three images. The performance of the algorithms are close since the number of unknowns is almost the same (only 5 unknown less for the intrinsics-free method). However, the intrinsics-free bundle adjustment is slightly faster (7 %) than the standard bundle adjustment and the error on the recovered structure is almost the same. The results obtained with simulated data are confirmed by this experiment. Table 1 shows the results obtained for the focal length and the principal point position for the ground truth, the linear algorithm, the standard and the intrinsics-free bundle adjustment. We observe that the standard bundle adjustment performs globally worse than the intrinsics-free. However, the gap is negligible compared to the difference with the ground truth. The reprojection error of both algorithms is a hundred times lower than a pixel, which is normal since the accuracy of image point position is very high.

<table><caption>Table 1: Comparison of the static calibration results, taken as ground truth, the linear solution and the different bundle adjustment techniques.</caption><thead><tr><th></th><th>ground truth</th><th>linear</th><th>intrinsics-free</th><th>standard</th></tr></thead><tbody><tr><th>focal length</th><td>757.5</td><td>749.9</td><td>754.3</td><td>753.1</td></tr><tr><th>principal point, u<sub>0</sub></th><td>259.6</td><td>256.0</td><td>255.1</td><td>253.4</td></tr><tr><th>principal point, v<sub>0</sub></th><td>327.0</td><td>320.0</td><td>319.6</td><td>321.8</td></tr></tbody></table>

### 5.2.2 Varying focal length and principal point.

In this experiments, we use 7 images of a calibration grid taken with a single zooming camera (see as an example the three images in Figure 15). The model of the grid is used as ground truth.

<figure><img src="image_21.png" /><figcaption>Figure 15: Three images of a calibration grid amongst the 7 used in the experiment.</figcaption></figure>

With respect to the previous experiment, the intrinsics-free bundle adjustment need much less unknowns than the standard method since the camera is zooming. As a consequence,
the intrinsics-free method is 25 % faster and requires less iterations to converge (9 instead of 13). Again, the error on the recovered structure is almost the same. The results given in Table 2 prove that, for all the seven images, the intrinsics-free method is able to recover “a posteriori” the unknown intrinsic parameters of the camera better than the standard bundle adjustment.

<table><caption>Table 2: Comparison of the static calibration results, taken as ground truth, the linear solution and the different bundle adjustment techniques.</caption><thead><tr><th></th><th>f<sub>1</sub></th><th>f<sub>2</sub></th><th>f<sub>3</sub></th><th>f<sub>4</sub></th><th>f<sub>5</sub></th><th>f<sub>6</sub></th><th>f<sub>7</sub></th></tr></thead><tbody><tr><td>ground truth</td><td>751.9</td><td>754.1</td><td>757.2</td><td>765.5</td><td>1161.7</td><td>1992.5</td><td>1943.6</td></tr><tr><td>intrinsic-free</td><td>771.6</td><td>777.2</td><td>772.9</td><td>777.6</td><td>1201.9</td><td>2176.0</td><td>2100.5</td></tr><tr><td>standard</td><td>807.7</td><td>799.8</td><td>789.6</td><td>795.9</td><td>1240.3</td><td>2185.9</td><td>2229.5</td></tr></tbody><thead><tr><th></th><th>u<sub>1</sub></th><th>u<sub>2</sub></th><th>u<sub>3</sub></th><th>u<sub>4</sub></th><th>u<sub>5</sub></th><th>u<sub>6</sub></th><th>u<sub>7</sub></th></tr></thead><tbody><tr><td>ground truth</td><td>324.3</td><td>302.9</td><td>294.9</td><td>341.3</td><td>316.2</td><td>321.1</td><td>237.2</td></tr><tr><td>intrinsic-free</td><td>334.7</td><td>311.7</td><td>298.2</td><td>350.4</td><td>328.1</td><td>318.9</td><td>317.1</td></tr><tr><td>standard</td><td>311.9</td><td>282.9</td><td>267.3</td><td>334.7</td><td>265.2</td><td>112.9</td><td>186.8</td></tr></tbody><thead><tr><th></th><th>v<sub>1</sub></th><th>v<sub>2</sub></th><th>v<sub>3</sub></th><th>v<sub>4</sub></th><th>v<sub>5</sub></th><th>v<sub>6</sub></th><th>v<sub>7</sub></th></tr></thead><tbody><tr><td>ground truth</td><td>271.3</td><td>254.7</td><td>250.9</td><td>249.8</td><td>273.5</td><td>293.9</td><td>286.9</td></tr><tr><td>intrinsic-free</td><td>259.6</td><td>244.8</td><td>247.0</td><td>242.8</td><td>255.6</td><td>268.7</td><td>261.4</td></tr><tr><td>standard</td><td>269.5</td><td>246.4</td><td>238.5</td><td>249.1</td><td>237.4</td><td>110.1</td><td>348.0</td></tr></tbody></table>

# 6 Conclusion

This report has presented a new refinement technique for metric reconstruction which is particularly adapted to computer vision applications close to real-time. Indeed, the proposed bundle adjustment uses much less unknowns in the optimization process allowing for a faster reconstruction algorithm. At the same time, the accuracy of the solution is slightly better than the accuracy obtained with standard bundle adjustment techniques. Future work will concern the test of our algorithm with several uncalibrated zooming cameras.

# References

[1] A. Bartoli and P. Sturm. Three new algorithms for projective bundle adjustment with minimum parameters. Technical Report 4236, INRIA, August 2001.

[2] O. Faugeras, and Q-T. Luong. *Geometry of multiple images*. MIT Press, 2001.

[3] P.E. Gill, W. Murray, and M.H. Wright. *Practical Optimization*. Academic Press, 1981.
[4] R. Hartley. In defence of the 8-point algorithm. In *Proceedings of the 5th International Conference on Computer Vision, Cambridge, Massachusetts, USA*, pages 1064–1070, June 1995.

[5] R. Hartley and P. Sturm. Triangulation. *Computer Vision and Image Understanding*, 68(2):146–157, 1997.

[6] R. I. Hartley and A. Zisserman. *Multiple View Geometry in Computer Vision*. Cambridge University Press, ISBN: 0521623049, 2000.

[7] F. Kahl and A. Heyden. Euclidean reconstruction and auto-calibration from continuous motion. In *IEEE International Conference on Computer Vision*, volume 2, Vancouver, Canada, July 2001.

[8] M. Pollefeys, R. Koch, and L. Van Gool. Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. *International Journal of Computer Vision*, 32(1):7–25, 1999.

[9] E. Malis. Visual servoing invariant to changes in camera intrinsic parameters. In *International Conference on Computer Vision*, volume 1, pages 704–709, Vancouver, Canada, July 2001.

[10] C.C. Slama, editor. *Manual of Photogrammetry, Fourth Edition*. American Society of Photogrammetry and Remote Sensing, Falls Church, Virginia, USA, 1980.

[11] S. Maybank and O. Faugeras. A theory of self-calibration of a moving camera. *International Journal of Computer Vision*, 8(2):123–151, 1992.

[12] B. Triggs. Autocalibration and the absolute quadric. In *Proceedings of the Conference on Computer Vision and Pattern Recognition, Puerto Rico, USA*, pages 609–614. IEEE Computer Society Press, June 1997.

[13] B. Triggs, P.F. McLauchlan, R.I. Hartley, and A. Fitzgibbon. Bundle adjustment — a modern synthesis. In B. Triggs, A. Zisserman, and R. Szeliski, editors, *Vision Algorithms: Theory and Practice*, volume 1883 of *Lecture Notes in Computer Science*, pages 298–372. Springer-Verlag, 2000.

[14] L. Van Gool, T. Moons, E. Pauwels, and A. Oosterlinck. Vision and lie’s approach to invariance. *Image and vision computing*, 13(4):259–277, May 1995.

[15] M. Werman, S. Banerjee, S. Dutta Roy, and M. Qiu. Robot localization using uncalibrated camera invariants. In *IEEE International Conference on Computer Vision and Pattern Recognition*, volume II, pages 353–359, Fort Collins, CO, June 1999.
<figure><img src="image_22.png" /></figure>

Unité de recherche INRIA Sophia Antipolis
2004, route des Lucioles - BP 93 - 06902 Sophia Antipolis Cedex (France)

Unité de recherche INRIA Lorraine : LORIA, Technopôle de Nancy-Brabois - Campus scientifique
615, rue du Jardin Botanique - BP 101 - 54602 Villers-lès-Nancy Cedex (France)

Unité de recherche INRIA Rennes : IRISA, Campus universitaire de Beaulieu - 35042 Rennes Cedex (France)

Unité de recherche INRIA Rhône-Alpes : 655, avenue de l'Europe - 38330 Montbonnot-St-Martin (France)

Unité de recherche INRIA Rocquencourt : Domaine de Voluceau - Rocquencourt - BP 105 - 78153 Le Chesnay Cedex (France)

Éditeur

INRIA - Domaine de Voluceau - Rocquencourt, BP 105 - 78153 Le Chesnay Cedex (France)

http://www.inria.fr

ISSN 0249-6399