# Estimation of Homography Dynamics on the Special Linear Group

Ezio Malis, Tarek Hamel, Robert Mahony, Pascal Morin

► To cite this version:

Ezio Malis, Tarek Hamel, Robert Mahony, Pascal Morin. Estimation of Homography Dynamics on the Special Linear Group. Springer London, 401, pp.133-150, 2010, Lecture Notes in Control and Information Sciences, 10.1007/978-1-84996-089-2_8. hal-04655112

HAL Id: hal-04655112

https://hal.science/hal-04655112v1

Submitted on 1 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Estimation of homography dynamics on the special linear group

Ezio Malis, Tarek Hamel, Robert Mahony, and Pascal Morin

## 1 Introduction

Visual servo control schemes use visual information obtained by one or multiple cameras as the primary measurement to regulate the motion of a robot [21, 11, 12, 6]. In the last decade, a number of visual servo control schemes have been proposed that extract homography image transformations between images of a planar scenes and use these as the primary visual information for a servo control problem [17, 7, 8]. A homography can be decomposed to explicitly reconstruct the pose (the translation and the rotation in Cartesian space) of the camera [9, 18] and the associated servo control task undertaken in Cartesian space [25, 16, 2, 24]. Alternatively, the control task can be defined in both image and Cartesian space; the rotation error is estimated explicitly and the translation error is expressed in the image [17, 7, 22, 8]. The resulting visual servo control algorithms are stable and robust [16] and do not depend on tracking of individual image features. Some recent work has been done on direct servo control of the homography matrix [1], an approach which offers considerable advantages in situations where the homography decomposition is ill-conditioned. A key component of this work is the identification of the group of homographies as a Lie-group isomorphic to the special linear group SL(3), an observation that has been known for some time in the computer vision community but had not been exploited before in the visual servo community.

In all cases, the performance of the closed-loop system depends on the quality of the homography estimates used as input to controller. In the case of visual servo control applications, the homographies must be computed in real-time with minimal computational overhead. Moreover, in such applications the homographies vary continuously and usually smoothly. It is natural, then, to consider using a dynamical observer (or filter) process in the closed-loop system to achieve temporal smoothing and averaging of the homography measurements. Such a process will reduce noise in the homography estimates, smoothing resulting closed-loop inputs and leading to improved performance, especially in visual servo applications. There has been a surge of interest recently in nonlinear observer design for systems with certain invariance properties [23, 5, 10, 15, 3] that have mostly been applied to applications in robotic vehicles [19, 20]. From these foundations there is an emerging framework for observer design for invariant systems on Lie groups [13, 4, 14]. The special linear group structure of the homographies [1] makes the homography observer problem an ideal application of these recent developments in observer theory.

In this chapter, we exploit the special linear Lie-group structure of the set of all homographies to develop a dynamic observer to estimate homographies on-line. The proposed homography observer is based on constant velocity invariant kinematics on the Lie group. We assume that the velocity is unknown and propose an integral extension of the nonlinear observer to obtain estimates for both the homography and the velocity. We prove the existence of a Lyapunov function for the system, and use this to prove almost global stability and local exponential stability around the desired equilibrium point. The proposed algorithm provides high quality temporal smoothing of the homography data along with a smoothed homography velocity estimate. The estimation algorithm has been extensively tested in simulation and on real data.

The chapter is organised into five sections followed by a short conclusion. The present introduction section is followed by Section 2 that provides a recap of the Lie group structure of the set of homographies. The main contribution of the paper is given in Section 3. Sections 4 and 5 provide an experimental study with simulated and real data.

Ezio Malis and Pascal Morin are with INRIA-Sophia Antipolis, France, Ezio.Malis@sophia.inria.fr and Pascal.Morin@sophia.inria.fr.

Tarek Hamel is with I3S-CNRS, Nice-Sophia Antipolis, France, thamel@i3s.unice.fr.

Robert Mahony is with School of Engineering, ANU, ACT, 0200, Australia, Robert.Mahony@anu.edu.au.
## 2 Theoretical background

A homography is a mapping between two images of a planar scene P. Let $p = (u, v)$ represent the pixel coordinates of a 3D point $\xi \in P$ as observed in the normalized image plane of a pinhole camera. Let $\mathcal{A}$ (resp. $\mathcal{B}$) denote projective coordinates for the image plane of a camera A (resp. B), and $\{A\}$ (resp. $\{B\}$) denote its frame of reference. A $(3 \times 3)$ homography matrix $H : \mathcal{A} \to \mathcal{B}$ defines the following mapping: $p^B = w(H, p^A)$, where

$$w(H, p) = \begin{bmatrix} (h_{11}u + h_{12}v + h_{13})/(h_{31}u + h_{32}v + h_{33}) \\ (h_{21}u + h_{22}v + h_{23})/(h_{31}u + h_{32}v + h_{33}) \end{bmatrix}$$

The mapping is defined up to a scale factor. That is, for any scaling factor $\mu \neq 0$, $p^B = w(\mu H, p^A) = w(H, p^A)$. The Lie-group $SL(3)$ is the set of real matrices $SL(3) = \{H \in \mathbb{R}^{3 \times 3} | \det(H) = 1\}$. If we suppose that the camera continuously observes the planar object, any homography can be represented by a homography matrix $H \in SL(3)$ such that:

$$H = \gamma K \left( R + \frac{tn^{\top}}{d} \right) K^{-1} \quad (1)$$

where $K$ is the upper triangular matrix containing the camera intrinsic parameters, $R$ is the rotation matrix representing the orientation of $\{B\}$ with respect to $\{A\}$, $t$ is the translation vector of coordinates of the origin of $\{B\}$ expressed in $\{A\}$, $n$ is the normal to the planar surface $P$ expressed in $\{A\}$, $d$ is the orthogonal distance of the origin of $\{A\}$ to the planar surface, and $\gamma$ is a scaling factor:

$$\gamma = \det \left( R + \frac{tn^{\top}}{d} \right)^{-\frac{1}{3}} = \left( 1 + \frac{n^{\top} R^{\top} t}{d} \right)^{-\frac{1}{3}}$$

Correspondingly, knowing the camera intrinsic parameters $K$, any full rank $3 \times 3$ matrix with unitary determinant can be decomposed according to (1) (see [9] for a numerical decomposition and [18] for the analytical decomposition). Note that there exist two possible solutions to the decomposition. The planar surface $P$ is parametrized by

$$P = \{ \xi \in \{A\} | n^{\top} \xi = d \}$$

For any two frames $\{A\}$ and $\{B\}$ whose origins lie on the same side of the planar surface $P$ then $n^{\top} R t > -d$ by construction and the determinant of the associated homography $\det(H) = 1$.

The map $w$ is a group action of $SL(3)$ on $\mathbb{R}^2$:

$$w(H_1, w(H_2, p)) = w(H_1 H_2, p)$$

where $H_1, H_2$ and $H_1 H_2 \in SL(3)$. The geometrical meaning of this property is that the 3D motion of the camera between views $\{A\}$ and $\{B\}$, followed by the 3D motion between views $\{B\}$ and $\{C\}$ is the same as the 3D motion between views $\{A\}$ and $\{C\}$.

**Remark 1.** The local parametrization given by (1) is singular when $\{A\}$ and $\{B\}$ are collocated. That is, when $t = 0$, the differential of the mapping defined by (1) is degenerate. Indeed, in this case the normal to the plane $n$ is not observable. The singularity of the parametrization does not affect the validity of the correspondence $\mathcal{H} \equiv SL(3)$, however, it does mean that the parametrization (1) is very poorly conditioned for homography matrices close to $SO(3)$. This is fundamental reasons why it is preferable to do both image based visual servo control and temporal smoothing directly on the homography group rather than extracting structure variables explicitly.

The Lie-algebra $sl(3)$ for $SL(3)$ is the set of matrices with trace equal to zero: $sl(3) = \{X \in \mathbb{R}^{3 \times 3} | \text{tr}(X) = 0\}$.

The adjoint operator is a mapping $\text{Ad} : SL(3) \times sl(3) \to sl(3)$ defined by

$$\text{Ad}_H X = HXH^{-1}, \quad H \in SL(3), X \in sl(3).$$

For any two matrices $A, B \in \mathbb{R}^{3 \times 3}$ the Euclidean matrix inner product and Frobenius norm are defined as

$$\langle\langle A, B \rangle\rangle = \text{tr}(A^T B), \quad \|A\| = \sqrt{\langle\langle A, A \rangle\rangle}$$

Let $\mathbb{P}$ denote the unique orthogonal projection of $\mathbb{R}^{3 \times 3}$ onto $sl(3)$ with respect to the inner product $\langle\langle \cdot, \cdot \rangle\rangle$
$$ \mathbb{P}(H) := \left( H - \frac{\mathrm{tr}(H)}{3} I \right) \in \mathrm{sl}(3). \qquad (2) $$

The projection onto the complementary subspace (the span of $I$ in $\mathbb{R}^{3 \times 3}$) is defined by

$$ \mathbb{P}^{\perp}(H) := H - \mathbb{P}(H) = \frac{\mathrm{tr}(H)}{3} I. \qquad (3) $$

Clearly one has $\langle\langle\mathbb{P}(H), \mathbb{P}^{\perp}(H)\rangle\rangle = 0$

### 3 Nonlinear observer on SL(3)

Consider the left invariant kinematics defined on SL(3)

$$ \dot{H} = HA \qquad (4) $$

where $H \in SL(3)$ and $A \in \mathrm{sl}(3)$. A general framework for nonlinear filtering on the special linear group is introduced. The theory is developed for the case where $A$ is assumed to be unknown and constant. The goal is to provide a set of dynamics for an estimate $\hat{H}(t) \in SL(3)$ of $H(t)$ and an estimate $\tilde{A}(t) \in \mathrm{sl}(3)$ of $A$ to drive the estimation error $\tilde{H} = \hat{H}^{-1}H$ to the identity matrix $I$, and the estimation error $\tilde{A} = A - \hat{A}$ to zero.

The estimator filter equation of $\hat{H}$ is posed directly on SL(3). It includes a correction term derived from the error $\tilde{H}$. We consider an estimator filter of the form

$$ \left\{ \begin{array}{l} \dot{\hat{H}} = \hat{H} (\mathrm{Ad}_{\tilde{H}} \hat{A} + \alpha(\hat{H}, H)), \quad \hat{H}(0) = \hat{H}_0, \\ \hat{A} = \beta(\hat{H}, H), \quad \hat{A}(0) = \hat{A}_0. \end{array} \right. \qquad (5) $$

This yields the following expression for the dynamics of the estimation error $(\tilde{H}, \tilde{A}) = (\hat{H}^{-1}H, A - \hat{A})$:

$$ \left\{ \begin{array}{l} \dot{\tilde{H}} = \tilde{H} (\tilde{A} - \mathrm{Ad}_{\tilde{H}^{-1}} \alpha) \\ \tilde{A} = -\beta \end{array} \right. \qquad (6) $$

with the arguments of $\alpha$ and $\beta$ omitted to lighten the notation. The main result of the paper is stated next.

**Theorem 1.** Assume that the matrix $A$ in (4) is constant. Consider the nonlinear estimator filter (5) along with the innovation $\alpha$ and the estimation dynamics $\beta$ defined as

$$ \left\{ \begin{array}{l} \alpha = -k_H \mathrm{Ad}_{\tilde{H}} \mathbb{P}(\tilde{H}^T (I - \tilde{H})), \quad k_H > 0 \\ \beta = -k_A \mathbb{P}(\tilde{H}^T (I - \tilde{H})), \quad k_A > 0 \end{array} \right. \qquad (7) $$

with the projection operator $\mathbb{P}: \mathbb{R}^{3 \times 3} \to \mathrm{sl}(3)$ defined by (2). Then, for the estimation error dynamics (6),

i) All solutions converge to $E = E_s \cup E_u$ with:

$$ \begin{aligned} E_s &= (I, 0) \\ E_u &= \{(\tilde{H}_0, 0)|\tilde{H}_0 = \lambda(I + (\lambda^{-3} - 1)vv^\top), v \in \mathbb{S}^2\} \end{aligned} $$

where $\lambda \approx -0.7549$ is the unique real solution of the equation $\lambda^3 - \lambda^2 + 1 = 0$.

ii) The equilibrium point $E_s = (I, 0)$ is locally exponentially stable.

iii) Any point of $E_u$ is an unstable equilibrium. More precisely, for any $(\tilde{H}_0, 0) \in E_u$ and any neighborhood $\mathcal{U}$ of $(\tilde{H}_0, 0)$, there exists $(\tilde{H}_1, \tilde{A}_1) \in \mathcal{U}$ such that the solution of System (6) issued from $(\tilde{H}_1, \tilde{A}_1)$ converges to $E_s$.

### Proof of Theorem 1:
*Proof of Part i)* : Let us consider the following candidate Lyapunov function

$$ 
\begin{aligned}
V(\tilde{H}, \tilde{A}) &= \frac{1}{2} \|I - \tilde{H}\|^2 + \frac{1}{2k_A} \|\tilde{A}\|^2 \\
&= \frac{1}{2} \mathrm{tr}((I - \tilde{H})^T (I - \tilde{H})) + \frac{1}{2k_A} \mathrm{tr}(\tilde{A}^T \tilde{A}),
\end{aligned}
\quad (8) $$

The derivative of $V$ along the solutions of System (6) is

$$ 
\begin{aligned}
\dot{V} &= -\mathrm{tr}((I - \tilde{H})^T \dot{\tilde{H}}) + \frac{1}{k_A} \mathrm{tr}(\tilde{A}^T \tilde{A}) \\
&= -\mathrm{tr}((I - \tilde{H})^T \tilde{H} \tilde{A} - (I - \tilde{H})^T \tilde{H} \mathrm{Ad}_{\tilde{H}^{-1}} \alpha) - \frac{1}{k_A} \mathrm{tr}(\tilde{A}^T \beta)
\end{aligned}
$$

Knowing that for any matrices $G \in SL(3)$ and $B \in sl(3)$, $\mathrm{tr}(B^T G) = \mathrm{tr}(B^T \mathbb{P}(G)) = \langle\langle B, \mathbb{P}(G) \rangle\rangle$, one obtains:

$$ \dot{V} = \langle\langle \mathbb{P}(\tilde{H}^T(I - \tilde{H})), \mathrm{Ad}_{\tilde{H}^{-1}} \alpha \rangle\rangle - \langle\langle \tilde{A}, \mathbb{P}(\tilde{H}^T(I - \tilde{H})) + \frac{1}{k_A} \beta \rangle\rangle $$

Introducing the expressions of $\alpha$ and $\beta$ (Eq. (7)) in the above equation yields

$$ \dot{V} = -k_H ||\mathbb{P}(\tilde{H}^T(I - \tilde{H}))||^2 \quad (9) $$

The derivative of the Lyapunov function is negative semi-definite, and equal to zero when $\mathbb{P}(\tilde{H}^T(I - \tilde{H})) = 0$. The dynamics of the estimation error is autonomous, i.e. it is given by

$$ \left\{ \begin{array}{l} \dot{\tilde{H}} = \tilde{H} (\tilde{A} + k_H \mathbb{P}(\tilde{H}^T(I - \tilde{H}))) \\ \dot{\tilde{A}} = k_A \mathbb{P}(\tilde{H}^T(I - \tilde{H})) \end{array} \right. \quad (10) $$

Therefore, we deduce from LaSalle's theorem that all solutions of this system converge to the largest invariant set contained in $\{ (\tilde{H}, \tilde{A}) | \mathbb{P}(\tilde{H}^T(I - \tilde{H})) = 0 \}$.

We now prove that, for System (10), the largest invariant set $E$ contained in $\{ (\tilde{H}, \tilde{A}) | \mathbb{P}(\tilde{H}^T(I - \tilde{H})) = 0 \}$ is equal to $E_s \cup E_u$.

We need to show that the solutions of System (10) belonging to $\{ (\tilde{H}, \tilde{A}) | \mathbb{P}(\tilde{H}^T(I - \tilde{H})) = 0 \}$ for all $t$ consist of all fixed points of $E_s \cup E_u$. Note that $E_s = (I, 0)$ is clearly contained in $E$. Let us thus consider such a solution $(\tilde{H}(t), \tilde{A}(t))$. First, we deduce from (10) that $\tilde{A}(t)$ is identically zero since $\mathbb{P}(\tilde{H}^T(t)(I - \tilde{H}(t)))$ is identically zero on the invariant set $E$ and therefore $\tilde{A}$ is constant. We also deduce from (10) that $\tilde{H}$ is solution to the equation $\tilde{H} = \tilde{H} \tilde{A}$. Note that at this point one cannot infer that $\tilde{H}$ is constant. Still, we omit from now on the possible time-dependence of $\tilde{H}$ to lighten the notation.

Since $\mathbb{P}(\tilde{H}^T(I - \tilde{H})) = 0$, we have that

$$ \tilde{H}^T(I - \tilde{H}) = \frac{1}{3} \mathrm{trace}(\tilde{H}^T(I - \tilde{H}))I \quad (11) $$

which means that $\tilde{H}$ is a symmetric matrix. Therefore, it can be decomposed as:

$$ \tilde{H} = UDU^T \quad (12) $$

where $U \in SO(3)$ and $D = \mathrm{diag}(\lambda_1, \lambda_2, \lambda_3) \in SL(3)$ is a diagonal matrix which contains the three real eigenvalues of $\tilde{H}$. Without loss of generality let us suppose that the eigenvalues are in increasing order: $\lambda_1 \le \lambda_2 \le \lambda_3$. Plugging equation (12) into equation (11), one obtains:

$$ D(I - D) = \frac{1}{3} \mathrm{trace}(D(I - D))I $$

Knowing that $\mathrm{det}(D) = 1$, the $\lambda_i$'s satisfy the following equations:
$$ \lambda_1(1 - \lambda_1) = \lambda_2(1 - \lambda_2) \qquad (13) $$

$$ \lambda_2(1 - \lambda_2) = \lambda_3(1 - \lambda_3) \qquad (14) $$

$$ \lambda_3 = 1/(\lambda_1\lambda_2) \qquad (15) $$

which can also be written as follows:

$$ \lambda_1 - \lambda_2 = (\lambda_1 - \lambda_2)(\lambda_1 + \lambda_2) \qquad (16) $$

$$ \lambda_1 - \lambda_3 = (\lambda_1 - \lambda_3)(\lambda_1 + \lambda_3) \qquad (17) $$

$$ \lambda_3 = 1/(\lambda_1\lambda_2) \qquad (18) $$

First of all, let us remark that if $\lambda_1 = \lambda_2 = \lambda_3$ then $\lambda_1 = \lambda_2 = \lambda_3 = 1$. This solution is associated with the equilibrium point $E_s = (I, 0)$.

If $\lambda_1 = \lambda_2 < \lambda_3$ then:

$$ 1 = \lambda_2 + \lambda_3 \qquad (19) $$

$$ \lambda_3 = 1/(\lambda_2^2) \qquad (20) $$

where $\lambda_2 \approx -0.7549$ is the unique real solution of the equation $\lambda_2^3 - \lambda_2^2 + 1 = 0$. This solution is associated with the equilibrium set $E_u$.

If $\lambda_1 < \lambda_2 = \lambda_3$ then:

$$ 1 = \lambda_1 + \lambda_2 \qquad (21) $$

$$ \lambda_1 = 1/\lambda_2^2 \qquad (22) $$

so that $\lambda_2$ is also solution of the equation $\lambda_2^3 - \lambda_2^2 + 1 = 0$. But this is impossible since we supposed $\lambda_1 < \lambda_2$ and the solution of the equation is such that $-1 < \lambda_2 < 0$ and $0 < \lambda_1 = 1/\lambda_2^2 < 1$.

If $\lambda_1 \neq \lambda_2 \neq \lambda_3$, then:

$$ 1 = \lambda_1 + \lambda_2 \qquad (23) $$

$$ 1 = \lambda_1 + \lambda_3 \qquad (24) $$

$$ \lambda_3 = 1/(\lambda_1\lambda_2) \qquad (25) $$

which means that $\lambda_2 = \lambda_3$. This is in contradiction with our initial hypothesis.

In conclusion, $\tilde{H}$ has two equal negative eigenvalues $\lambda_1 = \lambda_2 = \lambda$ ($\lambda \approx -0.7549$ is the unique real solution of the equation $\lambda^3 - \lambda^2 + 1 = 0$) and the third one is $\lambda_3 = 1/\lambda^2$. Writing the diagonal matrix D as follows:

$$ D = \lambda(I + (\lambda^{-3} - 1)e_3 e_3^\top) $$

and plugging this equation into equation (12), the homography for the second solution ($\lambda_1 = \lambda_2$) can be expressed as follows:

$$ \tilde{H} = \lambda(I + (\lambda^{-3} - 1)(Ue_3)(Ue_3)^\top) $$

Setting $v = Ue_3$, we finally find that $\tilde{H}$ must have the following form:

$$ \tilde{H} = \lambda(I + (\lambda^{-3} - 1)vv^\top) $$

where $v$ is a unitary vector: $\|v\| = 1$ and $\lambda$ is the unique real constant value that verifies the equation $\lambda^3 - \lambda^2 + 1 = 0$.

It remains to show that $\tilde{A} = 0$. The inverse of $\tilde{H}$ is

$$ \tilde{H}^{-1} = \lambda^{-1}(I + (\lambda^3 - 1)vv^\top) $$

The derivative of $\tilde{H}$ is

$$ \dot{\tilde{H}} = \lambda(\lambda^{-3} - 1)(\dot{v}v^\top + v\dot{v}^\top) $$
so that

$$ \tilde{A} = \tilde{H}^{-1} \tilde{\dot{H}} = (\lambda^{-3} - 1)(I + (\lambda^3 - 1)\nu\nu^{\top})(\dot{\nu}\nu^{\top} + \nu\dot{\nu}^{\top}) $$

Knowing that $\nu^{\top}\dot{\nu} = 0$, this equations becomes:

$$ \tilde{A} = (\lambda^{-3} - 1)(\dot{\nu}\nu^{\top} + \nu\dot{\nu}^{\top} + (\lambda^3 - 1)\nu\dot{\nu}^{\top}) $$

and knowing that $\lambda^3 = \lambda^2 - 1$, we obtain:

$$ \tilde{A} = (\lambda^{-3} - 1)(\dot{\nu}\nu^{\top} + \nu\dot{\nu}^{\top} + (\lambda^2 - 2)\nu\dot{\nu}^{\top}) \quad (26) $$

$$ = (\lambda^{-3} - 1)(\dot{\nu}\nu^{\top} - \nu\dot{\nu}^{\top} + \lambda^2\nu\dot{\nu}^{\top}) \quad (27) $$

Since $\dot{\nu}\nu^{\top} - \nu\dot{\nu}^{\top} = [\nu \times \dot{\nu}]_{\times} = [[\nu]_{\times}\dot{\nu}]_{\times}^{1}$, we finally obtain

$$ \tilde{A} = (\lambda^{-3} - 1)([[\nu]_{\times}\dot{\nu}]_{\times} + \lambda^2\nu\dot{\nu}^{\top}) $$

Since $[[\nu]_{\times}\dot{\nu}]_{\times}$ is a skew-symmetric matrix, the diagonal elements of $\tilde{A}$ are $a_{ii} = (\lambda^{-3} - 1)\lambda^2\nu_i\dot{\nu}_i$. Knowing that each $a_{ii}$ is constant we have two possible cases. The first one is $a_{ii} = 0$ for each $i$. Then $\nu$ is constant so that $\tilde{H}$ is also constant and $\tilde{A} = 0$. If there exists $i$ such that $a_{ii} \neq 0$, then there exists $i$ such that $a_{ii} < 0$. This is due to the fact that $\tilde{A} \in \mathfrak{sl}(3)$ and therefore $\sum_i a_{ii} = 0$. In this case, the corresponding $\nu_i$ diverges to infinity because $\nu_i\dot{\nu}_i$ is a strictly positive constant. This contradicts the fact that $\|\nu\| = 1$. This concludes the proof of Part i) of the theorem.

*Proof of Part ii)*: We compute the linearization of System (10) at $E_s = (I,0)$. Let us define $X_1$ and $X_2$ as elements of sl(3) corresponding to the first order approximations of $\tilde{H}$ and $\tilde{A}$ around (I,0):

$$ \tilde{H} \approx (I + X_1), \quad \tilde{A} \approx X_2 $$

Substituting these approximations into (10) and discarding all terms quadratic or higher order in $(X_1, X_2)$ yields

$$ \begin{pmatrix} \dot{X}_1 \\ \dot{X}_2 \end{pmatrix} = \begin{pmatrix} -k_H I_3 & I_3 \\ -k_A I_3 & 0 \end{pmatrix} \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} \quad (28) $$

Since $k_H, k_A > 0$, the linearized error system is exponentially stable. This proves the local exponential stability of the equilibrium (I,0).

*Proof of Part iii)*: First, we remark that the function $V$ is constant and strictly positive on the set $E_u$. This can be easily verified from (8) and the definition of $E_u$, using the fact that on this set $\tilde{A} = 0$, $\tilde{H}^T \tilde{H} = \tilde{H}^2 = \lambda^2 I + (\frac{1}{\lambda^2} - \lambda)\nu\nu^T$, and tr($\nu\nu^T$) = 1 since $\|\nu\| = 1$. We denote by $V_u$ the value of $V$ on $E_u$. The fact that $V_u$ is strictly positive readily implies (in accordance with Part ii)) that $E_s$ is an asymptotically stable equilibrium, since $V$ is non-increasing along the system's solutions, and each of them converges to $E_s \cup E_u$. Using the same arguments, the proof of Part iii) reduces to showing that for any point $(\tilde{H}_0, 0) \in E_u$, and any neighborhood $\mathcal{U}$ of this point, one can find $(\tilde{H}_1, \tilde{A}_1) \in \mathcal{U}$ such that

$$ V(\tilde{H}_1, \tilde{A}_1) < V_u \quad (29) $$

Let $\tilde{H}(\cdot)$ denote a smooth curve on SL(3), solution of $\tilde{H} = \tilde{H}C$ with $C$ a constant element of sl(3) that will be specified latter on. We also assume that $(\tilde{H}(0), 0) \in E_u$. Let $f(t) = \|I - \tilde{H}(t)\|^2/2$ so that, by (8), $f(0) = V_u$. The first derivative of $f$ is given by

$$ \begin{aligned} \dot{f}(t) &= -\mathrm{tr}((I - \tilde{H}(t))^T \tilde{H}(t)) \\ &= -\mathrm{tr}((I - \tilde{H}(t))^T \tilde{H}(t)C) \\ &= -\langle \mathbb{P}(\tilde{H}^T(t)(I - \tilde{H}(t))), C \rangle \end{aligned} $$

For all elements $(\tilde{H}_0, 0) \in E_u$, one has $\mathbb{P}(\tilde{H}_0^T(I - \tilde{H}_0)) = 0$, so that $\dot{f}(0) = 0$. We now calculate the second order derivative of $f$:

<footnote n="1">¹ $[\nu]_{\times}$ represents the skew-symmetric matrix associated with the cross-product by $\nu \in \mathbb{R}^3$, i.e. $[\nu]_{\times}y = \nu \times y, \forall y \in \mathbb{R}^3$.</footnote>
$$ \begin{aligned} \ddot{f}(t) &= \text{tr}(\tilde{H}(t)^T \tilde{H}(t)) - \text{tr}((I - \tilde{H}(t))^T \tilde{H}(t)) \\ &= \text{tr}(\tilde{H}(t)^T \tilde{H}(t)) - \text{tr}((I - \tilde{H}(t))^T \tilde{H}(t)C) \end{aligned} $$

where we have used the fact that C is constant. Evaluating the above expression at $t = 0$ and replacing $\tilde{H}(0)$ by its value $\tilde{H}(0)C$ yields

$$ \ddot{f}(0) = \|\tilde{H}(0)C\|^2 - \text{tr}((I - \tilde{H}(0))^T \tilde{H}(0)C^2) \quad (30) $$

When $(\tilde{H}_0, 0) \in E_u$, one has

$$ \tilde{H}_0^2 = \lambda^2 I + \left(\frac{1}{\lambda^2} - \lambda\right) v v^T = \tilde{H}_0 + (\lambda^2 - \lambda) I $$

Therefore, we deduce from (30) that

$$ \ddot{f}(0) = \|\tilde{H}(0)C\|^2 + \lambda(\lambda - 1)\text{tr}(C^2) \quad (31) $$

Since $(\tilde{H}(0), 0) \in E_u$, there exists $v \in \mathbb{S}^2$ such that $\tilde{H}(0) = \lambda I + (\frac{1}{\lambda^2} - \lambda)v v^T$. From this expression and using the fact that $\lambda^3 - \lambda^2 + 1 = 0$, one verifies that

$$ \|\tilde{H}(0)C\|^2 = \lambda^2 \|C\|^2 + (\frac{1}{\lambda^2} - \lambda)\text{tr}(C^T v v^T C) \quad (32) $$

Now let us set $C = [v]_\times$, the skew-symmetric matrix associated with the vector $v$. Clearly, $C \in \mathfrak{sl}(3)$. Then, it follows from (31) and (32) that

$$ \begin{aligned} \ddot{f}(0) &= \lambda^2 \|C\|^2 + \lambda(\lambda - 1)\text{tr}(C^2) \\ &= \lambda^2 \text{tr}(v_\times^T v_\times) + \lambda(\lambda - 1)\text{tr}((v_\times)^2) \\ &= -\lambda^2 \text{tr}((v_\times)^2) + \lambda(\lambda - 1)\text{tr}((v_\times)^2) \\ &= -\lambda \text{tr}((v_\times)^2) = 2\lambda \|v\|^2 = 2\lambda < 0 \end{aligned} $$

Therefore, there exists $t_1 > 0$ such that for any $t \in (0, t_1)$,

$$ \begin{aligned} f(t) &\approx f(0) + t \ddot{f}(0) + t^2/2 \ddot{f}(0) \\ &\approx v_u + t^2/2 \ddot{f}(0) < v_u \end{aligned} $$

Eq. (29) follows by setting $(\tilde{H}_1, A_1) = (\tilde{H}(t), 0)$ with $t \in (0, t_1)$ chosen small enough so as to have $(\tilde{H}(t), 0) \in \mathcal{W}$. This concludes the proof of Part iii) and the proof of the theorem.

## 4 Simulations with ground truth

We validated the proposed observer with several simulations. In this section, we illustrate and discuss two simulations results. We use the known ground truth to assess the quality of the homography and velocity estimations.

In order to simulate a real experiment, we build a sequence of reference homographies, starting from an initial homography $H_0 \in SL(3)$. The reference set of homographies was built using the following formula:

$$ H_{k+1} = H_k \exp(A \Delta t + Q_k \Delta t) $$

where $A \in \mathfrak{sl}(3)$ is a constant velocity, $Q_k \in \mathfrak{sl}(3)$ is a random matrix with Gaussian distribution, and $\Delta t$ is the sampling time (in the simulation we set the variance to $\sigma = 0.1$). By building the homographies in this way, we guarantee that the measured $H_k \in SL(3), \forall k$.

We implemented a discretized observer in order to process the data. In all examples the gains of the observer were set to $k_H = 2$ and $k_A = 1$.

Fig. 1 shows the elements of the measured and estimated homography matrices. Fig. 2 shows the elements of the associated homography velocities. In this simulation the initial “error” for the homography is chosen at random and it is very large. The
initial velocity estimate $\hat{A}_0$ is set to zero. Fig. 1 shows that after a fast transient the estimated homography converges towards the measured homography. Fig. 2 shows that the estimated velocity also converges towards the true one.

Fig. 3 and 4 illustrate the instability of the critical points. In this simulation, the initial estimation error for the homography matrix is chosen at a critical point. The velocity estimate is again set to zero. Fig. 3 shows that the critical point is unstable: a small noise allows the estimated homography to converge towards the measured homography. Fig. 4 shows that the estimated velocity converges towards the true velocity.

# 5 Experiments with real data

In this section, we present results obtained with real data. In the first image the user selects a rectangular area of interest. The homographies that transform the area of interest in the current image are measured using the ESM visual tracking software<fnref n="2" /> [1]. Fig. 5 shows four images extracted from the sequence Corkes. The first image in the figure shows a rectangle containing the area of interest that must be tracked in all the images of the video sequence. For each image of the sequence, the output of the ESM visual tracking algorithm is the homography that encodes the transformation of each pixels of the rectangular area from the current to the first image.

The measured homographies are the input of the proposed nonlinear observer. In this experiment the gains were $k_H = 5$ and $k_A = 1$. The filtering effect of the observer on the estimated homography are visible in Fig. 6.

In this experiment with real data, the velocity $A$ is unknown and not constant. Nevertheless, the observer provides a smoothed estimation of the homography velocity, as illustrated on Fig. 7.

# 6 Conclusion

In this paper, we proposed an observer for the homographies defined on $SL(3)$ and their velocities defined on $sl(3)$. We proved that the observer is almost globally stable. We also proved that isolated critical points exist but that they are far from the equilibrium point and unstable. We performed several simulations with ground truth to validate the theoretical results. Experiments with real data show that the observer performs well even when the constant velocity assumption does not hold.

**Acknowledgments:** The authors gratefully acknowledge the contribution of INRIA, CNRS and ANU. This research was partly supported by the Australian Research Council through discovery grant DP0987411 and the PEGASE EU project.

<footnote n="2">² Available for download at http://esm.gforge.inria.fr.</footnote>
References

1. S. Benhimane and E. Malis. Homography-based 2d visual tracking and servoing. *International Journal of Robotic Research*, 26(7): 661–676, 2007.

2. S. Benhimane, E. Malis, P. Rives, and J. R. Azinheira. Vision-based control for car platooning using homography decomposition. *IEEE Conf. on Robotics and Automation*, pages 2173–2178, 2005.

3. S. Bonnabel, P. Martin, and P. Rouchon. Symmetry-preserving observers. *IEEE Trans. on Automatic Control*, 53 (11): 2514–2526, 2008.

4. S. Bonnabel, P. Martin, and P. Rouchon. Non-linear observer on lie groups for left-invariant dynamics with right-left equivariant output. *IFAC World Congress*, pages 8594–8598, 2008.

5. S. Bonnabel and P. Rouchon. *Control and Observer Design for Nonlinear Finite and Infinite Dimensional Systems*, volume 322 of *Lecture Notes in Control and Information Sciences*, chapter On Invariant Observers, pages 53–67. Springer-Verlag, 2005.

6. F. Chaumette and S. Hutchinson. Visual servo control, part ii: Advanced approaches. *IEEE Robotics and Automation Magazine*, 14(1): 109–118, 2007.

7. K. Deguchi. Optimal motion control for image-based visual servoing by decoupling translation and rotation. *IEEE/RSJ Conf. on Intelligent Robots and Systems*, pages 705–711, 1998.

8. Y. Fang, W. Dixon, D. Dawson, and P. Chawda. Homography-based visual servoing of wheeled mobile robots. *IEEE Trans. on Systems, Man, and Cybernetics - Part B*, 35(5): 1041–1050, 2005.

9. O. Faugeras and F. Lustman. Motion and structure from motion in a piecewise planar environment. *International Journal of Pattern Recognition and Artificial Intelligence*, 2(3): 485–508, 1988.

10. T. Hamel and R. Mahony. Attitude estimation on SO(3) based on direct inertial measurements. *IEEE Conf. on Robotics and Automation*, pages 2170–2175, 2006.

11. Hashimoto, editor. *Visual Servoing: Real Time Control of Robot manipulators based on visual sensory feedback*, volume 7 of *World Scientific Series in Robotics and Automated Systems*. World Scientific Press, 1993.

12. S. Hutchinson, G. D. Hager, and P. I. Corke. A tutorial on visual servo control. *IEEE Trans. on Robotics and Automation*, 12(5): 651–670, 1996.

13. C. Lageman, R. Mahony, and J. Trumpf. State observers for invariant dynamics on a lie group. *Conf. on the Mathematical Theory of Networks and Systems*, 2008.

14. C. Lageman, J. Trumpf, and R. Mahony. Gradient-like observers for invariant dynamics on a lie group. *IEEE Trans. on Automatic Control*, to appear.

15. R. Mahony, T. Hamel, and J.-M. Pflimlin. Non-linear complementary filters on the special orthogonal group. *IEEE Trans. on Automatic Control*, 53(5): 1203–1218, 2008.

16. E. Malis and F. Chaumette. Theoretical improvements in the stability analysis of a new class of model-free visual servoing methods. *IEEE Trans. on Robotics and Automation*, 18(2): 176–186, 2002.

17. E. Malis, F. Chaumette, and S. Boudet. 2 1/2 d visual servoing. *IEEE Trans. on Robotics and Automation*, 15(2):234–246, 1999.

18. E. Malis and M. Vargas. Deeper understanding of the homography decomposition for vision-based control. Research Report 6303, INRIA, 2007.

19. P. Martin and E. Salaün. Invariant observers for attitude and heading estimation from low-cost inertial and magnetic sensors. *IEEE Conf. on Decision and Control*, pages 1039–1045, 2007.

20. P. Martin and E. Salaün. An invariant observer for earth-velocity-aided attitude heading reference systems. *IFAC World Congress*, pages 9857–9864, 2008.

21. C. Samson, M. Le Borgne, and B. Espiau. *Robot Control: the Task Function Approach*, volume 22 of *Oxford Engineering Science Series*. Clarendon Press, 1991.

22. D. Suter, T. Hamel, and R. Mahony. Visual servo control using homography estimation for the stabilization of an X4-flyer, *Conference on Decision and Control, CDC-2002*, 2002.

23. J. Thienel and R. M. Sanner. A coupled nonlinear spacecraft attitude controller and observer with an unknown constant gyro bias and gyro noise. *IEEE Trans. on Automatic Control*, 48(11): 2011–2015, 2003.

24. M. Vargas and E. Malis. Visual servoing based on an analytical homography decomposition. *IEEE Conf. on Decision and Control and European Control Conf.*, pages 5379–5384, 2005.

25. W. J. Wilson, C. C. W. Hulls, and G. S. Bell. Relative end-effector control using cartesian position-based visual servoing. *IEEE Trans. on Robotics and Automation*, 12(5): 684–696, 1996.
<figure><img src="image_2.png" /><figcaption>Fig. 1 Plain line: the measured homography matrix $H$. Dashed line: the observed homography $\hat{H}$.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>**Fig. 2** Plain line: the true homography velocity $A$. Dashed line the observed homography velocity $\hat{A}$.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 3 Unstability of critical points. Plain line: the measured homography matrix $H$. Dashed line: the observed homography $\hat{H}$.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Fig. 4 Unstability of critical points. Plain line: the true homography velocity $A$. Dashed line the observed homography velocity $\hat{A}$.</figcaption></figure>
image 1

<figure><img src="image_6.png" /></figure>

image 50

<figure><img src="image_7.png" /></figure>

image 100

<figure><img src="image_8.png" /></figure>

image 180

<figure><img src="image_9.png" /><figcaption>**Fig. 5** Images from the Corkes sequence. The quadrilateral represents the tracked area. The visual tracking is correctly performed in real-time. However, the noise in the images and modeling errors affect the accuracy of the measured homographies.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>**Fig. 6** Corks sequence. Each plot represents an element of the (3×3) homography matrix. Plain line: the measured homography matrix *H*. Dashed line: the observed homography *H*.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 7 Corks sequence. Each plot represents an element of the observed homography velocity matrix $\hat{A}$.</figcaption></figure>