# HYBRID VISION-BASED ROBOT CONTROL ROBUST TO LARGE CALIBRATION ERRORS ON BOTH INTRINSIC AND EXTRINSIC CAMERA PARAMETERS

Ezio Malis

I.N.R.I.A.

2004, route des Lucioles - B.P. 93,

06902 Sophia Antipolis Cedex, France.

fax: +33 (0)4 92 38 76 43

e-mail: Ezio.Malis@sophia.inria.fr

**Keywords:** Vision, Robotics, Control, Stability, Robustness.

## Abstract

This paper concerns the control of a camera, mounted on the end-effector of a robot manipulator, using a hybrid visual servoing scheme. Hybrid vision-based control consists in using information available directly at the image level jointly with the information reconstructed from two views of a rigid object (i.e. the displacement of the camera). The aim of this paper is to design a simple control law and to provide sufficient conditions for its global asymptotic stability. The control law is proved to be robust to a large amount of calibration errors. Contrarily to previous work, we suppose that calibration errors concern both intrinsic and extrinsic camera parameters.

## 1 Introduction

Visual servoing is a promising method to control dynamic systems using the information provided by visual sensors. In this paper, we focus on the control of a camera mounted on the end-effector of a robot manipulator. The computer vision system control the robot's end-effector in order to position it with respect to an object. Many different visual servoing approaches have been proposed [6] [7] and the corresponding control laws are generally stable in absence of calibration errors on the parameters of the system. In general, it can be observed experimentally that the control laws are robust in presence of calibration errors especially when the initial camera displacement is not too large. However, the theoretical analysis of the stability and robustness of a control law is generally impossible. For example, if we consider the standard image-based approach proposed in [4], it is extremely difficult to find analytical stability conditions, which can be exploited in practice, since the system is highly coupled [3]. Even if the behaviour of the image features is generally good, it is possible to reach local minima (which means that the final robot position does not correspond to the desired one) and/or the image Jacobian can become singular during the servoing (which of course leads to an unstable behaviour) [2]. Position-based approaches are naturally decoupled

since they are based on the reconstruction of the camera position with respect to the object. In this case the control of the rotation is decoupled from the control of the translation and the stability analysis could be simplified. However, when the position of the camera is computed using a CAD model of the object [12], it is extremely difficult to have an analytical expression of the estimated information as a function of the calibration errors. Even if it is possible to provide a stable control law in the presence of small calibration errors [13], it seems hard to know how small they should be in order to ensure the convergence of the control law.

Thanks to recent results in computer vision, it is possible to estimate the camera displacement between two views without knowing any CAD model of the considered object [5] [9]. Visual servoing approaches exploiting these results will be called model-free in order to distinguish them from previous ones which are model-dependent. With model-free approaches it is sufficient to know that several features correspond in the two images to recover the rotation of the camera and the translation up to a scale factor. As a consequence, the control of the rotation of the camera can be decoupled from the control of the translation. The translation of the camera can be controlled directly, as it was proposed in [1] (using thus a model-free position-based approach), or using a hybrid approach, called 2 1/2 D visual servoing, as it was proposed in [8] (it consists in combining visual features obtained directly from the image with features expressed in the Euclidean space). One interesting property of the 2 1/2 D visual servoing is that it is possible to prove the stability and the robustness of the control law. It is also possible to have the analytical expression of the robustness domain (i.e. the exact knowledge of the amount of calibration errors that can be tolerated by the system). More particularly, in [8] we proved the necessary and sufficient conditions for local asymptotic stability in the presence of calibration errors on the intrinsic parameters of the camera. In this paper, the sufficient conditions for the global asymptotic stability of the 2 1/2 D visual servoing are obtained by using a different control law. Contrarily to the previous work, we suppose here that calibration errors concern both intrinsic and extrinsic camera parameters.
## 2 Model of the eye-in-hand system

An eye-in-hand system is composed by a camera mounted on the end-effector of a robot manipulator. The model of the system consists of two different sets of parameters. The camera intrinsic parameters concern the geometry of the vision sensor while the camera extrinsic parameters concern the position of the camera with respect to the absolute and the end-effector frames.

### 2.1 Camera intrinsic parameters

Pinhole cameras perform a perspective projection. Let the point $C^*$ be a center of projection (see Figure 1). Suppose that $C^*$ coincide with the origin $O^*$ of the absolute frame $F^*$. Frame $F^*$ will be called the “reference” frame. Let the plane of projection be parallel to the plane $(\vec{x}, \vec{y})$. The distance $f^*$ between $O^*$ and the plane of projection is called the focal length.

<figure><img src="image_1.png" /><figcaption>Figure 1: Perspective projection of a point. The parameters involved in the projection are roughly known.</figcaption></figure>

Let the point $C$ be a different centre of projection coinciding with the origin of frame $F$ (see Figure 1). The rotation and the translation between frames $F^*$ and $F$ are respectively $\mathbf{R}$ and $\mathbf{t}$. A 3D point $\mathcal{P}$, with homogeneous coordinates $\mathbf{X} = [X \ Y \ Z \ 1]^T$ is projected to the image points $\mathbf{p}^* = [u^* \ v^* \ 1]^T$ and $\mathbf{p} = [u \ v \ 1]^T$ as follows:

$$ \zeta^* \mathbf{p}^* = \mathbf{A} \begin{bmatrix} \mathbf{I} & \mathbf{0} \end{bmatrix} \mathbf{X} \quad (1) $$

$$ \zeta \mathbf{p} = \mathbf{A} \begin{bmatrix} \mathbf{R} & \mathbf{t} \end{bmatrix} \mathbf{X} \quad (2) $$

where $\zeta^* = Z/f^*$, $\zeta = Z/f$ and:

$$ \mathbf{A} = \begin{bmatrix} f k_u & -f k_u \cot(\theta) & u_0 \\ 0 & f k_v / \sin(\theta) & v_0 \\ 0 & 0 & 1 \end{bmatrix} \quad (3) $$

is the triangular matrix containing the intrinsic parameters of the camera: $u_0$ and $v_0$ are the coordinates in pixels of the principal point (i.e. the intersection of the $\vec{z}$ axis and the projection plane), $f$ is the focal length (in meters), $k_u$ and $k_v$ are the magnifications respectively in the $\vec{u}$ and $\vec{v}$ direction (in pixels/meters), and $\theta$ is the angle between these axes. The intrinsic parameters of the camera are only roughly known. Instead of the true matrix $\mathbf{A}$, an approximation $\hat{\mathbf{A}}$ will be used in the control law.

### 2.2 Camera extrinsic parameters

Eliminating $\mathcal{X}$ from equations (1) and (2) we obtain the fundamental equation which links the perspective projections of the same 3D point in two different images:

$$ \frac{\zeta}{\zeta^*} \mathbf{p} = \mathbf{G}_{\infty}(\mathbf{p}^* - \frac{1}{\zeta^*} \mathbf{e}^*) \quad (4) $$

where $\mathbf{G}_{\infty} = \mathbf{A}\mathbf{R}\mathbf{A}^{-1}$ is the collineation of the plane at infinity and $\mathbf{e}^*$ is the epipole in the reference image. Let us suppose that it is possible to estimate $\mathbf{G}_{\infty}$ from a sequence of images of a static object [9]. Using the approximation $\hat{\mathbf{A}}$ of the camera intrinsic parameters, the estimated rotation of the camera is $\hat{\mathbf{R}} = \hat{\mathbf{A}}^{-1}\mathbf{G}_{\infty}\hat{\mathbf{A}} = \tilde{\mathbf{A}}\hat{\mathbf{R}}\tilde{\mathbf{A}}^{-1}$ where $\tilde{\mathbf{A}} = \hat{\mathbf{A}}^{-1}\mathbf{A}$. Since $\hat{\mathbf{R}}$ is similar to $\mathbf{R}$, it is possible to estimate the angle of rotation $\hat{\theta} = \theta$ and the axis of rotation $\hat{\mathbf{u}} = \tilde{\mathbf{A}}\mathbf{u}/\|\tilde{\mathbf{A}}\mathbf{u}\|$. From equation (4) it is also possible to compute $\frac{\zeta}{\zeta^*} = \frac{\mathbf{z}}{\mathbf{z}^*}$. These extrinsic parameters of the camera will be used in the control law.

Another set of camera extrinsic parameters consist of the position of the camera with respect the end-effector of the robot manipulator. Let $\mathcal{F}_e$ the frame attached to the end-effector. The rotation and the translation between frame $\mathcal{F}_e$ and $\mathcal{F}$ are respectively $\mathbf{R}_e$ and $\mathbf{t}_e$ (see Figure 2).

<figure><img src="image_2.png" /><figcaption>Figure 2: The displacement between the camera frame $\mathcal{F}$ and the end-effector frame $\mathcal{F}_e$ is roughly known.</figcaption></figure>

Since the movement of the camera is generated by the movement of the robot, the control law must be expressed in the end-effector frame. In our previous work [8], $\mathbf{R}_e$ and $\mathbf{t}_e$ were supposed to be exactly known. In this paper, the extrinsic parameters of the camera are roughly known and approximations $\hat{\mathbf{R}}_e$ and $\hat{\mathbf{t}}_e$ are used. We will suppose that the Jacobian of the robot is always full rank and perfectly known. Thus, the control input will be the velocity of the end-effector $\mathbf{v}_e = [\nu \ \omega]^T$ where $\nu$ is the velocity of translation and $\omega$ the velocity of rotation.

## 3 Hybrid vision-based robot control

Visual servoing can be achieved by regulating to zero a task function [11] built from the information issued by the camera. Since the rotation is decoupled from the translation, in our case the task function $\mathbf{e} = [\mathbf{e}_\nu \ \mathbf{e}_\omega]^T$ is composed by the following two $(3 \times 1)$ vectors:

$$ \mathbf{e}_\nu = [u - u^* \ v - v^* \ w - w^*]^T \quad (5) $$

$$ \mathbf{e}_\omega = \mathbf{u}\theta \quad (6) $$
where $u$ and $v$ are the coordinates of a point in the image, $w = \log(Z)$, $\mathbf{u}$ and $\theta$ are respectively the axis and the angle of rotation obtained from $\mathbf{R}$. The time derivative of the task function is:

$$ \dot{\mathbf{e}} = \mathbf{L} \mathbf{v}_c = \mathbf{L} \mathbf{W} \mathbf{v}_e \quad (7) $$

where:

* $\mathbf{L}$ is called the “task Jacobian matrix”. It links the variation of the task function with respect to the camera velocity $\mathbf{v}_c$. This matrix is upper block-triangular and depends on the camera parameters, the image coordinates of a point of the object and its depth $Z = e^w$:

$$ \mathbf{L} = \begin{bmatrix} e^{-w} \mathbf{L}_{\nu} & \mathbf{L}_{\nu} \mathbf{L}_{\nu\omega} \\ 0 & \mathbf{L}_{\omega} \end{bmatrix} \quad (8) $$

where:

$$ \mathbf{L}_{\nu} = \begin{bmatrix} -1 & 0 & u \\ 0 & -1 & v \\ 0 & 0 & -1 \end{bmatrix} \mathbf{A} \quad (9) $$

$$ \mathbf{L}_{\nu\omega} = [\mathbf{A}^{-1}\mathbf{p}]_{\times}^{T} \quad (10) $$

$$ \mathbf{L}_{\omega} = \mathbf{I} - \frac{\theta}{2} [\mathbf{u}]_{\times} + \left( 1 - \frac{\theta}{\tan(\frac{\theta}{2})} \right) [\mathbf{u}]_{\times}^{2} \quad (11) $$

* $\mathbf{W}$ is the matrix transforming the end-effector velocity $\mathbf{v}_e$ in the camera velocity: $\mathbf{v}_c = \mathbf{W} \mathbf{v}_e$. This matrix is upper block-triangular and contains the rotation $\mathbf{R}_e$ and the translation $\mathbf{t}_e$ between the camera frame $\mathcal{F}$ and the end-effector frame $\mathcal{F}_e$:

$$ \mathbf{W} = \begin{bmatrix} \mathbf{W}_{\nu} & \mathbf{W}_{\nu} \mathbf{W}_{\nu\omega} \\ 0 & \mathbf{W}_{\omega} \end{bmatrix} \quad (12) $$

where:

$$ \mathbf{W}_{\nu} = \mathbf{R}_e \quad (13) $$

$$ \mathbf{W}_{\nu\omega} = [\mathbf{R}_e^T \mathbf{t}_e]_{\times} \quad (14) $$

$$ \mathbf{W}_{\omega} = \mathbf{R}_e \quad (15) $$

Equation (7) can be separated into two equations:

$$ \dot{\mathbf{e}}_{\nu} = \mathbf{L}_{\nu} \mathbf{W}_{\nu} (e^{-w}\nu + \mathbf{M} \omega) \quad (16) $$

$$ \dot{\mathbf{e}}_{\omega} = \mathbf{L}_{\omega} \mathbf{W}_{\omega} \omega \quad (17) $$

where $\mathbf{M} = e^{-w}\mathbf{W}_{\nu\omega} + \mathbf{W}_{\nu}^{-1}\mathbf{L}_{\nu\omega}\mathbf{W}_{\omega}$. The control of the rotation of the camera is thus decoupled from the control of the translation.

## 3.1 Control of the rotation

The orientation of the camera is controlled using the estimated rotation between $\mathcal{F}$ and $\mathcal{F}^*$ (which has to reach the identity matrix). The control law is:

$$ \omega = -\lambda_{\omega} \widehat{\mathbf{W}}_{\omega}^{-1} \widehat{\mathbf{L}}_{\omega}^{-1} \hat{\mathbf{e}}_{\omega} \quad (18) $$

Thanks to the form of $\mathbf{L}_\omega$, given in equation (11), we have the following nice property:

$$ \widehat{\mathbf{L}}_{\omega}^{-1} \hat{\mathbf{e}}_{\omega} = \hat{\mathbf{e}}_{\omega} $$

It is important to notice that the estimated task function $\hat{\mathbf{e}}_\omega$ can be written as a function of $\mathbf{e}_\omega$:

$$ \hat{\mathbf{e}}_{\omega} = \mu \tilde{\mathbf{A}} \mathbf{e}_{\omega} \quad (19) $$

where $\mu = 1/\|\tilde{\mathbf{A}}\mathbf{u}\|$. From equations (17), (18) and (19), we obtain the following closed-loop differential equation:

$$ \dot{\mathbf{e}}_{\omega} = -\lambda_{\omega} \mu \mathbf{L}_{\omega} \tilde{\mathbf{R}} \tilde{\mathbf{A}} \mathbf{e}_{\omega} \quad (20) $$

where $\tilde{\mathbf{R}} = \mathbf{W}_\omega \widehat{\mathbf{W}}_\omega^{-1} = \mathbf{R}_e \widehat{\mathbf{R}}_e^T$. The stability of this system and its robustness domain are proved by the two following theorems:

**Theorem 1** The equilibrium point $\mathbf{e}_\omega = 0$ of the differential system (20) is locally asymptotically stable if and only if $\tilde{\mathbf{R}}\tilde{\mathbf{A}}$ has eigenvalues with positive real part. Simple sufficient conditions to ensure that are $\tilde{\mathbf{R}} > 0$ and $\tilde{\mathbf{A}} > 0$.

**Theorem 2** The equilibrium point $\mathbf{e}_\omega = 0$ of the differential system (20) is globally asymptotically stable if $\tilde{\mathbf{R}}\tilde{\mathbf{A}} > 0$. In that case, $\|\mathbf{e}_\omega\|$ decreases at each iteration of the control law.

Short proofs of these two theorems are in Appendix A. The sufficient conditions to ensure the local stability are very easily satisfied in practice since $\tilde{\mathbf{R}} > 0$ if the error on the angle of rotation is $\tilde{\phi} < \frac{\pi}{2}$ and $\tilde{\mathbf{A}} > 0$ if the conditions given in [8] are satisfied. The condition $\tilde{\mathbf{R}}\tilde{\mathbf{A}} > 0$ is stronger than the conditions $\tilde{\mathbf{R}} > 0$ and $\tilde{\mathbf{A}} > 0$ (i.e. if $\tilde{\mathbf{R}} > 0$ and $\tilde{\mathbf{A}} > 0$, it is not sure that $\tilde{\mathbf{R}}\tilde{\mathbf{A}} > 0$). The conditions to ensure $\tilde{\mathbf{R}}\tilde{\mathbf{A}} > 0$ are:

$$ |\tilde{\phi}| + |\tilde{\alpha}| < \frac{\pi}{2} $$

where:

$$ \tilde{\alpha} = \arccos \left( \frac{\rho^{-1}((\tilde{\mathbf{A}} + \tilde{\mathbf{A}}^T)^{-1})}{2\sqrt{\rho(\tilde{\mathbf{A}}^T \tilde{\mathbf{A}})}} \right) $$

$\rho(\mathbf{S})$ being the spectral radius of matrix $\mathbf{S}$. These conditions are in general easily satisfied using, for example, the camera parameters given by the manufacturer instead of the real ones. Furthermore, $\mu \ge 1/\|\tilde{\mathbf{A}}\|$ and the solution of the differential equation (20) is bounded as follows:

$$ \|\mathbf{e}_\omega(t)\| \le \|\mathbf{e}_\omega(0)\| e^{-\lambda'_\omega t} \quad (21) $$

where $\lambda'_\omega = \lambda_\omega \sigma / \|\tilde{\mathbf{A}}\|$, $\sigma$ being the unknown minimum singular value of $\frac{1}{2}(\tilde{\mathbf{R}}\tilde{\mathbf{A}} + \tilde{\mathbf{A}}^T\tilde{\mathbf{R}}^T)$. Thus, $\|\mathbf{e}_\omega(t)\|$ will converge exponentially to zero. Finally, since $\mathbf{e}_\omega(t)$ is completely defined by the differential system (20) and by its initial condition $\mathbf{e}_\omega(0)$, we can compute $\hat{\mathbf{e}}_\omega(t)$ from equation (19) and plug it in equation (16) obtaining a differential system where the only unknown is $\mathbf{e}_\omega$.
## 3.2 Control of the translation

The translation of the camera is controlled by regulating to zero the task function $e_\nu$. In [8] we proposed the following control law:

$$ \nu = -\lambda_{\nu} e^{\hat{w}} \hat{W}_{\nu}^{-1} \hat{L}_{\nu}^{-1} e_{\nu} - e^{\hat{w}} \hat{M} \omega \quad (22) $$

In the ideal case when $\hat{w} = w$ (i.e. $\hat{Z} = Z$), $\hat{A} = A$, $\hat{t}_e = t_e$ and $\hat{R}_e = R_e$ the closed-loop differential equation is decoupled and linearised: $\dot{e}_\nu = -\lambda_\nu e_\nu$. In the general case, we have the following closed-loop equation:

$$ \dot{e}_{\nu} = -\lambda_{\nu} \eta \mathbf{L}_{\nu} \tilde{\mathbf{R}} \tilde{\mathbf{A}} \mathbf{L}_{\nu}^{-1} e_{\nu} + \varphi(e_{\nu}, t) \quad (23) $$

where $\eta = e^{\hat{w}-w} > 0$ and:

$$ \varphi(e_{\nu}, t) = \lambda_{\omega} \mathbf{L}_{\nu} \mathbf{R}_{e} (\hat{\mathbf{M}} - \eta \mathbf{M}) \hat{\mathbf{R}}_{e}^{T} \hat{e}_{\omega}(t) $$

The following theorem defines the local stability and the robustness of the control law:

**Theorem 3** The equilibrium point $e_\nu = 0$ of the differential system (23) is locally asymptotically stable if and only if $\tilde{\mathbf{R}}\tilde{\mathbf{A}}$ has eigenvalues with positive real part. Simple sufficient conditions to ensure that are $\tilde{\mathbf{R}} > 0$ and $\tilde{\mathbf{A}} > 0$.

The proof of the theorem is given in the Appendix A. As we already noticed, the sufficient conditions are very easily satisfied in practice. Unfortunately, the local stability is only valid in a small (and “a priori” unknown) neighbourhood of the equilibrium point. In order to keep the error small, one can sample the error and follow a path as proposed in [10]. In our case, we can simply force a point to follow a straight line in the image by choosing:

$$ \begin{aligned} u^*(t) &= u(0) + \gamma(t)(u^* - u(0)) \\ v^*(t) &= v(0) + \gamma(t)(v^* - v(0)) \\ w^*(t) &= w(0) + \gamma(t)(w^* - w(0)) \end{aligned} \quad (24) $$

where $\gamma(t)$ is a function tuning the sampling such that $\gamma(0) = 0$ and $\gamma(t) = 1 \ \forall t > \bar{t}$. The initial rotation error can also be sampled as follows:

$$ \theta(t) = \theta(0) - \gamma(t)\theta(0) \quad (25) $$

Obviously, the tracking error $\frac{\partial e}{\partial t}$ should be taken into account in the control law as in [10]. Even if the control law is locally stable, in the presence of a large amount of calibration errors we are not sure that the error will remain in a sufficient small neighbourhood of the equilibrium point. It is thus necessary to have a control law which is stable in a known region around the equilibrium point (or even better which is globally stable). On the other hand, the global asymptotic stability of the system controlled with the control law (22) is not easy to prove. Furthermore, it is useless to use a decoupling control law in the presence of large calibration errors since the system will be coupled in any case. In order to provide a larger stability domain we propose a different control law for which it is possible

to provide sufficient conditions to ensure the global asymptotic stability. Consider the following control law:

$$ \nu = -\lambda_{\nu} e^{\hat{w}} \hat{W}_{\nu}^{T} \hat{L}_{\nu}^{T} e_{\nu} - e^{\hat{w}} \hat{M} \omega \quad (26) $$

The closed-loop equation is now:

$$ \dot{e}_{\nu} = -\lambda_{\nu} \eta \mathbf{L}_{\nu} \tilde{\mathbf{R}} \tilde{\mathbf{A}}^{-T} \mathbf{L}_{\nu}^{T} e_{\nu} + \varphi(e_{\nu}, t) \quad (27) $$

Choosing $\lambda_\nu > 0$, $\lambda_\omega > 0$ if $\|e_\nu\| \le \delta$ and $\lambda_\omega = 0$ if $\|e_\nu\| > \delta$ (where $\delta$ is a positive scalar tuning the bound of $\|e_\nu\|$), the stability of the system and its robustness domain are proved in the following theorem:

**Theorem 4** The equilibrium point $e_\nu = 0$ of the differential system (27) is globally stable if $\tilde{\mathbf{R}}\tilde{\mathbf{A}} > 0$.

A short proof of the theorem is given in the Appendix A. The sufficient conditions given in the theorems 4 and 2 are the same. Thus, they are sufficient conditions for the global stability of the 2 1/2 D visual servoing scheme.

## 4 Simulations

In the simulations, a camera mounted on a robot manipulator is positioned with respect to an object composed of 16 points randomly distributed in a volume of the workspace. The calibration error on the camera intrinsic parameters is about 50 %. The calibration error on the camera extrinsic parameters is 20° for the rotation and 20cm for the translation. The point 9 of the object is chosen to define the task function $e_\nu$. In the first simulation (see Figure 3), the camera is displaced from the reference position, then repositioned using the control law (26). In the second simulation (see Figure 4), the camera starts from the same initial position but the trajectory of the point 9 is constrained to follow a straight line in the image according to equations (24) and (25). For both simulations, the figures are organised as follows:

* Figure (a) shows the trajectory of the points in the image (numbered from 1 to 16). The points of the initial image are marked with a triangle. The points of the reference image are marked with a circle.

* Figure (b) shows the 3D points in the Cartesian space. The trajectory of the camera frame $\mathcal{F}$ is plotted at each iteration. The segments represents the axis of the frame.

* Figures (c) and (d) plot respectively the task function $e_\nu$ and the task function $e_\omega$.

* Figures (e) and (f) plot respectively the error of translation and the error of rotation between the current camera frame $\mathcal{F}$ and the reference frame $\mathcal{F}^*$.

* Figures (g) and (h) plot respectively the velocity of translation and the velocity of rotation of the end-effector (i.e the control law).
The first simulation shows that the control law is globally stable (see Figures 3(g) and (h)). Indeed, the initial position of the camera is far from the reference position (the initial error of translation is about 1 *m* and the initial error of rotation is about 120°, see Figures 3(e) and (f)). As expected, the task function converges exponentially to zero (see Figures 3(c) and (d)). Even if the control law is stable, the trajectory of the points in the image is not strongly constrained. Thus, some features may leave the field of view of the camera. In order to avoid that, in

the second simulation the initial error is sampled such that the point 9 follows a straight line in the image (see Fig. 4). When compared to the previous simulation, the trajectory of the point 9 is now constrained. Thus, it is unlikely that the points around the point 9 can leave the field of view of the camera. The small tracking error (see Figures 4(c) and (d)) is due to the large calibration errors. Finally, let us notice that if one want be really sure that all the points stay in the image, the trajectory of each point should be planned properly as proposed in [10].

<figure><img src="image_3.png" /><figcaption>Figure 3: Simulation with large calibration errors and large initial displacement of the end-effector of the robot.</figcaption></figure>

<caption>Figure 4: Simulation with large calibration errors, with the same initial displacement but with path planning.</caption>
# 5 Conclusion

In this paper, we proved that the 2 1/2 D visual servoing scheme is robust to large calibration errors on both intrinsic and extrinsic camera parameters. Since the proposed control law is globally stable, the convergence is obtained from any starting position in the workspace. However, the initial error can be sampled in order to keep some features of the observed object in the field of view of the camera. In this case, the global stability of the control law ensures that the tracking error is bounded.

# Appendix A

*Proof: [Theorem 1]* The local asymptotic stability of the non-linear system (20) can be deduced from the stability analysis of the linearised system. In this case, we have $L_\omega|_{e_\omega=0} = I$ and the linear system is:

$$ \dot{e}_\omega = -\lambda_\omega \mu \tilde{R} \tilde{A} e_\omega \quad (28) $$

It is well known that $\forall \lambda_\omega > 0$ and $\forall \mu > 0$ the system is stable if and only if the matrix $\tilde{R}\tilde{A}$ has eigenvalues with positive real part. The eigenvalues $\lambda$ of $\tilde{R}\tilde{A}$ and their correspondent eigenvectors $\mathbf{x}$ are related by the equation: $\tilde{R}\tilde{A}\mathbf{x} = \lambda\mathbf{x}$, $\forall \mathbf{x} \neq 0$. Multiplying the left and right side of this equation by $\mathbf{x}^T\tilde{R}\tilde{R}^T \neq 0$ we obtain the eigenvalue: $\lambda = \frac{\mathbf{x}^T\tilde{A}\mathbf{x}}{\mathbf{x}^T\tilde{R}\tilde{A}\mathbf{x}}$, which is positive if $\tilde{A} > 0$ and $\tilde{R} > 0$.

*Proof: [Theorem 2]* Consider the following Lyapunov function:

$$ V(e_\omega) = \frac{1}{2} e_\omega^T e_\omega \quad (29) $$

which is positive $\forall e_\omega \neq 0$. The derivative of $V(e_\omega)$ is:

$$ \dot{V}(e_\omega) = e_\omega^T \dot{e}_\omega \quad (30) $$

Using equation (20) and knowing that $e_\omega^T L_\omega = e_\omega^T$:

$$ \dot{V}(e_\omega) = -\lambda_\omega \mu e_\omega^T L_\omega \tilde{R} \tilde{A} e_\omega = -\lambda_\omega \mu e_\omega^T \tilde{R} \tilde{A} e_\omega \quad (31) $$

Then $\dot{V}(e_\omega) < 0$ if $\tilde{R}\tilde{A} > 0$ $\forall \lambda_\omega > 0$ and $\forall \mu > 0$.

*Proof: [Theorem 3]* The local stability analysis is obtained linearising the nonlinear system. In this case we have $L_\nu|_{e_\nu=0} = L_\nu^*$, $\varphi(0, t) = 0$ and the linear system is:

$$ \dot{e}_\nu = -\lambda_\nu \eta^* L_\nu^* \tilde{R} \tilde{A} L_\nu^{*-1} e_\nu \quad (32) $$

The matrix $L_\nu^* \tilde{R} \tilde{A} L_\nu^{*-1}$ is similar to the matrix $\tilde{R}\tilde{A}$, thus they have the same eigenvalues. Again, the matrix will have positive eigenvalues if $\tilde{R} > 0$ and $\tilde{A} > 0$.

*Proof: [Theorem 4]* Consider the following Lyapunov function:

$$ V(e_\nu) = \frac{1}{2} e_\nu^T e_\nu \quad (33) $$

which is positive $\forall e_\nu \neq 0$. The derivative of $V(e_\nu)$ is:

$$ \dot{V}(e_\nu) = e_\nu^T \dot{e}_\nu \quad (34) $$

Using equation (27) we obtain:

$$ \dot{V}(e_\nu) = -\lambda_\nu \eta e_\nu^T L_\nu \tilde{R} \tilde{A}^{-T} L_\nu^T e_\nu + \varphi(e_\nu, t) \quad (35) $$

If $\|e_\nu\| \le \delta$ then $\lambda_\omega > 0$ and from equation (21):

$$ \lim_{t \to \infty} \varphi(e_\nu, t) \le \psi(e_\nu) \|e_\omega(t)\| = 0 \quad (36) $$

which means that, if $\tilde{R}\tilde{A}^{-T} > 0$ then $\lim_{t \to \infty} \dot{V}(e_\nu) < 0$. On the other hand, if $\|e_\nu\| > \delta$ then $\varphi(e_\nu, t) = 0$ and $\|e_\nu\|$ decreases to $\delta$ since $\dot{V}(e_\nu) < 0$ if $\tilde{R}\tilde{A}^{-T} > 0$. Finally, $\tilde{R}\tilde{A}^{-T} > 0$ is equivalent to $\tilde{R}\tilde{A} > 0$.

# References

[1] R. Basri, E. Rivlin, and I. Shimshoni. Visual homing: Surfing on the epipoles. In *IEEE Int. Conf. on Computer Vision*, pages 863–869, Bombay, India, January 1998.

[2] F. Chaumette. Potential problems of stability and convergence in image-based and position-based visual servoing. In D. Kriegman, G. Hager, and A. Morse, editors, *The confluence of vision and control*, volume 237 of *LNCIS Series*, pages 66–78. Springer Verlag, 1998.

[3] B. Espiau. Effect of camera calibration errors on visual servoing in robotics. In *3rd International Symposium on Experimental Robotics*, Kyoto, Japan, October 1993.

[4] B. Espiau, F. Chaumette, and P. Rives. A new approach to visual servoing in robotics. *IEEE Trans. on Robotics and Automation*, 8(3):313–326, June 1992.

[5] O. Faugeras. *Three-dimensional computer vision: a geometric viewpoint*. MIT Press, Cambridge, MA, 1993.

[6] K. Hashimoto. *Visual Servoing: Real Time Control of Robot manipulators based on visual sensory feedback*, volume 7 of *World Scientific Series in Robotics and Automated Systems*. World Scientific Press, Singapore, 1993.

[7] S. Hutchinson, G. D. Hager, and P. I. Corke. A tutorial on visual servo control. *IEEE Trans. on Robotics and Automation*, 12(5):651–670, October 1996.

[8] E. Malis, F. Chaumette, and S. Boudet. 2 1/2 d visual servoing. *IEEE Trans. on Robotics and Automation*, 15(2):234–246, April 1999.

[9] E. Malis, F. Chaumette, and S. Boudet. 2 1/2 d visual servoing with respect to unknown objects through a new estimation scheme of camera displacement. *International Journal of Computer Vision*, 37(1):79–97, June 2000.

[10] Y. Mezouar and F. Chaumette. Path planning in image space for robust visual servoing. In *IEEE Int. Conf. on Robotics and Automation*, volume 3, pages 2759–2764, San Francisco, CA, April 2000.

[11] C. Samson, M. Le Borgne, and B. Espiau. *Robot Control: the Task Function Approach*, vol. 22 of *Oxford Engineering Science Series*. Clarendon Press, Oxford, UK, 1991.

[12] W. J. Wilson, C. C. W. Hulls, and G. S. Bell. Relative end-effector control using cartesian position-based visual servoing. *IEEE Trans. on Robotics and Automation*, 12(5):684–696, October 1996.

[13] P. Zanne, G. Morel, and F. Plestan. Robust vision based 3d trajectory tracking using sliding mode control. In *IEEE Int. Conf. on Robotics and Automation*, volume 3, pages 2088–2093, San Francisco, CA, April 2000.