# Vision-based rotational control of an agile observation satellite

Maxime Robic, Renaud Fraisse, Eric Marchand, François Chaumette

► To cite this version:

Maxime Robic, Renaud Fraisse, Eric Marchand, François Chaumette. Vision-based rotational control of an agile observation satellite. IROS 2022 - IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct 2022, Kyoto, Japan. pp.1-8. hal-03734033

HAL Id: hal-03734033

https://inria.hal.science/hal-03734033v1

Submitted on 21 Jul 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Vision-based rotational control of an agile observation satellite

Maxime Robic, Renaud Fraisse, Eric Marchand, François Chaumette

**Abstract**— Recent Earth observation satellites are now equipped with new instrument that allows image feedback in real-time. Problematic such as ground target tracking, moving or not, can now be addressed by precisely controlling the satellite attitude. In this paper, we propose to consider this problem using a visual servoing approach. While focusing on the target, the control scheme has also to take into account the satellite motion induced by its orbit, Earth rotational velocities, potential target own motion, but also rotational velocities and accelerations constraints of the system. We show the efficiency of our system using both simulation (considering real Earth image) and experiments on a robot that replicates actual high resolution satellite constraints.

## I. INTRODUCTION

Earth-pointing satellites for observation missions, or satellite imagery, have increased drastically in number and performance since the beginning of the space age. For example, Pléiades HR, a constellation of two imagery satellites on a sun-synchronous quasi-circular orbit (695km), acquires Earth images (with 70-cm resolution) [11]. Pléiades image acquisition system uses a push-broom technology (see Fig. 1), i.e., a sensor line allows to scan of a 20-km image swath of high-quality to cover both Defense and civilian observation needs. Pléiades Néo [18] aims to ensure the continuity and improve the current constellation in terms of satellite agility and acquisition resolution. To perform an acquisition mission, a desired satellite attitude is maintained thanks to an on-board closed-loop attitude control law [19]. However, Pléiades control is not achieved in real-time, i.e., the requested attitude is not supposed to change during the acquisition. Still, real-time attitude adjustments might have relevant interest for ground targeting (e.g. to track in real-time a moving target, or to improve the 3D reconstruction of Earth surface using satellite images [4], ...).

A motionless ground target tracking using a real-time control loop is studied in [10]. The tracking error is regulated using a model predictive control approach based on satellite dynamics. Yet, the target is fixed on Earth and the image acquisition is considered only as an output, as push-broom technologies are incompatible with real-time image processing and control as in the robotics field. Airbus Defence & Space is developing a new satellite system with a "Starer" acquisition principle (see Fig. 1). The acquisition is made by a matrix of sensors that allows real-time acquisition of the

images. This way, the satellite "camera" can be used as an input sensor for real-time attitude control process. Airbus DS plans to embed this new technology into the future LION constellation composed of several agile Earth observation satellites with 50-cm resolution. Real-time attitude control using such sensors will then allow the tracking of motionless and moving ground targets.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Push-broom and starer technologies</figcaption></figure>

In this paper, we show that this precise control of the satellite could be achieved using a closed loop visual servoing (VS) control scheme [5].

Visual servoing is widely used for space applications as it provides autonomous and efficient control for solving precision tasks, with real-time pose adjustments. Space Rendezvous [17] has been considered using 3D model-based tracking together with a 2 1/2 D visual servoing control law. Satellite captures have also been studied [20][2] with various image-based visual servoing control laws for free-floating space manipulators. In these works, the target is either considered to have a known motion [2] or an unknown motion [20]. In [1], a visual servoing of a multi-arm system is performed to improve on-orbit servicing by combining visual servoing with other robotic tasks to reduce attitude disturbance. Another image-based control process for observing tumbling objects has been studied in [16].

The main difference between these studies and our application is that they do not have to deal with large velocities difference between the target and the sensor. Indeed, for the above works, the two considered objects are moving in almost the same orbit, which is not the case for our work, while the orbital velocity of the satellite is one of the main issues for solving the tracking regulation.

This work is focused on establishing a visual control law that would allow to precisely control a LION satellite attitude

<footnote>This work was supported by BPI France Lichie project.</footnote>

<footnote>M. Robic and F. Chaumette are with Inria, Univ Rennes, CNRS, IRISA, Rennes, France Firstname.Name@inria.fr</footnote>

<footnote>R. Fraisse is with Airbus Defense and Space, Toulouse, France renaud.fraisse@airbus.com</footnote>

<footnote>E. Marchand is with Univ Rennes, Inria, CNRS, IRISA, Rennes, France eric.marchand@irisa.fr</footnote>
using images provided by the starer sensor. The goal is to perform acquisition missions devoted to focus on an object of interest (see Fig. 2). The sensor is fixed wrt. the satellite that is supposed to evolve on a sun-synchronous circular orbit of 500km. We have full control over the three rotational DoF of the satellite subject to dynamic constraints, while the satellite is evolving in an orbit that only influences its position (that is not controlled by our VS scheme). We

<figure><img src="image_3.png" /><figcaption>Fig. 2. LION mission: a satellite $\mathcal{F}_s$ directed by its orbital translations and servoed in rotations to track a terrestrial object of interest $\mathcal{F}_o$ thanks to its camera $\mathcal{F}_c$. World frame $\mathcal{F}_w$ is here the equatorial geocentric frame of reference.</figcaption></figure>

suppose that the target is visible in the image before starting the visual servoing. Any previous rotation to orientate the satellite toward a requested area is done by an open-loop attitude guidance, which is not studied in this paper (this is the classical way to control such satellite). Compensating the robot dynamics [12] [8] [21] by explicitly including it in the control scheme becomes essential when its own motion is significant. In our case, the orbit of the satellite is known, so we can accurately estimate its translation motion, and compensate it in the control law. When it comes to the target motion, literature shows various approaches [6]. We propose to decompose the target motion with known displacement caused by Earth's dynamics and residual motion due to the target own motion.

The contributions of this paper are a visual servoing scheme able to control the attitude of an agile Earth observation satellite for target tracking. The control law allows for dealing with the satellite's high translational velocity induced by its orbit and other external motions including Earth's rotation and target own motion. A velocity saturation algorithm dealing with dynamic constraints that does not alter the nature of the target trajectory in the image is also proposed.

## II. VISION-BASED CONTROL OF A SATELLITE

### A. Visual servo control in space environment

According to [5], we recall that the goal of a visual servoing is to minimize an error function $e$ that can be defined by:

$$e(t) = s(t) - s^* \quad (1)$$

where $s(t)$ represents a set of $k$ visual features derived from image measurements, and $s^*$ is their desired values. In our

application, the camera is embedded on an agile satellite whose full orientation can be controlled. Thus, to design an accurate velocity controller of the 3 rotational velocity components, the correct theoretical model for such a situation is [6]:

$$\dot{s} = \mathbf{L}_{\omega_c} \omega_c + \frac{\partial s}{\partial t} \quad (2)$$

where $\mathbf{L}_{\omega_c}$ is the $k \times 3$ interaction matrix that links the features time-derivative to the rotational velocity $\omega_c$. The term $\frac{\partial s}{\partial t}$ represents the feature variation due to the other motions. In our case, first, the satellite is moving on a circular trajectory around the Earth called orbit, with a significant orbital speed far superior to any terrestrial or aerial vehicle. Second, a terrestrial target, or object of interest, has a motion induced by Earth's rotation. Finally the target may have its own velocity in case it is a mobile vehicle. Therefore, $\frac{\partial s}{\partial t}$ is expressed by:

$$\frac{\partial s}{\partial t} = \mathbf{L}_s \mathbf{c} \mathbf{v}_s - \mathbf{L}_o \mathbf{c} \mathbf{v}_{oE} + \frac{\partial s_o}{\partial t} \quad (3)$$

where

* $\mathbf{c} \mathbf{v}_s$ is the satellite translational velocity (order of magnitude 7 km/s) expressed in camera frame $\mathcal{F}_c$ induced by orbital motion, and $\mathbf{L}_s$ its $k \times 3$ interaction matrix,

* $\mathbf{c} \mathbf{v}_{oE}$ is the target translational velocity expressed in camera frame $\mathcal{F}_c$ induced by Earth's rotation (order of magnitude 0.4 km/s), and $\mathbf{L}_o$ its $k \times 3$ interaction matrix,

* $\frac{\partial s_o}{\partial t}$ is an additional term induced by target -generally unknown- own motion (order of magnitude less than 0.3 km/s).

From now on, we present a classical velocity controller designed to provide an exponential decrease of the error function, altogether with a compensation of external motions [6]:

$$\omega_c = -\lambda \hat{\mathbf{L}}_{\omega_c} \hat{\mathbf{e}} - \hat{\mathbf{L}}_{\omega_c} \frac{\partial \mathbf{s}}{\partial t} \quad (4)$$

where $\hat{\mathbf{L}}$ represents an estimation of the true expression of $\mathbf{L}$ and $\mathbf{L}^+$ the pseudo-inverse of $\mathbf{L}$. Since, from (1), $\dot{\mathbf{e}} = \dot{s}$ as $s^*$ is fixed, substituting (4) in (2) allows obtaining the closed-loop behaviour of the system:

$$\dot{\mathbf{e}} = -\lambda \mathbf{L}_{\omega_c} \hat{\mathbf{L}}_{\omega_c}^{+} \hat{\mathbf{e}} - \mathbf{L}_{\omega_c} \hat{\mathbf{L}}_{\omega_c}^{+} \frac{\partial \mathbf{s}}{\partial t} + \frac{\partial \mathbf{s}}{\partial t} \quad (5)$$

The control scheme is thus globally asymptotically stable in the sense of Lyapunov if:

1) $\hat{\mathbf{L}}_{\omega_c}^+$ is such that $\mathbf{L}_{\omega_c} \hat{\mathbf{L}}_{\omega_c}^+ > \mathbf{O}_k$. In practice, we will see that it is possible to perfectly estimate $\hat{\mathbf{L}}_{\omega_c}^+$ so that $\mathbf{L}_{\omega_c} \hat{\mathbf{L}}_{\omega_c}^+ = \mathbf{I}_k$

2) $\frac{\partial \mathbf{s}}{\partial t}$ is well estimated so that $\frac{\partial \mathbf{s}}{\partial t} = \frac{\partial \mathbf{s}}{\partial t}$, which allows canceling any tracking error.

The next sections will show what the visual features $s$ that we have selected are, and what the corresponding interaction matrices are, and how the translational velocities involved in (5) are estimated regarding orbital mechanics.
## B. Image-Based Visual Servoing

We choose image-based visual features for our application. Note that only 3 independent visual features are needed to control the 3 rotational degrees of the system. The first visual features are simply the 2D image coordinate $\mathbf{x} = (x, y)$ of a point $\mathbf{P}$ belonging to the target, typically its center of gravity. $\mathbf{x}^* = (x^*, y^*)$ is then the desired image position of the target. This first set of visual features defines a focusing task, and a centering task if $\mathbf{x}^* = \mathbf{y}^* = 0$. Note that if $\dot{\mathbf{e}} = -\lambda\mathbf{e}$ is ensured, then the trajectory in the image of point $\mathbf{P}$ is a straight line from its initial position to the desired one. A third visual feature $\alpha$ is selected for a second task called the orientation task. Adding this visual feature allows a specific orientation of the acquisition. For instance, to obtain images with the sky on the top and the ground on the bottom, a second point $\mathbf{P}'$ is selected above the target, i.e., with a different altitude, and the corresponding segment $[\mathbf{P}, \mathbf{P}']$ has to appear as vertical in the image (see Fig. 3). Thus, the additional visual feature $\alpha$ is defined as $\arctan(\frac{y-y'}{x-x'})$, that is the angle of $[\mathbf{x}, \mathbf{x}']$, the projected segment $[\mathbf{P}, \mathbf{P}']$ onto the image plane, with the horizontal axis. $\alpha^*$ is set as $\frac{\pi}{2}$ so that the desired segment is vertical. Note that $\alpha$

<figure><img src="image_4.png" /><figcaption>Fig. 3. The three visual features to be regulated</figcaption></figure>

is well-defined when the satellite is not at the zenith of the target, i.e., when the length of the projected segment $l = \sqrt{(x-x')^2 + (y-y')^2}$ is not null. In these situations, the orientation task is disabled until $\alpha$ can be computed again. The error $\mathbf{e}$ to be regulated to 0, combining both the focusing and the orientation task, is defined by:

$$ \mathbf{e} = \begin{pmatrix} x - x^* \\ y - y^* \\ \alpha - \alpha^* \end{pmatrix} \qquad (6) $$

The 3 d.o.f. interaction matrix $\mathbf{L}_{\omega_c}$ for rotations considering an image point and the angle of a projected segment as set of visual features is expressed by [7]:

$$ \mathbf{L}_{\omega_c} = \begin{pmatrix} xy & -1 - x^2 & y \\ 1 + y^2 & -xy & -x \\ -xs^2 + ycs & -yc^2 + xcs & -1 \end{pmatrix} \quad (7) $$

with $s = \sin \alpha$ and $c = \cos \alpha$. All the parameters involved in $\mathbf{L}_{\omega_c}$ are directly available as image measurements so it is possible to use $\tilde{\mathbf{L}}_{\omega_c} = \mathbf{L}_{\omega_c}$ in (4) and we have $\mathbf{L}_{\omega_c} \tilde{\mathbf{L}}_{\omega_c}^+ = \mathbf{L}_{\omega_c} \mathbf{L}_{\omega_c}^{-1} = \mathbf{I}_3$. When the third visual feature $\alpha$ can not be computed because the satellite is near the zenith of the target, the orientation task is disabled by removing $\alpha - \alpha^*$ from $\mathbf{e}$ and by switching from $\mathbf{L}_{\omega_c}$ to $\mathbf{L}_{\omega_{xy}}$ with:

$$ \mathbf{L}_{\omega_{xy}} = \begin{pmatrix} xy & -1 - x^2 & y \\ 1 + y^2 & -xy & -x \end{pmatrix} \quad (8) $$

The three rotational DoF are then only used for the focusing task. In that case, we have $\tilde{\mathbf{L}}_{\omega_c} = \mathbf{L}_{\omega_{xy}}$ and $\mathbf{L}_{\omega_c} \tilde{\mathbf{L}}_{\omega_c}^+ = \mathbf{L}_{\omega_{xy}} \mathbf{L}_{\omega_{xy}}^+ = \mathbf{I}_2$.

## C. Motion compensation

In this subsection, we define and compute the external motions induced by the system and explain how to compensate for them in (4).

1) Satellite motion: The studied satellite is a high-speed system whose uncontrolled motion is induced by its orbit. Its position is determined by its orbital parameters, i.e., knowing these parameters allows us to compute an accurate estimation of the satellite translation speed. The satellite position for a circular orbit (i.e., argument of periapsis is set to 0) with respect to the world frame $\mathcal{F}_w$ is given by [9]:

$$ w \mathbf{t}_s = \begin{pmatrix} r \cos \theta \cos \Omega - r \sin \theta \sin \Omega \cos \iota \\ r \cos \theta \sin \Omega + r \sin \theta \cos \Omega \cos \iota \\ r \sin \theta \sin \iota \end{pmatrix} \quad (9) $$

where

* $\iota$ is the orbit inclination,

* $\Omega$ is the longitude of the ascending node; it determines where the two intersections of the orbit with the equatorial plane are located,

* $\theta$ is the true anomaly. For a circular orbit $\theta = \frac{\mu^2}{\hbar^2} t$ with $\hbar$ the angular momentum and $\mu$ the geocentric gravitational constant,

* $r = R_T + z_{alt}$ with $R_T$ the terrestrial radius and $z_{alt}$ the orbit altitude.

Let us note that for a given satellite orbit, all these parameters are perfectly known. The translational velocity $w \mathbf{v}_s$ is directly deduced from (9) and we obtain:

$$ w \mathbf{v}_s = \begin{pmatrix} r \dot{\theta} (-\sin \theta \cos \Omega - \cos \theta \sin \Omega \cos \iota) \\ r \dot{\theta} (-\sin \theta \sin \Omega + \cos \theta \cos \Omega \cos \iota) \\ r \dot{\theta} \cos \theta \sin \iota \end{pmatrix} \quad (10) $$

This velocity has to be expressed in the camera frame to be injected in (5). For that we use:

$$ c \mathbf{v}_s = c \mathbf{R}_w w \mathbf{v}_s \quad (11) $$

with $c \mathbf{R}_w = s \mathbf{R}_c^T s \mathbf{R}_w$ where:

* $s \mathbf{R}_c$, the orientation of the camera on the satellite, is known, constant and fixed by the constructor,

* $s \mathbf{R}_w$ is the orientation of the satellite with respect to the world frame $\mathcal{F}_w$ induced by visual servoing.
The interaction matrix $\mathbf{L}_s$ is thus the 3 dof interaction matrix for translations considering the three selected visual features. It is given by [7]:

$$ \mathbf{L}_s = \begin{pmatrix} -\frac{1}{Z} & 0 & \frac{x}{Z} \\ 0 & -\frac{1}{Z} & \frac{y}{Z} \\ -\frac{d}{l}s & \frac{d}{l}c & \frac{d}{l}(xs - yc) \end{pmatrix} \quad (12) $$

where $d = 1/Z' - 1/Z$ and $Z$ and $Z'$ are respectively the depth of $\mathbf{P}$ and $\mathbf{P}'$.

2) Earth rotational motion: The target is considered as a terrestrial object. In case it is static, its position $^w\mathbf{t}_o$ can be determined by spherical coordinates, as long as we hypothesise that the Earth is spherical and not an actual geoid:

$$ ^w\mathbf{t}_o = \begin{pmatrix} R_T \cos \delta \cos \phi \\ R_T \cos \delta \sin \phi \\ R_T \sin \delta \end{pmatrix} \quad (13) $$

where $\delta$ is the latitude of the target and $\phi$ its longitude. As we choose to work in the geocentric equatorial frame, and not in the terrestrial frame, Earth's rotation must be considered for any object standing on the Earth's surface. Earth's rotation can be described by a time variation of the longitude $\phi$:

$$ \phi = \omega_E t + \phi_0 \quad (14) $$

with $\omega_E = 7.29217 \times 10^{-5}$ rad/s Earth's rotation speed. Thus we obtain from (13) and (14):

$$ ^w\mathbf{v}_{oE} = \begin{pmatrix} -\omega_E R_T \cos \delta \sin \phi \\ \omega_E R_T \cos \delta \cos \phi \\ 0 \end{pmatrix} \quad (15) $$

Once again, this translational velocity is expressed in the camera frame $\mathcal{F}_c$ using:

$$ ^c\mathbf{v}_{oE} = ^c\mathbf{R}_w ^w\mathbf{v}_{oE} \quad (16) $$

Finally, $\mathbf{L}_o$ has the same definition as $\mathbf{L}_s$ and we have $\mathbf{L}_o = \mathbf{L}_s$.

3) Target own motion: In the case where the target has an unknown motion, the term $\frac{\partial s_o}{\partial t}$, unlike previous velocities, can not be determined analytically. This term can be estimated by various approaches, but if the target moves with a constant speed, a simple solution to compensate for that motion is to consider an integrator in the control law, which will eventually eliminate the drag error induced by a motion with constant velocity. So, reminding (3), $\frac{\partial s_o}{\partial t}$ will be compensated by:

$$ \frac{\partial \widehat{s_o}}{\partial t} = \mu \sum_j e(j) \quad (17) $$

where $\mu$ is the integral gain that must be tuned according to the drag error.

### D. Improving control behaviour

1) Adaptive gain: The gain $\lambda$ associated with the exponential decrease of the error $e$ is usually constant. However, $\lambda$ can be modulated to decrease high velocities when the visual error is high, and increase the convergence speed when it is

low [13]. An adaptive gain is then introduced, $\lambda$ becoming error-dependent:

$$ \lambda(\|\mathbf{e}\|) = (\lambda_0 - \lambda_\infty)e^{-\frac{\lambda'_0}{\lambda_0 - \lambda_\infty}}\|\mathbf{e}\| + \lambda_\infty \quad (18) $$

where

* $\lambda_0$ is the gain tuned for small values of $\|\mathbf{e}\|$,

* $\lambda_\infty$ is the gain for high values of $\|\mathbf{e}\|$,

* $\lambda'_0$ is the slope of $\lambda$ at $\|\mathbf{e}\| = 0$.

Such tuning increases the convergence speed of the system together with its stability when the error is high. Furthermore, in case the target has a own motion, the integrator gain $\mu$ in (17) is tuned in the same way.

2) Velocity constraints: A satellite is a critical system and its rotational axes commonly called pitch, roll and yaw, are limited to a certain maximal speed and acceleration:

* On $x$ and $y$ axes, speed is limited to 3 deg/s and acceleration to 0.6 deg/s²,

* On the $z$ axis, speed is limited to 1.2 deg/s and acceleration to 0.25 deg/s².

Those constraints raise that $z$ axis can be decoupled from the others, and a different gain can be used on this axis. We define $\mathbf{\Lambda}$ by:

$$ \mathbf{\Lambda} = \begin{pmatrix} \lambda & 0 & 0 \\ 0 & \lambda & 0 \\ 0 & 0 & \lambda_\alpha \end{pmatrix} \quad (19) $$

with $\lambda$ and $\lambda_\alpha$ adaptive gains following (18) with $\lambda_\alpha$ tuned with significantly below parameters than $\lambda$ to cope with stronger constraints around $z$ axis, and (4) becomes:

$$ \omega_c = -\mathbf{L}_{\omega_c}^{+} \mathbf{\Lambda} \mathbf{e} - \mathbf{L}_{\omega_c}^{+} \frac{\widehat{\partial s}}{\partial t} \quad (20) $$

For ensuring $\dot{\mathbf{e}} = -\mathbf{\Lambda}\mathbf{e}$ induced by (20), if the velocity or the instant acceleration becomes too high, a velocity reduction is operated on all components of $\omega_c$ for not altering as much as possible the nature of the image trajectory, i.e., a straight line. When a constraint is activated, a reduction factor is computed on the axis concerned. Reduction factors $(r_x^\omega, r_y^\omega, r_z^\omega)$ for velocity constraints and $(r_x^a, r_y^a, r_z^a)$ for instant acceleration constraints are computed according to the following rules:

* $\forall i \in [x, y, z]$, if $|\omega_i| > \omega_{max_i}$ then $r_i^\omega = \frac{\omega_{max_i}}{|\omega_i|}$ else $r_i^\omega = 1$,

* $\forall i \in [x, y, z]$, if $|a_i| > a_{max_i}$ where $a_i = \frac{\omega_i(t) - \omega_i(t - dt)}{dt}$ then $r_i^a = \frac{a_{max_i}}{|a_i|}$ else $r_i^a = 1$.

Then, as the $z$ axis has different constraints than the others, and for not penalizing the time to convergence of the focusing task, we adopt the following strategy:

* raising velocity or instant acceleration constraints on $x$ or $y$ axis induces a velocity reduction on all axes i.e., $r_x = r_y = r_z$,

* raising constraints on $z$ axis induces a velocity reduction on $z$ axis only, i.e., $r_x = r_y = 1$,

Raising constraints simultaneously on multiple axes induces that $r_x = r_y = \min(r_x, r_y)$ and $r_z = \min(r_x, r_y, r_z)$.
Finally, when a velocity constraint is raised, the rotational velocity $\omega_{sat}$ sent to the low-level controller is given by:

$$ \omega_{sat} = \begin{pmatrix} r_{xy}^{\omega} & 0 & 0 \\ 0 & r_{xy}^{\omega} & 0 \\ 0 & 0 & r_{z}^{\omega} \end{pmatrix} \omega_{c} \quad (21) $$

while, for an acceleration constraint:

$$ \omega_{sat} = \begin{pmatrix} r_{xy}^{a} & 0 & 0 \\ 0 & r_{xy}^{a} & 0 \\ 0 & 0 & r_{z}^{a} \end{pmatrix} \omega_{c} + \begin{pmatrix} 1-r_{xy}^{a} & 0 & 0 \\ 0 & 1-r_{xy}^{a} & 0 \\ 0 & 0 & 1-r_{z}^{a} \end{pmatrix} \omega_{c}(t-dt) \quad (22) $$

During a saturation on x or y axis, we have $\omega_{sat} = r\omega_{c}$ for a velocity constraint and $\omega_{sat} = r\omega_{c} + (1-r)\omega_{c}(t-dt)$ for an acceleration constraint. In both cases, the time-variation of the error is given by:

$$ \dot{\mathbf{e}} = -r\mathbf{\Lambda}\mathbf{e} + \mathbf{K} \quad (23) $$

with $\mathbf{K} = -r\frac{\partial s}{\partial t} + \frac{\partial s}{\partial t}$ for velocity saturation, and $\mathbf{K} = -r\frac{\partial s}{\partial t} + \frac{\partial s}{\partial t} + (1-r)\mathbf{L}_{\omega_c}\omega_c(t-dt)$ for acceleration saturation. The term $\mathbf{K}$ in (23) shows that $\frac{\partial s}{\partial t}$ is compensated partially by $-r\frac{\partial s}{\partial t}$, which necessitates the residual external motion $(1-r)\frac{\partial s}{\partial t}$ to be compensated by the feedback term $\mathbf{\Lambda}\mathbf{e}$ and thus reduces the time-to-convergence of the visual tasks. However, this constraint management allows the servoing scheme to keep a straight-line trajectory of the target point in the image during a possible saturation of axis x or y. Indeed, from (23) and assuming $\mathbf{K}$ is constant during the saturation, we can express $x(t)$ and $y(t)$ components of $\mathbf{e}$ by:

$$ \begin{pmatrix} x(t) - x^* \\ y(t) - y^* \end{pmatrix} = \begin{pmatrix} x(0)e^{-r\lambda t} + \frac{K_1}{r\lambda} \\ y(0)e^{-r\lambda t} + \frac{K_2}{r\lambda} \end{pmatrix} \quad (24) $$

which leads to:

$$ \frac{x(t)}{x(0)} - \frac{y(t)}{y(0)} - \frac{\frac{K_1}{r\lambda} + x^*}{x(0)} + \frac{\frac{K_2}{r\lambda} + y^*}{y(0)} = 0 \quad (25) $$

showing that the image trajectory of point P is a straight line. However, as soon as $K_1 \neq 0$ or $K_2 \neq 0$, which occurs during a saturation, the image point does not converge anymore to its desired position $(x^*, y^*)$ but to $(x^* + \frac{K_1}{r\lambda}, y^* + \frac{K_2}{r\lambda})$. This means that saturation on x or y will make the trajectory deviate to a set of successive segments. That is why such saturation must not occur during all the servo, and, for that, $\lambda$ is tuned so that constraint raising happens rarely. Finally, let us note that the straight-line trajectory is also lost when z axis is saturated. That is why $\lambda_\alpha$ is tuned to not reach a saturation until the focusing task is not completed. Indeed, a saturation of axis z will be unseen on the trajectory of point P once this point is already at its desired position.

3) Continuous transition: Our application is focused on the acquisition part when the object of interest is in the camera field of view, but, as already noticed in the introduction, other attitude control might be applied before the acquisition begins, such as open-loop rotations to point the satellite in the area of interest. We thus propose a smooth beginning for the visual control, i.e., a continuous transition between the attitude control task and the visual servoing one. To ensure

this, we introduce a nonhomogeneous first-order differential equation inspired from [14]:

$$ \dot{\mathbf{e}} = -\mathbf{\Lambda}\mathbf{e} + \rho(t) \quad (26) $$

with

$$ \rho(t) = e^{-\gamma t}(\dot{\mathbf{e}}_p(0) + \mathbf{\Lambda}\mathbf{e}(0) + \frac{\widehat{\partial s}}{\partial t}(0)) \quad (27) $$

where $\mathbf{e}_p$ is the previous task related to the open-loop attitude control at time $t=0$ when the visual servo starts. From (26) and (2) and reminding that $\dot{\mathbf{e}} = \dot{\mathbf{s}}$ we obtain:

$$ \omega_c = -\mathbf{L}_{\omega_c}^{+}(-\mathbf{\Lambda}\mathbf{e} - \frac{\partial \mathbf{s}}{\partial t} + \rho(t)) \quad (28) $$

Finally, in case the open-loop attitude control allows for compensating the external motions at time $t=0$, i.e., $\dot{\mathbf{e}}_p(0) = -\frac{\partial \mathbf{s}}{\partial t}(0)$, (27) can be simplified to:

$$ \rho(t) = e^{-\gamma t} \mathbf{\Lambda}\mathbf{e}(0) \quad (29) $$

The transition gain $\gamma$ is tuned so that rotational velocity induced by visual feedback is smooth at the beginning, avoiding an abrupt start and so acceleration peak.

4) System overview: Finally, the 3 dof IBVS scheme aiming to control an Earth-pointing imagery satellite in rotation is given by:

$$ \omega_c = -\mathbf{L}_{\omega_c}^{+} (\mathbf{\Lambda}\mathbf{e} - \rho(t) - \mathbf{L}_s (c\mathbf{v}_s - c\mathbf{v}_{oE}) + \mu \sum_j \mathbf{e}(j)) \quad (30) $$

This velocity $\omega_c$ is then sent to the velocity saturation process described previously before being applied by the low-level controller of the system.

### III. RESULTS AND DISCUSSION

The proposed control law was tested using the ViSP framework [15]. To achieve a realistic experiment, we have considered a Cartesian robot performing a downscaled satellite movement. A sun-synchronous satellite has also been simulated. Our control law has been tested using actual satellite images. Tracking a fast moving target has also been considered. A companion video illustrates the results described below.

#### A. Implementation on a Cartesian robot

<figure><img src="image_5.png" /><figcaption>Fig. 4. The Cartesian robot positioned above a satellite image. Fig. 5. The Intel D435 camera with a pan-tilt configuration.</figcaption></figure>

The robot is a 6 dof Gantry robot (see Fig. 4). Its effector is equipped with a RGB-D Intel D435 camera mounted with a pan-tilt configuration (see Fig. 5). It allows the robot to move with a downscaled satellite trajectory while
the camera is controlled by visual servoing. The camera is pointing at a scaled satellite image reflecting our application scenario. Two configurations are presented in Fig. 6: a planar configuration where two points determine the desired orientation (the two points have the same altitude), and a non-planar configuration where the two points share the same longitude and latitude but have different altitudes. The robot is moving in translation with a typical satellite trajectory whose translational velocities are reduced by a factor $Sc$ related to the ratio between the altitude of the simulated satellite, which is fixed at 1 meter, and the real one, which is 500km at the zenith ($Sc = 2 \times 10^{-6}$). Additionally, the depth $Z$ of the target is fixed to 1 meter for every configuration and for any camera inclination. An initial template is selected manually since object detection is not in the scope of this paper. It is then tracked by computing a homography using a SSD inverse compositional template tracker [3][15]. Visual features are then computed from the template: its center is the target while the middle of its first segment defines the second point from which angle $\alpha$ is computed (see Fig. 6).

In each situation, the control law succeeds in compensating for the robot translational motion and in minimizing the visual features errors (see Figs. 7 and 8). Moreover, the image trajectory of the target point is a straight line and saturation on $x$ or $y$ axis have no visible effects. When convergence is reached, the target is maintained at the center

<figure><img src="image_6.png" /><figcaption>Fig. 6. Image trajectory of points $\mathbf{P}$ and $\mathbf{P}'$ induced by the visual control law in both plane (on the left) and relief (on the right) configurations: at the beginning (on the top), at the middle when centering errors are low (on the middle), and at the end (on the bottom). In each situation, an almost perfect straight-line trajectory is observed while completing the centering task, then the target is kept at the center of the image while the orientation task ends.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 7. Angular rate and visual feature error for the visual control law applied in the planar configuration.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 8. Angular rate and visual feature error for the visual control law applied in the non-planar configuration.</figcaption></figure>

of the image thanks to the compensation of robot motion. We can note that the velocity $\omega_z$ is saturated during a long time to complete the *orientation* task as fast as possible.

## B. Satellite simulation

In this section, we simulate an actual satellite moving in a sun-synchronous circular orbit of 500km, targeting a terrestrial target, motionless or not. The satellite is supposed to react in a free-floating mode, i.e., without any delay when executing the control law. The control loop is supposed to be synchronized with the camera frequency embedded on a LION satellite, which is 5Hz.
1) *Real images for fixed target tracking:* A 50cm-resolution image of Brest harbor (France) has been used to perform a simulated acquisition close to reality. The distance to the target is computed thanks to the knowledge of the satellite trajectory and the exact longitude and latitude of the area of interest. Template tracking is operated as in the previous experiment. The acquisition rendering presented in Fig. 9 shows different images acquired during the satellite motion induced by its orbital translations and its rotations

<figure><img src="image_9.png" /><figcaption>Fig. 9. Image trajectory of points P and P' induced by the visual control law in satellite simulation targeting an object of interest in the Brest's harbour. Satellite motion can be observed as it gets closer to the target, distance to the target is reduced to 500km and the image is enlarged. After reaching the zenith, the satellite moves away, distance to the target increases and the image shrinks.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 10. Angular rate and visual feature error for the visual control law applied for satellite simulation in a planar configuration.</figcaption></figure>

induced by visual servoing. The trajectory of the target while performing the *centering* task is still a straight line in the image, and the control scheme succeeds in keeping the target at the center of the image with the desired orientation, in spite of satellite translations. Few oscillations appear on z-axis angular rate when the tracked zone becomes too small in the image since visual feature $\alpha$ becomes unstable when the two points used to compute it becomes too near (see Fig. 10).

2) *Tracking a fast moving target:* To evaluate the tracking efficiency for a moving target, a simplified control scheme considering only the centering task is operated on a simulated terrestrial target. The target is assumed to move with a fast constant speed (1000km/h), and an integrator is set in the control loop to compensate the dragging error induced by this motion (see SectionII-C.3). As mentioned in Section II-D.1, the integrator gain $\mu$ is tuned to provide a smooth convergence, i.e., the integrator influence is moderated for large values of $||e||$ and higher near the convergence. We can observe in Fig. 11 the large effect of the target motion: for the first segment that corresponds to the first iteration of the control scheme, until $t = 0.2s$, the visual servo has no effect due to the transition term described in Section II-D.3. Then, we can notice from $t = 0.2s$ to $t = 3.4s$ the influence of this transition term on angular rates, visual errors, and on the target point trajectory, which slows the visual feedback. Finally, from $t = 3.4s$, we can see the influence of the integrator that reduces the dragging error. From $t = 13.2s$, all the residual error is totally eliminated by the integrator.

This use case shows that any terrestrial object that moves at a constant speed can eventually be tracked by our control law, as soon as the target does not move away from the image at the beginning when visual servoing effect is reduced by the transition and the integrator has no influence yet.
<figure><img src="image_11.png" /><figcaption>Fig. 11. Image trajectory of target point for a moving target of 1000km/h speed.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 12. Angular rate and visual feature error for the visual control law applied for satellite simulation for a moving target of 1000km/h speed.</figcaption></figure>

IV. CONCLUSION

In this paper, a visual servoing scheme has been proposed to precisely control the full attitude of a low-orbit Earth observation satellite, aiming to target terrestrial objects on the Earth surface. A new control law has been developed while taking into account translational velocities of the satellite induced by its orbit, possible target own speed, and rotational velocity constraints. Our image-based controller has been evaluated on a Cartesian robot simulating a downscaled satellite movement and realistic image acquisition. Simulations with real images have also been considered. Results demonstrated the ability to track any terrestrial object, motionless or not.

REFERENCES

[1] A.H. Abdul Hafez, P. Mithun, V.V. Anurag, S.V. Shah, and K. Madhava Krishna. Reactionless visual servoing of a multi-arm space robot combined with other manipulation tasks. *Robotics and Autonomous Systems*, 91:1–10, 2017.

[2] J.P. Alepuz, M. R. Emami, and J. Pomares. Direct image-based visual servoing of free-floating space manipulators. *Aerospace Science and Technology*, 55:1–9, 2016.

[3] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unifying framework. *Int. Journal of Computer Vision*, 56(3):221–255, 2004.

[4] M. Bernard, D. Decluseau, L. Gabet, and P. Nonin. 3d capabilities of pleiades satellite. *Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.*, 39:B3, 2012.

[5] F. Chaumette and S. Hutchinson. Visual servo control, Part I: Basic approaches. *IEEE Robotics and Automation Magazine*, 13(4):82–90, December 2006.

[6] F. Chaumette and S. Hutchinson. Visual servo control, Part II: Advanced approaches. *IEEE Robotics and Automation Magazine*, 14(1):109–118, March 2007.

[7] F. Chaumette, P. Rives, and B. Espiau. Classification and realization of the different vision-based tasks. In K. Hashimoto, editor, *Visual Servoing*, volume 7, pages 199–228. World Scientific Series in Robotics and Automated Systems, Singapour, 1993.

[8] A. Crétual and F. Chaumette. Dynamic stabilization of a pan and tilt camera for sub-marine image visualization. *Computer Vision and Image Understanding*, 79(1):47–65, July 2000.

[9] H Curtis. *Orbital mechanics for engineering students*. Butterworth-Heinemann, 2013.

[10] A.M. Elbeltagy, A. Youssef, A. Bayoumy Aly, and Y. Elhalwagy. Fixed ground-target tracking control of satellites using a nonlinear model predictive control. *Mathematical Modelling of Engineering Problems*, 5(1):11–20, 2018.

[11] M. A. Gleyzes, L. Perret, and P. Kubik. Pleiades system architecture and main performances. *Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, 39(1):537–542, 2012.

[12] K. Hashimoto and H. Kimura. Visual servoing with non linear observer. In *Proc. IEEE Int. Conf. on Robotics and Automation*, ICRA’95, pages 484–489, Nagoya, 1995.

[13] O Kermorgant and F Chaumette. Dealing with constraints in sensor-based robot control. *IEEE Trans. on Robotics*, 30(1):244–257, 2013.

[14] N. Mansard and F. Chaumette. Tasks sequencing for visual servoing. In *IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, IROS’04*, volume 1, pages 992–997, 2004.

[15] E. Marchand, F. Spindler, and F. Chaumette. ViSP for visual servoing: a generic software platform with a wide class of robot control skills. *IEEE Robotics and Automation Magazine*, 12(4):40–52, December 2005.

[16] P. Mithun, H. Pandya, A. Gaud, S. V Shah, and K. M. Krishna. Image based visual servoing for tumbling objects. In *IEEE/RSJ Int. Conf. on Intelligent Robots and Systems*, pages 2901–2908, 2018.

[17] A. Petit, E. Marchand, and K. Kanani. Vision-based space autonomous rendezvous : A case study. In *IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, IROS’11*, pages 619–624, San Francisco, USA, September 2011.

[18] J. Soubirane. Shaping the future of earth observation with pléiades neo. In *2019 9th International Conference on Recent Advances in Space Technologies (RAST)*, pages 399–401, 2019.

[19] A. Thieuw and H. Marcille. Pleiades-HR CMGs-based attitude control system design, development status and performances. *IFAC Proceedings Volumes*, 40(7):834–839, 2007.

[20] H. Wang, D. Guo, H. Xu, W. Chen, T. Liu, and K.K. Leang. Eye-in-hand tracking control of a free-floating space manipulator. *IEEE Trans. on Aerospace and Electronic Systems*, 53(4):1855–1865, 2017.

[21] H. Wang, Y. Liu, W. Chen, and Z. Wang. A new approach to dynamic eye-in-hand visual tracking using nonlinear observers. *IEEE/ASME Trans. on Mechatronics*, 16(2):387–394, 2010.