# Clevis and Tenon Assembly Using Visual Guiding Fields

Gilles Chabert, François Chaumette, Adolfo Suarez Roos

► To cite this version:

Gilles Chabert, François Chaumette, Adolfo Suarez Roos. Clevis and Tenon Assembly Using Visual Guiding Fields. IROS 2025 - IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct 2025, Hangzhou, China. pp.1-8. hal-05163570

HAL Id: hal-05163570

https://hal.science/hal-05163570v1

Submitted on 15 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Clevis and Tenon Assembly Using Visual Guiding Fields

Gilles Chabert<fnref n="1" />, François Chaumette<fnref n="2" />, Adolfo Suarez Roos<fnref n="1" />

*Abstract*— This paper presents a visual servoing strategy for a clevis and tenon joint assembly as typically performed in aircraft manufacturing. The desired velocity in the image acquired by the cameras observing the parts is extracted from a vector field that guides the motion of the moving part so that it follows an accurate 3D straight line trajectory. The vector field is designed so that it is continuous and globally exponentially stable whatever the initial configuration. The strategy has been successfully implemented and validated on a full-scale demonstrator, with coarse camera localization, while ensuring an assembly without any collision thanks to a maximal error less than 1 mm along a 10 cm trajectory.

## I. INTRODUCTION

This article describes an innovative solution to a central problem in aircraft construction, namely the assembly of tenon and clevis joints. These joints are used for connecting structural elements such as beams, which play a crucial role regarding rigidity of the aircraft or load distribution. For this reason, the assembly of beams require a perfect junction of their clevis-tenon interfaces, that is, with a millimeter accuracy.

The traditional way of proceeding consists in using a mechanically referenced fixture designed for a specific assembly task [1], [2], [3]. This kind of system is however not at all compatible with a flexible production, where a minimum of general-purpose tooling should permit assembly of variable products [4], [5]. Today, manufacturers' goal is to perform beam assembly directly with jigs, which can be viewed as a standard robotic platform for manipulating aircraft sections. Several aspects make this a challenge: the weights of sections (around a ton), their size (several meters) and the fact that clevis and tenons must be interlocked with each other without any collision, constraining the trajectory as in a peg-in-hole problem. Above all, the motion is now controlled through hoisting points which are located on fuselage ribs, far away from the beam. The geometry between the jigs and the clevis-tenon joint is not mastered to the millimeter. Since the clevis-tenon joint is hidden inside the aircraft sections, large-scale metrology techniques like those described in [6], [7] are not applicable. This challenge therefore opens the door to servoing approaches based on local sensors, and especially vision sensors.

Visual servoing is a well known technique to control the motion of a robot from visual data [8]. However, main

developments in this area are concerned with reaching a desired configuration without taking care about the transient phase. In short, the choice of the visual features used as inputs of the closed-loop control scheme have a strong effect on the performed trajectories, either in the image space (for image-based visual servoing), or the Cartesian space (for pose-based visual servoing), or even in both spaces (for hybrid/2 1/2D visual servoing), but the explicit specification and tracking of these trajectories are usually ignored. This means that any perturbation in the system will affect the achievement of these trajectories, which is not compatible with our task of following accurately a specified path. Few exceptions exist and concern the coupling between path planning and visual servoing for improving the robustness of visual servoing and for taking into account external constraints such as preserving the target visibility or avoiding obstacles [9], [10], [11]. Once these trajectories are specified, simple control strategies can be used to track them [12], [13]. However, these strategies require managing the time evolution along the trajectory tracking, which may be problematic for preserving the stability of the system as recalled at the end of Section II. Furthermore, all methods based on collision avoidance cannot be employed since the distances between the parts to be assembled are not available in our case. That is why another more appealing strategy relies on vector-field guided path-following [14], [15] since it is particularly adequate to accurately follow a specified path. Up to our knowledge, this strategy has been applied to visual servoing only in [16] for general purpose.

### A. Contributions

In this paper, we apply the vector-field control strategy in the context of an assembly application that requires to track accurately a 3D straight line trajectory. The vector field that we have designed does not solely enforce path straightness. It also allows the effector to respect a nominal speed up to some completion level of the task (say, 90% of parts are overlapping) and then decelerate towards the final pose. If the effector deviates from the assembly axis due to modelling or calibration errors, the velocity vector supplied by the vector field simultaneously allows decreasing the gap and progressing along the axis. The balance between both actions (correction versus progression) is adjustable through a single gain allowing the system to avoid contact while ensuring a fast enough task execution.

The first nice property of our vector field is that it encapsulates the different steps of the process while being globally exponentially stable whatever the initial configuration and ensuring transitions in a continuous way. The control law is

<footnote n="1">¹G. Chabert and A. Suarez Roos are with Nantes Université, IRT Jules Verne, Nantes, France, gilles.chabert@irt-jules-verne.fr, adolfo.suarez-roos@irt-jules-verne.fr</footnote>

<footnote n="2">²F. Chaumette is with Inria, Univ Rennes, CNRS, IRISA, Rennes, France, Francois.Chaumette@inria.fr</footnote>
smooth and avoids infinite accelerations. Second, it does not require 3D calculations, except for determining the images of the starting and ending configurations. The latter are the endpoints of a segment that can be viewed as the counterpart of the assembly axis in the sensor space. All images in this space are then mapped to velocity vectors by a simple closed-form expression. Also, the overall strategy does not require a precise knowledge of the camera locations. This point allows a fast deployment of cameras on the shopfloor, a decisive advantage from the industrial viewpoint.

The rest of the paper is organized as follows. In the next section, we first describe the task to be solved and the few setup requirements for our vision-based strategy. A quick remainder about visual servoing is then given. Section III details the design of our vector field and Section IV reports experimental results before we conclude.

## II. BACKGROUND

### A. Task description

The increasing production rates in aircraft industry are pushing manufacturers to develop auto-rigging solutions for the positioning and assembly of components. This work originates in this context, with the vision-based assembly of multiple clevis-tenon joints as the task to be performed. We start by listing the few assumptions that are actually required on the task.

First, we assume stiff actuators, low velocities and absence of contact, so that efforts and dynamics can be safely neglected. This will allow us to consider a purely kinematic model and to limit ourselves to proportional feedback laws. Second, one part is fixed and one is moving. For simplicity, we will identify the moving part with the robot effector in the sequel. Third, the fixed part is progressively occluded by the moving part during assembly, and not the other way around (see Fig. 1). Finally, the motion of the moving part should be a pure translation during assembly. The tolerance on this translational motion is quite stringent. In our application, the gap between tenons and clevis is only 1 mm, while parts overlap up to 10 cm.

These assumptions relate to the task itself. The visual servoing strategy introduces a few additional constraints. The first are on the number of cameras and their positioning. Although one camera is enough in theory for controlling a rigid body, some degrees of freedom (dofs), especially the depth, are more sensitive to errors. To get a good control of all dofs, we impose a set of $m$ cameras with different points of view so that the scene is well surrounded. We took in our experiments $m = 3$ cameras, two pointing horizontally and one vertically (see Fig. 1). The precise location does not matter except for the optical axis of each camera that must be approximately orthogonal to the assembly direction.

The other constraints are on the parts observability. Each part is represented by a set of $n$ interest points, that we assume to be precisely measured in the images acquired by the cameras. Since 3 cameras are used, we require each camera to detect at least 1 point to control all the 6 dofs, but the more the better. In our tenon-clevis example, a natural

<figure><img src="image_2.png" /><figcaption>Fig. 1. Setup example. This setup describes our experimental test-case. Two parts are assembled using multiple clevis-tenon joints. The male side (tenons) are on the fixed part and the female side (clevis) on the moving part. Pins are inserted in the corresponding holes of the clevis and tenons to lock the assembly. Three cameras observe the scene, two from the sides and one from the top. The cameras can be fixed at approximate positions without impacting the final results (their optical axis must be approximately orthogonal to the assembly axis).</figcaption></figure>

choice would be the centers of each hole, that could be measured with a sub-pixelic precision by averaging the pixels of the circle contours. However, for simplifying the image processing, markers have been painted around each hole during part manufacturing.

### B. Visual servoing

We recall here some basic facts about visual servoing that will serve as the basis for our solution.

Let us denote by $\mathbf{s}$ the vector representing sensor data, that is, the concatenation of the $x$ and $y$ pixel coordinates of all interest points, in the normalized image plane ($z = 1$) of their corresponding camera. The sensor space is therefore $\mathbb{R}^{2n}$. The simplest scheme of visual servoing, called image-based visual servoing, tries to impose a velocity $\dot{\mathbf{s}}$ in the sensor space using a proportional feedback law [17], [8]:

$$ \dot{\mathbf{s}} = \lambda(\mathbf{s}^* - \mathbf{s}) \qquad (1) $$

where $\mathbf{s}^*$ denotes the desired vector of sensor data, the one of the target pose, and $\lambda$ some positive gain factor. This sensor-space desired velocity is then converted to the kinematic screw $\mathbf{v}$ of the robot effector by calculating first a task-to-sensor Jacobian matrix $\mathbf{L}_s$ (see details in Section III-D) and then using its pseudo-inverse:

$$ \mathbf{v} = \mathbf{L}_s^+ \dot{\mathbf{s}}. \qquad (2) $$

If only 3 points of interest are controlled, and unless a singularity occurs, the prescribed velocity $\dot{\mathbf{s}}$ is indeed enforced and the 3 points follow straight lines with exponentially decreasing speed in the images; but the Cartesian motion of the effector is usually not a translation. Otherwise, $\dot{\mathbf{s}}$ is usually not realizable and the resulting trajectories are hardly predictable for the image points and for the effector. This happens because $(\mathbf{s}^* - \mathbf{s})$ is usually not anymore spanned by $\mathbf{L}_s$. In the worst case, this vector lies in $\mathbf{L}_s^\perp$ and no motion occurs [18]. So, when $n > 3$, only local asymptotic stability
is granted. In practice however, the stability domain is quite
large [8] and this approach works well in a free space, i.e.,
when the precise trajectory of the effector does not matter.
This is exactly what we use to place the effector in the initial
position before assembly.

Of course, this simple approach does not fit for the
assembly, where the moving part must follow a straight
line. A possible alternative is to use pose-based visual
servoing [19]<fnref n="1" />. In this case, the pose $^cT_e$ of the effector
(e) in the camera frame (c) is directly controlled and
images are just used to estimate this pose. It corresponds
to using a parameterization of $^cT_e$ as visual features $\mathbf{s}$
in (1) and (2). The first drawback with this approach is that
the pose estimation resorts to nonlinear optimization, which
may be highly sensitive to image noise, especially from flat
surfaces [18]. The second and main drawback is that merging
the pose estimations $^c_1T_e, \dots, ^c_mT_e$ obtained from different
cameras requires accurately knowing their relative locations.
This would require an accurate calibration of the complete
system while only a coarse localization is sufficient with the
strategy we propose. Notice that resorting to several cameras
for the pose estimation would be necessary here for basically
the same reasons as for control purposes, that is, to get a
precise estimation in all dofs. For this reason, obtaining the
relative pose $^c_1T_{c2}$ using $^c_1T_e$ $^eT_{c2}$ makes no sense as this
accumulates the errors that we precisely want to reduce.

Whatever the visual servoing approach is selected, i.e.,
image-based or pose-based, when a trajectory in feature
space $\mathbf{s}^*(t)$ has been specified, the classical control scheme
for following this trajectory is given by [8], [12]:

$$ \mathbf{v} = -\lambda \mathbf{L}_{\mathbf{s}}^{+} (\mathbf{s}(t) - \mathbf{s}^{*}(t)) + \mathbf{L}_{\mathbf{s}}^{+} \dot{\mathbf{s}}^{*}(t) \quad (3) $$

where the feedforward term $\mathbf{L}_{\mathbf{s}}^{+} \dot{\mathbf{s}}^{*}(t)$ is introduced for reducing the tracking error. However, if the system dynamics is not sufficiently reactive, the discrepancy between the current value $\mathbf{s}(t)$ and its desired value $\mathbf{s}^{*}(t)$ may become very large, which may destabilize the system. This is why removing the time-dependency for following a path, as done using the vector-field approach, is particularly adequate. Finally, standard control loops like (3) induce a somehow uniform system dynamics (namely, the integral curves of the differential equation (1)), whereas the desired behavior in our case has rather a case-wise nature: *the system must move straight at some speed in this region, slow down here, and stop there.* This kind of behavior cannot be cast in a standard ordinary differential equation.

### III. METHODOLOGY

#### A. *Outline of the proposed method*

Our method is based on two ideas. The first idea is
to design the control law explicitly under the form of a
vector field in the sensor space $\mathbf{s} \mapsto f(\mathbf{s})$, built to fulfill
the task requirements while being time-independent. By
creating a velocity vector field in the sensor space, the

<figure><img src="image_3.png" /><figcaption>Fig. 2. **Computation of the sensor-space path.** The red points on the image plane of the camera correspond to the sensor data vector $\mathbf{s}$. The sensor-space path is the segment joining the data of the initial ($e_1$) and final ($e_2$) pose of the effector. Both are obtained by simulating the measurements from the reference frame whose pose is computed using the green points.</figcaption></figure>

key advantage of image-based servoing is maintained while
naturally integrating task specifications. We recall that the
concept of guiding fields in control theory dates back to the
90s and has been applied to many fields including visual
servoing [16]. Our contribution is in the particular design of
this field with regard to the targeted task. The second idea
deals with the stability of motion. As said above, a desired
velocity $\dot{\mathbf{s}}$ may not be realizable. In practice, using an error-
based formula like (1), besides the non-uniform velocity,
results in a zigzagging effector around the assembly axis,
with a high risk of collision. Our goal is to create a vector
field of sensor data velocities that are close to be realizable.
To this end, we first focus on the target Cartesian trajectory
of the effector and build a corresponding ideal trajectory in
the sensor space. Then, we build the vector field around it.

It is convenient to consider the fixed part ($f$) as the refer-
ence frame. The target Cartesian trajectory of the effector is
a pure translation from an initial pose $^fT_{e_1}$ to a final pose
$^fT_{e_2}$ selected as the 4 × 4 identity matrix $\mathbf{I}_4$. The poses
match in all dofs but the translation along the axis so that
$^{e_1}T_{e_2} = ^{e_1}T_f$ $^fT_{e_2}$ is indeed a pure translation (see Fig. 2).

The expected sensor data $\mathbf{s}_1^*$ and $\mathbf{s}_2^*$ at these poses are computed in two steps. First, we calculate the pose of each camera: $^fT_{c_1}, \dots, ^fT_{c_m}$ using the image measurements of points located on the fixed part (the green points in Fig. 2). Then, the expected images of the moving part are obtained by virtually placing the effector in each camera frame using $^{e_1}T_f$ $^fT_{c_1}$ and $^{e_2}T_f$ $^fT_{c_2}$, and by simulating the corresponding images.<fnref n="2" /> We do not fall here into the pitfalls of camera pose estimation mentioned above, as the pose computation $^fT_{c_i}$ is only performed once (off-line) and the relative poses between the cameras are not involved. In other words, some dofs of a given camera pose may not be well-estimated but this does not matter: the combination of

<footnote n="1">¹In this paper, any pose $^aT_b$ between frames (a) and (b) is represented by a 4 × 4 homogeneous matrix.</footnote>

<footnote n="2">²The images of the moving part in the final pose $e_2$ cannot match the images of the fixed part because of the part thickness (look at the difference between the green and red points in Fig. 2).</footnote>
the *m* images will compensate for it. This point is further
clarified in Section III-D.

Now, since a straight line is preserved by perspective
projection, when the effector translates at constant speed
from $fT_{e_1}$ to $fT_{e_2}$, each point describes a segment in its own
image. However, the point velocity is not constant unless its
depth in the camera frame is constant. This is why we require
the camera optical axis to be approximately orthogonal to
the assembly axis. Under this assumption, all image point
velocities are almost constant, which means that the path of
the whole sensor data vector *s* is very close to the segment
[$s_1^*, s_2^*$] ⊂ $\mathbb{R}^{2n}$. So the idea is to use this segment in $\mathbb{R}^{2n}$ as
the sensor-space counterpart of the assembly axis.

The next section details the way the vector field is built
from it.

B. Vector field construction

Let *D* denotes the assembly distance, *i.e.*, the norm of the
translation vector in $e_1 T_{e_2}$, and let $V_0^*$ denotes the desired
Cartesian velocity. The corresponding (average) velocity in
sensor space is:

$$
v_0 = \frac{\| \mathbf{s}_2^* - \mathbf{s}_1^* \|}{D} V_0^* . \quad (4)
$$

As said before, by the way cameras are positioned, the actual
velocity of *s* in the ideal trajectory only slightly deviates from
its mean value, and the deviation is thus well absorbed by
the pseudo-inverse of *L*<sub>s</sub> in (2).

We call *baseline* the line passing through $s_1^*$ and $s_2^*$ and denote by $d_0$ the direction vector, that we orient from $s_2^*$ to $s_1^*$ on purpose:

$$
\mathbf{d}_0 = \frac{\mathbf{s}_1^* - \mathbf{s}_2^*}{\|\mathbf{s}_1^* - \mathbf{s}_2^*\|}. \qquad (5)
$$

Note that even if cameras are perfectly orthogonal to the
assembly axis, the velocity of each image point will be
constant but different, depending on the depth of each
camera. This just results in a vector $\mathbf{d}_0$ with components
of different amplitudes.

To build the vector field, the sensor space is split into three
areas (see Fig. 3)

1) the *outside area* where *f* brings *s* back to the baseline,
following a proportional law based on the gap (the
distance to the baseline) with gain $\lambda$.

2) the *corridor* where *f* brings *s* back to the baseline
while making it progress towards $s_2^*$. The velocity
inside the corridor has constant norm: $||f(s)|| = v_0$.

3) the *braking area* where *f* makes *s* converge to $s_2^*$ along the baseline. As we will see below, convergence is bounded by a decreasing exponential, either governed by a factor named $\alpha$ (bounds the projection onto the baseline), or $\lambda$ (bounds the projection orthogonally to the baseline).

Thus, the vector field depends on 3 parameters:

• the desired velocity $v_0$ in the corridor,

* a gain factor $\lambda$ modeling the attraction strength of the baseline,

<figure><img src="image_4.png" /><figcaption>Fig. 3. Partition of the sensor space. The space is split into three areas. The figure is in the 2D case $n = 1$.</figcaption></figure>

• a gain factor $\alpha$ modeling the attraction strength of $s_2$ along the baseline (in the braking area).

The outside area has no practical usage around the corridor
where it is defined rather for mathematical completeness.
However, its definition is important around the braking area
since both areas meet at the point $s_2^*$ (see Fig. 3). The braking
area is depicted by the shaded butterfly quadrilateral in Fig. 3.
Note that if we exceed the target point $s_2^*$ (something which
happens in practice) the braking area allows the system to
move backward (red vector in the figure).

The field is built as follows (see Fig. 4). We start by
decomposing the vector *s* into a scalar *d* and a vector *e*:

$$
\mathbf{s} = \mathbf{s}_2^* + d \mathbf{d}_0 + \mathbf{e} \quad (6)
$$

where *d* represents the projection of *s* onto the baseline with
$s_2^*$ as origin (*d* standing for “distance”), and *e* the remainder,
that is necessarily orthogonal to this line (*e* standing for
“error”). They can be quickly obtained if we compute once
for all the projection matrix onto the baseline:

$$
\mathbf{P} = \mathbf{d}_0 \mathbf{d}_0^T. \tag{7}
$$

<figure><img src="image_5.png" /><figcaption>Fig. 4. Decomposition of s.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Fig. 5. Plot of the vector field using Scilab$^{®}$. This plot is made in the 2D case ($n = 1$) with $s_1^* = (2, 1)$, $s_2^* = (4, 5)$, $\lambda = 0.8$, $\alpha = 0.2$ and $v_0 = 1$.</figcaption></figure>

We have then $d = d_0^T P(s - s_2^*)$ and $e = (I - P)(s - s_2^*)$. Note that $\sigma(d)$, the sign of $d$, tells us if the target $s_2^*$ has not been reached yet ($\sigma = 1$) or if it has been exceeded ($\sigma = -1$).

We finally define the mapping $v(d)$, a linear mapping with saturation, that represents the desired velocity along the baseline:

$$v(d) = \begin{cases} \sigma(d)v_0 & \text{if } |d| > \frac{v_0}{\alpha} \\ \alpha d & \text{otherwise} \end{cases} \qquad (8)$$

The condition $|d| > \frac{v_0}{\alpha}$ actually delimits the braking area from the corridor. By construction, the velocity $v(d)$ is continuous when crossing the two areas.

The field $f(s) = f(d, e)$ is finally defined as follows:

$$f(s) = \begin{cases} -\lambda e & \text{if } \lambda \|e\| \ge |v(d)| \quad (\text{outside area}) \\ -\lambda e - \sigma \sqrt{v^2(d) - \lambda^2 \|e\|^2} d_0 & \text{otherwise} \end{cases} \quad (9)$$

An illustration in the 2D case is shown in Fig. 5.

The main point in this definition actually lies in the inequality $\lambda \|e\| \ge |v(d)|$ as it shapes the boundary between the two areas. The field being continuous inside each, the global continuity of the field thus only depends on the choice for this inequality. It turns out that more intuitive choices like $\|s - s_2^*\| \le \frac{v_0}{\alpha}$ do not grant continuity.

### C. Vector field properties

We shall now exhibit the main properties of the field.

**Proposition 1:** The vector field is continuous.

**Proof:** First, the two formulas for $f(s)$ are continuous so only transition between the two cases have to be considered. When a point $s$ inside the corridor/braking area tends to the boundary, $\lambda \|e\|$ tends to $|v(d)|$ (by definition of the boundary) so the second formula for $f(s)$ tends to $-\lambda e$ (as the term under the square root is nullified), i.e., matches the first formula. So the field varies continuously when the boundary is crossed. ■

Note that the continuity of the field relies on the triangular shape of the braking area.

We now conclude by showing how the convergence is bounded. We denote for convenience $e(t) = \|e(t)\|$.

**Proposition 2:** The target point $s_2^*$ is globally exponentially stable, with a convergence rate bounded by $\min(\alpha, \lambda)$, except if $e(0) = 0$ in which case the rate is always $\alpha$.

**Proof:** First of all, it is clear from the field construction that any trajectory starting outside the braking area will converge in finite time to the braking area and then remains trapped inside forever. So only convergence inside the latter has to be considered.

For each $s(t) = s_2^* + d(t)d_0 + e(t)$, we have from the field definition $\dot{s} = f(d, e)$, in one hand $\dot{e} = -\lambda e$ so

$$\dot{e} = -\lambda e \qquad (10)$$

and in the other hand

$$\dot{d} = -\sqrt{v^2(d) - \lambda^2 e^2} = -\sqrt{\alpha^2 d^2 - \lambda^2 e^2}. \qquad (11)$$

We assume without loss of generality that $d > 0$, the case $d < 0$ being symmetrical. We also rule out the trivial case $e(0) = 0$ which directly entails $e(t) = 0$ and $d(t) = d(0)e^{-\alpha t}$. So, let us focus on the general case $e(0) \ne 0$ inside the braking area.

The idea is then to consider the dynamics of $r = \frac{e}{d}$. We have

$$\begin{aligned} \dot{r} &= \frac{\dot{e}d - \dot{e}e}{d^2} &= \frac{-\lambda ed + \sqrt{\alpha^2 d^2 - \lambda^2 e^2}e}{d^2} \\ &= \frac{-\lambda ed + e\sqrt{\alpha^2 d^2 - \lambda^2 (dr)^2}}{d^2} \\ &= -\lambda r + r\sqrt{\alpha^2 - \lambda^2 r^2} \end{aligned} \qquad (12)$$

so $\dot{r} = g(r)$ with $g(r) = -\lambda r + r\sqrt{\alpha^2 - \lambda^2 r^2}$. The domain of $g$ is $[0, \frac{\alpha}{\lambda}]$.

If $\alpha > \lambda$, the two roots of $g$ are 0 and $\hat{r} = \frac{\sqrt{\alpha^2 - \lambda^2}}{\lambda}$ but $g$ is positive on $(0, \hat{r})$ and negative on $(\hat{r}, \frac{\alpha}{\lambda}]$ so $\hat{r}$ is a globally stable fixpoint on $(0, \frac{\alpha}{\lambda}]$. So, when $t \to \infty$ we have $r(t) \to \hat{r}$, i.e., $d(t) \sim e(t) \sim e^{-\lambda t}$.

If $\alpha < \lambda$, $g$ is strictly negative on $(0, \frac{\alpha}{\lambda}]$ and $\frac{\partial g}{\partial r} = \alpha - \lambda$ so 0 is a globally exponentially stable fixpoint with an asymptotic convergence rate of $\alpha - \lambda$. So, when $t \to \infty$ we have $d(t) = e(t)/r(t) \sim e^{-\lambda}/e^{(\alpha-\lambda)t} = e^{-\alpha t}$.

Finally, when $\alpha = \lambda$, $r(t)$ still converges to zero but the first (and second) derivative of $g$ is null at 0, which means that $r(t)$ is slower than any decreasing exponential, that is, $\forall \beta > 0, \lim_{t \to \infty} e^{-\beta t}/r(t) = 0$, i.e., $\lim_{t \to \infty} e^{-\beta t} \frac{d(t)}{e(t)} = 0$. Taking the limit case $\beta = 0$ entails that $d$ is dominated by $e(t) \sim e^{-\lambda t} = e^{-\alpha t}$. ■

### D. Main control loop

It only remains to detail how the matrix $L_s$ is built for designing the main control loop. To this end, let us denote by $s_i$ the restriction of $s$ to the points observed by the $i^{th}$ camera and $v_i$ the kinematic screw of the camera with respect to the observed part and expressed in the camera frame.
We can derive from perspective geometry the following relation (see, e.g. [8])

$$ \dot{\mathbf{s}}_i = \mathbf{L}_i \mathbf{v}_i, \qquad (13) $$

where $\mathbf{L}_i$ is called the interaction matrix of the points observed by the $i^{th}$ camera.

It should be pointed out that this interaction matrix can also be interpreted as the sensitivity matrix of the camera pose estimation. Intuitively, a dof of the camera is well-estimated if a slight change in this dof entails a big change in s. Passing to the limit, i.e., replacing discrete changes by velocities, we see that a parameter of the pose is well-estimated if and only if the corresponding column of $\mathbf{L}_i$ has a large magnitude. Conversely, columns of $\mathbf{L}_i$ with low magnitude correspond to the parameters that are ill-estimated.

Now, to use all cameras simultaneously in the control loop, we first need to express the kinematic screw of each camera in a common frame. We naturally choose the frame of the effector. This is a classical step, done by post-multiplying $\mathbf{L}_i$ with the twist matrix $\mathbf{V}_i$ [8] whose value depends of the camera pose with respect to the effector. Next, using the kinematic screw of the effector with respect to the cameras requires a minus sign as for all eye-to-hand systems [8]. So, finally, the overall Jacobian matrix $\mathbf{L}_s$ is:

$$ \mathbf{L}_s = - \begin{pmatrix} \mathbf{L}_1 \mathbf{V}_1 \\ \vdots \\ \mathbf{L}_m \mathbf{V}_m \end{pmatrix}. \qquad (14) $$

Cameras are “complementary” if for each parameter of the effector pose, at least one vector $\mathbf{s}_i$ is sensitive to this parameter, i.e., no column of $\mathbf{L}_s$ has low magnitude. We may even require $\mathbf{L}_s^T \mathbf{L}_s$ to be numerically well-conditioned, a more stringent condition. One can check this is indeed the case with the camera positioning of Fig. 1, when observing the fixed part.

The control scheme is finally the following simple sequence, repeated while some convergence threshold is not reached, i.e., while $\|\mathbf{s} - \mathbf{s}_2^*\| > \epsilon$ for some given $\epsilon$:

1) Acquire the sensor data s.

2) From this, extract the desired sensor-data velocity using the vector field: $\dot{\mathbf{s}}_d = \mathbf{f}(\mathbf{s})$.

3) Compute $\mathbf{L}_s$ and get the desired Cartesian velocity $\mathbf{v}$ of the effector using $\mathbf{v} = \mathbf{L}_s^+ \dot{\mathbf{s}}_d$.

4) Send $\mathbf{v}$ to a robot Cartesian velocity controller.

If the $j^{th}$ parameter of the $i^{th}$ camera pose is ill-estimated, $(\dot{\mathbf{s}}_d)_i$ should have less influence in the control of the $j^{th}$ dof of the effector because, by construction of the vector field, this velocity depends on $(\mathbf{s}_1)_i$ and $(\mathbf{s}_2)_i$ which both depend on the camera pose. This is indeed the case: by definition of the pseudo-inverse, we have

$$ \mathbf{v} = (\mathbf{L}_s^T \mathbf{L}_s)^{-1} \mathbf{L}_s^T \dot{\mathbf{s}}_d = -(\mathbf{L}_s^T \mathbf{L}_s)^{-1} \sum_i \mathbf{V}_i^T \mathbf{L}_i^T (\dot{\mathbf{s}}_d)_i \quad (15) $$

and since the $j^{th}$ column of $\mathbf{L}_i$ has low magnitude, the dot product with $(\dot{\mathbf{s}}_d)_i$ in (15) will cancel the contribution of the

$i^{th}$ camera for the $j^{th}$ dof. This simple argument illustrates how the pseudo-inverse gives naturally more weight to the data with less error. Of course, matrix $\mathbf{L}_s$ in the control loop depends on s, and is not exactly the same as in the pose estimation step, but the orders of magnitudes are similar.

## IV. EXPERIMENTS

The tests presented here were carried out on a demonstrator for the assembly of the keel beam, a rigid part located on the bottom of the plane which plays an important role in structural integrity. The manufacturer's desire was to demonstrate the feasibility of an automated assembly of the beam with modules (fuselage sections) of representative size and weight, without necessarily seeking to reproduce exactly a production case. The module we used as the moving one was a front lower shell of an Airbus aircraft. The immobile module did not need to be a real one since only its interface matters. So this module was represented by a simple template or iron bird. It was also unnecessary and costly to install full beams. We only built the interfaces, that is, the tenons and clevis blocks and mounted them on the modules at positions that seemed roughly adequate. The precise design of these parts cannot be disclosed but a fairly accurate sketch is shown in Fig. 1. The interface includes 5 tenons and clevis materialized by a total of 80 markers. As already said, resorting to markers allowed us to focus on the visual servoing part by avoiding complex image processing.

Figs. 6 and 7 show the demonstrator. The actioners were 3 Cartesian positioners, each positioner comprising 3 prismatic joints made by ball screws. The actionning system was therefore a hyper-static parallel robot with a total of 9 controlled joints. In the one hand, the high gear ratio of the motors gave us a very good mechanical resolution and torque to support the module weight, while allowing large enough speed (up to 1 centimeter per second). In the other hand, the model of the whole kinematics chain had large uncertainties because some elements could not be measured, such as the actual geometry of the module (versus the CAD model), the geometry of the hoisting interfaces or the position where clevis were clamped. Visual servoing is precisely expected to compensate for that.

Our visual servoing strategy produces an input kinematic screw $\mathbf{v}$ for the female part each time new images are acquired and processed. The way this screw is translated

<figure><img src="image_7.png" /><figcaption>Fig. 6. Full-scale demonstrator.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 7. Zoom on the junction area.</figcaption></figure>

to actual joint velocities was done using classical control techniques and is out of the scope of the paper.

The 3 cameras we used were standard 12 Mp industrial cameras with pixel size 1.85 µm, calibrated by ourselves. Note that distortion could not be neglected and was also calibrated. One can see in Fig. 7 that the cameras were installed on adjustable supports, allowing us to vary one dof in position and orientation. The camera poses are initially unknown, as explained in Section III-A, so our algorithm starts by calculating them, a step usually referred to as extrinsic calibration. Changing their pose was therefore important for assessing the robustness of our algorithm.

The extrinsic calibration gives the camera poses w.r.t. the iron bird (noted $^fT_{c_i}$ above). These poses are calculated once from the markers of the tenons seen by each camera (either 8 or 16) and we used for that the `vpPose::computePose(...)` routine of the VISP library [20]. This routine implements first the POSIT method [21] and then an iterative non-linear minimization technique.

We made 20 assembly cycles. Each cycle includes two steps: the first step is a pre-positioning step that reaches the initial sensor target called $s_1^*$ above. We chose $s_1^*$ far enough from $s_2^*$ so that a traditional image-based servoing strategy could be implemented for achieving this step with no risk of collision. The second step makes the assembly and reaches $s_2^*$ using the vector field. The first 10 cycles were performed with fixed camera poses, the following 5 cycles with other fixed poses and the last 5 cycles were carried out by changing the poses of the cameras each time, both in translation and orientation. This could easily be done thanks to our adjustable supports. The extrinsic calibration had to be run each time the poses were changed. The gain values in the vector field were set empirically to $\lambda = 0.2$ and $\alpha = 0.05$. The ground truth was obtained thanks to photogrammetry, performed by a metrology certified company. Measured points were spread all over the module, including the clevis. The repeatability of the latter was our performance indicator.

<figure><img src="image_9.png" /><figcaption>Fig. 8. Images of a clevis. Six pictures extracted from the video stream of the first camera at different stage of the assembly (zoom on the upper clevis). The numbers and target paths of the 8 markers are plotted. These paths are actually not perfectly straight because distortion is taken into account.</figcaption></figure>

<table><caption>TABLE I</caption><thead><tr><th rowspan="2"></th><th colspan="2">X (mm)</th><th colspan="2">Y (mm)</th><th colspan="2">Z (mm)</th></tr><tr><th>range</th><th>stdev</th><th>range</th><th>stdev</th><th>range</th><th>stdev</th></tr></thead><tbody><tr><td>Cycles 1 to 10</td><td>0.50</td><td>0.17</td><td>0.14</td><td>0.04</td><td>0.37</td><td>0.11</td></tr><tr><td>Cycles 11 to 15</td><td>0.66</td><td>0.28</td><td>0.12</td><td>0.05</td><td>0.19</td><td>0.08</td></tr><tr><td>Cycles 16 to 20</td><td>0.51</td><td>0.21</td><td>0.04</td><td>0.02</td><td>0.11</td><td>0.04</td></tr></tbody></table>

Finally, the speed was set to 1 mm/s leading to a full cycle time of 3 minutes, a competitive result as compared to today.

The main result is that the assembly was successful for the 20 cycles: no contact at all occurred between the parts, that is, no friction whatsoever. Each time, pins of tight-fitting diameter (the type of pins used in real production) could be introduced through the holes at the end of each cycle with no effort, attesting a good alignment from the manufacturer's point of view, hence a valid 6-dof positioning of the part. Fig. 8 shows a sequence of images acquired by the first camera during an assembly cycle and illustrating the correct tracking of the image trajectories thanks to the vector field. These images are extracted from the video accompanying this article.

We could also make a repeatability analysis by comparing the final 3D positions of the photogrammetry targets on the female part. Results of the worst cases (among the targets) are reported in Table I. The first column reports the range of the measurements (distance between the maximal and minimal value) and the second the standard deviation. We can see first that the repeatability is much better for the Y and Z direction. This is due to the fact the assembly axis is roughly aligned with the X direction. So the final accuracy on this dimension is mainly governed by the convergence stopping criterion of our algorithm, whereas the other dimensions are somehow refined all along the trajectory. The stopping criterion we set in the sensor space was chosen by considering several factors, such as the camera resolution, the number of pixels covered by a visual marker and the performance of image processing. It results in a precision around 0.6 mm for the 3D position of a single visual marker
at roughly 40 cm from the camera. The averaging effect with the 80 markers explains why we obtain a standard deviation under this value. Second, we can see that last cycles seem better than the previous ones, although 5 cycles give no evidence but a tendency. A possible explanation is the following: in the 10 first cycles, the pose calculation of the cameras was only performed once at the very beginning (and similarly for the 5 following cycles), whereas the last 5 cycles imposed poses to be re-calculated each time. Consequently, all potential perturbation in the poses of the cameras in the lapse of the experiments (because of temperature variation, vibrations, etc.) were therefore never compensated for in the first cycles.

The general conclusion we can draw from these experiments is a repeatability of the process about 0.2 mm with a cycle time of a few minutes. This is rather a good result for an assembly of parts with a millimeter of clearance that takes currently 20-30 minutes.

## V. CONCLUSION

We have shown in this paper an innovative solution addressing the practical problem of automated clevis-tenon joint assembly that takes into account both an uncertain actuation model and a strong requirement for precision, with no globally referenced measuring system. The method has been developed and tested on a full-scale demonstrator performing a structural assembly task in aeronautics. There is no doubt that this method can be easily declined in other industrial contexts. For instance, a similar type of assembly is called mortise and tenon in the woodwork joinery [22] and is used among others for furniture making, timber framing, boat building or musical instruments manufacturing.

The method gathers several well-known ideas in control theory, such as image-based visual servoing and guiding vector fields. Our contribution lies in the way these concepts have been combined and adapted in a novel and effective way. For example, the vector field has been designed to integrate the different stages of the task in a smooth way. Inherent properties of the task have also been exploited to stick to sensor data as much as possible and avoid resorting to 3D calculations. In particular, the linearity of the path allows us to keep a pure image-based control loop once the initialization step is done. Extension of the method to curved trajectory is probably possible but makes us lose this advantage, such extension being anyway fairly irrelevant in practice.

This work is currently pursued in a new project with new challenges. One is to get rid of the markers and use the bare parts. Pose estimation and control will take advantage of the circle-shape of a clevis hole using image moments [23]. Another is to integrate additional tasks on the aircraft module, such as untwisting and orbital panel junction.

## VI. ACKNOWLEDGMENT

This work was part of the HAPPY project managed by IRT Jules Verne. The authors wish to associate the other

partners: Airbus Atlantic, Acssystème, and CNRS.

## REFERENCES

[1] P. Joshi, *Jigs and fixtures, third edition*. McGraw-Hill, 2017.

[2] G. Burley and J. Corbett, "Flyaway tooling for higher quality, more cost-effective, aerostructure," *SAE Technical Paper*, no. 981843, 1998.

[3] J. Hartmann, C. Meeker, M. Weller, and N. Izzard, "Determinate assembly of tooling allows concurrent design of Airbus wings and major assembly fixtures," *SAE Technical Paper*, no. 2004-01-2832, 2004.

[4] Y. Koren and M. Shpitalni, "Design of reconfigurable manufacturing systems," *Journal of Manufacturing Systems*, vol. 29, no. 4, pp. 130-141, 2010.

[5] K. Jackson, K. Efthymiou, and J. Borton, "Digital manufacturing and flexible assembly technologies for reconfigurable aerospace production systems," *Procedia CIRP*, vol. 52, pp. 274-279, 2016.

[6] P. Maropoulos, J. Muelaner, M. Summers, and O. Martin, "A new paradigm in large-scale assembly - Research priorities in measurement assisted assembly," *The Int. Journal of Advanced Manufacturing Technology*, vol. 70, 2014.

[7] R. S. Fernandez, A. Olabi, and O. Gibaru, "On-line accurate 3D positioning solution for robotic large-scale assembly using a vision system and a 6Dof tracking unit," in *3rd IEEE Advanced Information Technology, Electronic and Automation Control Conf. (IAEAC)*, 2018, pp. 682-688.

[8] F. Chaumette and S. Hutchinson, "Visual servoing and visual tracking," in *Handbook of Robotics*. Springer, 2008, pp. 563-583.

[9] Y. Mezouar and F. Chaumette, "Path planning for robust image-based control," *IEEE Trans. on Robotics and Automation*, vol. 18(4), pp. 534-549, 2002.

[10] G. Chesi, "Visual servoing path planning via homogeneous forms and LMI optimizations," *IEEE Trans. on Robotics*, vol. 25(2), pp. 281-291, 2009.

[11] M. Kazemi, K. Gupta, and M. Mehrandezh, "Randomized kinodynamic planning for robust visual servoing," *IEEE Trans. on Robotics*, vol. 29(5), pp. 1197-1211, 2013.

[12] F. Berry, P. Martinet, and J. Gallice, "Trajectory generation by visual servoing," in *IEEE Int. Conf. on Robotics and Automation (ICRA'97)*, 1997, pp. 1066-1072.

[13] Z. Zake, F. Chaumette, N. Pedemonte, and S. Caro, "Robust 2 1/2D visual servoing of a cable-driven parallel robot thanks to trajectory tracking," *IEEE Robotics and Automation Letters*, vol. 5(2), pp. 660-667, 2020.

[14] E. Rimon and D. Koditschek, "Exact robot navigation using artificial potential functions," *IEEE Trans. on Robotics and Automation*, vol. 8(5), pp. 501-518, 1992.

[15] L. Pimenta, G. Pereira, and R. Mesquita, "Fully continuous vector fields for mobile robot navigation on sequences of discrete triangular regions," in *IEEE Int. Conf. on Robotics and Automation (ICRA'07)*, 2007, pp. 1992-1997.

[16] J. Pomares and F. Torres, "Time independent tracking using 2D movement flow-based visual servoing," in *IEEE Int. Conf. on Robotics and Automation (ICRA'05)*, 2005, pp. 2541-2546.

[17] J. T. Feddema and O. Mitchell, "Vision-guided servoing with feature-based trajectory generation (for robots)," *IEEE Trans. on Robotics and Automation*, vol. 5, no. 5, pp. 691-700, 1989.

[18] F. Chaumette, "Potential problems of stability and convergence in image-based and position-based visual servoing," in *The Confluence of Vision and Control, LNCIS Series, No 237, Springer-Verlag*, 1998, pp. 66-78.

[19] W. J. Wilson, C. C. Hulls, and G. S. Bell, "Relative end-effector control using Cartesian position-based visual servoing," *IEEE Trans. on Robotics and Automation*, vol. 12, pp. 684-696, 1996.

[20] E. Marchand, F. Spindler, and F. Chaumette, "ViSP for visual servoing: a generic software platform with a wide class of robot control skills," *IEEE Robotics and Automation Magazine*, vol. 12, no. 4, pp. 40-52, 2005.

[21] D. DeMenton and L. S. David, "Model-based object pose in 25 lines of code," in *Int. Journal of Computer Vision*, vol. 15, 02 1998.

[22] J. Glath, R. Mesnil, M. Mimram, and O. Baverel, "Non-sequential assembly of mortise and tenon joints," *Automation in Construction*, vol. 154, 2023.

[23] O. Tahri and F. Chaumette, "Point-based and region-based image moments for visual servoing of planar objects," *IEEE Trans. on Robotics*, vol. 21, no. 6, pp. 1116-1127, 2005.