# Human-Robot Handovers using Task-Space Quadratic Programming

Mohamed Djeha, Antonin Dallard, Ahmed Zermane, Pierre Gergondet, Abderrahmane Kheddar

► To cite this version:

Mohamed Djeha, Antonin Dallard, Ahmed Zermane, Pierre Gergondet, Abderrahmane Kheddar. Human-Robot Handovers using Task-Space Quadratic Programming. RO-MAN 2022 - 31st IEEE International Conference on Robot and Human Interactive Communication, Aug 2022, Napoli, Italy. pp.518-523, 10.1109/RO-MAN53752.2022.9900626 . hal-03694798

HAL Id: hal-03694798

https://hal.science/hal-03694798

Submitted on 14 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Human-Robot Handovers using Task-Space Quadratic Programming

Mohamed Djeha, Antonin Dallard, Ahmed Zermane, Pierre Gergondet
and Abderrahmane Kheddar, Fellow, IEEE

*Abstract*— Bidirectional object handover between a human and a robot enables an important functionality skill in robotic human-centered manufacturing or services. The problem in achieving this skill lies in the capacity of any solution to deal with three important aspects: (i) synchronized timing for the handing over phases; (ii) the handling of object pose constraints; and (iii) understanding the haptic exchanging to seamlessly achieve some steps of the (i). We propose a new approach for (i) and (ii) consisting in explicitly formulating the handover process as constraints in a task-space quadratic programming control framework to achieve implicit time and trajectory encounters. Our method is implemented on Panda robotic arm taking objects from a human operator.

# I. INTRODUCTION

Human-centered robotics trends aim at close contact inter-
actions in different areas: hospitals, industry, malls, homes,
restaurants, etc. This cohabitation implies a large taxonomy
of human-robot interactions. Among the most challenging
one, seamlessly exchanging objects between a human and
robot as *giver* and *receiver* agents through (bi-directional)
handovers. Indeed, handing objects between humans is a very
frequent daily form of interaction in almost all domains [1].
Therefore, embedding human-centered robotic systems in
automation and services with handovers capabilities, is a
key enabler for rich cognitive interaction [2], [3]. Although
human-human handover is an intuitive behavior, it is a
result of complex and sophisticated learned social-cognitive
communication channels [4]. Grasping such a complexity in
an equivalently robot control formulation strategy is not a
trivial problem.

Indeed, how to codify the human-robot handover process
has been thoroughly addressed in the research community.
The state-of-the-art studies split the handover process in
two main phases: the pre-grasping phase [5]–[12] and the
grasping phase [13]–[17]. Such a breakdown is simply
chronological: the former focuses on detecting the object,
estimating/predicting the human movement and planning the
robot reaching motion toward the meeting point, whereas
the latter consists of understanding the interaction forces
(haptics) applied at the object by the two handing over agents
during the exchange such that to ensure a stable grasping.
In [15] a retraction phase is considered, which describes how
the giver and receiver move away after the latter gets the full
control of the object. Yet, for an exhaustive handover survey,
please refer to the excellent recent review in [1].

The authors are with the CNRS-University of Montpellier
LIRMM Interactive Digital Humans group, Montpellier, France.
mohamed.djeha@lirmm.fr

A. Kheddar and P. Gergondet are also with the CNRS-AIST Joint
Robotics Laboratory, IRL, Tsukuba, Japan.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Proposed seamless handover process executed without any prior knowledge about the HOL and final object orientation.</figcaption></figure>

Despite the huge amount of works addressing human-robot
handover, one issue remains open: how to ensure/codify an
adaptive behavior of the robot w.r.t to a human versatile
intention on *where* to exchange the object, aka *HandOver
Location* (HOL), with the robot, and in which configuration
(*orientation*). A large amount of existing works required
the knowledge of the HOL as a precondition either for
motion planning or control formulations of the pre-grasping
phase. In particular, the HOL is often considered either
as fixed (the robot moves systematically to a fixed spot
to pick-up the object) or pre-planned (real-time detection
and estimation of the object fixed-location) [1, Tables 1-2].
There are works that provide HOL estimation and on-line
prediction methods [15], [18]–[22]. However, such existing
approaches have at least one of the following shortcomings:

* Collecting data sets to train motion prediction models;

* Time-consuming computations leading to a hashed or a slow handover;

* Magic-numbers of meta-parameters to be tuned;

* Conservatism in the HOL prediction policy;

* Non-systematic success of the handover performance;

* The object orientation at the HOL is often kept the same during the experiments;

* Knowledge of the HOL is very often required.

In this work, we formulate the premises of a generic robot
control-strategy that ensures high fidelity and reliability of
reaching motion for bi-directional handover scenarios and
guarantees adaptability w.r.t the HOL and object orienta-
tion (Fig. 1). Particularly, our control formulation gathers
the advantages from both methods in [15], [17]. Indeed,
our method is HOL knowledge free. We only assume that
the object structural properties and dimensions are known,
and there exists a sensor that provides the pose (Cartesian
position and orientation) of the object of interest, e.g. [23].
Conversely to [17], our method ensures/codifies a real-time
proactive and adaptive robot motion w.r.t to the human versatile intention on where to exchange the object with the robot and in which configuration (orientation) without the need of offline acquired data. Our aim is to build on our existing multi-objective task-space optimization Quadratic-Programming (QP) framework [24], [25] that prove to be very powerful in an industrial context [26] and a human-assistance context [27]. In particular, our proposed idea relies on the concept of coupled tasks where the state of one task is forwarded as the reference for another task. More precisely, we formulate an observation task that estimates either human hand or object full-state in terms of pose, velocity and acceleration which are then forwarded as references for a trajectory-tracking task for the robot end-effector to track. Leveraging multi-objective QP control paradigm, these two tasks run concurrently in a leader-follower fashion: the object (leader) moves and converges to the HOL while the robot end-effector (follower) moves proactively toward the object while adjusting its orientation accordingly. From this standpoint, our approach is similar to the coupled Dynamical Systems (DS) in [15]. Except that our approach is computationally cheap and does not require the estimation of the HOL nor to separately learn human and robot DS models. To sum-up, our contributions to coupled task-space observer-controller are as follows:

* Systematic approach (no conservative policy of HOL prediction) with less parameter to tune (task gains only);

* No need for planning and time consuming processes of collecting data and model training;

* Requires no time-consuming computations resulting in a smooth (fluid) and seamless handover motion;

* Codifies rich behaviors with multi-objective QP control paradigm: proactive position and orientation control while accounting for safety constraints (collision avoidance, joint and actuation limits, etc.);

* Generalizes to bi-directional human-robot handovers;

* Needs only a sensor or estimator that provides the object pose, e.g. [23], [28], [29].

**Notations:** $x^* \in \mathbb{R}^3$ is expressed in frame $\mathcal{R}_*$. $R_*^* \in \text{SO}(3)$ denotes the rotation matrix of the frame $\mathcal{R}_{**}$ w.r.t $\mathcal{R}_*$. If $\mathcal{R}_* = \mathcal{R}_*$, the world-frame, then $R_*^* = R_{**}$. $[\omega \times] = \text{skew}(\omega)$ denotes the skew-symmetric matrix associated with the vector $\omega \in \mathbb{R}^3$. $\alpha_x$ denotes the velocity of $x \in \mathcal{X}$. If $\mathcal{X}$ is an Euclidean space then $\alpha_x = \dot{x}$. $\mathbf{q}_*^T = [\bar{\mathbf{q}}_* \ \hat{\mathbf{q}}_*] \in \mathbb{R}^4$ is the unit quaternion representation of $R_*$ where $\bar{\mathbf{q}}_* \in \mathbb{R}$ and $\hat{\mathbf{q}}_* \in \mathbb{R}^3$ with $\|\mathbf{q}_*\| = 1$, and $\mathbf{q}_*^{-1} = [\bar{\mathbf{q}}_* \ \hat{\mathbf{q}}_*]$. The vector part of the quaternion product $\mathbf{q}_* \otimes \mathbf{q}_{**}$ is denoted as $\hat{\mathbf{q}}_* \ominus \hat{\mathbf{q}}_{**} = \bar{\mathbf{q}}_{**} \hat{\mathbf{q}}_* + \bar{\mathbf{q}}_* \hat{\mathbf{q}}_{**} + [\hat{\mathbf{q}}_* \times] \hat{\mathbf{q}}_{**} \in \mathbb{R}^3$. If $\mathbf{q}_{**} = \mathbf{q}_*^{-1}$ then $\hat{\mathbf{q}}_* \ominus \hat{\mathbf{q}}_{**} = 0$. $P_* \in \mathbb{R}^3$ denotes the Cartesian coordinates of $\mathcal{R}_*$ origin. $\dot{P}_*, \ddot{P}_* \in \mathbb{R}^3$ are is linear velocity and acceleration. The angular velocity and acceleration of $\mathcal{R}_*$ frame are $\omega_*, \dot{\omega}_* \in \mathbb{R}^3$, respectively.

## II. HANDOVER PARADIGM

### A. Handover Control Problem

In order to perform object handover efficiently, the robot reaching-motion toward the HOL shall be synchronized

with the object's motion to achieve a one-shot continuous and smooth motion. If the HOL is known in advance, the control formulation resumes to perform simply a set-point reaching task for the end-effector. Nevertheless, the HOL and its timing as well as the object orientation are highly versatile: they may change during the handover motion or between two successive processes depending on the human intention and her/his posture. Handing over an object can even be aborted on the human decision while moving. Hence, planning the robot motion is likely to fail in such conditions. Alternatively, a reactive control method is more suitable for its ability to adjust the robot motion in real-time. We denote by end-effector the robot terminal link used to pick-up the object. It encompasses two-finger grippers, multi-finger hand and even a non-human like devices as a suction-cup or an electromagnetic gripper [30].

Our approach is to tackle the human-robot handover problem from a reactive closed-loop control perspective. We shall explain our approach when the human is the giver, then we show how it applies when s/he is the receiver. The HOL can be seen as an attractor toward which the object is converging (steered by the human). Since the HOL is not known a priori, the robot end-effector can track the object trajectory leading to a proactive motion such that when the object converges to the HOL, its trajectory becomes a point to which the end-effector converges. The same reasoning can be applied for the orientation. The remaining question is how to obtain the object trajectory in terms of pose $X_{obj} \in \mathbb{R}^7$, linear and angular velocity $\alpha_{X_{obj}} \in \mathbb{R}^6$ and linear and angular acceleration $\dot{\alpha}_{X_{obj}} \in \mathbb{R}^6$? These terms are encompassed as the object full-state

$$s_{\text{obj}} = \begin{bmatrix} X_{\text{obj}} \\ \alpha_{X_{\text{obj}}} \\ \dot{\alpha}_{X_{\text{obj}}} \end{bmatrix} \in \mathbb{R}^{19}. \quad (1)$$

Often, $s_{obj}$ cannot be directly obtained by the sensors as generally they provide only $X_{obj}$. Hence, an estimation of $s_{obj}$ needs to be constructed from $X_{obj}$ and it is denoted as

$$s_{\text{obs}} = \begin{bmatrix} X_{\text{obs}} \\ \alpha_{X_{\text{obs}}} \\ \dot{\alpha}_{X_{\text{obs}}} \end{bmatrix} \in \mathbb{R}^{19}. \quad (2)$$

To this end, the observation task is formulated as a PD controller that drives $X_{obs}$ toward $X_{obj}$, e.g., [23]. While converging, the observation task outputs also $\alpha_{X_{obs}}$ and $\dot{\alpha}_{X_{obs}}$, e.g. [31]. $s_{obs}$ terms are then used as references for the trajectory-tracking task formulated for the end-effector. Note that the observation task allows to generate trajectories for the orientation tracking without the need for offline planning methods [32]. The same approach can be adapted if the human is a receiver. In such case, the end-effector already holds the object and its full-state known by forward kinematics. Instead, the observation task constructs the full-state of the human hand [31].

Multi-objective task-space QP control has been extensively used as it enables to specify various tasks objectives while explicitly accounting for a set of convex constraints [33].
In [25], one single QP controller can be formulated for multi-robots systems either decoupled or interacting with each other using contact forces. In particular, the pre-grasping handover phase can be suitably formulated using the multi-robots QP by considering the robotic arm and the object as two decoupled robots where the former is a redundant multi-body system where all the joints are actuated, and the latter as a floating-base rigid-body system. Moreover, the observation task is formulated for the object to compute $s_{\text{obs}}$ and the trajectory-tracking task is formulated for the end-effector. The following subsections explicit in details our approach.

## B. Background

Consider a $n$ degree-of-freedom redundant robotic arm such that its joint configuration is defined by $q \in \mathbb{R}^n$. The robot is governed by the following equation of motion:

$$M(q)\ddot{q} + N(q, \dot{q}) = \tau + J_c^T f, \quad (3)$$

where $M(q) \in \mathbb{R}^{n \times n}$ is the inertia matrix, $N(q, \dot{q}) \in \mathbb{R}^n$ encompasses Coriolis, centrifugal and gravity torques, $\tau \in \mathbb{R}^n$ is the joint-torque, $J_c \in \mathbb{R}^{6 \times n}$ is the Jacobian at the contact point and $f \in \mathbb{R}^6$ is the contact wrench. Let us consider the frame $\mathcal{R}_{ee}$ rigidly attached to the end-effector and which pose is

$$X_{ee} = \begin{bmatrix} P_{ee} \\ \mathbf{q}_{ee} \end{bmatrix} \in \mathbb{R}^7. \quad (4)$$

Its velocity and acceleration are given as

$$\alpha_{X_{ee}} = \begin{bmatrix} \dot{P}_{ee} \\ \omega_{ee} \end{bmatrix} = J_{ee} \dot{q} \in \mathbb{R}^6, \quad (5)$$

$$\dot{\alpha}_{X_{ee}} = \begin{bmatrix} \ddot{P}_{ee} \\ \dot{\omega}_{ee} \end{bmatrix} = J_{ee} \ddot{q} + \dot{J}_{ee} \dot{q} \in \mathbb{R}^6, \quad (6)$$

where $J_{ee} \in \mathbb{R}^{6 \times n}$ is the end-effector Jacobian. Let us consider the object as a one-rigid-link robot with 6 DoF to which a frame $\mathcal{R}_{obj}$ is rigidly attached and which pose is

$$X_{\text{obj}} = \begin{bmatrix} P_{\text{obj}} \\ \mathbf{q}_{\text{obj}} \end{bmatrix} \in \mathbb{R}^7, \quad (7)$$

where $X_{\text{obj}}$ is assumed to be provided by a sensor. Its velocity and acceleration are given as

$$\alpha_{X_{\text{obj}}} = \begin{bmatrix} \dot{P}_{\text{obj}} \\ \omega_{\text{obj}} \end{bmatrix} \in \mathbb{R}^6, \quad \dot{\alpha}_{X_{\text{obj}}} = \begin{bmatrix} \ddot{P}_{\text{obj}} \\ \dot{\omega}_{\text{obj}} \end{bmatrix} \in \mathbb{R}^6. \quad (8)$$

The object structural and dimension properties are known.

## C. Observation Task

Let us consider an observed object to which a frame $\mathcal{R}_{\text{obs}}$ is rigidly attached which pose is

$$X_{\text{obs}} = \begin{bmatrix} P_{\text{obs}} \\ \mathbf{q}_{\text{obs}} \end{bmatrix} \in \mathbb{R}^7. \quad (9)$$

Assuming that the object (linear and angular) velocity and acceleration (8) are not provided by the sensor (which is likely the case), the observation task aims at constructing these non-measured states by estimating $s_{\text{obs}}$ in (2). This is achieved by the observation task that steers $X_{\text{obs}}$ toward

$X_{\text{obj}}$ by keeping the observation error $e_{\text{obs}}$ as small as possible such that

$$e_{\text{obs}} = \begin{bmatrix} P_{\text{obs}} - P_{\text{obj}} \\ \hat{\mathbf{q}}_{\text{obs}} \ominus \hat{\mathbf{q}}_{\text{obj}}^{-1} \end{bmatrix} \in \mathbb{R}^6. \quad (10)$$

The observation error velocity and acceleration are given as

$$\alpha_{e_{\text{obs}}} = \alpha_{X_{\text{obs}}}, \quad \dot{\alpha}_{e_{\text{obs}}} = \dot{\alpha}_{X_{\text{obs}}}. \quad (11)$$

The observation task state is defined as

$$\eta_{\text{obs}} = \begin{bmatrix} e_{\text{obs}} \\ \alpha_{e_{\text{obs}}} \end{bmatrix} \in \mathbb{R}^{12}. \quad (12)$$

Thus, the observation task is formulated as follows

$$\dot{\eta}_{\text{obs}} = \begin{bmatrix} 0 & I \\ 0 & 0 \end{bmatrix} \eta_{\text{obs}} + \begin{bmatrix} 0 \\ I \end{bmatrix} \mu_{\text{obs}}, \quad \mu_{\text{obs}} = \dot{\alpha}_{X_{\text{obs}}} \quad (13)$$

Choosing $\mu_{\text{obs}}$ in (13) a linear state feedback

$$\mu_{\text{obs}} = -[K_{s_{\text{obs}}} \ K_{d_{\text{obs}}}] \eta_{\text{obs}} = -K_{\text{obs}} \eta_{\text{obs}}, \quad (14)$$

with $K_{s_{\text{obs}}}, K_{d_{\text{obs}}} \in \mathbb{R}^{6 \times 6}$ are diagonal positive-definite matrices denoting the stiffness and damping gains; it yields to the observation-task closed-loop dynamics

$$\dot{\eta}_{\text{obs}} = A_{\text{obs}}\eta_{\text{obs}}, \quad A_{\text{obs}} = \begin{bmatrix} 0 & I \\ -K_{s_{\text{obs}}} & -K_{d_{\text{obs}}} \end{bmatrix}, \quad (15)$$

with $A_{\text{obs}}$ Hurwitz [34]. Note that $\eta_{\text{obs}}$ only converges asymptotically to zero if the object is static ($X_{\text{obs}}$ constant). However, choosing high gain values typically enables a fast convergence and keeps $\|e_{\text{obs}}\|$ sufficiently small.

The benefits of the observation task are three folds: (i) allows a bounded estimation of $s_{\text{obs}}$ in (2) given (13)–(15); (ii) $X_{\text{obj}}$ is low-pass filtered by the closed-loop observation task dynamics (15); (iii) enables an online generation of a smooth twice-differentiable trajectory<fnref n="1" /> for the Cartesian and orientation coordinates of $\mathcal{R}_{\text{obs}}$ without the needs of an offline planning methods [32]. The latter advantage is the core idea presented in this paper: the observation task outputs trajectory references required by the trajectory-tracking task in order to achieve a proactive handover process. More importantly, these two tasks are integrated in the multi-objective QP controller as shown in Section II-F.

Once having $s_{\text{obs}}$, we can compute the full-state for any other frame $\mathcal{R}_*$ attached to the object (for which the local pose $X_*^{\text{obs}}$ is known) by simply applying the classical kinematic relations for position, velocity and acceleration:

$$P_* = P_{\text{obs}} + R_{\text{obs}} P_*^{\text{obs}}, \quad (16)$$

$$\dot{P}_* = \dot{P}_{\text{obs}} + [\omega_{\text{obs}} \times] R_{\text{obs}} P_*^{\text{obs}}, \quad (17)$$

$$\ddot{P}_* = \ddot{P}_{\text{obs}} + [\dot{\omega}_{\text{obs}} \times] R_{\text{obs}} P_*^{\text{obs}} + [\omega_{\text{obs}} \times]^2 R_{\text{obs}} P_*^{\text{obs}}, \quad (18)$$

$$R_* = R_{\text{obs}} R_*^{\text{obs}}. \quad (19)$$

Note that the rigid body assumption implies that $\dot{P}_*^{\text{obs}} = 0$, $\ddot{P}_*^{\text{obs}} = 0$, $\omega_* = \omega_{\text{obs}}$ and $\dot{\omega}_* = \dot{\omega}_{\text{obs}}$.

<footnote n="1">¹This is also known as a reference model-based approach for trajectory references generation [34, Chapter 13]. In addition, the trajectory feedforward terms are generated in real-time since the observation task is updated at the same frequency of QP no matter the sampling-frequency of the sensor.</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2. Illustrative scheme showing the frames $\mathcal{R}_w$, $\mathcal{R}_{ee}$, $\mathcal{R}_{obj}$, $\mathcal{R}_{obs}$ and $\mathcal{R}_{grasp}$. The observed object is tracking the actual object giving the data from sensor thanks to the observation task. The end-effector tracks the observation task state yielding an anticipatory motion toward the HOL where it ultimately meets the object. The colored unit vectors in $\mathcal{R}_{ee}$ track their corresponding in $\mathcal{R}_{grasp}$. For instance, if the object is cylindrical, only the yellow unit vectors must be aligned.</figcaption></figure>

### D. Trajectory-Tracking Task

Let $\mathcal{R}_{grasp}$ be the grasping frame rigidly attached to the object and which pose is defined as

$$X_{\text{grasp}} = \begin{bmatrix} P_{\text{grasp}} \\ \mathbf{q}_{\text{obs}} \end{bmatrix} \in \mathbb{R}^7, \qquad (20)$$

such that $P_{\text{grasp}} \in \mathbb{R}^3$ denotes the coordinates of the point where the end-effector grasps the object and is computed from (16), and $\mathbf{q}_{\text{obs}}$ is obtained from $\mathcal{R}_{\text{grasp}}$ in (19). $\mathcal{R}_{\text{grasp}}$ velocity and acceleration are computed as

$$\alpha_{X_{\text{grasp}}} = \begin{bmatrix} \dot{P}_{\text{grasp}} \\ \omega_{\text{obs}} \end{bmatrix} \in \mathbb{R}^6, \quad \dot{\alpha}_{X_{\text{grasp}}} = \begin{bmatrix} \ddot{P}_{\text{grasp}} \\ \dot{\omega}_{\text{obs}} \end{bmatrix} \in \mathbb{R}^6. \quad (21)$$

Let us define the trajectory-tracking task error as

$$e_{tt} = \begin{bmatrix} P_{ee} - P_{\text{grasp}} \\ \dot{\mathbf{q}}_{ee} \ominus \dot{\mathbf{q}}_{\text{grasp}}^{-1} \end{bmatrix} \in \mathbb{R}^6, \qquad (22)$$

and which derivative is given as

$$\alpha_{e_{tt}} = \begin{bmatrix} \dot{P}_{ee} - \dot{P}_{\text{grasp}} \\ \omega_{ee} - \omega_{\text{obs}} \end{bmatrix} = \alpha_{X_{ee}} - \alpha_{X_{\text{grasp}}} \in \mathbb{R}^6. \quad (23)$$

Let us denote the trajectory-tracking task state

$$\eta_{tt} = \begin{bmatrix} e_{tt} \\ \alpha_{e_{tt}} \end{bmatrix} \in \mathbb{R}^{12}. \qquad (24)$$

Hence, the trajectory-tracking task dynamics is formulated

$$\dot{\eta}_{tt} = \begin{bmatrix} 0 & I \\ 0 & 0 \end{bmatrix} \eta_{tt} + \begin{bmatrix} 0 \\ I \end{bmatrix} \mu_{tt}, \qquad (25)$$

$$\mu_{tt} = \dot{\alpha}_{X_{ee}} - \dot{\alpha}_{X_{\text{grasp}}} = J_{ee}\ddot{q} + \dot{J}_{ee}\dot{q} - \dot{\alpha}_{X_{\text{grasp}}}. \qquad (26)$$

Then, choosing $\mu_{tt}$ in (26) as

$$\mu_{tt} = -[K_{s_{tt}} K_{d_{tt}}] \eta_{tt} = -K_{tt} \eta_{tt}, \qquad (27)$$

with $K_{s_{tt}}$, $K_{d_{tt}} \in \mathbb{R}^{6 \times 6}$ are diagonal positive-definite matrices denoting the stiffness and damping gains; it yields to the following trajectory-tracking task closed-loop dynamics

$$\dot{\eta}_{tt} = A_{tt}\eta_{tt}, \quad A_{tt} = \begin{bmatrix} 0 & I \\ -K_{s_{tt}} & -K_{d_{tt}} \end{bmatrix}. \qquad (28)$$

where $A_{tt}$ is Hurwitz. This enables a global asymptotic convergence of $\eta_{tt}$ to the origin [34]. Commonly, the trajectory is planned such that the initial reference trajectory pose is as close as possible to the current end-effector pose $X_{ee}(t_0)$ which indeed ensures that the tracking error $e_{tt}(t_0)$ is small and thereby enforces the end-effector pose to stick on trajectory forward in time. However, when the trajectory starts far from the initial end-effector pose, the latter converges to the trajectory with an asymptotic decay of the tracking error $\eta_{tt}$. This property enables the anticipatory motion of the end-effector which moves proactively toward the object grasping position (see Fig. 2).

### E. Posture Task

The posture task is mainly intended to solve the remaining redundant states $(q, \dot{q})$ and keep them bounded. Let $q_{\text{ref}} \in \mathbb{R}^n$ be a given reference posture designed generally to represent a suitable robot posture (e.g., elbow up). Let us define the posture task state as

$$\eta_{\text{pos}} = \begin{bmatrix} e_{\text{pos}} \\ \dot{e}_{\text{pos}} \end{bmatrix} = \begin{bmatrix} q - q_{\text{ref}} \\ \dot{q} \end{bmatrix}. \qquad (29)$$

The posture task dynamics is thereby

$$\dot{\eta}_{\text{pos}} = \begin{bmatrix} 0 & I \\ 0 & 0 \end{bmatrix} \eta_{\text{pos}} + \begin{bmatrix} 0 \\ I \end{bmatrix} \mu_{\text{pos}}, \quad \mu_{\text{pos}} = \ddot{q}. \qquad (30)$$

Choosing $\mu_{\text{pos}}$ in (30) as

$$\mu_{\text{pos}} = -[K_{s_{\text{pos}}} \quad K_{d_{\text{pos}}}] \eta_{\text{pos}} = -K_{\text{pos}}, \qquad (31)$$

leads to the posture task closed-loop dynamics

$$\dot{\eta}_{\text{pos}} = A_{\text{pos}}\eta_{\text{pos}}, \quad A_{\text{pos}} = \begin{bmatrix} 0 & I \\ -K_{s_{\text{pos}}} & -K_{d_{\text{pos}}} \end{bmatrix}, \qquad (32)$$

where $A_{\text{pos}}$ is Hurwitz given that the stiffness and damping gains $K_{s_{\text{pos}}}$, $K_{d_{\text{pos}}}$ are diagonal positive-definite. Eq. (32) yields asymptotic convergence of $\eta_{\text{pos}}$ to the origin. However, this is only the case when the posture task is not in conflict with the other tasks and constraints among which the posture task has the lowest priority. Alternatively, $e_{\text{pos}}$ is only ensured to be uniformly ultimately bounded [24].

### F. Multi-Tasks QP Formulation

Let us denote $\chi^T = [\ddot{q}^T \dot{\alpha}_{\text{obs}}^T f^T] \in \mathbb{R}^{n+6+3}$ the vector encompassing the linear and angular acceleration of the observed rigid-body object, the joint-acceleration of the robotic arm, and the contact force. Given the affinity of (13) w.r.t. $\dot{\alpha}_{X_{\text{obs}}}$, that of (26)(30) w.r.t. $\ddot{q}$ and that of (3) w.r.t. $\ddot{q}$ and $f$, then we can combine all the tasks and constraint in a single weighted-prioritized QP formulation

$$\min_{\chi} \sum_{i} w_i \| E_{\chi}^i \chi + F_{\chi}^i \|^2 \qquad (33a)$$

$$\text{S.t.: } C_{\chi} \chi \le d_{\chi} \qquad (33b)$$

where $w_i > 0$ is the associated weight to each task $i$. QP formulation (33) enables to: (i) combine the different competing tasks in (33a) by settling a soft prioritization scheme; (ii) account for unilateral constraints (33b) that
<figure><img src="image_4.png" /><figcaption>Fig. 3. Perception Neuron motion capture sensor-suit used for the handover experiments and mounted on the left arm. Three IMUs (highlighted in colored squares) mounted on the left hand are used to provide an estimation of the hand pose (the orange square).</figcaption></figure>

embed limitations like joint-position, velocity, acceleration and torque limits, as well as safety aspects like (self)-collision avoidance to ensure safe grasping [35]; (iii) find optimal $\ddot{q}$ that generates the robotic arm motion that achieve at best all the tasks while fulfilling all the constraints; and (iv) handle multi-robot control where the robotic arm and the object are considered as two distinct robots entities which opens the possibility to add other robots to achieve the handover (e.g., bi-arm handover) while still using one compact formulation [25].

### III. EXPERIMENTAL RESULT

For demonstration, we implemented our approach on the open-source code implementation of the QP controller interface `mc_rtc<fnref n="2" />`. Based on the embedded sensors data, `mc_rtc` builds and solves the QP problem at each control cycle (1 ms). Experiments are conducted using 7-DoF robotic arm Panda. Perception Neuron<fnref n="3" /> sensor-suit has been used for the pose measurement (Fig. 3). An API provides the hand pose in an arbitrarily fixed-frame at a frequency of 60 Hz. A calibration step is necessary to correctly coincides the sensor frame with $R_w$. The calibration process is repeated at the beginning of each handover experiment to avoid drifting. When the human is a giver, the object is assumed to be rigidly attached to the hand, and thereby its pose can be computed based on the hand's one. Several handover scenarios have been performed where the robot starts from different configurations. In addition, multiple HOLs have been tried (Fig. 4). The observation task gains have been set to high values $K_{s_{obs}} = 1500I$, $K_{d_{obs}} = 2\sqrt{K_{s_{obs}}}$ to ensure an accurate object pose estimation $s_{obj}$ (1). Fig. 5 shows how $X_{obs}$ tracks $X_{obj}$. The high observation task gains allow an accurate pose tracking. Concurrently, it allows obtaining a good estimation of $\alpha_{X_{obj}}$ and $\dot{\alpha}_{X_{obj}}$ by computing $\alpha_{X_{obs}}$ and $\dot{\alpha}_{X_{obs}}$ from (15). $s_{obj}$ is then forwarded to the trajectory-tracking task where the gains of the orientation part have been chosen twice bigger than those of the translation to ensure that end-effector reaches the object while its orientation has already converged to the target. The stiffness of the translation part is fixed at $I$ and the corresponding

<figure><img src="image_5.png" /><figcaption>Fig. 4. Different HOLs (red) desired by the human operator. The object is a black cylindrical container. The transparent robots represent the initial configuration.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Position (left) and orientation (right with RPY angles) of $R_{obj}$ (dashed) and its estimation $R_{obs}$ (solid) obtained by the observation task.</figcaption></figure>

damping $2\sqrt{I}$. From Fig. 6, the object and the end-effector meet seamlessly and smoothly at the HOL without being specified *a priori*, see Fig. 7. Furthermore, the end-effector adapts reactively to the variation of the grasping point as shown in the attached video.

### IV. CONCLUSION

We propose an original new approach for human-robot handovers formulated using task-space optimization controller. The core idea is based on a novel implementation of tasks interdependency, which consists in providing the state of a task (of estimation nature) as an input to another task so that both meet at the HOL without explicit time or object configuration specification. Our experimental results confirmed very promising performances of simple handovers focusing mainly on the reaching phase. This new approach raised very promising novel features of the task-space optimization control schemes. Extension in terms of functionalities and theoretical investigation on how observer-tasks can

<figure><img src="image_7.png" /><figcaption>Fig. 6. Position (left) and orientation (right with RPY angles) of the frames $R_{grasp}$ (dashed) and $R_{ee}$ (solid) obtained by the trajectory-tracking task. The sudden variation in the yaw angle is sound because the angle is bounded between $-\pi$ and $\pi$. This does not induce any singularity issue since the orientation is implemented using unit quaternions.</figcaption></figure>

<footnote n="2">²https://jrl-umi3218.github.io/mc_rtc/index.html</footnote>

<footnote n="3">³https://neuronmocap.com/</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 7. 3D trajectories of $R_{grasp}$ (blue) and $R_{ee}$ (orange). The initial and terminal points are denoted with a square and a circle, respectively.</figcaption></figure>

be embedded in a unified task-space observer/control framework through task interdependencies and constraints between task errors opens perspectives in embedding scheduling in the task-space control formalism. Our on-going and future research is focusing on this issue together with complete-phases and more complex handovers considering force cues.

## REFERENCES

[1] V. Ortenzi, A. Cosgun, T. Pardi, W. P. Chan, E. Croft, and D. Kulić, "Object handovers: A review for robotics," *IEEE Transactions on Robotics*, vol. 37, no. 6, pp. 1855-1873, 2021.

[2] A. Ajoudani, A. M. Zanchettin, S. Ivaldi, A. Albu-Schäffer, K. Kousuge, and O. Khatib, "Progress and prospects of the human-robot collaboration," *Autonomous Robots*, vol. 42, pp. 957-975, Jun 2018.

[3] A. Billard and D. Kragic, "Trends and challenges in robot manipulation," *Science*, vol. 364, no. 6446, p. eaat8414, 2019.

[4] K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, S. S. Srinivasa, M. Cakmak, and V. Micelli, "Toward seamless human-robot handovers," *J. Hum.-Robot Interact.*, vol. 2, p. 112-132, feb 2013.

[5] S. Shibata, K. Tanaka, and A. Shimizu, "Experimental analysis of handing over," in *Proceedings 4th IEEE International Workshop on Robot and Human Communication*, pp. 53-58, 1995.

[6] J. Waldhart, M. Gharbi, and R. Alami, "Planning handovers involving humans and robots in constrained environment," in *IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp. 6473-6478, 2015.

[7] J. Mainprice, M. Gharbi, T. Siméon, and R. Alami, "Sharing effort in planning human-robot handover tasks," in *IEEE International Symposium on Robot and Human Interactive Communication*, pp. 764-770, 2012.

[8] M. Huber, M. Rickert, A. Knoll, T. Brandt, and S. Glasauer, "Human-robot interaction in handing-over tasks," in *IEEE International Symposium on Robot and Human Interactive Communication*, pp. 107-112, 2008.

[9] C. Shi, M. Shiomi, C. Smith, T. Kanda, and H. Ishiguro, "A model of distributional handing interaction for a mobile robot," in *Robotics: Science and Systems*, (Berlin, Germany), June 2013.

[10] A. Koene, S. Endo, A. Remazeilles, M. Prada, and A. M. Wing, "Experimental testing of the coglaboration prototype system for fluent human-robot object handover interactions," in *IEEE International Symposium on Robot and Human Interactive Communication*, pp. 249-254, 2014.

[11] A. Remazeilles, M. Prada, A. Garzo, I. Rasines, M. Controzzi, C. Cipriani, I. Strazzulla, C. Peccia, J. Canseco, D. Cabaneros, et al., "Towards human-robot object exchange lessons learned," in *Workshop on Human-Robot handover in RSS*, 2015.

[12] M. Bdiwi, J. Suchý, and A. Winkler, "Handing-over model-free objects to human hand with the help of vision/force robot control," in *International Multi-Conferences on Systems, Signals Devices*, pp. 1-6, 2013.

[13] K. Nagata, Y. Oosaki, M. Kakikura, and H. Tsukune, "Delivery by hand between human and robot based on fingertip force-torque information," in *IEEE/RSJ International Conference on Intelligent Robots and Systems*, vol. 2, pp. 750-757 vol.2, 1998.

[14] W. P. Chan, C. A. Parker, H. M. V. der Loos, and E. A. Croft, "A human-inspired object handover controller," *International Journal of Robotics Research*, vol. 32, no. 8, pp. 971-983, 2013.

[15] J. R. Medina, F. Duvallet, M. Karnam, and A. Billard, "A human-inspired controller for fluid human-robot handovers," in *IEEE-RAS International Conference on Humanoid Robots*, pp. 324-331, 2016.

[16] G. Solak and L. Jamone, "Learning by demonstration and robust control of dexterous in-hand robotic manipulation skills," in *IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp. 8246-8251, 2019.

[17] M. Costanzo, G. De Maria, and C. Natale, "Handover control for human-robot and robot-robot collaboration," *Frontiers in Robotics and AI*, vol. 8, 2021.

[18] M. Prada, A. Remazeilles, A. Koene, and S. Endo, "Implementation and experimental validation of dynamic movement primitives for object handover," in *IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp. 2146-2153, 2014.

[19] N. Vahrenkamp, H. Arnst, M. Wächter, D. Schiebener, P. Sotiropoulos, M. Kowalik, and T. Asfour, "Workspace analysis for planning human-robot interaction tasks," in *IEEE-RAS International Conference on Humanoid Robots*, pp. 1298-1303, 2016.

[20] G. J. Maeda, G. Neumann, M. Ewerton, R. Lioutikov, O. Kroemer, and J. Peters, "Probabilistic movement primitives for coordination of multiple human-robot collaborative tasks," *Autonomous Robots*, vol. 41, pp. 593-612, Mar 2017.

[21] D. Widmann and Y. Karayiannidis, "Human motion prediction in human-robot handovers based on dynamic movement primitives," in *European Control Conference*, pp. 2781-2787, 2018.

[22] H. Nemlekar, D. Dutia, and Z. Li, "Object transfer point estimation for fluent human-robot handovers," in *International Conference on Robotics and Automation*, pp. 2627-2633, 2019.

[23] A. Paolillo, K. Chappellet, A. Bolotnikova, and A. Kheddar, "Inter-linked visual tracking and robotic manipulation of articulated objects," *IEEE Robotics and Automation Letters*, vol. 3, no. 4, pp. 2746-2753, 2018.

[24] K. Bouyarmane and A. Kheddar, "On weight-prioritized multitask control of humanoid robots," *IEEE Transactions on Automatic Control*, vol. 63, pp. 1632-1647, June 2018.

[25] K. Bouyarmane, K. Chappellet, J. Vaillant, and A. Kheddar, "Quadratic programming for multirobot and task-space force control," *IEEE Transactions on Robotics*, vol. 35, pp. 64-77, Feb 2019.

[26] A. Kheddar, S. Caron, P. Gergondet, A. Comport, A. Tanguy, C. Ott, B. Henze, G. Mesesan, J. Englberger, M. A. Roa, P.-B. Wieber, F. Chaumette, F. Spindler, G. Oriolo, L. Lanari, A. Escande, K. Chappellet, F. Kanehiro, and P. Rabaté, "Humanoid robots in aircraft manufacturing: The airbus use cases," *IEEE Robotics Automation Magazine*, vol. 26, no. 4, pp. 30-45, 2019.

[27] A. Bolotnikova, P. Gergondet, A. Tanguy, S. Courtois, and A. Kheddar, "Task-space control interface for softbank humanoid robots and its human-robot interaction applications," in *IEEE/SICE International Symposium on System Integration*, (Iwaki, Japan), pp. 560-565, 11-14 January 2021.

[28] M. Rana and V. Mittal, "Wearable sensors for real-time kinematics analysis in sports: A review," *IEEE Sensors Journal*, vol. 21, no. 2, pp. 1187-1207, 2021.

[29] M. Jongeneel, A. Bernardino, N. Van De Wouw, and A. Saccon, "Model-Based Visual Object Tracking with Impact Collision Models," in *International Conference on Intelligent Robots and Systems*, (Prague, Czech Republic), Sept. 2021.

[30] M. K. X. J. Pan, E. A. Croft, and G. Niemeyer, "Exploration of geometry and forces occurring within human-to-robot handovers," in *IEEE Haptics Symposium*, pp. 327-333, 2018.

[31] T.-H. Pham, N. Kyriazis, A. A. Argyros, and A. Kheddar, "Hand-object contact force estimation from markerless visual tracking," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 40, no. 12, pp. 2883-2896, 2018.

[32] L. Sciavicco, B. Siciliano, and B. Sciavicco, *Modelling and Control of Robot Manipulators*. Berlin, Heidelberg: Springer-Verlag, 2nd ed., 2000.

[33] A. Bolotnikova, S. Courtois, and A. Kheddar, "Autonomous initiation of human physical assistance by a humanoid," in *IEEE International Conference on Robot and Human Interactive Communication*, pp. 857-862, 2020.

[34] H. K. Khalil, *Nonlinear systems; 3rd ed.* Upper Saddle River, NJ: Prentice-Hall, 2002.

[35] M. Djeha, A. Tanguy, and A. Kheddar, "Adaptive-gains enforcing constraints in closed-loop qp control," *IEEE Robotics and Automation Letters*, vol. 5, no. 4, pp. 6504-6511, 2020.