# A Shared-control Teleoperation Architecture for Nonprehensile Object Transportation

Mario Selvaggio, Jonathan Cacace, Claudio Pacchierotti, Fabio Ruggiero, Paolo Robuffo Giordano

► To cite this version:

Mario Selvaggio, Jonathan Cacace, Claudio Pacchierotti, Fabio Ruggiero, Paolo Robuffo Giordano. A Shared-control Teleoperation Architecture for Nonprehensile Object Transportation. IEEE Transactions on Robotics, 2022, 38 (1), pp.569-583. 10.1109/TRO.2021.3086773. hal-03246629v2

HAL Id: hal-03246629

https://inria.hal.science/hal-03246629v2

Submitted on 8 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# A Shared-control Teleoperation Architecture for Nonprehensile Object Transportation

Mario Selvaggio<fnref n="1" />, Member, IEEE, Jonathan Cacace<fnref n="1" />, Claudio Pacchierotti<fnref n="2" />, Senior Member, IEEE, Fabio Ruggiero<fnref n="1" />, Senior Member, IEEE, and Paolo Robuffo Giordano<fnref n="2" />, Senior Member, IEEE

**Abstract**—This article proposes a shared-control teleoperation architecture for robot manipulators transporting an object on a tray. Differently from many existing studies about remotely operated robots with firm grasping capabilities, we consider the case in which, in principle, the object can break its contact with the robot end-effector. The proposed shared-control approach automatically regulates the remote robot motion commanded by the user and the end-effector orientation to prevent the object from sliding over the tray. Furthermore, the human operator is provided with haptic cues informing about the discrepancy between the commanded and executed robot motion, which assist the operator throughout the task execution.

We carried out trajectory tracking experiments employing an autonomous 7 degree-of-freedom (DoF) manipulator and compared the results obtained using the proposed approach with two different control schemes (i.e., constant tray orientation and no motion adjustment). We also carried out a human-subjects study involving eighteen participants, in which a 3-DoF haptic device was used to teleoperate the robot linear motion and display haptic cues to the operator. In all experiments, the results clearly show that our control approach outperforms the other solutions in terms of sliding prevention, robustness, commands tracking, and user's preference.

**Index Terms**—Telerobotics and Teleoperation, Dexterous manipulation, Grasping, Shared Control.

## SUPPLEMENTARY MATERIAL

Video is available at: https://www.youtube.com/watch?v=5_eReIS7Ku4. Code can be found at: https://github.com/prisma-lab/nonprehensile-object-transp.

## I. INTRODUCTION

FULLY autonomous robots are still out of reach in many real-world contexts. This is the case, for instance, of unstructured and safety-critical environments (e.g., medical [1] and nuclear [2] scenarios), where complex and highly-dexterous movements are required, and where any failure imputable to autonomy cannot be tolerated. In such environments, teleoperation is still the unique viable solution to safely and

<figure><img src="image_1.png" /><figcaption>Fig. 1. A teleoperated 7-DoF robot manipulator transports an object in a nonprehensile configuration while autonomously modulating the user-specified inputs and the object orientation to prevent it from sliding and possibly falling under the action of gravity.</figcaption></figure>

reliably accomplish difficult tasks [3]. Teleoperation allows combining the precision and the robustness of robots with the superior scene understanding and cognitive capabilities of humans. In this respect, shared control allows a human operator and an autonomous controller to simultaneously and collaboratively control the robotic system to achieve a common goal [4], [5]. Shared-control strategies are devised to reduce the human operator's workload when performing a difficult task (requiring skills/precision that exceed those of a human operator) through a robotic system [6]. Examples range from grasping and manipulating objects using remote manipulator arms [7] (possibly accounting for post-grasping objectives [8]), to collaborative transportation of large objects using a team of mobile robots [9], [10]. Employing shared-control techniques is particularly useful when dealing with complex tasks and/or many degree-of-freedom (DoF) robotic systems [11], as direct control would result in cumbersome, time-consuming, and cognitively-demanding task execution.

In this paper, we address the problem of semi-autonomously transporting an object manipulated by a tray mounted on a remote robotic arm, preventing any relative movement during the motion (see Fig. 1). Transporting an object in a nonprehensile configuration (i.e., without any form- or force-closure grasp [12]–[14]) is representative of many situations in which the robot cannot firmly hold the object and constrain its motion induced by inertial/external forces. In these cases, the object is free to slide or break contact with the robot end-effector, which may sometimes lead to a complete failure of the task. A solution to such a nonprehensile manipulation problem is known as *dynamic grasp*, formally defined in [15],

<footnote n="1">¹Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples, Italy.</footnote>

<footnote n="2">²CNRS, Univ Rennes, Inria, IRISA, Rennes, France.</footnote>

<footnote>Corresponding author e-mail: mario.selvaggio@unina.it</footnote>

<footnote>The research leading to these results has been partially supported by the WELDON project, in the frame of Programme STAR, financially supported by UniNA and Compagnia di San Paolo, and by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101017008.</footnote>

<footnote>This work involved human subjects in its research. The authors confirm that all human subject research procedures and protocols are exempt from review board approval.</footnote>
[16] as the condition in which friction forces prevent the object from moving relative to the manipulator. This is achieved by regulating the robot motion such that the object remains stationary with respect to the end-effector despite the action of inertial or external forces (such as gravity).

Differently from the sliding nonprehensile manipulation [17], which exploits friction forces to guide the motion of objects sliding on the surface of the manipulator and finds relevant applications within industries, the dynamic grasp (or non-sliding nonprehensile manipulation) finds relevant applications within field and service robotics. An example is remotely manipulating contaminated objects with very different sizes and shapes for achieving a faster decommissioning of nuclear sites. Using non-sliding nonprehensile manipulation, such as the one proposed in this paper, the range of objects that can be manipulated through the same robotic system can be enlarged. Besides this, telepresence systems for elderly assistance in a domestic environment, as well as in hospitals (e.g., for telemedicine and decontamination), are envisioned applications of our system. This is an active area of research, as teleoperated systems that, for instance, facilitate interactive communication between patients isolated in hospitals' ward and their relatives are currently being developed [18]. The case study of a robot able to carry the meal on a tray to a patient is the perfect example of a non-sliding nonprehensile manipulation task. A dish, a glass, a bottle, and pieces of cutlery are placed on the tray, and they must be safely transported to the patient.

However, while planning a trajectory with the appropriate time scale has proven to be a viable option for autonomously executing nonprehensile manipulation tasks in structured environments [19], this cannot be directly applied to unstructured and/or to dynamic environments (e.g. populated by humans), which require reactive control strategies able to deal with the inherent uncertainty of the environment. This is also valid for teleoperation scenarios where the user online specifies the desired motion trajectory which is, thus, not known beforehand. In this case, it is unrealistic to expect that a user may be capable of respecting/enforcing these dynamic non-sliding conditions for the object while directly commanding the manipulator motion without any external assistance.

For these reasons, we propose a shared-control teleoperation architecture able to modify the operator's commands to prevent the transported object from sliding relative to the manipulator (see Fig. 1). Besides altering the user's commands, the proposed shared-control architecture autonomously regulates the object orientation for both increasing the performance, in terms of user's commands tracking, and being more robust with respect to any uncertainty in the friction parameter. In addition, information about the discrepancy between the user's commands and those actually applied to the remote robot are provided to the user via haptic feedback. Force cues create a tactile sensation which convey high-level information and can be used by the user to infer the state of the system. In this context, they help the operator specifying motion commands which comply with the non-sliding constraints. To our best knowledge, there are no other shared-control teleoperation architectures explicitly designed for nonprehensile object manipulation such as the one proposed in this work.

## A. Related works

Teleoperation is one of the oldest research fields in robotics, dating back to 1950 when the first remotely operated robot was built to handle radioactive materials [3]. Since then, several teleoperation methods have been developed tackling different problems: enforcing closed-loop system stability [20], [21], overcoming large time delays [22]–[24], increasing telepresence feeling [25], [26], and so on. Among others, shared-control teleoperation architectures have the primary goal of making the remote task execution less tiring for the operator by combining their commands with those of an autonomous controller. The benefits introduced by sharing the control over a task on the human operator's physical and cognitive workload have already been demonstrated in multiple contexts, such as remote control of a group of aerial robots [4], [27], [28], dual-arm telemanipulation [7], [29], [30], or execution of space exploration tasks using multi-robot systems [31]. Haptics is sometimes used in shared control as a means of communication between the user and the autonomous controller [6], [32]. Several haptic-based shared-control teleoperation architectures (specifically designed for grasping and manipulation tasks) can be found within the literature. A vision-based shared-control teleoperation architecture was proposed in [33] and extended to dual-arm systems in [30]: some DoFs of the robotic system are autonomously regulated exploiting visual information, while the user has control over the remaining ones. Haptic guidance is used to avoid inter-robot collisions, joint limits, and singularities. Haptic-based shared control has been applied to both multi-target grasping in a cluttered and unknown environment [34], as well as to single-target grasping, accounting for post-grasping manipulation objectives [8], [35]. None of the above works has, however, explicitly addressed the problem of remotely transporting an object that is not firmly grasped by the robot.

Nonprehensile manipulation is, instead, a relatively more recent field in robotics research [36]. In nonprehensile manipulation scenarios, it is not always possible to prevent the motion of the object caused by external/inertial forces. Considering the problem addressed by this paper, when the tray is horizontal, it can counteract the gravity force applied to the object, thus preventing it from dropping; however, the tray cannot resist to forces lifting up the object. A conventional way to cope with a nonprehensile manipulation task is to split it into simple subtasks. These are usually referred to as *nonprehensile manipulation primitives*. Each primitive should be endowed with an appropriate motion planner and a controller, while a high-level supervisor can switch between the various subtasks suitably [37]. A list of such possible primitives can be found in [15], [36], [38]. They include throwing [39], dynamic catching [40], batting [41], pushing [42], holonomic/nonholonomic rolling [43], [44], and so on. We restrict our focus to the so-called dynamic grasp (or non-sliding) nonprehensile manipulation primitive, which consists in preventing the object from sliding by exploiting inertial and frictional forces [15], [16]. Such a nonprehensile manipulation primitive is not extensively investigated in the literature. Many works, instead, are focused on the sliding
primitive, in which inertial and frictional forces are exploited to properly move an object on a tray [17], [45]–[47]. Motion planning for nonprehensile object transportation is addressed in [48], where an admissible velocity propagation algorithm is adopted to take into account the dynamic constraints imposed by the task. A waiter humanoid robot is developed in [49], where the zero-moment-point constraint of the object transported on a tray is considered: this allows the possibility for the object to rotate on one edge during the motion, thus exploiting the so-called pivoting nonprehensile manipulation primitive [50]. Non-sliding nonprehensile manipulation, on the contrary, aims at keeping the object fixed on the tray at all times. A task priority control scheme for object transportation on a tray is designed in [51], where a human is directly in contact with the robot through a force sensor mounted on its end-effector. However, dynamic constraints due to the object/tray interaction are not considered. Moreover, it is worth remarking that most of the works addressing nonprehensile dynamic manipulation problems are considering fully autonomous robots executing the task. A teleoperation architecture, specifically designed to suppress sloshing dynamics in liquid handling systems, was proposed in [52]: the authors combine a filtering technique to suppress liquid oscillation and active feed-forward compensation of lateral accelerations through container re-orientation. However, the container is fixed to the tray and no operator’s haptic cueing methods were used. Therefore, to the best of our knowledge, there are no previous works explicitly dealing with nonprehensile rigid object transportation tasks executed by teleoperated robots.

## B. Contribution

To fill the above-mentioned gap in the robotic teleoperation literature, we propose a haptic shared-control teleoperation architecture explicitly designed for non-sliding nonprehensile manipulation. In our intentions, this work aims at advancing the state-of-art in several aspects, which are summarized in the following.

* We propose the first shared-control teleoperation architecture specifically designed for remote nonprehensile rigid object transportation tasks. The novelties of this architecture are:

- an optimal autonomous regulation of the remote robot commanded motion that prevents the object from sliding with respect to the tray;

- an autonomous regulation of the object orientation for both improving robustness and achieving better task performances;

- a kinesthetic haptic guidance method that informs the users about the discrepancy between the commanded and executed motions, assisting them in accomplishing the task.

* The literature review in [36] suggested that a way to pursue advancements within the nonprehensile manipulation field is to design methods to control the contact forces directly. This is precisely what is carried out in this paper. In similar related works, the typical approach is instead to mathematically model a nonprehensile system

<figure><img src="image_2.png" /><figcaption>Fig. 2. An illustration of the nonprehensile manipulation robotic system: the object (in orange) is manipulated via a tray-like robotic end-effector (in grey). Technical explanation of the symbols are given within Sec. II</figcaption></figure>

as a prehensile one. The control design is performed more straightforwardly, and the proof that the designed controller does not violate the given assumptions is often performed a-posteriori. We instead overturn this concept and directly apply a control action through the contact forces, guaranteeing the satisfaction of the friction constraints on-line.

* We extensively validate the proposed shared-control architecture and compare its performances with no motion overriding and constant orientation control approaches. Trajectory tracking, teleoperation, and a human-subjects study enrolling eighteen participants are used to show the benefits introduced by our proposed solution.

## II. PRELIMINARIES

Let us consider an object transported by a tray-like robotic end-effector. We set the following assumptions to derive our model:

(i) the object shape is known, and considered to be a rectangular cuboid<fnref n="1" />;

(ii) the initial object’s pose relative to the tray can be observed;

(iii) the tray/object interaction is realized through a finite set of contact points located on the corresponding object vertexes facing the tray;

(iv) the set of wrenches that are transmitted across the contacts is constituted only by linear forces (point contacts with friction model [53]);

(v) the Coulomb static friction coefficient between the object and the tray is uniform and measurable.

The main reference frames and quantities used in this work are illustrated in Fig. 2. Let us denote with $q_{s,b} = (p_{s,b}, R_{s,b}) \in SE(3)$ the current pose of the object frame $\{B\}$, which is attached to the object’s center of mass (CoM), in the inertial reference frame $\{S\}$, where $p_{s,b} \in \mathbb{R}^3$ denotes the position vector of the object, and $R_{s,b} \in SO(3)$ the orientation of $\{B\}$ with respect to $\{S\}$. The object dynamics can be written in body coordinates as [53]

$$M_b \dot{V}^b + C_b(V^b)V^b + N_b(R_{s,b}) = \mathcal{F}^b, \quad (1)$$

where $M_b \in \mathbb{R}^{6 \times 6}$ is the constant and positive-definite object’s mass matrix, constructed from $m_b \in \mathbb{R}_{\ge 0}$, being the object’s

<footnote n="1">¹This assumption is not binding and can be conveniently used with more complex-shaped objects. Indeed, the video contains experiments performed with a cup, whose shape was roughly approximated to a cuboid.</footnote>
mass, and $\mathcal{I}_b \in \mathbb{R}^{3 \times 3}$, the constant symmetric and positive-definite inertia matrix; $C_b \in \mathbb{R}^{6 \times 6}$ is the matrix accounting Centrifugal/Coriolis effects; $N_b (R_{s,b}) \in \mathbb{R}^6$ encodes the gravity force; $\mathcal{V}^b = (v^b, \omega^b) \in \mathbb{R}^6$ is the object twist, with $v^b, \omega^b \in \mathbb{R}^3$ linear and angular velocity vectors, respectively; and $\mathcal{F}^b = (f^b, \tau^b) \in \mathbb{R}^6$ is the body applied wrench, with $f^b, \tau^b \in \mathbb{R}^3$ force and torque vectors, respectively, all specified with respect to $\{B\}$.

The body wrench $\mathcal{F}^b$ can be realized by means of a set of wrenches exerted at the $n_c$ contact points located along the object perimeter. In view of the assumptions (i), (ii), and (iii) the pose of the $n_c = 4$ contact points in $\{B\}$ and $\{S\}$ is known. The $i$-th contact point is identified by a contact frame $\{C_i\}$ whose pose is expressed in $\{B\}$ by $q_{b,c_i} = (p_{b,c_i}^b, R_{b,c_i}^b) \in \text{SE}(3)$. To mathematically describe the tray/object interaction behavior, a suitable contact model is adopted. In general, the set of wrenches that can be transmitted across the $i$-th contact is described by a wrench basis $B_{c,i} \in \mathbb{R}^{6 \times m_i}$, where $m_i$ denotes the dimension of the generalized forces at the contact. $B_{c,i}$ maps the components of the contact forces, which are transmissible through the contact point, into the 6-dimensional space. As stated in assumption (iv), only the linear forces $f_{c_i} \in \mathbb{R}^3$ can be transmitted through the $i$-th contact, thus $m_i = 3$. The body wrench $\mathcal{F}^b$ can be expressed as

$$ \mathcal{F}^b = GF_c, \quad G = \left[ \text{Ad}_{q_{b,c_1}^{-1}}^T B_{c,1}, \dots, \text{Ad}_{q_{b,c_n}^{-1}}^T B_{c,n_c} \right], \quad (2) $$

where $G \in \mathbb{R}^{6 \times 3n_c}$, usually referred to as grasp matrix, maps the stacked vector of independent contact forces $F_c = [f_{c_1}^T, \dots, f_{c_{n_c}}^T]^T \in \mathbb{R}^{3n_c}$ to the body wrench $\mathcal{F}^b$ exerted at the object's center of mass. Using the assumption (iv), the matrices involved in the calculation of $G$ can be expressed as follows [53]

$$ \text{Ad}_{q_{b,c_i}^{-1}}^T = \begin{bmatrix} R_{b,c_i}^b & 0 \\ \hat{p}_{b,c_i}^b & R_{c_i,b}^b \\ \hat{p}_{b,c_i}^b & R_{b,c_i}^b \end{bmatrix}, \quad B_{c,i} = \begin{bmatrix} I_{3 \times 3} \\ 0_{3 \times 3} \end{bmatrix}, \quad (3) $$

where $\hat{p}_{b,c_i}^b \in \text{so}(3)$ denotes the skew-symmetric matrix associated with the vector $p_{b,c_i}^b \in \mathbb{R}^3$ (i.e., the position of the $i$-th contact point expressed in $\{B\}$).

Another essential component of the contact model is the friction cone. At the $i$-th contact, the friction cone can be defined as the set of generalized contact forces realizable given the friction coefficient. In our case, it can be written as [53]

$$ FC_{c_i} = \{ f_{c_i} \in \mathbb{R}^3 : \sqrt{f_{c_i,x}^2 + f_{c_i,y}^2} \le \mu f_{c_i,z}, f_{c_i,z} \ge 0 \}, $$

where $\mu \in \mathbb{R}_{\ge 0}$ denotes the friction coefficient that, in view of the assumption (v), is known and equal for all the contacts. The complete friction cone space is the Cartesian product of the $n_c$ cones, i.e. $FC = FC_{c_1} \times \dots \times FC_{c_{n_c}} \subset \mathbb{R}^{3n_c}$. Whenever $F_c \in FC$, the object can be manipulated through the contact forces while preventing sliding with respect to the tray. To enforce this constraint, hereafter we use a linearly approximated representation of the friction cone, which considers the $i$-th circular friction cone conservatively approximated by a polyhedral cone generated by a finite set of vectors $\hat{f}_{c_i,j} \in \mathbb{R}^3$ [53]. This is a common approximation which allows treating the friction cone as a (otherwise quadratic) linear constraint in the

following developments. Indeed, the condition $f_{c_i} \in FC_i$ can be conveniently formulated expressing $f_{c_i}$ as a nonnegative linear combination of unit vectors $\hat{f}_{c_i,1} \dots \hat{f}_{c_i,k} \in \partial FC_i$, with $\partial FC_i$ denoting the boundary of the $i$-th friction cone manifold

$$ FC_{c_i} = \left\{ f_{c_i} \in \mathbb{R}^3 : f_{c_i} = \sum_{j=1}^{k} \lambda_{c_i,j} \hat{f}_{c_i,j}, \lambda_{c_i,j} \ge 0 \right\}, \quad (4) $$

where $k \in \mathbb{N}_{>0}$ is the positive number of unit vectors chosen to linearly approximate the friction cone. In this work, $k = 4$ is chosen, i.e. the circular friction cone is approximated by an inscribed pyramid [53].

For a given $\mathcal{F}^b$ (e.g., designed to achieve a desired object motion), the vector of contact forces $F_c$ that prevents sliding can be obtained by solving the following optimization problem

$$ \underset{F_c}{\text{minimize}} \quad \|F_c\|^2 \quad (5) $$

$$ \text{subject to} \quad GF_c = \mathcal{F}^b, \quad (6) $$

$$ F_c = \hat{F}_c \Lambda, \quad (7) $$

$$ \Lambda_i \ge 0 \quad \forall i = 1, \dots, kn_c, \quad (8) $$

where $\|\cdot\|^2$ denotes the two-norm, $\Lambda = [\lambda_{c_1,1}, \dots, \lambda_{c_{n_c},k}]^T \in \mathbb{R}^{kn_c}$ and $\hat{F}_c = \text{blockdiag}(\hat{F}_{c,1}, \dots, \hat{F}_{c,n_c})$, with $\hat{F}_{c,i} = [\hat{f}_{c_i,1}, \dots, \hat{f}_{c_i,k}]$, are used to compactly write the condition (4) for all the contacts. In the problem (5), the constraint (6) expresses the object wrenches mapping through the grasp matrix $G$ introduced in (2), while (7) and (8) enforce the (linear) non-sliding constraint $F_c \in FC$.

### III. PROBLEM STATEMENT

The problem in (5) is always feasible when any instantaneous wrench $\mathcal{F}^b$ is realizable at the object's CoM through $F_c$: this means that the object can be manipulated or, equivalently, any inertial/external force can be resisted. This happens when a force closure grasp is realized, that is when the range of the grasp matrix covers all the wrench space $(\mathcal{R}(G) = \mathbb{R}^6)$ [54]. On the contrary, when nonprehensile manipulation tasks are considered, the problem in (5) may not admit a feasible solution (e.g., the contact forces between the object and the tray cannot resist to pure lateral forces applied to the object). This has to be carefully considered when the task is to transport (without sliding) the object between any two configurations.

One possible approach to solve this problem at the planning level is to find, if it exists, a trajectory for the object such that problem (5) always admits a feasible solution. However, this possibility is not practical/viable when considering teleoperation scenarios. Online (re-)planning object/robot trajectories can be time-consuming and lead to losing the precious feeling of being directly in control of the system at every time instant. Alternatively, one can rely on the operator experience, even if, in practice, it is clearly very demanding (if not impossible) for the user to specify set-points that are consistent with the non-sliding conditions (7)-(8). For this reason, it is convenient to adopt shared-control solutions in nonprehensile teleoperation scenarios, where the robotic system must be able to process the
human's inputs while guaranteeing the satisfaction of the non-sliding constraints. We also hypothesize that, in transportation tasks, for a human operator it is more natural to command the linear motion of the object, and, in most cases, this is the only objective she/he is concerned about. Therefore, in our setting, the object's orientation is left free to be adjusted by the autonomous controller. Our question is then: in which way the system motion and, in particular, the object's orientation can be autonomously regulated for achieving the best performance in terms of command tracking, while being robust to friction parameter uncertainties?

We are now ready to formally state our problem.

**Problem:** Given an object, transported in a nonprehensile way on a tray, and an online specified position set-point $p_{s,b}^*(t) : t \ge 0 \to \mathbb{R}^3$, find a controller that calculates the commanded wrench $\mathcal{F}^b$ that satisfies the non-sliding constraint (7)-(8) while, simultaneously, minimizing the user-specified commands tracking error, and being robust to friction parameter uncertainties.

The solution to this problem would result in a shared-control teleoperation architecture in which a human operator online specifies the linear motion for the object while the robot autonomously modulates the user's input and regulates the orientation of its end-effector for maximizing performances and improving robustness.

## IV. SHARED-CONTROL TELEOPERATION ARCHITECTURE

In this section we describe the proposed shared-control teleoperation architecture designed to solve the problem introduced above. For the sake of clarity, the overall architecture is split up into three parts, namely: the object non-sliding controller (Sec. IV-A), the robot manipulator controller (Sec. IV-B), and the teleoperation controller (Sec. IV-C).

### A. Object non-sliding controller

In order to move an object between any two configurations, possibly following a given trajectory, the simplest approach consists in developing a control law for the object which does not take into account the non-sliding constraint (see Sec. II). Let us consider that, given the current object state $(\mathcal{V}^b, q_{s,b})$ and its desired motion/configuration $(\dot{\mathcal{V}}^{b*}, \mathcal{V}^{b*}, q_{s,b}^*)$, a body wrench $\mathcal{F}^{b*}$ can be obtained using an inverse dynamics control law, which can be derived from (1) as follows

$$ 
\begin{aligned}
\mathcal{F}^{b*} &= \begin{bmatrix} f_{\tau^{b*}}^{b*} \\ \tau^{b*} \end{bmatrix} = M_b y + C_b (\mathcal{V}^b) \mathcal{V}^b + N_b (R_{s,b}), \quad (9) \\
y &= \dot{\mathcal{V}}^{b*} + K_d \dot{\mathcal{E}}^b + K_p \mathcal{E}^b,
\end{aligned}
$$

where $\mathcal{E}^b = [e_p^{bT}, e_o^{bT}]^T \in \mathbb{R}^6$ includes the position error $e_p^b = R_{s,b}^T [p_{s,b}^* - p_{s,b}] \in \mathbb{R}^3$ and the orientation error $e_o^b = R_{s,b}^T \Delta\varphi \in \mathbb{R}^3$, with $\Delta\varphi$ being the exponential coordinates of the rotation<fnref n="2" />, both expressed in body frame; $K_d$,

$K_p \in \mathbb{R}^{6 \times 6}$ are diagonal positive-definite gain matrices. If $\mathcal{F}^b = \mathcal{F}^{b*}$ (computed from (9)) is realizable (i.e., there exists a $F_c$ computed from (5)), this controller guarantees asymptotic convergence to the desired trajectory with prescribed performances that depend on the choice of gain matrices $K_d$ and $K_p$ [55].

However, because of what previously said, the choice $\mathcal{F}^b = \mathcal{F}^{b*}$ does not guarantee that problem (5) admits a solution, that is $F_c \in \mathcal{F}C$ might not exist. In turn, this implies that when trying to realize $\mathcal{F}^{b*}$ the object may slide on the tray. Our goal is, then, to find a commanded body wrench $\mathcal{F}^b$ that is as close as possible to $\mathcal{F}^{b*}$ while satisfying the constraints in (5). A solution to this problem can be found by solving the following optimization problem

$$ 
\begin{array}{ll}
\underset{\mathcal{F}_b, F_c}{\text{minimize}} & \|\mathcal{F}^{b*} - \mathcal{F}^b\|_H^2 + \|F_c\|^2 \quad (10) \\
\text{subject to} & (6), (7), (8),
\end{array}
$$

where the cost function is the sum of two-norms and the diagonal positive-definite matrix $H \in \mathbb{R}^{6 \times 6}$ can be used to specify the relative weight between the components of $\mathcal{F}^{b*} - \mathcal{F}^b$ in the solution. The problem in (10) allows the simultaneous calculation of the optimal body wrench $\mathcal{F}_b$ and the corresponding contact force vector $F_c$ satisfying the non-sliding constraint. Compared to the previous solution (9), this controller solves the problem of specifying a desired object motion that is consistent with the system constraints (which would represent a highly demanding task for the user teleoperating the system).

It is worth remarking here that the control law introduced above requires full specification of the desired object motion and configuration. Our goal, however, is to design a teleoperation architecture in which the desired linear motion $(\dot{\mathcal{V}}^{b*}, \mathcal{V}^{b*}, p_{s,b}^*)$ is specified online by the user, while the rotational motion is regulated by the system autonomously.

One possible way to control the rotational motion consists in enforcing a constant desired orientation $R_{s,b}^*$ which keeps the tray horizontal. However, enforcing the system to track a constant orientation might degrade the task execution performance and the system's robustness. Indeed, to execute high horizontal accelerations, large $\mathcal{F}^b$ are required. In this situation, the contact forces $F_c$, which realize $\mathcal{F}^b$ through (10), tend to exit the friction cone boundaries. Thus, realizing $\mathcal{F}^b = \mathcal{F}^{b*}$, may cause the object to slide. On the contrary, using $\mathcal{F}^b$ from (10), the object's sliding is always prevented but tracking performances may be degraded, as it will be shown later in this paper. This behavior is expected since the optimization-based controller (10) operates limiting lateral contact forces. Moreover, in this last case, the contact forces may often operate at the boundary of the friction cones and, thus, any overestimation of the friction coefficient may cause the object to slide, ultimately making this approach not robust.

These drawbacks can be mitigated by autonomously rotating the tray/object system in response to the required lateral accelerations. To accomplish this, we first introduce a positive-definite function $\mathcal{H}(F_c) : \mathcal{F}C \to \mathbb{R}_{\ge 0}$ that represents a measure of the distance between the contact forces $F_c$ and the friction cones boundaries. Denoting by $\hat{Z} = [\hat{z}_1, \dots, \hat{z}_{n_c}]^T \in \mathbb{R}^{3n_c}$ the stacked vector of the object contact normals, the function $\mathcal{H}(F_c)$

<footnote n="2">²i.e., $\Delta\varphi = \Delta\theta\hat{n}$, with $\hat{n}$ being the rotation axis and $\Delta\theta$ being the corresponding angle, both extracted from the rotation matrix $R_e = R_{s,b}^* R_{s,b}^T$ through the logarithmic map [53].</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 3. Drawing of the object and related quantities involved in the tilting torque ($\tau_b$) calculation. Symbols are explained in Sec. IV-A.</figcaption></figure>

is chosen such that it satisfies the following two properties

$$F_c^T \hat{Z} \rightarrow \|F_c\| \quad \Rightarrow \quad \mathcal{H}(F_c) \rightarrow \min \mathcal{H}, \quad (11)$$

$$F_c \rightarrow \partial FC \quad \Rightarrow \quad \mathcal{H}(F_c) \rightarrow +\infty, \quad (12)$$

where $\partial FC$ is used to denote the boundary of the contact force space FC. The rationale behind these properties is to have a measure of how close the contact forces are to the friction cones' borders (indeed, $\mathcal{H}(F_c)$ is designed so as to rapidly increase as one contact force approaches the limit of its relative friction cone). To derive the autonomous orientation control input, it is convenient to parameterize the manifold FC through the chart $\alpha(FC) = [\alpha_1(FC_1), \dots, \alpha_{n_c}(FC_{n_c})]^T$, representing the stacked vector of the angles that each contact force $f_{c,i}$ forms with the corresponding contact normal $\hat{z}_i$ (see Fig. 3). In this way, the i-th friction cone manifold boundary $\partial FC_i$ can be conveniently expressed as the set $\partial FC_i = \{f_{c,i} : |\alpha_i| = \theta_i\}$, where $\theta_i = \arctan \mu \ge 0$. The angle $\alpha_i$ can be easily calculated given $f_{c,i}$ as $\alpha_i = \arccos(\hat{z}_i^T f_{c,i} / \|f_{c,i}\|) \ge 0$. The function $\mathcal{H}(\alpha)$ is, thus, designed as [11]

$$\mathcal{H}(\alpha) = \sum_{i=1}^{n_c} \mathcal{H}_i(\alpha_i) = \frac{1}{\lambda} \sum_{i=1}^{n_c} \frac{1}{(\theta_i - \alpha_i)(\alpha_i + \theta_i)}. \quad (13)$$

At this point, the autonomous controller torque $\tau_b \in \mathbb{R}^3$ should be designed such that (13) is minimized. To achieve this goal, we first compute the fictitious contact torque $\tau_{c,i} \in \mathbb{R}^3$ that would align $\hat{z}_i$ to $f_{c,i}$ at the i-th contact as (see Fig. 3)

$$\tau_{c_i} = - \frac{\partial \mathcal{H}_i \hat{z}_i \times f_{c_i}}{\partial \alpha_i \|f_{c_i}\|}, \quad (14)$$

which consists in a locally optimal gradient-descent solution for minimizing $\alpha_i$. From the properties (11)-(12), it is possible to see that the intensity of $\tau_{c_i}$ is zero when $f_{c,i}$ is aligned to $\hat{z}_i$, while it grows when $f_{c,i}$ approaches the friction cone limits. This is a desirable feature that allows realizing a more intense rotating action when $f_{c,i}$ is approaching the cone limits. It is worth remarking that the torque $\tau_{c_i}$ is not realizable at the i-th contact point given our contact model (see Sec. II). Thus, the effect of $\tau_{c_i}$ must be realized through the corresponding body torque $\tau_b$ constructed as follows. Denoting by $\tau_c = [\tau_{c_1}, \dots, \tau_{c_{n_c}}]^T \in \mathbb{R}^{3n_c}$, the following mapping holds

$$\tau_b = G_o \tau_c, \quad G_o = H \left[ \mathrm{Ad}_{g_{b,c_1}^{-1}} B_o, \dots, \mathrm{Ad}_{g_{b,c_{n_c}}^{-1}} B_o \right], \quad (15)$$

where

$$H = \begin{bmatrix} 0_{3 \times 3} & I_{3 \times 3}, \end{bmatrix}, \quad B_o = \begin{bmatrix} 0_{3 \times 3} \\ I_{3 \times 3} \end{bmatrix}, \quad (16)$$

are selection matrices useful to construct the orientation grasp matrix $G_o \in \mathbb{R}^{3 \times 3n_c}$. At this point, $\tau_b$ in (15) is used to compute the $\mathcal{F}^{b*}$ using the following $y$ in (9)

$$y = \begin{bmatrix} \dot{v}^{b*} \\ 0 \end{bmatrix} + K_d \begin{bmatrix} (v^{b*} - v^b) \\ -\omega^b \end{bmatrix} + K_p \begin{bmatrix} (p_{s,b}^* - p_{s,b}) \\ K_\tau \tau_b \end{bmatrix} \quad (17)$$

where $K_\tau \in \mathbb{R}^{3 \times 3}$ is a gain matrix. Finally, to include the rotation of the object in the controller, both $\mathcal{F}^b$ and $F_c$ can be calculated solving the problem in (10), with $\mathcal{F}^{b*}$ obtained from (17).

## B. Robot manipulator controller

Let us consider that the motion of the tray/object is realized through a torque-controlled manipulator. The non-sliding optimal controller (10) guarantees that $F_c \in FC$, which is an essential condition to derive the robot manipulator control input. Given the calculated optimal $\mathcal{F}^b$ and the corresponding $F_c$, here the goal is to find the manipulator generalized control forces that realize the desired end-effector/object motion. Assuming that the tray dynamics is accounted for by a suitable augmentation of the last link dynamic parameters, the dynamics of the n-DoF manipulator can be written in joint coordinates $\theta \in \mathbb{R}^n$ as [55]

$$M_r(\theta) \ddot{\theta} + C_r(\theta, \dot{\theta}) \dot{\theta} + N_r(\theta, \dot{\theta}) = \tau - J_r^{bT}(\theta) F_c \quad (18)$$

where $M_r(\theta) \in \mathbb{R}^{n \times n}$ is the positive-definite joint-space inertia matrix, $C_r(\theta, \dot{\theta}) \in \mathbb{R}^{n \times n}$ is the matrix accounting for Coriolis/centrifugal terms, $N_r(\theta, \dot{\theta}) \in \mathbb{R}^n$ encodes gravity and frictional terms, $\tau \in \mathbb{R}^n$ is the vector of generalized joint forces (our control input), and $J_r^b(\theta) = [J_{c_i}^b(\theta), \dots, J_{c_{n_c}}^b(\theta)]^T \in \mathbb{R}^{m \times n}$ is the body Jacobian of the contact points. Each term $J_{c_i}^b(\theta) \in \mathbb{R}^{m_i \times n}$ can be calculated by pre-multiplying the manipulator body Jacobian of the tray end-effector $J_e^b(\theta) \in \mathbb{R}^{6 \times n}$ by the adjoint matrix between the contact location and the end-effector reference frame $\{E\}$ (see Fig. 4)

$$J_{c_i}^b(\theta) = B_c^T \mathrm{Ad}_{g_{c_i, \cdot}} J_e^b(\theta),$$

where $B_c$ is defined in (3). Thus, the manipulator control torque $\tau$ can be designed according to the inverse dynamics control paradigm as [55]

$$\tau = J_r^{bT}(\theta) F_c + C_r(\theta, \dot{\theta}) \dot{\theta} + N_r(\theta, \dot{\theta}) + M_r(\theta) u, \quad (19)$$

where $u \in \mathbb{R}^n$ denotes the additional control input. Plugging (19) into (18) yields

$$\ddot{\theta} = u. \quad (20)$$

At this point, the control input $u$ can be designed to track the desired object motion/trajectory. To this end, differentiating the velocity mapping between joint and object spaces $\mathcal{V}^b = J_b^b \dot{\theta}$ yields

$$\dot{\mathcal{V}}^b = J_b^b(\theta) \ddot{\theta} + J_b^b(\theta) \dot{\theta} \quad (21)$$
where $J_b^b(\theta) \in \mathbb{R}^{6 \times n}$ is the object body Jacobian. In addition, any solution of the problem (10) satisfies the condition $F_c \in FC$ that allows to exploit the fundamental grasp constraint [53]

$$J_r^b(\theta)\dot{\theta} = G^T \mathcal{V}^b, \qquad (22)$$

describing the fact that the end-effector and object contact points velocities are equal (this is, indeed, guaranteed when no sliding occurs, i.e., when $F_c \in FC$). Solving (22) for $\dot{\theta}$ and substituting in (21) suggests the choice of the control law [55]

$$u = J_b^{b\dagger}(\theta) (\dot{\mathcal{V}}_d^b - \dot{J}_b^b(\theta) \dot{\theta}) + N \dot{\theta}_0, \qquad (23)$$

where $J_b^{b\dagger}(\theta) \in \mathbb{R}^{n \times 6}$ denotes the Moore-Penrose inverse of $J_b^b(\theta)$ (assumed full row rank), and the second term on the right-hand side represents the null-space control input, which accomplishes a lower priority task, with $N = (I_{n \times n} - J_b^{b\dagger}(\theta) J_b^b(\theta))$. In this case, we chose the secondary task keeping the robot configuration as close as possible to initial one, i.e. $\dot{\theta}_0 = \theta(0) - \theta$. In (23), $\dot{\mathcal{V}}_d^b$ is the desired object acceleration which is realized by the optimal wrench $\mathcal{F}_b$, which can be calculated inverting (1) as

$$\dot{\mathcal{V}}_d^b = M_o^{-1} (\mathcal{F}^b - C_b (\mathcal{V}^b) \mathcal{V}^b - N_b (R_{s,b})) . \qquad (24)$$

Substituting (23) in (20) and using (24) yields $\dot{\mathcal{V}}^b = \dot{\mathcal{V}}_d^b$.

### C. Teleoperation controller

The previously introduced methods are used to build a haptic-guided shared-control teleoperation architecture. Let us consider a telerobotic system consisting of a local haptic device and a remote robot arm equipped with a tray-like end-effector. A non-sliding nonprehensile manipulation task is envisioned, where a human operator continuously specifies the set-point $(\dot{v}^{b*}, v^{b*}, p^{*}_{s,b})$ for transporting the object between any two configurations through the input device. Rate control is used as a forward teleoperation scheme [56]. In this scheme, the object linear velocity is proportional to the haptic device position. More specifically, the entire set-point is calculated as follows

$$\begin{aligned} \dot{v}^{b*} &= R_{s,b}^{\mathrm{T}} K_{o,h} R_{o,h} \dot{p}_h, \\ v^{b*} &= R_{s,b}^{\mathrm{T}} K_{o,h} R_{o,h} (p_h - p_{h,o}), \\ p^{*}_{s,b} &= \int_{0}^{T} R_{s,b} v^{b*} \mathrm{d}t, \end{aligned}$$

where $p_h, \dot{p}_h \in \mathbb{R}^3$ are the measured position and linear velocity of the haptic device handle, respectively; $p_{h,o}$ is a position offset (useful to implement a deadzone); $K_{o,h} \in \mathbb{R}^{3 \times 3}$ is a scaling matrix; and $R_{o,h} \in SO(3)$ is a rotation matrix that maps the haptic device reference frame into the object frame.

In the considered context, haptic guidance can be a very useful feature assisting the user in accomplishing the task. Indeed, it has the double purpose of guiding the operator movements and, at the same time, increasing its awareness of the system state. For this reason, haptic cues are designed based on the difference between the commanded and the executed forces, where any discrepancy is due to the modulation action of the autonomous controller which implements (10).

<figure><img src="image_4.png" /><figcaption>Fig. 4. Experimental setup for the teleoperation experiments. Left: picture of the remote side with main reference frames ({M}: measuring camera frame, {E}: end-effector frame, {B}: object frame) and task target areas (A, B, C) overlayed. Right: picture of the local side with the Novint Falcon haptic device operated by the user ({H}: haptic device reference frame).</figcaption></figure>

More specifically, the vector of forces rendered to the user is calculated as follows

$$f_h = -K_f R_{o,h}^{\mathrm{T}} (f^{b*} - f^b) - K_v \dot{p}_h, \qquad (25)$$

where $K_f, K_v \in \mathbb{R}^{3 \times 3}$ are gain matrices; $f^{b*}$ and $f^b$ are linear components of the wrenches $\mathcal{F}^{b*}$ and $\mathcal{F}^b$, respectively. The choice in (25) allows receiving haptic guidance forces when the commanded motion would violate the non-sliding constraint (first term on the right-hand side), plus damping forces (second term on the right-hand side) to reduce oscillations. In details, guidance forces were designed to act against any high (positive or negative) acceleration command helping the operator to slow down or accelerate to better cope with the non-sliding constraints.

## V. EXPERIMENTS AND RESULTS

This section presents several experiments meant to show the performance and applicability of the proposed shared-control teleoperation architecture to real-world applications. The experimental setup is described in Sec. V-A. To demonstrate the improvements introduced by our shared-control technique, we performed experiments considering different conditions, detailed in Sec. V-B. First, to favor an objective comparison among the control modes, we carried out two autonomous trajectory tracking experiments. The results of these experiments are shown and analyzed in Sec. V-C. All the control modes, also endowed with the haptic cues described in Sec. IV-C, are then used to perform teleoperation experiments. The results of these experiments are shown in Sec. V-D, while their quantitative assessment and analysis in a human-subjects study is described in Sec. VI.

### A. Experimental setup

The experimental setup for both the trajectory tracking and teleoperation experiments is shown in Fig. 4. The setup consists of a 12-cm-radius plastic tray attached to the end-effector of a 7-DoF KUKA LWR robotic manipulator. The robot was controlled as explained in Sec. IV-B, with the secondary task trying to keep the manipulator at the initial configuration $\theta(0) = [15, -90, 90, 90, 90, -90, 75]^T$ deg. A hollowed cuboid of dimensions $30 \times 30 \times 35$ mm, whose inertial properties are given in Table I, is placed on the tray in the initial pose $p_{e,b} = [-0.08, 0, 0.017]^T$ m, $R_{e,b} = I_{3 \times 3}$ with respect to
{E}. The Coulomb friction coefficient was estimated by quasi-statically ($\omega_e = 1$ deg/s) rotating the tray around its end-effector y-axis (Fig. 4), starting from the horizontal configuration, until the object started sliding. At this point, the system was stopped, and the angle $\theta_e$ around y was measured. Each measurement was scaled by a safety factor $s = 1.1$ to be more conservative and it was repeated in different object positions relative to the tray to validate assumption (v). The average of the scaled angles $\tilde{\theta}_e$ was used to computed the friction coefficient as $\mu = \tan\theta_e$. Due to the nonprehensile configuration, the cube is free to break contact with the tray and slide. Two fiducial markers are attached on the top and lateral sides of the object to retrieve its pose with respect to the tray and estimate its movements along it. In more details, an UEYE UI-5241LE-C<fnref n="3" /> gigabit camera is attached to one side of the tray, in a calibrated configuration, to retrieve any movement of the object during the experiment. The camera, running at a frame rate of 100 frames per second, has been set to acquire images at low resolution. To detect the position of the transported object in each frame, the recorded images are post-processed by means of an AprilTag fiducial marker detector [57]. An additional marker was initially attached at the center of the tray to estimate the initial relative position of the object with respect to the tray. For the teleoperation experiments, the local side is composed of a 3-DoF Novint Falcon haptic interface, which enables the user to control the position, velocity, and acceleration of the remote robot, and receive haptic feedback as explained in Sec. IV-C. To execute the task, the user exploits the visual feedback provided by a monitor at the local side. This is streamed by a standard universal serial bus colored camera mounted on the top of the experimental setup, which also online checks whether the transported object has reached the circular target areas A, B, or C. When the object is in a target area, its color on the screen changes from green to red. Performing all the computations in our control loop takes 2.2 (mean) $\pm 1.028 \times 10^{-1}$ ms (standard deviation) while the communication rate with the robot is running at 5 ms. The parameters set for the proposed experiments are given in Table I.

A video showing representative experiments, plus other cases not described in this paper, is available as supplemental material at https://youtu.be/5_eReIS7Ku4. As it can be noticed, a rough approximation of the tray friction coefficient and the contact geometry was sufficient to successfully perform repeatable experiments with multiple objects, having different contact geometries, and made by very diverse materials (ceramic cup, steel bottle, plastic cube, etc.).

<table><caption>TABLE I
OBJECT AND ROBOT CONTROL PARAMETERS</caption><tr><td>$m_b = 0.38$</td><td>[Kg]</td><td>$\mu = 0.3$</td><td></td></tr><tr><td>$\mathcal{I}_b = \text{diag}(4.096e^{-4})$</td><td>[Kgm<sup>2</sup>]</td><td>$K_\tau = 1e^{-3}I_3$</td><td>[(Nm)<sup>-1</sup>]</td></tr><tr><td>$K_p = \text{diag}(6e^2I_3, 1e^3I_3)$</td><td>[s<sup>-2</sup>]</td><td>$K_{o,h} = 20I_3$</td><td>[s<sup>-1</sup>]</td></tr><tr><td>$K_d = \text{diag}(40I_3, 20I_3)$</td><td>[s<sup>-1</sup>]</td><td>$K_v = 4I_3$</td><td>[Kgs<sup>-1</sup>]</td></tr><tr><td>$\lambda = 4$</td><td>[rad<sup>-1</sup>]</td><td>$K_f = 0.5I_3$</td><td></td></tr><tr><td>$H_1 = \text{diag}(2e^2I_3, 1e^3I_3)$</td><td>[1, m<sup>-1</sup>]</td><td></td><td></td></tr></table>

## B. Experimental conditions

We considered three different control modes. In all of them, the linear motion of the object placed on the robotic manipulator is commanded through its set points ($\dot{v}^{b*}, v^{b*}, p_{s,b}^*$), specified either from a pre-planned trajectory (autonomous) or from the haptic interface (teleoperation), as explained in Sec. IV-C. The three autonomous/teleoperated control modes, with reference to Sec. IV-A, are:

**T:** classic inverse dynamics control, where the commands are directly applied to the object through the robot, which in addition keeps a fixed orientation of the tray, that is $\mathcal{F}^b = \mathcal{F}^{b*}$ calculated from (9) with $R_{s,b}^* = I_{3 \times 3}$. In this condition, the contact forces are calculated as $F_c = G^\dagger \mathcal{F}^{b4}$;

**S:** a shared-control technique, where the optimization in (10) regulates the commanded robot motion to prevent the object from sliding, while the orientation of the tray tracks a constant one, i.e., $\mathcal{F}^b$ is calculated from (10) using $\mathcal{F}^{b*}$ from (9) with $R_{s,b}^* = I_{3 \times 3}$;

**SO:** the proposed shared-control technique, where the optimization in (10) adjusts the commanded robot motion and the optimization method in (14) and (15) computes the tray rotation to prevent the object from sliding, that is, $\mathcal{F}^b$ is calculated from (10) using $\mathcal{F}^{b*}$ derived from (17).

In each mode, the robot is always controlled as explained in Sec. IV-B. For the teleoperation experiments only, we also considered two haptic feedback modalities to assist the users and inform them about the state of the object:

**H:** haptic feedback, in which the operator receives haptic information about the controller motion overlaying action. The cues are proportional to the difference between what the user commanded and what it is actually performed by the robot (see Sec. IV-C);

**H̄:** no haptic feedback, in which the user receive no haptic information about the motion overlaying action of the controller (see Sec. IV-C).

## C. Trajectory tracking experiments

To show the improvements introduced by our proposed controller SO over S and T approaches, we evaluated and

<footnote n="3">³https://en.ids-imaging.com/store/products/cameras/ui-5241le.html</footnote>

<footnote n="4">⁴It is worth to note that, in the T case, this method might calculate unfeasible contact forces whenever the non-sliding condition is violated ($F_c \notin FC$). Moreover, the feasibility of the solution is also dependent upon the contact model: changing the number of contact points (or their location), which are used to discretize the continuous surface contact, may alter the solution. However, this method was empirically found to be reliable in discerning sliding from non-sliding situations, which is our primary goal. It is also worth to note that contact forces in the null-space of G are not considered since, in this case, these null-space internal forces cannot be generated in a non-prehensile manipulation systems such as the one considered in this work. Indeed, following the derivations in [58], the controllable contact forces $F_c$ belong to the range space of $J_r^b$, i.e., they are generated by contact point displacements. Instead, internal contact forces belong to the null space of G. Denoting by E a basis of the null space of G, it is easy to prove that $E^T J_r^b = O$, with O being the null matrix. This then shows that no internal forces can be generated by rigid contact points displacements, and therefore forces in the null space of G should not be considered in non-prehensile manipulation.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 5. Comparison of the norm of the position error $\mathcal{E}$, the robustness measure $\mathcal{R}$, and the object relative displacement $\mathcal{D}$ along a predefined linear trajectory.</figcaption></figure>

compared three metrics along trajectory tracking experiments, namely

* the norm of the positional tracking error

$$ \mathcal{E}(t) = \|p_{s,b}^*(t) - p_{s,b}(t)\|; \qquad (26) $$

* the robustness, evaluated as the distance of the contact forces from the friction cone limits<fnref n="5" />

$$ \mathcal{R}(t) = 1/\mathcal{H}(t); \qquad (27) $$

* the norm of the object linear displacement with respect to its initial position in the camera reference frame $\mathcal{M}$

$$ \mathcal{D}(t) = \|p_{m,b}(t) - p_{m,b}(0)\|. \qquad (28) $$

The first experiment consists of a pure lateral movement of 0.5 m in the negative x direction of the {$S$} frame. Results of this experiment are shown in Fig. 5.

The dotted and the dashed lines are associated with the conditions T and S, respectively, while the continuous lines are associated with the condition SO. In condition T, the object reaches a maximum norm of the displacement $\mathcal{D}$ of 0.08 m from the initial position and falls from the tray, while conditions S and SO always prevent the object from sliding with respect to the tray. This is evident from the bottom graph. The robustness measure $\mathcal{R}$ is overall higher in conditions SO compared to condition S. In particular, condition SO performs better on this metric as it never attains zero values. Values close to zero are achieved for the condition T when the object slides. The norm of the tracking error $\mathcal{E}$ would be the lowest in the T condition,

<figure><img src="image_6.png" /><figcaption>Fig. 6. Comparison of the executed path, norm of the position error $\mathcal{E}$, robustness measure $\mathcal{R}$ and object relative displacement $\mathcal{D}$ along a predefined circular trajectory.</figcaption></figure>

but the object slides and falls from the tray. However, it is lower in condition SO with respect to S (where non-sliding is enforced) showing enhanced system performances. It is also worth noting that condition SO allows achieving faster convergence to the final configuration, denoted by the norm of the tracking error $\mathcal{E} \to 0$ around 1.5 s.

The second experiment consists in tracking a circular trajectory, in the x-z plane, and its purpose is to show the applicability of our approach to execute motions in the vertical plane. The circular path has the center at $c = [0.105, 0.480, 0.415]$ m in {$S$} and the radius $r = 0.1$ m. The results of this experiment are shown in Fig. 6. For this experiment, we also show the executed path in the three conditions T, S, and SO (top graph). It is possible to note that, in condition S, the executed path substantially differs from the one desired one, and this can be attributed to the enforcement of the non-sliding constraints in (10). Considerations analogous to the previous trajectory tracking experiment can be made, i.e., SO outperforms the S and T approaches in terms of sliding prevention, performances, and robustness.

## D. Teleoperation experiments

We tested the three control modes T, S, and SO, combined with the two haptic modes H and $\mathcal{H}$ (see Sec. V-B) in a series

<footnote n="5">⁵The robustness measure $\mathcal{R}$ is imposed to be zero whenever $F_c \notin FC$ as it might happen in the T case during object sliding. This is done to perform a fair comparison of $\mathcal{R}$ among control modes T, S and SO.</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 7. Comparison of the transparency measure $\tilde{\mathcal{F}}$, the robustness measure $\mathcal{R}$, and the norm of the positional object displacement $\mathcal{D}$ along a teleoperation experiment.</figcaption></figure>

of representative teleoperation experiments. These two sets of modalities yield a total of six experimental conditions: T-H, T-H, S-H, S-H, SO-H, SO-H. For brevity, we only consider experimental conditions TH, SH, and SOH here, as they constitute the most complete examples. All conditions will be considered in the human-subjects study of Sec. VI. We consider a teleoperation task consisting in carrying the object among three setpoints, A, B, and C, placed in the horizontal plane (see Fig. 4). The goal is to move the object as fast as possible across the set points, following the sequence A-B-C-B-A. In addition to the robustness measure $\mathcal{R}$ and the object displacement $\mathcal{D}$, introduced in Sec. V-C, we show (in place of the tracking error $\mathcal{E}$) the quantity

$$ \tilde{\mathcal{F}} = \|f^{b*} - f^b\|, \qquad (29) $$

which is a measure of the system's transparency. Indeed, $\tilde{\mathcal{F}}$ tells us the difference between the commanded and the applicable body force (respecting the non-sliding constraints). It is used both to regulate the robot motion when shared control is active and to display haptic cues to the user. Lower values of $\tilde{\mathcal{F}}$ indicate better transparency, as the system more faithfully executes what is being intended/commanded by the user.

In Fig. 7, the dotted and the dashed lines are associated to the conditions TH and SH, respectively, while the continuous lines are associated to the condition SOH. It is possible to note that in condition TH, the object reaches a maximum norm of displacement $\mathcal{D} = 0.018$ m from the initial position, while the conditions SH and SOH always prevent the object from sliding with respect to the tray. The robustness measure $\mathcal{R}$ is overall higher in conditions SH and SOH compared to condition TH. In the considered case, the transparency of the system is always

high (as indicated by the low values attained by $\tilde{\mathcal{F}}$) in the two conditions SH and SOH, while it decreases in the condition TH when sliding occurs. Of course, the results presented here are highly subjective and might differ across users. For this reason, we analyzed the performance of our control/feedback conditions in a human-subjects experimental study, as detailed in the next section.

## VI. HUMAN-SUBJECTS STUDY

We carried out a human-subjects study to evaluate the effectiveness of the proposed shared-control approach as well as the role of providing haptic feedback to the user. The task is the same as the one proposed in Sec. V-D. Each participant performed six trials corresponding to the six conditions T, TH, S, SH, SO, SOH, yeilding a total of 108 teleoperation experiments. Participants were asked to teleoperate the robotic manipulator through the haptic interface and move the object, carried by the tray, as fast as possible between the three target areas, A, B, and C following the sequence A-B-C-B-A (Fig. 4). To this end, a countdown timer starting at 15 s was overlayed to the operator view of the remote side. The participant was told to perform the task within the 15 s time frame which was the baseline for the expert operator. The task started when the robot moved for the first time towards A, and it was considered completed when it reached again point A at the end of the sequence. To be aware of the completion of a motion segment, the user exploits the visual feedback provided by the monitor placed at the local side, showing the image in Fig. 4-left.

### A. Participants

Eighteen subjects (4 females, 14 males, average age of 31.4) participated in the study. Six of them had previous experience with robotic teleoperation systems. None of the participants reported any deficiencies in their visual or haptic perception abilities. The experimenter explained the procedures and spent about two minutes adjusting the setup to be comfortable before the subject began the experiment. Each subject then spent 5 minutes practicing the control of the telemanipulation system before starting the experiment. We did not record any sensitive, protected, or identifiable data from the participants.

### B. Results

We considered five metrics:

* the task completion time $\mathcal{T}$, i.e., the time the user employs to carry out the whole sequence A-B-C-B-A;

* the maximum displacement of the object on the tray relative to its initial position

$$ \bar{\mathcal{D}} = \max \{\mathcal{D}(t) : 0 \le t \le \mathcal{T}\}; \qquad (30) $$

* the average robustness along the task, i.e., the integral of the robustness measure defined in (27)

$$ \bar{\mathcal{R}} = \frac{1}{\mathcal{T}} \int_{0}^{\mathcal{T}} \mathcal{R} \, dt; \qquad (31) $$
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 8. Human-subjects study results. Mean and standard error of the mean of (a) completion time $T$, (b) maximum relative displacement of the object $\bar{D}$, (c) average robustness measure $\bar{R}$, (d) transparency $\bar{F}$, and (e) perceived effectiveness $S$ for six conditions T, TH, S, SH, SO, SOH.</figcaption></figure>

* the transparency of the system, evaluated as the amount of assistance of the autonomous controller

$$ \bar{\mathcal{F}} = \int_{0}^{T} \tilde{\mathcal{F}} \, dt, \qquad (32) $$

where $\tilde{\mathcal{F}}$ is defined in (29) as the norm of the difference between the forces commanded by the human operator and those applied by the robot. When S and SO modes are considered, the quantity $\tilde{\mathcal{F}}$ is used to regulate the robot

motion. When the H mode is considered, the quantity $\tilde{\mathcal{F}}$ is also involved in the calculation of the displayed haptic cues;

* the perceived effectiveness $S$ of each experimental condition, score reported by the subjects immediately after the experiment using bipolar Likert-type eleven-point scales.

To compare the different metrics, unless specified otherwise, we ran two-way repeated-measures ANOVA tests (significance level $\alpha = 0.05$). ANOVA is a very popular collection of statistical models and estimation tools used to analyze differences among group means in a sample [59]. As it is able to test whether two or more population means are equal, it can be seen as a generalization of the Student's t-test beyond two means. The control modality (T vs. S vs. SO) and feedback modality (H vs. $\mathcal{H}$) were the within-subject factors. All data passed the Shapiro-Wilk normality test, except condition T-$\mathcal{H}$ in metric $\tilde{\mathcal{F}}$. A Greenhouse-Geisser correction was used when the assumption of sphericity was violated. Sphericity was assumed for variables with only two levels of repeated measures (the feedback). Results of post hoc analysis with Bonferroni adjustments are reported in Table II (only significant $p$ values are shown). When simple main effects are considered, significant $p$ values are also reported in Fig. 8.

Fig. 8a shows the completion time $T$. Mauchly's Test of Sphericity [60] indicated that the assumption of sphericity had been violated for the control-feedback variables interaction ($\chi^2(2) = 7.985, p = 0.018$). A two-way repeated-measure ANOVA revealed no statistically significant change for this metric in the control technique, feedback modality, and their interaction.

Fig. 8b shows the maximum relative displacement of the object from its initial position $\bar{D}$. In all the performed experiments conditions S and SO were 100% successful in preventing object dropping. Mauchly's Test of Sphericity indicated that the assumption of sphericity had been violated for the control variable ($\chi^2(2) = 58.016, p < 0.001$) and the control-feedback variables interaction ($\chi^2(2) = 27.717, p < 0.001$). The two-way repeated-measure ANOVA revealed a statistically significant two-way interaction between control and feedback variables (F(1.0097, 18.649) = 17.073, $p = 0.007$). When a statistically significant interaction between variables is found, we need to analyze the simple main effects [61]. Interpreting the simple main effects for the control variable, we found a statistically significant difference (T-$\mathcal{H}$ vs. S-$\mathcal{H}$ vs. SO-$\mathcal{H}$: F(1.015, 17.253) = 38.158, $p < 0.001$; T-H vs. S-H vs. SO-H: F(1.032, 17.545) = 37.709, $p < 0.001$). For the feedback variable, we found a statistically significant difference between T-$\mathcal{H}$ vs. T-H and S-$\mathcal{H}$ vs. S-H.

Fig. 8c shows the robustness measure $\bar{R}$ evaluated as the distance of the contact forces applied to the object from the friction cone limits. A low value indicates high robustness of the approach. Mauchly's Test of Sphericity indicated that the assumption of sphericity had been violated for the control-feedback variables interaction ($\chi^2(2) = 9.107, p = 0.011$). The two-way repeated-measure ANOVA revealed a statistically significant change for this metric across control technique (F(2,34) = 30.086, $p < 0.001$).

Fig. 8d shows the transparency of the control $\tilde{\mathcal{F}}$, representing
how well the robot followed the commands imparted by the human operator. A low value indicates a high transparency of the approach. In this case, of course, T-H shows perfect transparency ($\bar{F} = 0$), since the operator's commands are imparted to the robotic manipulator directly and no feedback is involved. Normality plots revealed two subjects outliers, which registered $\bar{F}$ values up to 6 times higher than the rest: we did not consider them in the following analysis. Analyzing the simple main effects for the control variable, a Friedman test showed a statistically significant difference (T-H vs. S-H vs. SO-H: $\chi^2(2) = 28.500, p < 0.001$; T-H vs. S-H vs. SO-H: $\chi^2(2) = 11.175, p = 0.003$). For the feedback variable, a Wilcoxon Signed Rank test found a statistically significant difference between T-H vs. T-H. The Friedman and the Wilcoxon Signed Rank tests [62] are the non-parametric equivalents of the more popular repeated-measures ANOVA and paired t-test; the latter is not appropriate here since data T-H is, of course, non-normally distributed.

Finally, immediately after the experiment, participants were asked to fill in a questionnaire using bipolar Likert-type eleven-point scales. It asked the perceived effectiveness of each condition, where a score of 1 was described as “not effective at all” and a score of 11 as “very effective”. Fig. 8e shows the ratings given by the subjects. Mauchly’s Test of Sphericity on logn transformed data indicated that the assumption of sphericity had been violated for the control-feedback variables interaction ($\chi^2(2) = 7.543, p = 0.023$). The two-way repeated-measure ANOVA revealed a statistically significant change for this metric across control technique (F(2,34) = 25.754, $p < 0.001$) and feedback modality (F(1,17) = 13.985, $p = 0.002$). Nine subjects out of eighteen chose SO-H as their preferred condition, six chose SO-H, and one chose S-H, S-H, and T-H.

## VII. DISCUSSION AND CONCLUSION

Shared control teleoperation allows combining human operators' experience and cognitive capabilities with the precision, reactivity, and repeatability of autonomous systems. Our work introduces the first haptic-guided shared-control teleoperation architecture designed for the nonprehensile transportation of an object on a tray-like end-effector. The human operator controls the object's linear motion, while, at the same time, an autonomous controller alters her/his commands to prevent the object from sliding and falling. This is accomplished by regulating the robot's linear motion and autonomously inclining the tray. Haptic feedback informs the operator about the autonomous controller's actions, conveying forces proportional to the mismatch between the user-commanded forces and those applied to the robot.

We performed trajectory tracking experiments to quantitatively compare the improvements introduced by our proposed architecture with respect to constant orientation and no-motion overriding control approaches. The devised architecture performed better in terms of trajectory tracking error, robustness, and amount of object sliding along both linear and circular trajectory executions. The representative teleoperation experiments showed similar trends.

To more rigorously prove the viability and effectiveness of the proposed techniques, we carried out a human subject

<table><caption>TABLE II
EXPERIMENTAL EVALUATION</caption><tbody><tr><td><strong>Subjects</strong></td><td colspan="2">18 (4 females, 14 males)</td></tr><tr><td><strong>Task</strong></td><td colspan="2">Control the teleoperation system to move the object among a sequence of set points.</td></tr><tr><td><strong>Conditions</strong></td><td colspan="2"><u>Control</u><br/>T (classic teleoperation), S (shared control with constant tray orientation), SO (shared control with adapting tray orientation)<br/><u>Feedback</u><br/>H (haptic feedback), H (no haptic feedback)</td></tr><tr><td colspan="3"><strong>Statistical analysis</strong> (only significant <em>p</em> values are shown)</td></tr><tr><td><u>Completion time, T</u></td><td colspan="2">No statistical difference observed.</td></tr><tr><td><u>Object max. displacement, D</u></td><td colspan="2"></td></tr><tr><td>Simple main effect of control</td><td></td><td></td></tr><tr><td>T-H vs. S-H <em>p</em> &lt; 0.001</td><td>T-H vs. SO-H <em>p</em> &lt; 0.001</td><td></td></tr><tr><td>T-H vs. S-H <em>p</em> &lt; 0.001</td><td>T-H vs. SO-H <em>p</em> &lt; 0.001</td><td></td></tr><tr><td>Simple main effect of feedback</td><td></td><td></td></tr><tr><td>T-H vs. T-H <em>p</em> = 0.012</td><td>T-H vs. S-H <em>p</em> = 0.014</td><td></td></tr><tr><td><u>Robustness, R</u></td><td colspan="2"></td></tr><tr><td>Main effect of control</td><td></td><td></td></tr><tr><td>T vs. S <em>p</em> = 0.001</td><td>T vs. SO <em>p</em> &lt; 0.001</td><td></td></tr><tr><td>S vs. SO <em>p</em> = 0.005</td><td></td><td></td></tr><tr><td><u>Control transparency, F</u></td><td colspan="2"></td></tr><tr><td>Simple main effect of control</td><td></td><td></td></tr><tr><td>T-H vs. S-H <em>p</em> &lt; 0.001</td><td>T-H vs. SO-H <em>p</em> = 0.004</td><td></td></tr><tr><td>T-H vs. SO-H <em>p</em> = 0.014</td><td>S-H vs. SO-H <em>p</em> = 0.008</td><td></td></tr><tr><td>Simple main effect of feedback</td><td></td><td></td></tr><tr><td>T-H vs. T-H <em>p</em> &lt; 0.001</td><td></td><td></td></tr><tr><td><u>Subjective questionnaire</u></td><td colspan="2"></td></tr><tr><td>Main effect of control</td><td></td><td></td></tr><tr><td>T vs. S <em>p</em> &lt; 0.001</td><td>T vs. SO <em>p</em> &lt; 0.001</td><td></td></tr><tr><td>S vs. SO <em>p</em> = 0.028</td><td></td><td></td></tr><tr><td>Main effect of feedback</td><td></td><td></td></tr><tr><td>H vs. H <em>p</em> = 0.002</td><td></td><td></td></tr><tr><td><u>Most effective experimental condition as chosen by subjects</u></td><td colspan="2">Nine subjects out of eighteen chose SO-H, six chose SO-H, one chose S-H, S-H, and T-H.</td></tr></tbody></table>

study enrolling eighteen subjects. We tested the performance of six experimental conditions: classic teleoperation (T), shared control with fixed tray orientation (S), shared control with adapting tray orientation (SO), all tested with (H) and without (H) haptic guidance based on the difference between the commands imparted by the user and those adjusted by the autonomous controller. Results showed that in all the considered metrics but two (completion time and transparency), shared control (S, SO) outperformed pure teleoperation T. Moreover, the great majority of subjects preferred SO and S over T. Shared control with adapting tray orientation (SO) performed better than T in all metrics but one (completion time) and better than shared control with fixed tray orientation (S) in all metrics but two (completion time, object displacement). These results are in line with our expectations. By acting on the user's commands, both shared control approaches can significantly limit the object's movements on the tray. Moreover, they are both well appreciated by the users, who feel more confident when teleoperating the robot in these conditions. As SO also acts on the orientation of the tray, it allows achieving
far higher accelerations than S. This characteristic shows
in the transparency metric, where SO performs significantly
better than S, meaning that it alters significantly less the
user's commands. Haptic feedback was also deemed useful in
letting the users know how the autonomous system alters their
commands before imparting them to the robot. Its benefits
were most evident in the subjective metrics, i.e., perceived
effectiveness and most effective condition. Its role is particularly
crucial in T-H, where the autonomous controller does not
directly alter the user's commands. In this case, providing
haptic cues is the only way to communicate the user the need
to adjust their commands to prevent the object from moving.
This effect is shown in the object displacement metric, where
T-H performs significantly better than T-М.

One limitation of our approach is that it requires knowing the
shape and the physical characteristics of the object. However,
approximating objects having more complex shapes with
rectangular cuboids has already proven to be effective in
some other experimental cases. For instance, a cup was safely
transported by the manipulator using the same model, as shown
in the video. To what extend this approach would work will
be subject of future studies. Extending the proposed approach
to estimate the object's properties at runtime or during an
additional calibration step should also be possible.

The point contact with friction model adopted in this work
has proven to work satisfactorily throughout the proposed
experiments. However, there exists more accurate, yet more
involved, models (e.g., the *soft finger model* [53]) that can
be, in principle, used to capture frictional/sliding physical
phenomena more accurately. For instance, different approaches
can be adopted to account non-uniform and/or rotational friction
in the model. However, more elaborated models may depend
on parameters that are, in general, harder to be accurately
estimated (e.g., rotational friction).

Even if this paper only focuses on non-sliding manipulation,
the developed approach can be easily integrated with other
approaches to build more complex (autonomous or human-
operated) behaviours. An interesting future work might be
indeed combining non-sliding and re-positioning “in-hand”
manipulation features. This would allow seamless sliding only
along the desired directions during teleoperation or perform
autonomous object re-positioning before the user takes over.

Although no unstable behavior was detected during the exper-
iments performed with the proposed teleoperation architecture,
passivity can be formally guaranteed exploiting any passivity-
based control technique, such as the energy tanks method [24],
[28], [63]. This will be investigated in future works.

Finally, it is essential to notice that none of our subjects was
experienced in using the proposed shared control framework.
The recorded significant difference between T, S and SO
might change in the presence of experienced users. Indeed,
it might be argued that expert operators, who know the
system's dynamics, might be able to estimate how the robot's
movement will affect the object movements. However, even
if we employ a 6-DoF haptic interface to control both the
translation and orientation of the robot end-effector, it is not
easy to precisely and reactively control six DoFs simultaneously.
It is also important to highlight that our system alters the

user's command only when necessary. It is thus not activated
when the commanded forces stay within the imposed limits
making the autonomous control minimally invasive. The
autonomous controller actions could also be adapted to the
human operator's experience. For example, a system could use
a highly-autonomous shared-control approach (i.e., stiff haptic
feedback, more DoF managed by the autonomous algorithm)
when novices operate it. Contrarily, it could implement a
lowly-autonomous shared-control approach (i.e., compliant
or ungrounded haptic feedback, few DoF managed by the
autonomous algorithm) when instead experts operate it. This
flexible approach might also be useful when teaching new
operators, employing different levels of autonomy according
to the operator's experience.

REFERENCES

[1] S. Avgousti, E. G. Christoforou, A. S. Panayides, S. Voskarides, C. Novales, L. Nouaille, C. S. Pattichis, and P. Vieyres, "Medical telerobotic systems: current status and future trends," *Biomed. Eng. Online*, vol. 15, no. 1, p. 96, Aug 2016.

[2] I. Tsitsimpelis, C. J. Taylor, B. Lennox, and M. J. Joyce, "A review of ground-based robotic systems for the characterization of nuclear environments," *Prog. Nucl. Energy*, vol. 111, pp. 109–124, 2019.

[3] G. Niemeyer, C. Preusche, and G. Hirzinger, *Telerobotics*. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 741–757.

[4] A. Franchi, C. Secchi, M. Ryll, H. H. Bulthoff, and P. Robuffo Giordano, "Shared control : Balancing autonomy and human assistance with a group of quadrotor uavs," *IEEE Rob. Aut. Mag.*, vol. 19, no. 3, pp. 57–68, Sept 2012.

[5] K. Hauser, "Recognition, prediction, and planning for assisted teleoperation of freeform tasks," *Auton. Robot.*, vol. 35, no. 4, pp. 241–254, Nov 2013.

[6] H. Boessenkool, D. A. Abbink, C. J. M. Heemskerk, F. C. T. van der Helm, and J. G. W. Wildenbeest, "A task-specific analysis of the benefit of haptic shared control during telemanipulation," *IEEE Trans. Haptics*, vol. 6, no. 1, pp. 2–12, 2013.

[7] D. Rakita, B. Mutlu, M. Gleicher, and L. M. Hiatt, "Shared control–based bimanual robot manipulation," *Sci. Robot.*, vol. 4, no. 30, 2019.

[8] A. M. Ghalamzan E., F. Abi-Farraj, P. Robuffo Giordano, and R. Stolkin, "Human-in-the-loop optimisation: Mixed initiative grasping for optimally facilitating post-grasp manipulative actions," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 2017, pp. 3386–3393.

[9] D. Lee, O. Martinez-Palafox, and M. W. Spong, "Bilateral teleoperation of multiple cooperative robots over delayed communication networks: Application," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2005, pp. 366–371.

[10] S. Musić and S. Hirche, "Control sharing in human-robot team interaction," *Annu. Rev. Control*, vol. 44, pp. 342–354, 2017.

[11] M. Selvaggio, P. Robuffo Giordano, F. Ficuciello, and B. Siciliano, "Passive task-prioritized shared-control teleoperation with haptic guidance," in *Proc. IEEE Int. Conf. Robot. Autom.*, May 2019, pp. 430–436.

[12] K. M. Lynch and M. T. Mason, "Dynamic nonprehensile manipulation: Controllability, planning, and experiments," *Int. J. Robot. Res.*, vol. 18, no. 1, pp. 64–92, 1999.

[13] A. Bicchi, "On the closure properties of robotic grasping," *Int. J. Robot. Res.*, vol. 14, no. 4, pp. 319–334, 1995.

[14] J. R. Napier, "The prehensile movements of the human hand," *J Bone Joint Surg*, vol. 38-B, no. 4, pp. 902–913, 1956.

[15] M. T. Mason and K. M. Lynch, "Dynamic manipulation," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, vol. 1, 1993, pp. 152–159.

[16] K. M. Lynch and M. T. Mason, "Dynamic underactuated nonprehensile manipulation," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, vol. 2, 1996, pp. 889–896.

[17] M. Higashimori, R. Sakashita, and A. Shibata, "Single-actuator-based three-dof planar manipulation via a viscoelastic and nonparallel hybrid joint mechanism," *IEEE Trans. Robot.*, vol. 35, no. 3, pp. 602–617, 2019.

[18] M. R. Fossati, M. G. Catalano, M. Carbone, G. Lentini, D. Caporale, G. Grioli, M. Poggiani, M. Maimeri, M. Barbarossa, C. Petrocelli, et al., "Lhf connect: a diy telepresence robot against covid-19," *Strategic Design Research Journal*, vol. 13, no. 3, pp. 418–431, 2020.

[19] S. S. Srinivasa, M. A. Erdmann, and M. T. Mason, "Using projected dynamics to plan dynamic contact manipulation," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 2005, pp. 3618–3623.
[20] D. A. Lawrence, "Stability and transparency in bilateral teleoperation," *IEEE Trans. Robot. Autom.*, vol. 9, no. 5, pp. 624–637, 1993.

[21] G. Niemeyer and J. J. E. Slotine, "Stable adaptive teleoperation," *IEEE J. Ocean. Eng.*, vol. 16, no. 1, pp. 152–162, 1991.

[22] R. J. Anderson and M. W. Spong, "Bilateral control of teleoperators with time delay," *IEEE Trans. Autom. Control*, vol. 34, no. 5, pp. 494–501, 1989.

[23] J.-H. Ryu, D.-S. Kwon, and B. Hannaford, "Stable teleoperation with time-domain passivity control," *IEEE Trans. Robot. Autom.*, vol. 20, no. 2, pp. 365–373, 2004.

[24] M. Franken, S. Stramigioli, M. Misra, C. Secchi, and A. Macchelli, "Bilateral telemanipulation with time delays: a two-layer approach combining passivity and transparency," *IEEE Trans. Robot.*, vol. 27, no. 4, pp. 741–756, 2011.

[25] T. B. Sheridan, "Teleoperation, telerobotics and telepresence: A progress report," *Control Eng. Pract.*, vol. 3, no. 2, pp. 205–214, 1995.

[26] J. V. Draper, D. B. Kaber, and J. M. Usher, "Telepresence," *Hum. Factors*, vol. 40, no. 3, pp. 354–375, 1998.

[27] C. Secchi, A. Franchi, H. H. Bülthoff, and P. Robuffo Giordano, "Bilateral teleoperation of a group of uavs with communication delays and switching topology," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2012, pp. 4307–4314.

[28] A. Franchi, C. Secchi, H. I. Son, H. H. Bulthoff, and P. R. Giordano, "Bilateral teleoperation of groups of mobile robots with time-varying topology," *IEEE Trans. Robot.*, vol. 28, no. 5, pp. 1019–1033, 2012.

[29] M. Laghi, M. Maimeri, M. Marchand, C. Leparoux, M. Catalano, A. Ajoudani, and A. Bicchi, "Shared-autonomy control for intuitive bimanual telemanipulation," in *IEEE-RAS 18th Int. Conf. Human. Rob.*, 2018, pp. 1–9.

[30] M. Selvaggio, F. Abi-Farraj, C. Pacchierotti, P. Robuffo Giordano, and B. Siciliano, "Haptic-based shared-control methods for a dual-arm system," *IEEE Robot. Autom. Lett.*, vol. 3, no. 4, pp. 4249–4256, Oct 2018.

[31] R. L. Williams, F. W. Harrison, and D. L. Soloway, "Shared control of multiple-manipulator, sensor-based telerobotic systems," in *Proc. IEEE Int. Conf. Robot. Autom.*, vol. 2, 1997, pp. 962–967.

[32] M. K. O'Malley, A. Gupta, M. Gen, and Y. Li, "Shared control in haptic systems for performance enhancement and training," *J. Dyn. Syst. Meas. Control*, vol. 128, no. 1, pp. 75–85, 2006.

[33] F. Abi-Farraj, N. Pedemonte, and P. Robuffo Giordano, "A visual-based shared control architecture for remote telemanipulation," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 2016, pp. 4266–4273.

[34] F. Abi-Farraj, C. Pacchierotti, O. Arenz, G. Neumann, and P. Robuffo Giordano, "A haptic shared-control architecture for guided multi-target robotic grasping," *IEEE Trans. Haptics*, vol. 13, no. 2, pp. 270–285, 2020.

[35] M. Selvaggio, A. M. Ghalamzan E., R. Moccia, F. Cicuciello, and B. Siciliano, "Haptic-guided shared control for needle grasping optimization in minimally invasive robotic surgery," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, Nov 2019, pp. 3617–3623.

[36] F. Ruggiero, V. Lippiello, and B. Siciliano, "Nonprehensile dynamic manipulation: A survey," *IEEE Robot. Autom. Lett.*, vol. 3, no. 3, pp. 1711–1718, July 2018.

[37] J. Z. Woodruff and K. M. Lynch, "Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks," in *Proc. IEEE Int. Conf. Robot. Autom.*, May 2017, pp. 4066–4073.

[38] K. M. Lynch and T. D. Murphey, "Control of nonprehensile manipulation," in *Control Problems in Robotics*, ser. Springer Tracts in Advanced Robotics, A. Bicchi, D. Prattichizzo, and H. Christensen, Eds. Springer Berlin Heidelberg, 2003, vol. 4, pp. 39–57.

[39] A. Satici, F. Ruggiero, V. Lippiello, and B. Siciliano, "Coordinate-free framework for robotic pizza tossing and catching," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2016, pp. 3932–3939.

[40] M. M. Schill, F. Gruber, and M. Buss, "Quasi-direct nonprehensile catching with uncertain object states," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2015, pp. 2468–2474.

[41] C. Liu, Y. Hayakawa, and A. Nakashima, "Racket control and its experiments for robot playing table tennis," in *Proc. IEEE Int. Conf. Robot. Biomim.*, 2012, pp. 241–246.

[42] F. Bertoncelli, F. Ruggiero, and L. Sabattini, "Linear time-varying MPC for nonprehensile object manipulation with a nonholonomic mobile robot," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2020.

[43] D. Serra, F. Ruggiero, A. Donaire, L. Buonocore, V. Lippiello, and B. Siciliano, "Control of nonprehensile planar rolling manipulation: A passivity-based approach," *IEEE Trans. Robot.*, vol. 35, no. 2, pp. 317–329, 2019.

[44] D. Serra, J. Ferguson, F. Ruggiero, A. Siniscalco, A. Petit, V. Lippiello, and B. Siciliano, "On the experiments about the nonprehensile recon-

figuration of a rolling sphere on a plate," in *Proc. 26th Mediterranean Conference on Control and Automation*, Zadar, HR, 2019, pp. 13–20.

[45] M. Higashimori, K. Utsumi, Y. Omoto, and M. Kaneko, "Dynamic manipulation inspired by the handling of a pizza peel," *IEEE Trans. Robot.*, vol. 25, no. 4, pp. 829–838, 2009.

[46] T. Vose, P. Umbanhowar, and K. Lynch, "Sliding manipulation of rigid bodies on a controlled 6-dof plate," *Int. J. Robot. Res.*, vol. 31, no. 7, pp. 819–838, 2012.

[47] A. Gutierrez-Giles, F. Ruggiero, V. Lippiello, and B. Siciliano, "Closed-loop control of a nonprehensile manipulation system inspired by a pizza-peel mechanism," in *Proc. European Control Conference*, Naples, I, 2019, pp. 1580–1585.

[48] P. Lertkultanon and Q.-C. Pham, "Dynamic non-prehensile object transportation," in *Proc. Int. Conf. Control Autom. Robot. Vis.*, 2014, pp. 1392–1397.

[49] J. Garcia-Haro, S. Martinez, and C. Balaguer, "Balance computation of objects transported on a tray by a humanoid robot based on 3D dynamic slopes," in *IEEE-RAS Int. Conf. Human. Rob.*, 2018, pp. 704–709.

[50] Y. Aiyama, M. Inaba, and H. Inoue, "Pivoting: A new method of graspless manipulation of object by robot fingers," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, 1993, pp. 136–143.

[51] J. Solanes, L. Gracia, P. Muñoz Benavent, J. Miro, M. Carmichael, and J. Tornero, "Human-robot collaboration for safe object transportaion using force feedback," *Rob. Auton. Syst.*, vol. 107, pp. 196–208, 2018.

[52] L. Biagiotti, D. Chiaravalli, L. Moriello, and C. Melchiorri, "A plug-in feed-forward control for sloshing suppression in robotic teleoperation tasks," in *Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.*, Oct 2018, pp. 5855–5860.

[53] R. M. Murray, S. S. Sastry, and L. Zexiang, *A Mathematical Introduction to Robotic Manipulation*, 1st ed. USA: CRC Press, Inc., 1994.

[54] A. Bicchi, "On the problem of decomposing grasp and manipulation forces in multiple whole-limb manipulation," *Robot. Auton. Syst.*, vol. 13, no. 2, pp. 127–147, 1994.

[55] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, *Robotics: modelling, planning and control*. Springer Science & Business Media, 2010.

[56] J. M. Romano, R. J. Webster, and A. M. Okamura, "Teleoperation of steerable needles," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2007, pp. 934–939.

[57] E. Olson, "Apriltag: A robust and flexible visual fiducial system," in *Proc. IEEE Int. Conf. Robot. Autom.*, 2011, pp. 3400–3407.

[58] A. Bicchi, "Force distribution in multiple whole-limb manipulation," in *Proc. IEEE Int. Conf. Robot. Autom.*, 1993, pp. 196–201.

[59] E. R. Girden, ANOVA: Repeated measures. Sage, 1992, no. 84.

[60] J. W. Mauchly, "Significance test for sphericity of a normal n-variate distribution," *Ann. Math. Stat.*, vol. 11, no. 2, pp. 204–209, 1940.

[61] N. R. Draper and H. Smith, *Applied regression analysis*. John Wiley & Sons, 1998, vol. 326.

[62] S. Siegel, "Nonparametric statistics," *Am. Stat.*, vol. 11, no. 3, pp. 13–19, 1957.

[63] P. R. Giordano, A. Franchi, C. Secchi, and H. H. Bülthoff, "A passivity-based decentralized strategy for generalized connectivity maintenance," *Int. J. Rob. Res.*, vol. 32, no. 3, pp. 299–323, 2013.

<figure><img src="image_13.png" /></figure>

**Mario Selvaggio** (S'16, M'20) was born in Naples, Italy, on June 23, 1990. He received the bachelor's and master's degrees in mechanical engineering from University of Naples Federico II, Naples, Italy, in 2013 and 2015, respectively, and the Ph.D. in information technology and electrical engineering from the same institution in 2020. He has been Master Student with the Fraunhofer IGD, Darmstadt, Germany in 2014; Intern with Istituto Italiano di Technologia, Genova, Italy, in 2016; Visiting Student with IRISA, INRIA Rennes, Rennes, France, in 2017 and 2018; Visiting Student with the University of California Santa Barbara, Santa Barbara, California, US, in 2019. Since 2020 he is working as a Postdoctoral Researcher in the Electrical Engineering and Information Technology department of the University of Naples Federico II, Naples, Italy. He is Associate Editor of the IEEE ICAR and of the IEEE Robotics and Automation Letters. He has published more than 35 peer-reviewed journal articles, conference papers and book chapters.
<figure><img src="image_14.png" /></figure>

**Jonathan Cacace** was born in Naples, Italy, on December 13, 1987. He received the Master's degree (magna cum laude) in computer science from the University of Naples Federico II in 2012 and the Ph.D. degree in Robotics in 2016 from the same institution. Currently, he is working as Postdoctoral Researcher at the PRISMA Lab (Projects of Robotics for Industry and Services, Mechatronics and Automation Laboratory) at the University of Naples Federico II, where is involved in several research projects in the field of Human-Robot Interaction in Industry

4.0, autonomous control of UAV for inspection and maintenance and robotic manipulation. He serves as Associate Editor for IEEE ICRA conference and IEEE Robotics and Autonomous Letters.

<figure><img src="image_15.png" /></figure>

**Claudio Pacchierotti** (S'12, M'15, SM'20) is a tenured researcher at CNRS-IRISA in Rennes, France, since 2016. He was previously a postdoctoral researcher at Dept. of Advanced Robotics at the Italian Institute of Technology, Genova, Italy. Pacchierotti earned his PhD at the University of Siena in 2014. He visited the Penn Haptics Group at University of Pennsylvania in 2014, the Dept. of Innovation in Mechanics and Management at University of Padua in 2013, and the Institute for Biomedical Technology and Technical Medicine (MIRA) at University of

Twente in 2014. Pacchierotti received the 2014 EuroHaptics Best PhD Thesis Award for the best doctoral thesis in the field of haptics. He is Chair of the IEEE Technical Committee on Haptics and Secretary of the Eurohaptics Society. He has published more than 100 peer-reviewed papers on the topics of robotic teleoperation, cutaneous haptics, and wearable interfaces.

<figure><img src="image_16.png" /></figure>

**Fabio Ruggiero** (S'07–M'10–SM'19) received the M.Sc. degree in Automation Engineering from the University of Naples Federico II in 2007. He got the Ph.D. degree from the same institution in 2010. He spent seven months at Northwestern University as a visiting Ph.D. student from September 2009 to March 2010. After several PostDoctoral positions from 2011 to 2016, he has been holding an Assistant Professor position (tenure track) at the University of Naples Federico II. His research interests are focused on model-based control design of robotic systems. In

particular, his studies are specialized on control strategies for dexterous, dual-hand and nonprehensile robotic manipulation, aerial robots, aerial manipulators, and legged robots. He is associate editor of the IEEE Robotics and Automation Letters. He has participated to several European research projects. He has been principal investigator of a project funded by the Italian Ministry of Research and a project funded by Compagnia di San Paolo. He has co-authored more than 60 among journal papers, book chapters, and conference papers.

<figure><img src="image_17.png" /></figure>

**Paolo Robuffo Giordano** (M'08–SM'16) received his M.Sc. in Computer Science Engineering in 2001, and his Ph.D. in Systems Engineering in 2008, both from the University of Rome “La Sapienza”. In 2007 and 2008 he spent one year as a Postdoc at the Institute of Robotics and Mechatronics of the German Aerospace Center (DLR), and from 2008 to 2012 he was Senior Research Scientist at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany. He is currently a senior CNRS researcher head of the Rainbow group at IRISA and

Inria in Rennes, France. He received the 2018 IEEE Robotics and Automation Letters best paper award, and he is Editor of the IEEE Transactions on Robotics.