# Haptic-guided Grasping to Minimise Torque Effort during Robotic Telemanipulation

Rahaf Rahal, Amir M Ghalamzan, Firas Abi-Farraj, Claudio Pacchierotti,

Paolo Robuffo Giordano

► To cite this version:

Rahaf Rahal, Amir M Ghalamzan, Firas Abi-Farraj, Claudio Pacchierotti, Paolo Robuffo Giordano. Haptic-guided Grasping to Minimise Torque Effort during Robotic Telemanipulation. Autonomous Robots, 2023, 47, pp.405-423. 10.1007/s10514-023-10096-7 . hal-04002085

HAL Id: hal-04002085

https://inria.hal.science/hal-04002085

Submitted on 23 Feb 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Haptic-guided Grasping to Minimise Torque Effort during Robotic Telemanipulation

Rahaf Rahal* · Amir M. Ghalamzan-E.* · Firas Abi-Farraj ·

Claudio Pacchierotti · Paolo Robuffo Giordano

Received: date / Accepted: date

**Abstract** Teleoperating robotic manipulators can be complicated and cognitively demanding for the human operator. Despite these difficulties, teleoperated robotic systems are still popular in several industrial applications, e.g., remote handling of hazardous material. In this context, we present a novel haptic shared control method for minimising the manipulator torque effort during remote manipulative actions in which an operator is assisted in selecting a suitable grasping pose for then displacing an object along a desired trajectory. Minimising torque is important because it reduces the system operating cost and extends the range of objects that can be manipulated. We demonstrate the effectiveness of the proposed approach in a series of representative real-world pick-and-place experiments as well as in a human subjects study. The reported results prove the effectiveness of our shared control vs. a standard teleoperation approach. We also find that haptic-only guidance performs better than visual-only guidance, although combining them together leads to the best overall results.

**Keywords** Teleoperation, Shared Control, Grasping, Haptics, Post-grasp manipulation

This project was partially funded by EU H2020 RoMaNS, 645582, and EPSRC EP/M026477/1, National Centre for Nuclear Robotic.

R. Rahal
Univ Rennes, Inria, CNRS, IRISA
E-mail: rahaf.rahal@irisafr

A. Ghalamzan
University of Lincoln
E-mail: a.ghalamzanesfahani@lincoln.ac.uk

F. Abi Farraj, C. Pacchierotti, and P. Robuffo Giordano
CNRS, Univ Rennes, Inria, IRISA
E-mail: {firas.abi-farraj, claudio.pacchierotti, prg}@irisafr

## 1 INTRODUCTION

Currently-available autonomous robots are still far from the reliability and safety required in many applications, such as medical robotics (Panesar et al., 2019) or hazardous waste management (Pardi et al., 2020). In these industrial settings, robotic teleoperation has become an effective tool to combine the experience and cognitive capabilities of a human operator with the precision, strength, and repeatability of a robotic system. Indeed, robotic teleoperation has proven effective in many applications, such as in nuclear waste decommissioning, minimally-invasive surgery, demolition, and tissue palpation. However, controlling a dexterous robotic manipulator can be rather complicated as well as cognitively and physically demanding (Talha et al., 2016; Rahal et al., 2020). In this respect, shared control algorithms have been investigated as one of the main tools for designing effective yet intuitive robotic teleoperation systems, helping operators in carrying out increasingly challenging tasks. This approach enables to share the available degrees of freedom of the robotic system between the operator and an autonomous controller. Applications include surgical suturing, industrial pick-and-place, mobile robotics, and industrial robotics. Implementations of shared control can use variable admittance control or active constraints, often enforced through haptic force feedback guiding the operator along the preferred directions of motion.

The use of haptic guidance in shared control has been referred to as *haptic shared control* (Abbink et al., 2012) or, alternatively, as virtual fixtures, active constraints or haptic guidance (Abi-Farraj et al., 2019). Haptic shared control in robotic teleoperation has been employed to, e.g., guide the human operator towards a reference position or along a given path (Ly et al., 2021), to avoid certain areas of the robot workspace, to teach the operator a manual task (Zhu et al., 2020), to modulate the level of robotic guidance (Izadi et al., 2020), based on predictions of human motion intentions (Xi

<footnote>* Authors Ghalamzan and Rahal contributed equally to this work.</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1: Teleoperation setup. The human operator uses a Virtuose 6D grounded haptic interface (Haption, France) to control a 7 degree of freedom (DoF) Panda robotic manipulator (Franka Emika, Germany), used as a 6-DoF manipulator. The user receives haptic feedback through the haptic interface and visual feedback from an LCD screen</figcaption></figure>

et al., 2019), or to ease the grasp of irregularly-shaped objects (Yang and Liu, 2021).

However, haptic shared control has rarely been used to optimise a relevant metric over the future consequences of a local operator's action. Ghalamzan et al. (2017) studied this problem and proposed a shared control algorithm able to guide the human operator towards a grasping pose that minimises the proximity to robot singularities during the execution of the object future trajectory. Although the preliminary results were quite promising, we also realized that just directly providing the haptic cues may sometimes result confusing for the human operator since these cues (that follow the gradient of the chosen cost function w.r.t. the current pose) can easily end up guiding the operator towards unfeasible grasping poses.

In this work, inspired from the works of Ghalamzan et al. (2017); Pardi et al. (2018); Selvaggio et al. (2019a); Parsa et al. (2020), we present a haptic shared control approach that helps human operators grasping objects in a way that minimises the torque exerted by the robotic manipulator over a future post-grasp trajectory. Besides considering a torque-related metric, our system further improves on the previous work by making sure that the provided haptic cues will always guide the user towards a feasible grasping pose whereas, as explained, the cues proposed by Ghalamzan et al. (2017) do not have guarantee of pointing towards feasible grasps. This is obtained by first using a geometric grasping algorithm to find several good grasp poses on the target object, and by then evaluating, for each grasping candidate, the torques needed by the remote robot for picking up the object and moving it along a pre-planned trajectory. In addition, we propose to convey this torque-related information using haptic guidance, visual guidance, or a combination of the two to human operators, which helps them to choose a grasp candidate corresponding with the minimum robot torque effort. As a final contribution, we also demonstrate the feasibility and effectiveness of the proposed approach in a set of representative pick-and-place tasks and by conducting a human

subjects study where we compare the performance of the
above three feedback modalities.

The contributions of our paper can then be summarised
as follows:

- present a haptic shared control technique selecting the best local grasping pose to minimise the robot torques over a future post-grasp trajectory;

- devise three feedback techniques to guide the operator toward the best local grasp pose: haptic-only, visual-only, and combined visuo-haptic guidance;

- carry out real-world trials using three representative objects having different shapes, dimensions, and weights;

- carry out an extensive human subjects evaluation en-
  rolling 15 subjects, using statistical tools to compare
  the considered feedback conditions over six metrics.

The proposed framework is compatible with any robotic
teleoperation system and it only requires the 3-dimensional
model and an estimate of the dynamic parameters of the target
object(s). In this paper, as a proof of concept, we consider a
teleoperation system (Fig. 1) composed of a 6-DoF grounded
haptic interface and a 7-DoF robotic manipulator equipped
with a parallel jaw gripper.

## 2 RELATED WORKS

Robotic grasping and manipulative movements are key el-
ements to build a reliable robotic solution for many indus-
trial setups. In this regard, robotic teleoperation has proven
effective in many applications, such as in nuclear waste
decommissioning (Talha et al., 2016), minimally-invasive
surgery (Selvaggio et al., 2019a), demolition (Corucci and
Ruffaldi, 2016), tissue palpation (Pacchierotti et al., 2015),
and needle insertion (Meli et al., 2017). However, control-
ling a dexterous robotic manipulator can be rather compli-
cated as well as cognitive and physically demanding (Rahal
et al., 2020). Teleoperating a robot with complex and non-
intuitive kinematic poses a high cognitive load on human
operators (Talha et al., 2016). While conservative industries
do not sufficiently trust the autonomous systems, the emerg-
ing autonomous technologies can potentially ease the tele-
operation and help reduce the cognitive load on the human
operator.

Simple robotic manipulation tasks can be classified in
two categories: (i) synthesising grasp poses, and (ii) perform-
ing manipulative movements. There is a bulk of research on
synthesising grasp poses from given single/multiple point
clouds, e.g., by sim-to-real learning (James et al., 2019),
deep learning via domain randomisation (Tobin et al., 2018),
deep learning that learns hand-eye coordination for grasp-
ing unknown objects (Levine et al., 2018) and probabilistic
generative models of grasping configurations (Kopicki et al.,
2016). Other works, e.g. (Varley et al., 2017), reported im-
proved performance in synthesising grasping configuration
for unknown objects by shape completion, e.g., via primitive
shape fitting to point cloud (Baronti et al., 2019). However,
state-of-the-art autonomous grasping methods currently do
not provide *proof of stability/success* of the grasping actions
which is demanded by conservative industries, such as nu-
clear waste decommissioning. For instance, Kopicki et al.
(2019) presents an approach to improve some of these short-
comings as reported in (Kopicki et al., 2016): this approach
can partially generalise across different working conditions,
unknown objects and unknown environments with success
rate of 87.8%. Although the improved performance reported
in (Kopicki et al., 2019) (i.e. < 25% improvement in the
success w.r.t. the result reported in (Kopicki et al., 2016)) is
significant, the obtained success rate is still far from 100%
which is demanded by conservative industries (Talha et al.,
2016).

The works above are only concerned with (i) reach-to-grasp, (ii) forming grasping configuration on an object surface and (iii) lift the object (Pardi et al., 2018). Nonetheless, the robot may need to deliver complex manipulative movements after making a stable grasp. Other lines of research studied the planning and control of manipulative movements (Ratliff et al., 2009) to deliver complex (e.g. post-grasp) actions. For instance, a robot may need to grasp an air paint spray gun and follow a specific trajectory to deliver the quality painting. There is a range of different manipulative movements, e.g., they can be a simple pick-and-place task (Ghalamzan E. et al., 2016), suturing with a non-invasive surgical robot (Selvaggio et al., 2019a), robotic painting or robotic cutting (Pardi et al., 2020). The optimisation based manipulation planning algorithm (Ratliff et al., 2009; Zucker et al., 2013; Schulman et al., 2014) can encode the environmental constraints into a cost function where it can be used to find the optimum trajectory in a given environment. While the robot can optimally perform a task using trajectories computed by optimisation based approaches, they are usually computationally very expensive (Zucker et al., 2013) and may not be suitable if the robot needs to quickly adapt to the changes in its environment. Robot learning from demonstrations (LfD) (Abbeel et al., 2010) was proposed to overcome this shortcoming and quickly generalise demonstrated trajectories. For instance, Peternel et al. (2018) proposed a human-in-the-loop approach for teaching robots how to solve assembly tasks in unpredictable and unstructured environments. The haptic feedback is used both as a means for improving teacher demonstrations and as a human-robot interaction tool in (Rozo et al., 2013). Kormushev et al. (2011) presented method to learn positional and force profiles and reproduce robot force interactions in a human-robot interaction setting.

The two lines of research, namely grasp synthesising
and manipulative movements planning, have been studied
mostly in isolation. However, synthesised grasping configu-
rations may impose some constraints on feasible post-grasp

manipulative movements. For example, a certain grasping
configuration could lead to reaching singularities or joint
limits in the post-grasp task. Moreover, the desired manipu-
lative movements may limit the choice of suitable grasping
configuration, e.g., the screwing task determines that a robot
must form a stable grasp on the handle of a screwdriver. In
this regard, Detry et al. (2017) select the grasping pose with
the maximum corresponding affordance utility value.

There are a few numbers of studies on jointly considering
the problem of grasping an object, and delivering desired
post-grasp manipulative movements. For instance, (Pardi
et al., 2018) and (Ghalamzan E. et al., 2016) studied how
the choice of a grasp pose can be used to avoid singularities
and collision during manipulative movements. Two-phase
optimisations were used in (Horowitz and Burdick, 2012) to
generate the contact necessary for making a stable grasp on
an object and to find the optimal object path that can be fol-
lowed, given the optimal grasping configuration. In contrast,
(Vahrenkamp et al., 2011, 2012) studied the optimal grasps
resulting in a maximum manipulability at initial grasping con-
figuration. Similarly, Mavrakis et al. (2016, 2017) studied the
selection of the gasping configuration yielding the best safety
value and torque efforts during the post-grasp manipulative
movements. These works, however, assume a reliable planner
can generate several precise/stable grasp poses where there
does not yet exist a reliable autonomous system fully trusted
by conservative industries for different scenarios, e.g., for an
arbitrary object in a variety of lighting conditions, making
this an open research challenge (Zhou et al., 2017).

The conservative industries only trust fully teleoperated
robotic systems despite the significant recent advancements
in autonomous grasping systems (James et al., 2019; Levine
et al., 2018; Kopicki et al., 2019). In fact, a very basic tele-
operation system is still the only means of performing many
remote manipulation tasks. For example, the majority of
robots deployed in the nuclear industry are still teleoper-
ated by a human (Bogue, 2011). Teleoperation may become
very complex and impose a high cognitive load on a human
operator (Talha et al., 2016). The human operator may not
be fully aware of consequences of commanded manipulative
movements (Selvaggio et al., 2019a; Pardi et al., 2018; Ghala-
mzan E. et al., 2016; Detry et al., 2017; Mavrakis et al., 2016,
2017; Parsa et al., 2022), hence, they may use trial and error
procedures, e.g. procedures of reach-to-grasp movements,
grasping, moving, dropping the object (because a constraint
has occurred), changing the robot configuration to avoid the
local constraint, and so on. Controlling a complex robotic
system using human inputs alone is often difficult and may re-
quire special skills or dedicated training (Franchi et al., 2012).
Hence, assisted teleoperation has been proposed to facilitate
the teleoperation by, for example, allowing an autonomous
algorithm to perform parts of the task or by providing the
operator with visual and haptic guidance (Abi-Farraj et al.,
2016). This can potentially reduce the workload of the human
operators and improve their performance (Boessenkool et al., 2013).

Several assisted teleoperation frameworks have been proposed to tackle the problem of grasping or manipulation. For example, Achibet et al. (2014) introduced a paradigm for visual-haptic manipulation of objects and (Cipriani et al., 2008) discussed the impact of different force-feedback-based control strategies on the operator's performance during grasping. Similar studies for different manipulation tasks are presented in (Boessenkool et al., 2013; Nelson et al., 1996). In these works, the haptic feedback provided to the operator aims at transmitting the forces (which are sensed through tactile and/or force sensors on the remote-arm) to the operator. In contrast, Masone et al. (2014) proposed an approach in which the operator is informed about the feasibility of modifying an intended trajectory. However, these assisted teleoperation approaches are used only for solving either reach-to-grasp (RtG) or post-grasp manipulative movements (PGMM). Neither of those facilitates planning jointly for both RtG and PGMM. Hence, a selected grasp by a human may not be optimal for post-grasp motions. In other words, the human operators are not informed about the consequences of their actions in a receding time horizon, i.e. about the quality of their preferred grasp pose in terms of control effort or singularity over the post-grasp motions.

Haptic-guided shared control has recently caught the attention of the researchers to improve the telemanipulation experience which helps joint solution of RtG and PGMM. Applications include surgical suturing (Selvaggio et al., 2019a), a simple pick-and-place (Ghalamzan et al., 2017), mobile robotics (Bruemmer et al., 2005; Cognetti et al., 2020), and industrial robotics (Selvaggio et al., 2018, 2021). Implementations of shared control can use variable admittance control (Duchaine and Gosselin, 2007) or active constraints (Rahal et al., 2019; Pacchierotti et al., 2018), often enforced through haptic force feedback guiding the operator along the preferred directions of motion. For instance, in (Abi-Farraj et al., 2018), the haptic feedback cues are considered to avoid reaching the constraints of keeping the balance of the humanoid robot. Selvaggio et al. (2019b) present a passive task-prioritised shared-control method for remote telemanipulation of redundant robots. The proposed method fuses the task-prioritised control architecture with haptic guidance techniques to realise a shared-control framework that keeps the user away from the dangerous region of the workspace. Adjigble et al. (2019) presented a shared control system in which if the operator follows the desired trajectory generated by a grasp planner, zero force would be fed back to the leader robot, and if he/she deviates from the desired path, a returning force would be proportional to the geometric distance. Abi-Farraj et al. (2019) used haptic information to guide the user smoothly and continuously as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. Selvaggio

et al. (2019a) compute the optimal grasping configuration in a manifold of grasping configurations around a needle where the robot does not face any singularities or joint limits. Singh et al. (2020) used haptics (1) to feedback the force sensed at the remote-arm and (2) to inform the human operator of collision in robot workspace. In (Rahal et al., 2020), haptic force cues are generated to guide the users towards successful completion of the task, along with directions that improve their posture and increase their comfort. Moreover, the effect of inaccuracy in haptic shared control on human performance is studied in (van Oosterhout et al., 2015). Peternel et al. (2020) propose a method for improving the human operator's arm posture during bilateral teleoperation. Parsa et al. (2020) presents a shared control informing the operator of future expected collision by haptic force cues. However, the works above do not consider the joint effort of the manipulator during manipulative movements. Prior work (Ghalamzan et al., 2017) studied how the user can benefit from the predictive singularity cost computation. This is based on a Task-Oriented Velocity manipulability (TOV) cost function. The user can then select a grasping configuration so that the robot avoids singularities during manipulative movements. However, this optimisation was performed without accounting for whether the resulting optimised end-effector pose was compatible with the main task of grasping the object as the force cues based on TOV cost can disagree with the movements toward the object to grasp. Moreover, neither of the works (i) use a combination visual-haptic guidance, nor (ii) use a discretised grasping manifold to provide a space of possible grasp poses.

The present work proposes a shared control system for addressing the above shortcomings and informing the operator about the grasp choice with the minimum torque effort – using Task Oriented Torque Effort (TOTE) – that would be exerted by the robot during the manipulative movements. A grasp pose yielding the minimum joint effort is a very important factor as minimum joint effort correlates with the minimum robots running cost. It may also yield a safe distance to min/max joint effort limits imposed by the electrical motors at each joint. This can also result in an extended lifetime of the joint motors. We illustrate the effectiveness of our proposed visual-haptic shared control for grasping an object by a series of experiments with the real robotic system shown in Fig. 1 (consisting of a Haption device and a Panda robotic arm) and a comprehensive human subject study.

## 3 SHARED CONTROL METHOD

Whenever an operator grasps an object using a teleoperated robotic manipulator, the grasping actions can be divided into three phases (see Fig. 2): (a) *reach-to-grasp*, when the remote robot moves towards the object to be grasped; (b) *grasp*, when the robot securely grasps the object with its end-effector; and (c) *post-grasp movements*, when the manipulator carries out
<figure><img src="image_3.png" /><figcaption>Fig. 2: Three phases of grasping an object: (a) reach-to-grasp–moving the robotic arm towards the object to be grasped; local coordinate frames are attached to the end-effector (red), centre of mass (CoM) of the object (blue) and the selected grasp pose (green); (b) grasping the target object, and (c) performing the required post-grasp manipulation; the past local frames attached to the CoM are shown with blue dashed lines. In this paper, the first two phases are carried out via teleoperation, while the last one is performed autonomously. During the teleoperation, we provide visual-haptic feedback about the grasp poses minimising the robot torque during the future autonomous manipulation phase</figcaption></figure>

the desired manipulation (e.g., moving the object towards a desired location).

In this work, we decide to leave the human operators free to approach and grasp the target object according to their preferences and to then switch to the robot autonomy as soon as the object is grasped. In this way, the operator participates to the part of the task which is most demanding in terms of cognition while leaving the rest to the robot autonomy. While the operator approaches the object, we provide feedback information about the torque necessary for the robot during delivering the object in the post-grasp autonomous movement phase. In fact, as shown in Fig. 3, for the same autonomous post-grasp manipulation defined for the object, two different grasp poses lead to different robot configurations, resulting, in turn, to different torques required by the motors.

We summarise the architecture of the control system in Fig. 4 and describe it in details in the following Sections. A grasping algorithm generates a dense distribution of feasible grasp poses using the 3-dimensional model of the object. Any data-driven grasping approaches, e.g. (Kopicki et al., 2019), can be employed for this purpose, and we used GraspIt! (Miller and Allen, 2004) in this work. The torque-related metric is then calculated for each grasp pose

<caption>Fig. 3: Two choices of grasp pose shown in (a) and (c). The choice of grasp pose and the corresponding robotic arm configuration determines the following joint space trajectory of the robotic arm for the very same object trajectory. (b) and (d) show two shots of the robotic arm (corresponding to the grasping choices shown in (a) and (c), respectively) for moving the object to the predefined pose. The joint configurations during motion are different in the two cases, as well as the torques exerted by the robot for moving the object (due to different gravity, Coriolis and acceleration terms)</caption>

considering the object inertia matrix and the planned post-grasp trajectory. Finally, this information is used at runtime to guide the user towards the grasp poses locally minimising the torque metric. Combining this information with their own experience and perception of the environment, the human operators can make the choice on how to grasp the object informed about the future expected cost.

## 3.1 System details

In our teleoperation setup, we consider a 7-DoF manipulator–remote arm–and a 6-DoF haptic device–master arm–(see Fig. 5). We fix joint-3 of the Panda robot to reduce it to 6-DoF, as this allows us to avoid complexities of the inverse kinematics of redundant robots in Sec. 3.2. As such, the robot acts and is referred to as a 6-DoF serial manipulator.

Let us define three reference frames (see Fig. 5): $\mathcal{F}_g \in SE(3)$, attached to the robot end-effector, $\mathcal{F}_o \in SE(3)$, attached to the object centre of mass, and the robot base frame $\mathcal{F}_r \in SE(3)$. These frames are also shown in Fig. 2, along with $\mathcal{F}_{gr}$ representing the robot end-effector frame at the chosen grasping configuration. $\mathcal{F}_m \in SE(3)$ is the base frame of the master device, w.l.o.g. taken parallel to the robot base frame $\mathcal{F}_r \in SE(3)$, and $\mathcal{F}_w \in SE(3)$ is the world frame.
<figure><img src="image_4.png" /><figcaption>Fig. 4: Block diagram summarising the shared control framework. A set of feasible grasps {$^o x_i$} is generated by a grasping algorithm. Then, we calculate the torque cost function $H(^o x_i)$ at each of these poses to design the visuo-haptic feedback $\tau_{hf}$ to guide the human user. Finally, the operator controls the remote robot in velocity using the grounded haptic interface</figcaption></figure>

The configuration of the haptic interface and its Cartesian velocity are defined in $\mathcal{F}_w$ by $x_m \in \mathbb{R}^6$ and $v_m \in \mathbb{R}^6$, respectively. The device is modelled as a gravity-pre-compensated generic mechanical system,

$$M(x_m)\dot{v}_m + C(x_m, v_m)v_m = \tau_{hf} + \tau_{hu} + Bv_m, \quad (1)$$

where $M(x_m) \in \mathbb{R}^{6 \times 6}$ is the positive-definite symmetric inertia matrix, $C(x_m, v_m) \in \mathbb{R}^{6 \times 6}$ are the Coriolis/centrifugal terms, $\tau_{hf}$, $\tau_{hu} \in \mathbb{R}^6$ are the haptic feedback and the human operator's forces, respectively; $B \in \mathbb{R}^{6 \times 6}$ is a damping matrix for stabilizing the system. The velocity of the robot is defined as $v_g \in \mathbb{R}^6$, and a velocity-to-velocity coupling between the master and the remote system is implemented by setting $v_g = v_m$.

We let $^o x_g = \{^o t_g, ^o R_g\} \in SE(3)$ represent the grasp pose, which is defined as the relative pose between the gripper and the object to grasp. As we have already mentioned, we consider the post-grasp trajectory to be assigned and carried out autonomously after the user grasps the object. For instance, considering a pick-and-place task, we generate this trajectory based on the initial position of the object and a given target location. The trajectory is defined in the world frame as $^w x_o(t) = \{^w t_o(t), ^w R_o(t)\} \in SE(3)$, $0 \le t \le 1$, with $t$ being a time parametrisation such that $t=0$ is the starting point and $t=1$ is the endpoint of the trajectory. From the planned post-grasp trajectory, we calculate the corresponding trajectory for the robot end-effector w.r.t. $\mathcal{F}_w$,

$$\begin{aligned} ^w R_g(t) &= ^w R_o(t) ^o R_g \\ ^w t_g(t) &= ^w t_o(t) + ^w R_o(t) ^o t_g \end{aligned} \quad (2)$$

Using the inverse kinematics (IK) solver we then obtain the joint space trajectory $\theta(t, ^o x_g) \in \mathbb{R}^6$, which is function of the object trajectory over time $^w x_o(t)$ and of the grasp pose $^o x_g$. We want to inform the human operator about the quality of a candidate grasp pose $^o x_g$ w.r.t. the torque that the robot would exert when moving the object after the grasp.

<figure><img src="image_5.png" /><figcaption>Fig. 5: We consider several coordinate frames including (i) the world reference frame, and coordinate frames attached to (ii) the handle of master-arm, (iii) the end-effector of the remote-arm, base of the remote-arm, and (iv) the object</figcaption></figure>

We call this metric Task-Oriented Torque Effort (TOTE), and it is function of the previously calculated desired joint trajectory. Our shared control system provides the operators with feedback information guiding them towards poses that minimise the TOTE index and are also feasible grasp poses.

## 3.2 Manipulator dynamics under load

To evaluate Task Oriented Torque Effort (TOTE) for a given grasp pose, we need to calculate an augmented equation of motion which accounts for the grasped object as well as for the dynamics of the manipulator. Assuming a secure and steady grasp, the inertia tensor can then be expressed in a frame attached to the robot end-effector, as follows:

$$M_o(^o x_g) = E^{-T} (^o x_g) M_{obj} E^{-1} (^o x_g), \quad (3)$$

where $E(^o x_g) \in \mathbb{R}^{6 \times 6}$ is the matrix transforming the linear and angular velocities of the object CoM to generalized velocities in the frame $\mathcal{F}_g$ attached to the end-effector (Murray et al., 1994; Mavrakis et al., 2017). We can then compute the grasped object inertia matrix and, hence, the governing equation of motion of the augmented robot and grasped object in the joint space is:

$$M_o(^o x_g, \theta) = [J^T(\theta) M_o(^o x_g) J(\theta)], \quad (4)$$

where $J(\theta) \in \mathbb{R}^{6 \times 6}$ is the the robot Jacobian. We compute the effect of the object on the Coriolis and gravitational term of the robot dynamic equation using join-space robot governing equation of motion (JSR-GEM) and from (3) and (4), the following holds:

$$C_o(^o x_g, \theta, \dot{\theta}) = \frac{1}{2} \sum_{k=1}^{n} \left( \frac{\partial M_{o,ij}}{\partial \theta_k} + \frac{\partial M_{o,ik}}{\partial \theta_j} - \frac{\partial M_{o,kj}}{\partial \theta_i} \right) \dot{\theta}. \quad (5)$$

The gravitational term of the dynamics of the grasped object in the robot joint space can be defined using JSR-GEM:

$$N_o(^o x_g, \theta) = \frac{\partial V_o(^o x_g, \theta)}{\partial \theta}, \quad (6)$$
where $V_o(^{o}x_g, \theta) = mgh_o(^{o}x_g, \theta)$ and $h_o(^{o}x_g, \theta)$ can be computed using the forward kinematics of the robot. The resulting equation of motion accounting for the dynamics of the system including the external object dynamics can then be written as

$$ \bar{M}(^{o}x_g, \theta)\ddot{\theta} + \bar{C}(^{o}x_g, \dot{\theta}, \theta) + \bar{N}(^{o}x_g, \theta) = \bar{\tau}(^{o}x_g, \theta, \dot{\theta}, \ddot{\theta}), \quad (7) $$

where variables with a bar notation refer to the system augmented with the external object. $\bar{\tau}(^{o}x_g, \theta, \dot{\theta}, \ddot{\theta}) \in \mathbb{R}^6$ represents the joint torques needed to perform the target manipulative action while

$$ \begin{aligned} \bar{M}(^{o}x_g, \theta) &= M_o(^{o}x_g, \theta) + M(\theta), \\ \bar{C}(^{o}x_g, \theta, \dot{\theta}) &= C_o(^{o}x_g, \theta, \dot{\theta}) + C(\theta, \dot{\theta}), \\ \bar{N}(^{o}x_g, \theta, \dot{\theta}) &= N_o(^{o}x_g, \theta, \dot{\theta}) + N(\theta, \dot{\theta}). \end{aligned} $$

### 3.3 Task Oriented Torque Effort (TOTE) cost function

The grasp pose $^{o}x_g$ specifies the joint space trajectory of the robot needed to perform the post-grasp manipulation (see eq. (2)) and affects the dynamics of the post-grasp manipulative motions, which includes the robot and the grasped object. As a consequence, the grasp pose ultimately affects the robot joint torques, $\bar{\tau}(^{o}x_g, \theta, \dot{\theta}, \ddot{\theta})$ (hereafter referred to by $\bar{\tau}(^{o}x_g)$ for simplicity) needed to move the robot (and the object) over the desired trajectory. Torques $\bar{\tau}(t, ^{o}x_g)$ expected at each time t of the trajectory are defined following (7) as

$$ \bar{\tau}(t, ^{o}x_g) = \bar{M}(^{o}x_g, \theta(t, ^{o}x_g), \ddot{\theta}(t, ^{o}x_g)) + \bar{C}(^{o}x_g, \dot{\theta}(t, ^{o}x_g), \theta(t, ^{o}x_g)) + \bar{N}(^{o}x_g, \theta(t, ^{o}x_g)). \quad (8) $$

Let $H(^{o}x_g)$ be a cost function defining the torque effort the robot exerts to perform the post-grasp manipulation, i.e., moving the object along a pre-defined trajectory in our case. Of course, this cost function can be defined for any other type of post-grasp manipulation, depending on the task at hand. Given an object and a desired trajectory to follow $^{w}x_o(t)$, we can finally define task oriented torque effort (TOTE) cost function $H(^{o}x_g)$ as

$$ H(^{o}x_g) = \int_0^T \bar{\tau}(t, ^{o}x_g)dt, \quad (9) $$

where T is the duration of the post-grasp manipulation task.

## 4 NAVIGATION INFORMATION

During reach-to-grasp, we would like to guide the operator towards grasp poses locally minimising the TOTE index (which corresponds with the minimum torque effort during post-grasp manipulative motions), using a combination of haptic and visual navigation feedback.

<figure><img src="image_6.png" /><figcaption>Fig. 6: Feasible grasps and TOTE cost $H(^{o}x_g)$ for the three objects used in our experiments. The grasping candidates are colour-coded according to the corresponding cost value. Lighter green corresponds to a lower cost function</figcaption></figure>

### 4.1 Haptic navigation guidance

The objective of our haptic navigation cues $\tau_{hf}$ is to guide the user towards the direction minimising the TOTE metric $H(^{o}x_g)$. A first approach to achieve this objective would be to provide force cues along the negative gradient of $H(^{o}x_g)$ w.r.t. optimization variable $^{o}x_g$, as done in (Ghalamzan et al., 2017). However, this method might lead to undesired behaviours as it does not consider where the location of feasible grasp poses on the target object, i.e., the cues could lead towards a pose that does minimize $H(^{o}x_g)$ but from which it is impossible to grasp the object (or the ‘optimal’ pose could even end up lying inside or above the object itself). In this work, we instead combine the gradient descent approach with knowledge regarding the feasible grasp poses. From the 3-dimensional model of the object, we use GraspIt! to retrieve a dense set of grasp poses. Then, for each pose, we calculate the corresponding end-effector and joint post-grasp trajectories, excluding those for which no feasible inverse kinematic solution exists. This step leaves us with a set of n feasible grasp poses $^{o}x_i$ where $0 \le i \le n$. Finally, we compute the TOTE for each feasible grasp pose $^{o}x_i$, called $H_i(^{o}x_i)$. This information is used to inform the user about the direction to follow so as to minimise the robot effort during the post-grasp action.

The most straightforward way to provide the user with a meaningful feedback may be to drive her/him towards the grasp pose corresponding with the minimum cost. However, the operator may still consider that the grasping pose corresponding to the minimum value of TOTE is not the best choice because of other constraints important to task completions (or her/his own judgment). In general, robotic grasping/manipulation may involve different application specific objectives, e.g., the manipulation may be subject to a suitable
affordance of the object or collision-free movements (just to name a few). The goal, therefore, is to provide the users with a local, continuous and informative guidance such that they get informed on how to move in the vicinity of their current gripper pose so as to minimise the torques needed for the post-grasp action. To achieve this, we do not rely on the cost function value alone in the design of the haptic cues, but also take into consideration the distance of each proposed grasp pose to the current gripper pose.

To this end, we define the roto-translational distance between any grasp candidate $^o x_i$ and the current gripper pose $^o x_g$ as

$$|^o x_i - ^o x_g| = ||^o p_i - ^o p_g|| + \mu |^g \theta_i|, \quad (10)$$

where $^g \theta_i \in [-180, +180]$ is the angular part of the angle-axis representation of $^g R_i = ^g R_o ^o R_i$, and $\mu > 0$ is used to properly scale the angular component of the distance with respect to the linear one ($\mu = \pi/180$ in our experiments, which indicates an equal weight for angular and translation distances).

Finally, the force cues are calculated as the weighted average of multiple force vectors guiding the user towards nearby grasp poses having a low cost. The lower $H_i(^o x_i)$, the stronger the force guidance; moreover, each force vector is weighted inversely to the distance between its grasp pose $^o x_i$ and the current gripper pose $^o x_g$, i.e.,

$$\tau_{hf} = \frac{1}{n} \sum_{i=1}^{n} \frac{H(^o x_g) - H_i(^o x_i)}{1 + k|^o x_i - ^o x_g|^m} \left[ \frac{^o p_i - ^o p_g}{||^o p_i - ^o p_g||} \right]^{g \Delta_i} \quad (11)$$
if $H(^o x_g) > H_i(^o x_i)$

where $^g \Delta_i$ is the axis part of the angle-axis representation of $^g R_i = ^g R_o ^o R_i$, and $k$ and $m$ are positive control gains ($k = 6$ and $m = 8$ in our experiments).

## 4.2 Visual navigation guidance

In addition to the haptic feedback described above, we provide the user with a visual representation of the different possible grasping poses to give them a preliminary idea of how “good” the different grasps are, and allow them to plan ahead their grasping pose in a way to make the post-grasping manipulation more efficient for the robot. This is achieved by color coding the grasping configurations, as shown in Fig. 6, where grasps are represented with different shades of green depending on their TOTE cost values. In Sec. 6, we study the impact of haptic, visual and viso-haptic feedback on the human performance in an comprehensive human subject tests, and the effect of combining them together on the user performance.

<figure><img src="image_7.png" /><figcaption>Fig. 7: Objects used in our grasping experiments. From left to right, an L-shaped wooden object (LS), a lamp (LA), and a b-shaped wooden object (WO). These objects have been chosen following a discussion within the European H2020 project “Robotic Manipulation for Nuclear Sort and Segregation” (RoMaNS), which considered them as good representatives for the sort and segregation of nuclear waste</figcaption></figure>

# 5 PICK-AND-PLACE EXPERIMENT

To test the effectiveness of our proposed shared control approach, we carried out a first pick-and-placement experiment on three different objects.

## 5.1 Experimental setup

The experimental setup is shown in Fig. 1. The master side is composed of a 6-DoF grounded haptic interface (Virtuose 6D, Haption, France); while the remote side is composed of a 7-DoF robotic manipulator (Panda manufactured by Franka Emika) with one joint fixed, and therefore used as a 6-DoF manipulator. An LCD screen is placed in front of the human operator (i.e. subjects). The master interface is placed next to the remote robotic manipulator to provide the subjects with a direct view on the remote workspace. This simplifies our study as we avoid any complexities caused by the choice of camera views and screens showing the remote workspace.

The environment is composed of three different objects placed on a table in front of the robot:

* WO: a 275-g wooden object made of two rectangles having 10×2.6×4 cm and 11×2.6×9 cm dimensions (right of Fig. 7).

* LA: a 11×11×15 cm rectangular lamp of 1958 g with an handle (center of Fig. 7);

* LS: an 228-g L-shaped object made of two 14×1.2×6.5 cm rectangles (left in Fig. 7);

These objects have been chosen for their similarity with common objects used in nuclear decommissioning scenarios (Abi-Farraj et al., 2019), following a discussion within the European H2020 project “Robotic Manipulation for Nuclear Sort and Segregation” (RoMaNS). The use of a broader set of objects will be the subject of future work (see Sec. 8).
<figure><img src="image_8.png" /><figcaption>Fig. 8: Grasp poses teleoperated in T (fully teleoperated) and H (haptic-guided shared control teleoperated) mode of control system: 8a, 8c and 8e grasp poses for WO, LS, and LA chosen by the user in T control mode; 8b, 8d and 8f grasp poses for WO, LS, and LA chosen by the user in H control mode</figcaption></figure>

## 5.2 Task and methods

One expert operator carried out the pick-and-place task. The operator was asked to control the robotic manipulator with the grounded haptic interface to grasp the considered objects. We visualise the post-grasp trajectory before the start of the experiment and explain to the subjects that the aim of the experiments is to choose a grasping pose minimising the overall torque exerted by the robot during autonomous post-grasp motions. Once the object was grasped, the autonomous system moves the remote robot to pick the object up and deliver it at the planned pose, following a pre-planned autonomous trajectory. The pose of the object relative to a frame attached to robot hand is fixed throughout the post-grasp trajectory.

We employed the shared control method described in Sec. 3. We used velocity-to-velocity mapping for master-slave commanding. Since the workspace of the master device is smaller than that of the remote robot, we used a clutch button on the master interface to decouple the motion of the master and remote systems, allowing the user to reposition the haptic device in a comfortable position before controlling again the robot (Abi-Farraj et al., 2019). We considered two navigation feedback modalities: (T) standard human-in-the-

loop teleoperation, where the operator receives no guidance about suitable grasp poses; (H) our haptic shared control teleoperation approach, where the subject receives haptic guidance toward grasps minimising $H(^{o}x_g)$, as described in Sec. 4.1. The post-grasp trajectory was designed to include both a translational and rotational component and is different for each of the 3 objects, as shown in the video available at https://youtu.be/B3YCOZPpynQ.

The human operators started by grasping “WO”. As described in Sec. 3, the system generated the set of feasible grasp poses, calculating for each of them the TOTE cost $H(^{o}x_g)$ over the post-grasp trajectory. Then, haptic feedback guides the user towards the pose locally minimising the cost, in case of H. Fig. 6a shows $H(^{o}x_g)$ for the feasible grasps on the object used in our experiments. The feasible grasping candidates are colour coded where the darker the green colour of the grasping candidate, the higher the corresponding cost value. The user, then, grasps object LA, receiving the same type of guiding feedback. This object represented a particular challenge for the system, as it is quite heavy and, if not handled correctly, might require torques too high for the robot to exert during the post-grasp trajectory. Again, Fig. 6b illustrates $H(^{o}x_g)$ for the feasible grasps on this object using different shades of green. Finally, the user grasped object LS and Fig. 6c shows $H(^{o}x_g)$ for the feasible grasps on this object.

It is important to highlight that the navigation feedback guides the operator towards the pose locally minimising $H(^{o}x_g)$. This behaviour still enables the human operator to decide from where to approach the object, combining the knowledge of the system about $H(^{o}x_g)$ with the experience and additional environmental information brought by the operator. For example, from Fig. 6c, we can see that $H(^{o}x_g)$ is equally low if grasping the object from its right or left hand side. The user might prefer one or the other considering information unavailable to the system, e.g., one part looks damaged/corroded or it is too close to other objects.

## 5.3 Robustness against object parameters

Our method needs a rough 3-dimensional model of the object and of the corresponding approximated inertial properties. The rough 3D model of the object can be obtained using depth cameras mounted on the robot end-effector for example and the object mass and inertia parameters can be estimated using robot pushing actions, as per our previous work (Abi-Farraj et al., 2019) and (Mavrakis et al., 2020). These obtained 3D model and inertia parameters may be rough approximations of the real values: hence, we propose in this section a discussion on the effects of these uncertainties on our shared control method.

Starting with the effect of inertial properties, we observe that at low velocities, the torques on the robot joints
<table><caption>Table 1: Average error between the measured joint torques $\tau_g$ and the calculated ones using eq. (8) (with and without the effect of velocity and acceleration) for a sample task for our experiments</caption><thead><tr><th>Calculated (eq. (8))</th><th>Calculated (no acceleration)</th><th>Calculated (no acceleration or velocity)</th></tr></thead><tbody><tr><td>0.479 N.m</td><td>0.475 N.m</td><td>0.480 N.m</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Fig. 9: (a) Plot of $H(^{o}x_g)$, for grasps with high and low TOTE, based on different mass approximations. The x-axis represents the multiplier of the mass parameters used in the calculations, i.e., half the actual mass, the actual mass, and twice the actual mass. (b)-(d) Evolution of the norm of the joint torques $||\tau_g||$ for the actual mass (b), half the mass (c) and double the mass (d). We also show the curves for two grasping poses, with a high and low TOTE value</figcaption></figure>

$\bar{\tau}(t, ^{o}x_g)$ could be approximated solely using the gravity vector, $\bar{N}(^{o}x_g, \theta(t, ^{o}x_g))$, as the other two components are negligible. To verify this observation, we move an object of 1 Kg, grasped by the gripper of the Panda robot, along a sample pre-defined trajectory with low velocity similar to the ones we used in our experiments. We compare the measured torques over the post-grasp trajectory to the calculated torques using eq. (8), with and without the effect of the velocity and acceleration components. The results are summarised in Table 1, and show that the effect of adding these terms is, indeed, negligible. We thus used this approximation to simplify the calculation of the TOTE in eq. (9) that is used to compute the haptic feedback. Higher velocity tasks should of course include the dynamics components in the calculations.

We also tested the effect of accuracy in the mass estimation on the cost function calculation offline, using the object LA used in our experiments. Fig. 9a shows the value of the cost function $H(^{o}x_g)$ for different estimations of the object mass: its actual mass *m*, 0.5*m* and 2*m*. For each case, we show the value of the cost function for the grasping poses

<figure><img src="image_10.png" /><figcaption>Fig. 10: Time [s] vs. norm of the joint torques $||\tau_g||$ [Nm] during the post-grasp trajectory for T and H control mode and three objects: 10a WO (corresponding with 8a and 8b), 10b LS (corresponding with 8c and 8d), 10c and LA (corresponding with 8e and 8f)</figcaption></figure>

with the highest and the lowest TOTE value. Even though the value is different for each mass estimation, we notice that the three cases lead to the same ranking of the grasping poses in terms of cost function value. We can also see in Figs. 9b, 9c and 9d that the evolution of the torque norm over time steps in the post-grasp trajectory has the same trend. As such, an approximation of the mass of the object would lead to the same generated shared control forces in terms of direction, even if the strength of the force is different. Since in our setting the mass of the object is known, we used the exact value of the mass in the experiments.

## 5.4 Results

We registered the grasp pose chosen by the operator and the torques exerted by the robot over the post-grasp trajectory in the two different control modalities (T vs. H-full teleoperation and the haptic shared control) for the three considered objects (WO vs. LA vs. LS).

As expected, the grasp pose chosen by the user differed between T and H modalities. In condition T, the operator chose the most intuitive grasp poses, shown in Figs. 8a, 8c, 8e, which are usually along top edge/surface line of the object. However, the robot may require a large amount of torque effort to deliver desired manipulative movements during the corresponding post-grasp manipulative movements and those grasps are not the ones with the corresponding minimum
$H(^{o}x_g)$). In condition H, the shared control algorithm guides the user to grasp poses shown in Figs. 8b, 8d, 8f, which are less intuitive but more efficient in terms of $H(^{o}x_g)$.

After forming successful grasps, the operators stop teleoperating the remote robot and an autonomous controller picks up the object and moves it along a planned trajectory. Fig. 10 shows the evolution of the L2 norm of the joint torques $||\tau_g||$ over time during the post-grasp trajectory, for the three objects in case of H and T control mode. As expected, torques are minimised when the user is guided towards grasps with low $H(^{o}x_g)$.

It is interesting to see that our haptic-guided shared control allow the operator to freely choose different local optimal solution– it is shown in Fig. 10b that the operator can chose left and right hand side grasp pose, shown with H left and H right, which have $H(^{o}x_g)$ value lower than the intuitively selected grasp pose, shown with T.

# 6 HUMAN-SUBJECTS STUDY

The experiments above showcase the features of our proposed haptic-guided shared control. To further illustrate the efficiency of our approach, we present a series of human subject tests showing the effectiveness of our approach to minimise the TOTE cost in comparison with the basic teleoperation setup.

We consider again the robotic system described in Secs. 3, 5.1 and shown in Fig. 1. In this section, we only consider object WO (right hand side of Fig. 7) because it does not present any handling risk for novice users–the lamp mass, e.g., may be dropped by novice and damaging the experimental setup– and also provide a larger area of grasp choices for the user. This provides a good example to compare the use of basic teleoperation and our proposed haptic-guided shared control teleoperation setups.

## 6.1 Task and feedback conditions

Similar to the experiments in Sec. 5, participants were asked to control the motion of the robotic manipulator to grasp the object, following the guidance information provided by the system. We describe the experiments and the task for the user, i.e. the users need to teleoperate reach-to-grasp and form a stable grasp where the post-grasp movements are performed autonomously. They were also informed that we would ideally like to have the minimum joint torque efforts during the entire experiments including post-grasp movements. The task started when the manipulator moved for the very first time, and it was considered successful completion when the robot completed the planned post-grasp trajectory. We consider four different ways of enabling the user to control the system for completing the grasping task,

T: basic teleoperation, where the subject receives no haptic guidance about suitable grasp poses.

H: our proposed haptic shared control approach, where the subject receives haptic guidance toward grasp poses locally minimising $H(^{o}x_g)$, i.e., the torque the robot needs to exert along the post-grasp trajectory.

V: visual guidance, where the subject is shown a graphical representation of the scene indicating the value of $H(^{o}x_g)$ for each grasp pose using a colour scale (see Fig. 6a).

HV: a combination of H and V, where the subject receives haptic guidance toward grasp poses minimising $H(^{o}x_g)$ and is shown a graphical representation of the scene indicating the value of $H(^{o}x_g)$ for each grasp pose using a colour scale.

Conditions T and H are the same already employed in Sec. 5. Users always started by conditions T, so to not be influenced by the guidance. The other three conditions H, V and HV were performed in a random order. In all conditions, the user controls all the DoF of the robotic manipulator through the haptic interface, as described in Secs. 3 and 5.1.

A video showing trials in all experimental conditions is available in the video.

## 6.2 Participants

Fifteen right-handed subjects (average age 27.6, 12 males, 3 females) participated in the study. Seven of them had previous experience with haptic interfaces, three had previous experience with robotic teleoperation in general, none had ever tried the proposed shared control technique before. The experimenter explained the procedures and spent about two minutes adjusting the setup to be comfortable before the subject began the experiment. Each subject then spent about three minutes practising the control of the system before starting the experiment.

## 6.3 Results

To evaluate the effectiveness of our telemanipulation system and the usefulness of the proposed shared control approach, we recorded (i) the completion time, (ii) the mean torque exerted by the manipulator joints during the post-grasp phase, (iii) the peak torque exerted by the manipulator joints during the post-grasp phase, (iv) the linear trajectory followed by the robotic end-effector, and (v) the angular motion of the robotic end-effector. Moreover, immediately after the experiment, subjects were also asked to report (vi) the effectiveness of each feedback condition in completing the given task using bipolar Likert-type eleven-point scales.

To compare the different metrics, we ran one-way repeated-measures ANOVA tests (first five metrics) and Friedman tests (last metric) on the data. Four control modalities (standard
<figure><img src="image_11.png" /><figcaption>Fig. 11: Human subjects experiments: Mean and 95% confidence interval of (a) completion time, (b) mean torque and (c) peak torque over the post-grasp trajectory, (d) linear trajectory and (e) angular motion of the end effector for the four conditions (T, H, V, HV), and (f) subjective perceived effectiveness for conditions H, V, HV</figcaption></figure>

teleoperation vs. our haptic shared control vs. visual guidance vs. visuo-haptic shared control–T vs. H vs. V vs. HV) were considered. Data from the first five metrics (i) – (v) passed the Shapiro-Wilk normality test. Table 2 summarises this experiment.

Fig. 11a shows the average task completion time. One user took significantly more time than any other to grasp in the H condition (41.96 s, five times the average of the other users). We did not consider this outlier in calculating the mean and 95% confidence interval showed in the figure. However, we considered it in the following statistical analysis. Mauchly’s Test of Sphericity indicated that the assumption of sphericity had been violated ($\chi^2(5) = 16.255, p = 0.006$). The one-way ANOVA test with a Greenhouse-Geisser revealed a statistically significant change in the task completion time across the conditions ($F(1.868, 26.148) =$

3.572, $p = 0.045, a = 0.05$). Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between conditions T vs. V ($p = 0.004$) and T vs. HV ($p = 0.012$). The Bonferroni correction is used to reduce the chances of obtaining false-positive results when multiple pair-wise tests are performed on a single set of data.

Fig. 11b shows the mean torque applied by the manipulators’ joints along the post-grasp trajectory. Data passed the Mauchly’s Test of Sphericity. The one-way ANOVA test revealed a statistically significant change in the task mean torque across the conditions ($F(3, 42) = 160.778, p < 0.001, a = 0.05$). Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between all conditions (T vs. H, $p < 0.001$; T vs. V, $p < 0.001$; T vs. HV, $p < 0.001$; H vs. V, $p = 0.002$; H vs. HV, $p < 0.001$; V vs. HV, $p < 0.001$).

Fig. 11c shows the peak torque applied by the manipulators’ joints along the post-grasp trajectory. Data passed the Mauchly’s Test of Sphericity. The one-way ANOVA test revealed a statistically significant change in the task peak torque across the conditions ($F(3, 42) = 108.291, p < 0.001, a = 0.05$). Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between all conditions (T vs. H, $p < 0.001$; T vs. V, $p < 0.001$; T vs. HV, $p < 0.001$; H vs. V, $p = 0.002$; H vs. HV, $p = 0.004$; V vs. HV, $p < 0.001$).

Fig. 11d shows the linear trajectory described by the robot end-effector during the task. Mauchly’s Test of Sphericity indicated that the assumption of sphericity had been violated ($\chi^2(5) = 11.728, p = 0.039$). The one-way ANOVA test with a Greenhouse-Geisser correction revealed no statistically significant change in the task linear trajectory across the conditions ($F(2.085, 29.187) = 0.677, p > 0.05, a = 0.05$).

Fig. 11e shows the summed angular motion described by the robot end-effector during the task. Data passed the Mauchly’s Test of Sphericity. The one-way ANOVA test revealed no statistically significant change in the angular trajectory across the conditions ($F(3,42) = 2.203, p > 0.05, a = 0.05$).

At the end of the experiment, we asked the participants to rate the perceived effectiveness of the three conditions in guiding them in comparison to T (i.e., H vs. V vs. HV), asking “How do you rate the effectiveness of condition [XX] in completing the given task?” for each condition. The responses were given using bipolar Likert-type scales that ranged from 0 to 10, where a score of 0 meant “very low” and a score of 10 meant “very high”. The same question was also used in (Maisto et al., 2017; Meli et al., 2018) for evaluating the subjective/perceived effectiveness of haptics in robotic teleoperation. Fig. 11f shows the perceived effectiveness of the four experimental conditions. A Friedman test showed a statistically significant difference between the means of the three feedback conditions ($\chi^2(2) = 13.192, p = 0.001$). The Friedman test is the non-parametric equivalent of the more
<table><caption>Table 2: Summary of the experiment</caption><tr><td><b>Task</b></td><td>Control the haptic teleoperation system to grasp and lift an object.</td></tr><tr><td><b>Participants</b></td><td>15 subjects (12 males, 3 females)</td></tr><tr><td><b>Conditions</b></td><td>T (standard teleoperation), H (haptic shared control), V (visual guidance), HV (haptic shared control and visual guidance)</td></tr></table>

**Statistical analysis (one-way rm ANOVA or Friedman test)**

<table><tr><td><u>Completion time</u></td><td></td><td></td><td></td></tr><tr><td>T vs. V</td><td>p = 0.004</td><td>T vs. HV</td><td>p = 0.012</td></tr><tr><td><u>Mean torque</u></td><td></td><td></td><td></td></tr><tr><td>T vs. H</td><td>p &lt; 0.001</td><td>T vs. V</td><td>p &lt; 0.001</td></tr><tr><td>T vs. HV</td><td>p &lt; 0.001</td><td>H vs. V</td><td>p = 0.002</td></tr><tr><td>H vs. HV</td><td>p &lt; 0.001</td><td>V vs. HV</td><td>p &lt; 0.001</td></tr><tr><td><u>Peak torque</u></td><td></td><td></td><td></td></tr><tr><td>T vs. H</td><td>p &lt; 0.001</td><td>T vs. V</td><td>p &lt; 0.001</td></tr><tr><td>T vs. HV</td><td>p &lt; 0.001</td><td>H vs. V</td><td>p = 0.002</td></tr><tr><td>H vs. HV</td><td>p = 0.004</td><td>V vs. HV</td><td>p &lt; 0.001</td></tr></table>

**Linear trajectory**

No statistically significant difference between conditions.

**Angular motion**

No statistically significant difference between conditions.

**Perceived effectiveness**

H vs. HV p = 0.004 V vs. HV p = 0.032

**Most effective condition (chosen by subjects)**

Thirteen subjects out of fifteen chose HV, two chose V.

popular repeated-measures ANOVA. The latter is not appropriate here since the dependent variable was measured at the ordinal level. Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between H vs. HV (p = 0.004) and V vs. HV (p = 0.032).

We also asked whether subjects were tired at the end of the task, using again bipolar Likert-type scales that ranged from 0 to 10, where a score of 0 meant “not tired” and a score of 10 meant “very tired”. Results showed that subjects were not particularly tired at the end of the task (mean ± stand. dev., 2.07±1.03).

Finally, thirteen subjects out of fifteen found condition HV to be the most effective at completing the grasping task. Two subjects preferred condition V. Quite surprisingly, no subject indicated H as their preferred condition.

## 7 DISCUSSION

Shared control is becoming a popular technique to improve the performance and intuitiveness of robotic telemanipulation systems. Our work aims at improving the human operator's awareness of some future manipulation related metrics, which might be non-intuitive or difficult to predict at the master side during teleoperation. As a proof of concept, we evaluated the torques exerted by the robot over a future trajectory set to be autonomously executed after the object is grasped via teleoperation. This scenario is relevant, e.g., in the sort and segregation of dangerous waste, where it is impor-

tant to leave operators free to choose which object and how to grasp while providing them as much information as possible on the constraints and objectives relevant to the remote system and environment. Minimising the torque expenditure is not only relevant because it reduces the energy used by the robot–hence reduces the operation cost– but also because it extends the range of objects the system can manipulate. Indeed, as the robotic system can only provide limited torque, knowing where and how to grasp enables the manipulator to handle, e.g., heavier objects. It might be argued that expert operators know well the system dynamics and therefore can already estimate how the robot torques will evolve over a given post-grasp trajectory. However, especially when handling dangerous waste, operators do not have a direct and clear view of the environment and it might be rather difficult to estimate such a complex metric just from looking at the system from a distant window. Even if the operator has a good view of the environment, as in our experimental setup, receiving intuitive haptic cues can significantly reduce the cognitive load and speed up the process, which is already an important result. For example, the Sellafield (UK) nuclear site stores 140 tonnes of civil plutonium and 90,000 tonnes of radioactive graphite (Pearce, 2015). The robotic systems in use provide teleoperation capabilities through primitive master consoles (e.g., passive joystick or teach pendants), making the process too slow for processing the material in a reasonable time, hence the need for faster-teleoperation solutions. Finally, increasing the intuitiveness of these robotic systems is expected to flatten the learning curve, enabling more operators to become proficient in a shorter time (subjects in Sec. 6 were all novices).

Of course, our framework can be adapted to consider different metrics of predicted costs, e.g., relevant for the post-grasp phase, such as cost of maximising the robot workspace in a certain direction at the end of the manipulation or positioning the robot body to optimise the user's viewpoint on future actions. As long as the set of feasible grasp poses can be assigned with relevant cost values, our approach can be adapted with very little efforts.

First, we evaluated our shared control technique in a preliminary pick-and-place experiment on three different objects. We asked experienced users to grasp each object using classic teleoperation (T) in which they received no guidance about suitable grasp poses, and our haptic shared control (H) in which they received haptic feedback towards poses locally minimising $H(^{\circ}x_g)$. Shared control H takes into account all the feasible grasp poses evaluated by the grasping algorithm, weighted by their distance to the current pose of the robot. In this way, the feedback is never abrupt and is gently updated as the robot moves around the object. This characteristic enables the user to always know where the best local grasp is, while still being able to move away if needed, which is paramount for many applications. We want to leave operators free to ultimately choose where to grasp
because they might pick up on some information unknown to
the system. Results of this first experiment show that robot
torques during the autonomous part of the task are lower
when using the haptic shared control approach.

After this preliminary experiment, we carried out a hu-
man subject study enrolling fifteen human subjects. We tested
the performance of four experimental conditions: classic tele-
operation (T), haptic shared control (H), visual guidance
(V), where each grasp pose is color-coded to indicate its
$H(^{o}x_g)$ value, and combined visuo-haptic guidance (HV),
which provides both haptic and visual feedback as in H and
V. As a measure of performance, we evaluated the time-to-
completion for each experiment, robot mean and peak torque,
end-effector trajectory length and angular motion, as well
as users' perceived effectiveness. Results showed that, in all
the considered metrics but three (time-to-completion, linear
and angular motion), the proposed visuo-haptic guidance
HV outperformed more classic teleoperation T. Moreover,
the great majority of subjects preferred HV over the other
approaches. Results also show that in the two most relevant
metrics, mean and peak torque, both haptic approaches (H
and HV) perform significantly better than any other, proving
the effectiveness of haptic guidance in optimising the robot
torque during telemanipulation.

However, as expected, providing users with this type
of guidance led to a longer time-to-completion, which, of
course, needs to be taken into consideration. In this respect,
it is interesting to notice that condition H requires 12% and
31% less time than V and HV, respectively. At the same
time, it requires 8% less torque than V and only 9% more
torque than HV. In some scenarios, such an improvement
in time-to-completion might be worth a small increase in
torque demands. Moreover, of course, longer completion
time might lead to increasing user's fatigue, especially if
using the system for long periods. To minimize user effort,
we will consider including ergonomics-related consideration
in the computation of the costs (Rahal et al., 2020) Finally,
the role of visual guidance merits special attention. While
it clearly makes the task longer to complete (both V and
HV), it is very well appreciated by users, who chose HV and
V as the most preferred conditions. This preference is not
unexpected, as humans are generally rather used to follow
visual navigation feedback (e.g., turn-by-turn car navigation
systems based on road signs), while they are not used to
follow haptic navigation cues at all. For this reason, we expect
more training to significantly improve the performance of the
haptic modality in all the considered metrics.

It is also interesting to notice that human operators had
a good direct view of the environment. We can safely ex-
pect that, in conditions with degraded visual feedback on
the robot and remote environment, haptic feedback will play
an even bigger role with respect to the manipulation perfor-
mance (Pacchierotti, 2015).

We also want to highlight that our approach is indepen-
dent from the grasping algorithm. Although here we used
*GraspIt!* to generate the grasp poses, our framework is ca-
pable to work on top of any other similar approach. For the
same reason, as long as the algorithm can generate feasible
grasp poses, our framework is expected to work on objects
of any shape which are, e.g., sensed by RGB-D sensors.

Finally, as in our target application most waste consists of
single rigid bodies, the proposed framework currently does
not support the handling of deformable objects. This feature
could be added by considering the uncertainty on the relative
pose of the robot hand w.r.t. the object surface.

8 CONCLUSION AND FUTURE WORKS

This paper presented an innovative haptic shared control tech-
nique for robotic telemanipulation that provides the human
operator with information on how to grasp an object for op-
timising a metric accounting for the consequences on the
future object motion, which could be unintuitive or difficult
for the operator to predict. For example, in this work, we con-
sidered minimising the torques the manipulator will exert on
the object to carry out an autonomous manipulation after the
grasp. This metric is important because it helps reducing the
system operating costs while extending the range of objects
it can manipulate.

Exploiting a 3-dimensional model of the object, the sys-
tem identifies the feasible grasp poses and then, using a rough
approximation of the object inertial properties, it evaluates
for each grasp pose the torques needed by the robot for per-
forming the post-grasp manipulative action (i.e., in our case,
moving the grasped object along a pre-planned trajectory).
With this information, during the teleoperation the system
can provide the human operator with navigation guidance
towards the grasp that locally minimises the expected torque
effort. While this work focused on minimising the expected
torques, the framework can be easily adapted to consider
any other type of metric which is able to provide a cost for
the set of feasible grasps. Similarly, while our post-grasp
manipulation consisted in picking the object up and placing
it somewhere else, the framework can be adapted to consider
to any other type of manipulation action.

We demonstrated the effectiveness of the proposed ap-
proach in a series of representative real-world experiments
as well as a human subjects study enrolling 15 subjects. The
results proved the effectiveness of our shared control tech-
niques vs. standard human-in-the-loop teleoperation in most
performance metrics. Moreover, haptic-only guidance per-
formed better than visual-only guidance, although combined
visuo-haptic guidance led to the best overall results.

Future work will study how the strength of the haptic
guidance links to task performance. For example, we could
adapt the guidance strength to the operator's performance
or experience, e.g., a system could use a stiff navigation approach (i.e., less freedom for the operator) when operated by novices, while it could employ a soft navigation approach (i.e., more freedom for the operator) when operated by experts. This approach could be also useful when teaching new operators, employing different levels of autonomy according to the operator's experience and learning process. Of course, as the framework supports different cost functions and post-grasp manipulation actions, we will also consider how to extend our experimental evaluation in this direction, considering what might be relevant for other fields of application (e.g., medical robotics). In this respect, we also plan to compare the proposed techniques with other grasping and guidance methods currently employed in these different scenarios of application. Finally, we plan to extend our experimentation to consider a broader range of environments, moving closer to the type of objects we can find in high-impact target scenarios, e.g., in nuclear waste storage sites. In this respect, we should also consider the capabilities of the employed robotic system, e.g., maximum applicable grasping and lifting forces, with respect to the considered objects in the environment, so as to always ensure a feasible manipulation.

## References

Abbeel P, Coates A, Ng AY (2010) Autonomous helicopter aerobatics through apprenticeship learning. *The International Journal of Robotics Research* 29(13):1608–1639

Abbink DA, Mulder M, Boer ER (2012) Haptic shared control: smoothly shifting control authority? *Cognition, Technology & Work* 14(1):19–28

Abi-Farraj F, Pedemonte N, Robuffo Giordano P (2016) A visual-based shared control architecture for remote telemanipulation. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 4266–4273

Abi-Farraj F, Henze B, Werner A, Panzirsch M, Ott C, Roa MA (2018) Humanoid teleoperation using task-relevant haptic feedback. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 5010–5017

Abi-Farraj F, Pacchierotti C, Arenz O, Neumann G, Robuffo Giordano P (2019) A haptic shared-control architecture for guided multi-target robotic grasping. *IEEE Transactions on Haptics* 13(2):270–285

Achibet M, Marchal M, Argelaguet F, Lécuyer A (2014) The virtual mitten: A novel interaction paradigm for visuo-haptic manipulation of objects using grip force. In: IEEE Symposium on 3D User Interfaces (3DUI), pp 59–66

Adjigble M, Marturi N, Ortenzi V, Stolkin R (2019) An assisted telemanipulation approach: combining autonomous grasp planning with haptic cues. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 3164–3171

Baronti L, Alston M, Mavrakis N, Ghalamzan E, Amir M, Castellani M, et al. (2019) Primitive shape fitting in point clouds using the bees algorithm. *Applied Sciences* 9(23):5198

Boessenkool H, Abbink DA, Heemskerk CJM, van der Helm FCT, Wildenbeest JGW (2013) A task-specific analysis of the benefit of haptic shared control during telemanipulation. *Transactions on Haptics* 6(1):2–12

Bogue R (2011) Robots in the nuclear industry: a review of technologies and applications. *Industrial Robot: An International Journal* 38(2):113–118

Bruemmer DJ, Few DA, Boring RL, Marble JL, Walton MC, Nielsen CW (2005) Shared understanding for collaborative control. *IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans* 35(4):494–504

Cipriani C, Zaccone F, Micera S, Carrozza MC (2008) On the shared control of an emg-controlled prosthetic hand: Analysis of user prosthesis interaction. *IEEE Transactions on Robotics* 24(1):170–184

Cognetti M, Aggravi M, Pacchierotti C, Salaris P, Giordano PR (2020) Perception-aware human-assisted navigation of mobile robots on persistent trajectories. *IEEE Robotics and Automation Letters* 5(3):4711–4718

Corucci F, Ruffaldi E (2016) Toward autonomous robots for demolitions in unstructured environments. In: Intelligent Autonomous Systems 13, Springer, pp 1515–1532

Detry R, Papon J, Matthies L (2017) Task-oriented grasping with semantic and geometric scene understanding. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 3266–3273

Duchaine V, Gosselin CM (2007) General model of human-robot cooperation using a novel velocity based variable impedance control. In: 2007 IEEE World Haptics Conference, pp 446–451

Franchi A, Secchi C, Ryll M, Bulthoff HH, Robuffo Giordano P (2012) Shared control: Balancing autonomy and human assistance with a group of quadrotor uavs. *IEEE Robotics & Automation Magazine* 19(3):57–68

Ghalamzan EAM, Abi-Farraj F, Robuffo Giordano P, Stolkin R (2017) Human-in-the-loop optimisation: mixed initiative grasping for optimally facilitating post-grasp manipulative actions. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 3386–3393

Ghalamzan E AM, Mavrakis N, Kopicki M, Stolkin R, Leonardis A, et al. (2016) Task-relevant grasp selection: A joint solution to planning grasps and manipulative motion trajectories. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 907–914

Horowitz MB, Burdick JW (2012) Combined grasp and manipulation planning as a trajectory optimization problem. In: International Conference on Robotics and Automation, IEEE, pp 584–591
Izadi V, Bhardwaj A, Ghasemi AH (2020) Impedance modulation for negotiating control authority in a haptic shared control paradigm. In: 2020 American Control Conference (ACC), IEEE, pp 2478–2483

James S, Wohlhart P, Kalakrishnan M, Kalashnikov D, Ipan A, Ibarz J, Levine S, Hadsell R, Bousmalis K (2019) Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 12627–12637

Kopicki M, Detry R, Adjigble M, Stolkin R, Leonardis A, Wyatt JL (2016) One-shot learning and generation of dexterous grasps for novel objects. *The International Journal of Robotics Research* 35(8):959–976

Kopicki MS, Belter D, Wyatt JL (2019) Learning better generative models for dexterous, single-view grasping of novel objects. *The International Journal of Robotics Research* 38(10-11):1246–1267

Kormushev P, Calinon S, Caldwell DG (2011) Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. *Advanced Robotics* 25(5):581–603

Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. *The International Journal of Robotics Research* 37(4-5):421–436

Ly KT, Poozhiyil M, Pandya H, Neumann G, Kucukyilmaz A (2021) Intent-aware predictive haptic guidance and its application to shared control teleoperation. In: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), IEEE, pp 565–572

Maisto M, Pacchierotti C, Chinello F, Salvietti G, De Luca A, Prattichizzo D (2017) Evaluation of wearable haptic systems for the fingers in augmented reality applications. *IEEE Transactions on Haptics* 10(4):511–522

Masone C, Robuffo Giordano P, Bülthoff HH, Franchi A (2014) Semi-autonomous Trajectory Generation for Mobile Robots with Integral Haptic Shared Control. In: 2014 International Conference on Robotics and Automation (ICRA), pp 6468–6475

Mavrakis N, Ghalamzan E AM, Stolkin R, Baronti L, Kopicki M, Castellani M, et al. (2016) Analysis of the inertia and dynamics of grasped objects, for choosing optimal grasps to enable torque-efficient post-grasp manipulations. In: 2016 IEEE-RAS International Conference on Humanoid Robots (Humanoids), IEEE, pp 171–178

Mavrakis N, Ghalamzan EAM, Stolkin R (2017) Safe robotic grasping: Minimum impact-force grasp selection. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 4034–4041

Mavrakis N, E AMG, Stolkin R (2020) Estimating an object’s inertial parameters by robotic pushing: A data-driven approach. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 1–9

Meli L, Pacchierotti C, Prattichizzo D (2017) Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation. *The International Journal of Medical Robotics and Computer Assisted Surgery* 13(4):e1809

Meli L, Pacchierotti C, Salvietti G, Chinello F, Maisto M, De Luca A, Prattichizzo D (2018) Combining wearable finger haptics and augmented reality: User evaluation using an external camera and the microsoft hololens. *IEEE Robotics and Automation Letters* 3(4):4297–4304

Miller AT, Allen PK (2004) Graspit! a versatile simulator for robotic grasping. *IEEE Robotics & Automation Magazine* 11(4):110–122

Murray RM, Li Z, Sastry SS, Sastry SS (1994) *A mathematical introduction to robotic manipulation*. CRC press

Nelson B, Morrow J, Khosla P (1996) Robotic manipulation using high bandwidth force and vision feedback. *Mathematical and Computer Modelling* 24(5):11–29

van Oosterhout J, Wildenbeest JG, Boessenkool H, Heemskerk CJ, de Baar MR, van der Helm FC, Abbink DA (2015) Haptic shared control in tele-manipulation: Effects of inaccuracies in guidance on task execution. *IEEE Transactions on Haptics* 8(2):164–175

Pacchierotti C (2015) *Cutaneous haptic feedback in robotic teleoperation*. Springer

Pacchierotti C, Prattichizzo D, Kuchenbecker KJ (2015) Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery. *IEEE Transactions on Biomedical Engineering* 63(2):278–287

Pacchierotti C, Ongaro F, Van den Brink F, Yoon C, Prattichizzo D, Gracias DH, Misra S (2018) Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance. *IEEE Transactions on Automation Science and Engineering* 15(1):290–306

Panesar S, Cagle Y, Chander D, Morey J, Fernandez-Miranda J, Kliot M (2019) Artificial intelligence and the future of surgical robotics. *Annals of surgery* 270(2):223–226

Pardi T, Stolkin R, et al. (2018) Choosing grasps to enable collision-free post-grasp manipulations. In: 2018 IEEE-RAS International Conference on Humanoid Robots (Humanoids), IEEE, pp 299–305

Pardi T, Ortenzi V, Fairbairn C, Pipe T, Esfahani AMG, Stolkin R (2020) Planning maximum-manipulability cutting paths. *IEEE Robotics and Automation Letters* 5(2):1999–2006

Parsa S, Kamale D, Mghames S, Nazari K, Pardi T, Srini-vasan AR, Neumann G, Hanheide M, Amir GE (2020) Haptic-guided shared control grasping: collision-free manipulation. In: 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), IEEE, pp 1552–1557

Parsa S, Maior H, Thumwood A, Wilson M, Hanheide M, Amir GE (2022) The impact of motion scaling and haptic guidance on operators’ workload and performance in
teleoperation. In: Proceedings of Conference on Human Factors in Computing Systems, p accepted

Pearce F (2015) Shocking state of world's riskiest nuclear waste site. https://www.newscientist.com

Peternel L, Petrič T, Babič J (2018) Robotic assembly solution by human-in-the-loop teaching method based on real-time stiffness modulation. *Autonomous Robots* 42(1):1–17

Peternel L, Fang C, Laghi M, Bicchi A, Tsagarakis N, Ajoudani A (2020) Human arm posture optimisation in bilateral teleoperation through interface reconfiguration. In: 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), IEEE, pp 1102–1108

Rahal R, Abi-Farraj F, Giordano P, Pacchierotti C (2019) Haptic shared-control methods for robotic cutting under non-holonomic constraints. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 8151–8157

Rahal R, Matarese G, Gabiccini M, Artoni A, Prattichizzo D, Robuffo Giordano P, Pacchierotti C (2020) Caring about the human operator: haptic shared control for enhanced user comfort in robotic telemanipulation. *IEEE Transactions on Haptics* 13(1):197–203

Ratliff N, Zucker M, Bagnell JA, Srinivasa S (2009) Chomp: Gradient optimization techniques for efficient motion planning. In: 2009 International Conference on Robotics and Automation (ICRA), IEEE, pp 489–494

Rozo L, Jiménez P, Torras C (2013) A robot learning from demonstration framework to perform force-based manipulation tasks. *Intelligent service robotics* 6(1):33–51

Schulman J, Duan Y, Ho J, Lee A, Awwal I, Bradlow H, Pan J, Patil S, Goldberg K, Abbeel P (2014) Motion planning with sequential convex optimization and convex collision checking. *The International Journal of Robotics Research* 33(9):1251–1270

Selvaggio M, Abi-Farraj F, Pacchierotti C, Robuffo Giordano P, Siciliano B (2018) Haptic-based shared-control methods for a dual-arm system. *IEEE Robotics and Automation Letters* 3(4):4249–4256

Selvaggio M, Ghalamzan Esfahani A, Moccia R, Ficuciello F, Siciliano B (2019a) Haptic-guided shared control for needle grasping optimization in minimally invasive robotic surgery. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 3617–3623

Selvaggio M, Robuffo Giordano P, Ficuciello F, Siciliano B (2019b) Passive task-prioritized shared-control teleoperation with haptic guidance. In: 2019 International Conference on Robotics and Automation (ICRA), IEEE, pp 430–436

Selvaggio M, Cacace J, Pacchierotti C, Ruggiero F, Giordano PR (2021) A shared-control teleoperation architecture for nonprehensile object transportation. *IEEE Transactions on Robotics* 38(1):569–583

Singh J, Srinivasan AR, Neumann G, Kucukyilmaz A (2020) Haptic-guided teleoperation of a 7-dof collaborative robot arm with an identical twin master. *IEEE Transactions on Haptics* 13(1):246–252

Talha M, Ghalamzan E AM, Takahashi C, Kuo J, Ingamells W, Stolkin R (2016) Towards robotic decommissioning of legacy nuclear plant: Results of human-factors experiments with tele-robotic manipulation, and a discussion of challenges and approaches for decommissioning. In: IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, pp 166–173

Tobin J, Biewald L, Duan R, Andrychowicz M, Handa A, Kumar V, McGrew B, Ray A, Schneider J, Welinder P, et al. (2018) Domain randomization and generative models for robotic grasping. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 3482–3489

Vahrenkamp N, Przybylski M, Asfour T, Dillmann R (2011) Bimanual grasp planning. In: 2011 IEEE-RAS International Conference on Humanoid Robots (Humanoids), IEEE, pp 493–499

Vahrenkamp N, Asfour T, Metta G, Sandini G, Dillmann R (2012) Manipulability analysis. In: 2012 IEEE-RAS International Conference on Humanoid Robots (Humanoids), IEEE, pp 568–573

Varley J, DeChant C, Richardson A, Ruales J, Allen P (2017) Shape completion enabled robotic grasping. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 2442–2447

Xi B, Wang S, Ye X, Cai Y, Lu T, Wang R (2019) A robotic shared control teleoperation method based on learning from demonstrations. *International Journal of Advanced Robotic Systems* 16(4):1729881419857428

Yang D, Liu H (2021) Human-machine shared control: New avenue to dexterous prosthetic hand manipulation. *Science China Technological Sciences* 64(4):767–773

Zhou Y, Wang W, Guan W, Wu Y, Lai H, Lu T, Cai M (2017) Visual robotic object grasping through combining rgb-d data and 3d meshes. In: International Conference on Multimedia Modeling, Springer, pp 404–415

Zhu Y, Yang C, Wei Q, Wu X, Yang W (2020) Human-robot shared control for humanoid manipulator trajectory planning. *Industrial Robot: the international journal of robotics research and application*

Zucker M, Ratliff N, Dragan AD, Pivtoraiko M, Klingensmith M, Dellin CM, Bagnell JA, Srinivasa SS (2013) Chomp: Covariant hamiltonian optimization for motion planning. *The International Journal of Robotics Research* 32(9-10):1164–1193