# Human-Inspired Haptic-Enabled Learning from Prehensile Move Demonstrations

Aleksander Lillienskiold, Rahaf Rahal, Paolo Robuffo Giordano, Claudio

Pacchierotti, Ekrem Misimi

► To cite this version:

Aleksander Lillienskiold, Rahaf Rahal, Paolo Robuffo Giordano, Claudio Pacchierotti, Ekrem Misimi. Human-Inspired Haptic-Enabled Learning from Prehensile Move Demonstrations. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52 (4), pp.2061-2072. 10.1109/TSMC.2020.3046775. [hal-03084466](https://hal-inriaHal)

HAL Id: hal-03084466

https://inria.hal.science/hal-03084466

Submitted on 21 Dec 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Human-Inspired Haptic-Enabled Learning from Prehensile Move Demonstrations

Aleksander Lillienskiold¹,⁴,*, Rahaf Rahal²,⁵,*,
Paolo Robuffo Giordano³, Claudio Pacchierotti³, and Ekrem Misimi⁴

***Abstract***—Research on robotic manipulation of fragile, compliant objects, such as food items, is gaining traction due to its game-changing potential within the food production and retailing sectors, currently characterized by manually-intensive and highly repetitive tasks. Food products exhibit high levels of frailness, biological variation, and complex 3D shapes and textures. For these reasons, introducing greater levels of robotic automation in the food and agricultural sectors remains an important challenge. This paper addresses this challenge by developing a human-centred, haptic-based, Learning from Demonstration (LfD) policy that enables pre-trained autonomous grasping of food items using an anthropomorphic robotic system. The policy combines data from teleoperation and direct human manipulation of objects, embodying human intent and interaction areas of significance. We evaluated the proposed solution against a recent state-of-the-art LfD policy as well as against two standard impedance controller techniques. Results show that the proposed policy performs significantly better than the other considered techniques, leading to high grasping success rates while guaranteeing the integrity of the food at hand.

***Index Terms***—Haptics, Grasping, Learning from Demonstration, Teleoperation, Robotics, Food handling

## I. INTRODUCTION

AUTONOMOUS food processing and handling is gaining attention among researchers in the fields of robotics, automation, and machine learning. Reasons for this interest are three-fold: (i) compelling scientific and technological challenges linked to the handling of food, such as flexibility, frailness, friction, and high inter-object variability, (ii) increasing interest from public and private investors, and (iii) the possibility of impacting food quality and reducing waste. To address current challenges, the European Commission has recently funded the €7-million collaborative research project “SoMa,” which looks at innovative soft robotic manipulation for handling fragile objects such as fruits and vegetables [1]. Similarly, Amazon’s $13.7-billion acquisition of the food retailer “Whole Foods” aims to promote an automation revolution in the food delivery market. According to Bloomberg [2], Amazon will soon introduce robots in Whole Food warehouses as a cost-cutting initiative. However, not only distributors need to automate the manipulation of delicate food items. Currently, food production, harvesting, and processing continue to be dominated by labour-intensive tasks. In addition, vital global suppliers have a constant need for more efficient operations,

and improved HSSEQ (Health, Safety, Security, Environment, and Quality) practices linked to their workers and the safe handling of food. To address these issues, researchers need to focus on efficient and safe methods for the autonomous handling of food while preserving their quality.

Nowadays, (purely) vision-based techniques are the most widely applied approaches for autonomous robotic grasping and manipulation, either requiring a 3D-model of the item in question or constructing models using 3D-vision information, e.g. point clouds [3], [4], [5]. Although the knowledge of an object’s model may often be sufficient [6], the lack of precise information concerning its shape and mechanical properties may significantly affect the quality of the grasping [7], [8]. An issue of major importance when handling food is its frailness, which is an even more predominant problem when objects are moving, or when they are handled and grasped during harvesting, post-harvest processes, and on production lines. A representative example is strawberry harvesting, where even the smallest excessive force can degrade and spoil the product.

Haptic sensations have been proven to play a key role in enhancing human fine manipulation [9] and precision grasping [10]. In robotics, haptic feedback is widely believed to be a valuable tool in teleoperation, and it has been shown to enhance operators’ performance in a wide range of robotic applications, including microrobotics [11], [12], [13], mobile robotics [14], needle and catheter insertion [15], [16], surgical knot tying [17], assembly [11], and palpation [18], [16], [19]. The benefits of haptic feedback in this scenario include greater manipulation and perception accuracy, as well as lower levels of peak and mean force applied to food objects, thus lowering the impact on the food due to their handling.

A promising approach to autonomous grasping via robotic manipulators is Learning from Demonstration (LfD) [20]. This technique involves a robot learning a target policy from examples, or demonstrations, provided by an experienced human operator. LfD uses recorded datasets in the form of state-action pairs to derive policies that reproduce target behaviours. This methodology is in stark contrast to other learning techniques, such as Reinforcement Learning, where robots learn from direct self experience. The use of LfD to teach robots how to grasp is quite popular among researchers who make use of visual and/or haptic information to demonstrate tasks by means of teleoperation [21], [22], [23], [24], [25] or direct interaction [26], [27], [28]. However, all of the aforementioned works involve human operators driving robots using non-intuitive and somewhat basic master interfaces such as teach pendants, non-haptic joysticks, single-point haptic interfaces, or by directly moving the robots around. It is our belief that, in order for robots to learn correctly from humans

c. Corresponding author: rahaf.rahal@irisa.fr

* Rahaf Rahal and Aleksander Lillienskiold (f.k.a. Eilertsen) have contributed equally and are to be seen as co-first-authors.

1. REALTEK at NMBU, 1433 Ås, NORWAY.

2. Univ Rennes, Inria, CNRS, IRISA, 35000 Rennes, FRANCE.

3. CNRS, Univ Rennes, Inria, IRISA, 35000 Rennes, FRANCE

4. SINTEF Ocean AS, 7010 Trondheim, NORWAY.
<figure><img src="image_1.png" /><figcaption>Fig. 1. *Food items and Heat-map capturing.* a) The five food objects used in the experiments: average circumference 290 mm, height 75 mm. b) Heat-map generation set-up. A laptop connected to a camera, a lighting system, water paint with a brush for application, and a cloth for wiping the subject's hand after each trial.</figcaption></figure>

<figure><img src="image_2.png" /></figure>

how to grasp challenging, compliant, and fragile objects, it is necessary to provide an intuitive and natural tool for controlling the slave manipulators. In other words, we believe that by achieving a high degree of dexterity and telepresence, human operators will be able to approach and grasp target objects in the same way as they would do if grasping barehanded.

## II. CONTRIBUTION AND MOTIVATION

This paper presents a human-centered, haptic-based, LfD approach for teaching autonomous robotic systems the action of grasping delicate, fragile, and compliant food items.

First, five human subjects were asked to grasp five food items barehanded, recording the contact area between the users' hand and the objects' surface using paint transfer and machine vision techniques (see Sec. III-A and Fig. 1b). Paint-transfer approaches have already been used to detect contact areas when humans grasp objects [29]. However, to the best of our knowledge, this is the first time that this approach is used to learn an LfD policy. This first step enables us to study *how humans grasp* the considered food items when they are endowed with maximum dexterity and feedback. However, these data alone are not sufficient to design an autonomous grasping policy for a robotic manipulator, which has significantly lower sensing and actuation capabilities than a human. For this reason, afterwards, we conducted *teleoperated* prehensile move experiments involving the same food object types (see Sec. III-B). All relevant data on the grasping motion and exchanged forces were recorded. To achieve high levels of dexterity and telepresence, we devised a haptic-enabled high-degrees-of-freedom robotic teleoperation system, shown in Fig. 3a.

By using both direct interaction and teleoperation data, we then devised four Support Vector Regression (SVR) algorithms based on LfD policies, enabling a robotic manipulator to autonomously grasp a set of representative food items.

Finally, we validated our autonomous grasping policies by carrying out 2310 autonomous robotic grasping tasks on seven representative food items, not limited to those used during the training. We compared the proposed policies to each other,

as well as, to a recently published LfD-based approach [21] and two standard impedance controllers. Misimi et al. [21] combines RGB-D images and tactile data to learn the required pose of the gripper, the gripper finger configuration, and the exerted forces. However, unlike our approach, it neither uses information derived from direct human interaction nor data supplied by high-dexterity haptic-enabled teleoperation.

With reference to related published work [21], [30], [31], [32], [33], our study heavily focuses on human input. By providing an intuitive and rich interface to control the slave robot, we enabled human operators to carry out the tasks naturally and intuitively, close to what they would do with their own hands. To the best of our knowledge, this is the first time an LfD approach is applied to enable autonomous grasping by making use of both (i) rich haptic-endowed teleoperation data and (ii) the mapping of direct hand interaction. Moreover, this is also the first time that such an approach is used for grasping raw food items. Our approach combines teleoperation data with real human interaction to build a robust learning policy based on LfD. This policy enables a continuous understanding of the interactions taking place between the human and the robot during a grasping sequence.

We have focused our research and validation on food items because they are excellent examples of 3D compliant objects that should not be degraded during manipulation. Robotic manipulation of 3D compliant objects is indeed a research field that still requires significant research input to achieve a robust manipulation of such objects.

## III. GRASPING-DATA GATHERING

This Section provides details on the two data gathering experiments we carried out with the aim of building our LfD policies. In both cases, we asked our subjects to grasp the five different food items shown in Fig. 1a: an aubergine, a bell pepper, a lettuce, a navel orange, and a beef tomato. These items were selected as representative examples of 3D compliant objects because of their difference in size, shape, texture, and compliance properties, and as representatives of the most common fruits and vegetables consumed in global food
## GENERATING HEAT-MAP:

Per Heat-mapping:
5 people x 10 grasps

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 2. Example of a heat-map generation (aubergine). The intensity of the RGB-image is extracted, converted to binary, then mapped to a generic hand using Thin-Plate spline interpolation, creating an intensity image for accumulated binary images. Active areas in the hand change with different objects.</figcaption></figure>

markets. According to the Food and Agriculture Organization of the United Nations (FAO), tomatoes are one of the most cultivated fruits in the world, with more than 180M metric tonnes produced every year. Aubergines and oranges are also very popular, with 50M metric tonnes produced every year, bell peppers reach 35M metric tonnes, and lettuce 25M metric tonnes. Another important constraint for our choice of produce was the payload of our robotic system, which is only able to carry up to 2 kg. Employing a different robot could of course enlarge the set of food objects we are able to handle. However, it is important to highlight that our machine learning approach can easily be trained to handle any other type of compliant food objects, regardless of the robotic system being employed. A video of these experiments is available as supplemental material and at https://youtu.be/0LaDPbGwZlw. Table I summarizes the main symbols and variables used in the paper.

### A. Direct grasping by hand

Five subjects (four male and one female) were asked to grasp the five sample objects barehanded. Our objective was to understand how humans grasp objects when they are endowed with maximum dexterity and feedback (i.e. during direct hand interaction). Inspired by the work of Knopp et al. [29] and Kamakura et al. [34], we used a paint-transfer approach to identify the parts of the hand that are employed the most during this grasping.

**Task and data gathering**—Subjects were comfortably seated in front of a table. The experimenter placed the first object on the table and covered it in non-toxic water paint. Subjects were asked to grasp and lift the object as if they were moving it from one table to another. Then, they were asked to release the object and show their palm to a camera, which recorded the paint-transfer patterns (see Fig. 1a). Finally, subjects washed their hands and a new object was prepared for grasping. Each subject grasped each object ten times, yielding 50 grasps per subject, 50 grasps per object, and 250 paint-transfer trials in total.

**Heat-map generation**—Images of paint-transfer patterns were taken using a Grasshopper 3 camera (FLIR Systems, Canada), and the contact area was segmented using standard

<table><caption>TABLE I
List of main symbols and variables.</caption><thead><tr><th>Symbol</th><th>Definition/Description</th></tr></thead><tbody><tr><td>*<sub>m</sub>, *<sub>s</sub></td><td>related to the master or slave, respectively</td></tr><tr><td><b>x</b><sub>m</sub>, <b>x</b><sub>s</sub></td><td>end-effector pose</td></tr><tr><td><b>p</b><sub>m</sub>, <b>p</b><sub>s</sub></td><td>end-effector position</td></tr><tr><td><b>R</b><sub>m</sub>, <b>R</b><sub>s</sub></td><td>end-effector orientation</td></tr><tr><td><i>W</i></td><td>world coordinate system</td></tr><tr><td><b>v</b><sub>m</sub>, <b>v</b><sub>s</sub></td><td>end-effector linear velocity</td></tr><tr><td><i>ω</i><sub>m</sub>, <i>ω</i><sub>s</sub></td><td>end-effector rotational velocity</td></tr><tr><td>*MCP, *PIP</td><td>related to the fingers' metacarpophalangeal joint and proximal-interphalangeal joint, respectively</td></tr><tr><td><i>θ</i><sub>r<sub>i</sub></sub></td><td>angular command to the MCP robotic joint <i>i</i></td></tr><tr><td><i>θ</i><sub>e<sub>i</sub></sub></td><td>angular value of the MCP exoskeleton joint <i>i</i></td></tr><tr><td><i>τ</i><sub>MCP,i</sub>, <i>τ</i><sub>PIP,i</sub></td><td>haptic feedback applied to finger <i>i</i></td></tr><tr><td><i>m</i><sub>pro,i</sub>, <i>m</i><sub>dis,i</sub></td><td>average pressure on the proximal and distal phalanges of the robot finger <i>i</i>, respectively</td></tr><tr><td><b>g</b><sub>pos</sub></td><td>pose of the gripper</td></tr><tr><td><b>f</b><sub>prs</sub></td><td>tactile readings from the robotic hand's pressure sensors</td></tr><tr><td><b>f</b><sub>mot</sub></td><td>position of the motors of the robotic hand</td></tr><tr><td><b>f</b><sub>ang</sub></td><td>joint angles of the exoskeleton</td></tr><tr><td><b>s</b></td><td>observed robotic grasping action performed by the human teacher</td></tr><tr><td><b>a</b></td><td>action-state, including gripper pose and motor positions</td></tr><tr><td>α, β, γ</td><td>discrete time-series steps</td></tr><tr><td><b>H</b><sup>k</sup><sub>j</sub></td><td>heat-map values gathered during direct hand interaction for object <i>k</i> and action vector feature <i>j</i></td></tr></tbody></table>

background subtraction and light intensity filtering. We then mapped the resulting contact area and constructed a heat-map using Thin-Plate Spline interpolation [35]. A heat-map is a graphical representation of individual values contained within a 2D matrix represented by colours. Brighter colours indicate higher levels of accumulated contact, identifying which parts of the hand most used by the subjects when grasping the object in question. Establishing a human-grasp signature inferring
<figure><img src="image_10.png" /><figcaption>Fig. 3. Teleoperation set-up and control design. (a) The human operator teleoperates the slave system to grasp a tomato. (b) Master-slave control: $[x_s, y_s, z_s, q_s^T]^T$ are the end-effector coordinates defining the pose of the RHR hand. The difference between the previous and current master position is used to evaluate the velocity command, which is then sent to the slave manipulator. RHR finger movements are proportional to the positions registered by the H-Glove fingers.</figcaption></figure>

innate sensor contact importance. An example of data derived from this heat-mapping process is shown in Fig. 2. Heat-map data are used in Sec. IV as a basis for our human-inspired LfD grasping policy.

## B. Teleoperated grasping by an expert operator

Although the heat-map data are interesting, they are not sufficient on their own to define an autonomous grasping policy for a robotic manipulator. For this reason, we gathered additional data from 250 teleoperated grasping trials (50 grasps per food item) using our target robotic system as the slave of a haptic-enabled bilateral teleoperation system, shown in Fig. 3a. The master interface is composed of a 9-degrees-of-freedom (9-DoF) haptic hand exoskeleton (H-Glove by Haption, France), attached to a 6-DoF grounded haptic interface (Virtuose 6D by Haption, France). The slave system is composed of a 7-DoF robotic manipulator (Panda by Franka Emika, Germany), endowed with a 4-DoF 3-fingered robotic hand equipped with matrix pressure sensors on all fingers (ReFlex TakkTile by Right-Hand-Robotics, USA). This integrated system enables a human operator to grasp and manipulate compliant objects while receiving compelling haptic feedback of the forces applied by the slave system.

**Teleoperation of the 7-DoF robotic manipulator**—To achieve high motion fidelity, we mapped the velocity of the grounded haptic interface to that of the slave robotic end-effector, using an approach similar to that employed in [36], [37]. Let $x_m = (p_m, R_m) \in \mathbb{R}^3 \times SO(3)$ be the pose of the master interface end-effector, and $x_s = (p_s, R_s) \in \mathbb{R}^3 \times SO(3)$ the pose of the slave end-effector, both expressed w.r.t. a common world frame W (see Fig. 3a). The velocity of the slave robot $[v_s \omega_s]^T$ is then calculated as a function of the master's velocity $[v_m \omega_m]^T$ and the difference between the desired and current pose of the robot, i.e.

$$ \begin{bmatrix} v_s \\ \omega_s \end{bmatrix} = \begin{bmatrix} v_m \\ \omega_m \end{bmatrix} + \lambda \begin{bmatrix} p_{s,d} - p_s \\ R_s \, s \theta u_{s,d} \end{bmatrix}, \quad (1) $$

where $\lambda$ is a gain parameter, and $p_{s,d}$ the desired robot position derived from the current position of the haptic device, $p_{s,d} = (p_m - p_{m0}) + p_{s0}$, with $p_{m0}$ and $p_{s0}$ as the initial poses of the master and slave, respectively. Similarly, to establish orientation, $s\theta u_{s,d}$ is the angle-axis representation of the relative rotation between the desired and current orientation of the slave, i.e. $sR_{s,d} = s_0 R_s^T m_0 R_m$, where $s_0 R_s$ and $m_0 R_m$ define the current orientation of the slave and master, respectively, w.r.t their orientation at the start of a given experiment. Thus, the first term of (1) ensures an adequate responsiveness of the system, while the second prevents drifts.

Of course, there are other ways we can map the motion of the human operator into that of the slave robot [38], [39], [40]. In this work, we chose a direct and straightforward way to control the robotic manipulator, as one of our hypotheses states that a human-like natural control can improve the quality of recorded data and, thus, of the learning policies.

**Teleoperation of the 4-DoF robotic hand**—To control the robotic hand, we mapped the positions of the exoskeleton fingers to the positions of the slave's robotic fingers.

The robotic hand has four active DoF: one DoF for each finger's metacarpophalangeal (MCP) joint, and one to control the abduction of the two neighbouring fingers (see Fig. 3a). These fingers were chosen to correspond with the index and middle finger of the exoskeleton, together with an opposing thumb. The position command $\theta_{r_i}$ for the MCP joint in a robotic finger $i$ is proportional to the recorded position $\theta_{e_i}$ of the MCP joint in the corresponding exoskeleton finger:

$$ \theta_{r_i} = k_i \theta_{e_i} + \theta_{r_i, s}, \quad (2) $$

where $k_i$ is a proportional gain, and $\theta_{r_i, s}$ is a safety offset, ensuring that the commanded position value is always within safe intervals.

**Haptic feedback**—The robotic hand is endowed with six matrix pressure sensors (TakkTile, USA), located on the proximal and distal phalanges of the three fingers. Force feedback at the MCP joint of the exoskeleton, $\tau_{MCP,i}$, is
<figure><img src="image_11.png" /><figcaption>Fig. 4. *Generation of LfD policy from acquired data.* Finger tactile values coming from the RHR-hand were weighted according to heat-map data when generating the human-inspired LfD-policy. Temporal variation was captured by implementing time-series, where n previous values for the robot pose, finger configuration, and tactile stimuli were applied to the generation of predictions.</figcaption></figure>

computed by summing the pressure stimuli registered by sensors located at the proximal and distal phalanges in the corresponding slave fingers. At the proximal-interphalangeal (PIP) joint, forces $\tau_{PIP,i}$ are computed only from pressure stimuli on sensors located on the slave's distal phalanx. The total force feedback is computed as:

$$ \tau_i = \begin{bmatrix} \tau_{MCP,i} \\ \tau_{PIP,i} \end{bmatrix} = u_i \begin{bmatrix} m_{pro,i} + m_{dis,i} \\ m_{dis,i} \end{bmatrix}, \quad (3) $$

where $u_i$ is a proportional gain, and $m_{pro,i}$ and $m_{dis,i}$ are average pressures recorded on the proximal and distal phalanges of the slave finger i, respectively.

Each finger of the master exoskeleton is modeled as a gravity-compensated mechanical system with two active joints (MCP and PIP, see Fig. 3a), with force feedback $\tau_i$. This approach enables the human operator to feel, on each finger, the forces exerted by the robotic hand on the environment.

**Task and data collection**—An expert operator used the teleoperation system for grasping and lifting the same five food items described in Sec. III-A and shown in Fig. 1a. Each item was placed on a table in front of the slave robot and lifted 50 times as part of a total of 250 trials (see Fig. 3a). To enable the operator to view the environment, the master interface was placed one metre in front of slave robot's workspace. For each trial, both the pose of the slave system and the forces exerted on the food items were saved. At the end of each task, the operator rated the quality of the grasp on a continuous scale from 0 (worst) to 4 (best). Zero (0) indicates that the operator was not able to grasp the object in the first place, while 4 means that the object was held firmly during the entire grasp sequence (see Table II). This information was used to select the successful demonstration examples for training the LfD policy. The majority of the grasps were rated between 3 and 4. We also asked the expert operator to rate the quality of the food items after each grasp on a scale from 0 (worst) to 4 (best), carefully looking for bruises, cuts, changes in shape, and any other sign of degradation (see Table II). The operator was gentle when handling the produce, resulting in a consistent quality rating of 4, reflecting the rich feedback and dexterity capabilities of the employed haptic teleoperation system. In comparison, Misimi et al. [21] provided no haptic feedback

during their teleoperation trials, and operators reported that it was very challenging to not damage the items when not receiving any tactile information.

Finally, for each grasping repetition, we also registered the angle of approach, any significant synergy between the fingers, and the forces applied by the robotic hand on the object. The design of the robotic hand lead to most grasps being precision ball (enveloping) grasps [41]. This type of grasp enabled the operator to receive haptic feedback to all the three fingers of the exoskeleton.

<caption>TABLE II</caption>

Scales used in the evaluation of the teleoperated prehensile move and the quality of the food after the move, inspired from [42], [43]. Scale runs continuously from 0 (worst) to 4 (best), enabling the operator to select values between those indicated below.

<table><thead><tr><td>Prehensile move</td><td>Food items</td></tr></thead><tbody><tr><td>4 – Complete prehensile move</td><td>4 – No visible modifications</td></tr><tr><td>3 – Slips close to the landing</td><td>3 – Small, localized, light marks</td></tr><tr><td>2 – Slips mid-way</td><td>2 – Mid-sized, spread marks</td></tr><tr><td>1 – Slips right after lift-off</td><td>1 – Bruises, small tears in skin</td></tr><tr><td>0 – Robotic hand fails to clutch, action aborted</td><td>0 – Heavy bruises, spread marks and tears in skin</td></tr></tbody></table>

## IV. MACHINE LEARNING

Learning a task from scratch can be very challenging, even for humans. A good strategy for learning how to perform a brand new task is to be provided with a set of instructions by an expert, followed by a practical demonstration [44]. Indeed, most forms of machine learning (ML) applied in robotics are based on supervised learning (SL), typically using manually-crafted features as key to learning sought-after policies [21], [45]. Unsupervised learning methods, such as (deep) reinforcement learning, achieve objective functions by enabling learning agents to autonomously define actions that maximize specific reward functions [46]. Van Hoof et al. [47] illustrate such an approach with their tactile in-hand manipulation skills derived from reinforcement learning with tactile feedback. However, as SL can reduce the number of trials needed for sufficient implementation, i.e. also reducing food waste compared to a self-training algorithm, we opted for an LfD approach.
In this study, the prediction of a time-varying sequence of grasping features during the grasping task was performed using Support Vector Regression (SVR) time series. In addition, we tested the SVRs' response to the weighting of selected variables, and demonstrate how the imposition of human-inspired interaction indicators (heat-map weights) affects performance. SVR is one of the most powerful and popular machine learning techniques to deal with non-linear prediction problems. It has a strict, well understood mathematical description, and it is widely used in various applications. Compared to recent NN-based methods, SVR is significantly less data-greedy, meaning that it does not require huge amounts of training data-sets to develop prediction models. Moreover, SVR is very robust to local minima problems. These aspects led us to choose SVR as the machine learning method for this application.

A detailed description of the proposed three variants of SVR techniques is given in the following subsections. Fig. 4 summarizes the process of generating these policies from the acquired data.

### A. Standard SVR

Let $g_{\text{pos}} = [x_s, y_s, z_s, q_s^T]^T \in \mathbb{R}^3 \times \mathbb{H}$ represent the pose of the robotic gripper, where $q_s$ is a unit quaternion. Each robotic finger has nine tactile sensors, five on its proximal phalanx and four on the distal. Let $f_{\text{prs}} \in \mathbb{R}^{27}$ be the vector containing all these tactile readings, $f_{\text{mot}} \in \mathbb{R}^4$ the position of the motors in the robotic hand, and $f_{\text{ang}} \in \mathbb{R}^7$ the joint angles of the exoskeleton. We can now define $s \in \mathbb{S} \subseteq \mathbb{R}^3 \times \mathbb{H} \times \mathbb{R}^{27} \times \mathbb{R}^4 \times \mathbb{R}^7$ as an observed (demonstrated) robotic grasping action performed by a human teacher for a given object. Furthermore, let $a \in \mathbf{A} \subseteq \mathbb{R}^3 \times \mathbb{H} \times \mathbb{R}^4$ be the corresponding action-state, which consists of a gripper pose and motor positions. We define the set of $n$ demonstrations as ordered pairs $\mathcal{D} = \{(s, a)_i : \forall i \in \mathbb{I}_n, s \in \mathbf{S}, a \in \mathbf{A}\}$. The goal is to instruct a policy function $\Pi_0(s; \theta) : \mathbb{S} \to \mathbf{A}$, which is parameterized by $\theta \in \mathbb{R}^d$, using the set of human demonstrations. Let $l: \mathbf{A} \times \mathbb{H} \to \mathbb{R}$ be a loss function between two action-states and $\Pi^*: \mathbb{S}^* \to \mathbf{A}^*$ a robotic grasp policy demonstrated by a human. The objective is to find the parameterization $\theta$ that minimizes the expected loss across a set of human demonstrations, i.e. $\forall(s, a) \in \mathcal{D}^*$:

$$ \arg\min_{\theta} \mathbf{E}[l(\Pi_0(s; \theta), \Pi^*(s))]. \quad (4) $$

### B. SVR with time-series

For the implementation of an LfD policy based on SVR modelling, we used Vapnik's $\mathcal{E}$-insensitive time-series [48], where the aim is to identify a policy $\Pi_0$ by minimization of an $\mathcal{E}$-function. In our case, this function is defined as

$$ l_{\mathcal{E}}(s, a) = \sum_{k=1}^{d} l_{\mathcal{E}_k}(s_k, a_k^*). \quad (5) $$

An SVR with time-series was chosen to generate a policy able to predict grasping actions in the form of action vectors with use of memory,

$$ \hat{a}_i(t + \Delta_t) = f(a_i(t - \alpha), a_i(t - \beta), a_i(t - \gamma), \dots), \quad (6) $$

where $\hat{a}_i$ is the predicted value of the action vector $a_i$ derived from its discrete grasping time-series values, and $\alpha = 0, \beta = 2, \gamma = 5$ are selected time-steps brought into the prediction equation, i.e. action predictions use current time sequence $t$ together with shifted time sequences $(t-2)$ and $(t-5)$ as data input.

We organized and subdivided the data into predefined grasping stages: approach, grasp, lift, placement, and release (see [49], [50] for details). Then, we shifted $g_{\text{pos}}$ to make it relative to the position of the grasped object prior to finally normalizing all the data.

### C. SVR with heat-map weighting

We used heat-maps to represent action vector predictions by weighting the predicted tactile finger data using object-specific values generated by the heat-maps themselves. For a given object, we found little variability among the ten grasps performed by each subject (i.e. grasp practitioner). Although we registered a difference in heat-maps between subjects, this was negligible compared to the much larger differences identified between objects (i.e. the produce). Fig. 2 shows that the distal parts of the fingers experienced the most contact, whilst proximal parts yielded higher inter-object variation. As expected, the palm was more active in subjects with smaller hands, and when grasping larger objects such as the aubergine. However, since the RHR hand is not equipped with sensors on the palm, we did not use the palmar heat-map data.

In practice, we weighted each column of the sensor action vector ($a$ for $f_{\text{prs}} \in \mathbb{R}^{27}$) with its respective heat-map values, represented by the matrix $H$:

$$ \mathbf{a}_{ij}^{*k} = H_j^k \cdot \mathbf{a}_{ij}^k, \quad (7) $$

where $k$ is the object type, $i$ the sequence (row) values, and $j$ a feature of the action vector. As mentioned above, this heat-map weighting was performed only on the sensor-covered area of the RHR hand, i.e., the RHR's sensor placements were overlaid to the generic human hand heat-map generated during direct grasping. The human hand heat-map was thereby segmented into parcels which were averaged to create the matrix $H$ (see also Fig. 4). This procedure enabled us to create a map of human-weighted grasp actions. It privileges the activation of specific tactile sensors according to the areas of the hand most used by the operator during direct interaction with the objects. Of course, this mapping approach only works if the structure of the slave system resembles that of the human hand. Moreover, the more human-like the slave system is, the better we expect this mapping to perform.

## V. EXPERIMENTAL EVALUATION

This section presents the experimental evaluation of our LfD policy for autonomous grasping of compliant food. The execution-validation stage, conducted after the LfD-ML policy is learned, is illustrated on the right-hand side of Fig. 4.

### A. Setup

The setup is comprised of the same slave system described in Sec. III-B and illustrated on the right-hand side of Fig. 3b,
i.e. an RHR hand attached to a Panda manipulator. However, in this case, the slave robot is autonomously controlled by the learned LfD-ML policies and no human is involved in the loop. Feedback data derived from the RHR and the manipulator, such as tactile forces, joint positions, and velocity are continuously provided to the controller. The environment involves seven types of food objects: five (aubergine, bell pepper, lettuce, navel orange, and beef tomato) from the same categories used during training, and two (tangerine and plume tomato) which were not used in the training phase. These two latter objects are significantly different in size, approximately 50% smaller than the navel orange and beef tomato, respectively.

Food items were placed at a designated location within the robot's reachable working space. The location was carefully marked to ensure repeatability. The robot was placed in a starting position above the food item in question. Then, the grasping, lifting, and placement sequences were generated entirely from the autonomous LfD policy under consideration. While here we only focus on the interaction phase, in the future we also plan to consider the approaching phase by using, e.g., visual-servoing techniques to automatically reach the object from any position.

## B. Grasping policies

We carried out grasping experiments taking into account (i) standard human-in-the-loop teleoperation, (ii) standard SVR, (iii-v) three variants of our SVR-based policies, (vi) a recently-published haptic LfD approach, and (vii-viii) two standard impedance controllers:

(i) **Human teleoperation (no automation)**. This approach considers the ground truth grasping data recorded during human teleoperation with the haptic interface (see Sec. III-B). No machine learning is involved here.

(ii) **SVR**. This approach uses normalized data (i.e. forces, positions, velocities) as input and directly maps them into grasping actions through a standard Support Vector Regression algorithm (see Sec. IV-A).

(iii) **SVR-W (weighted as in [21])**. This approach is similar to the SVR above, but it privileges specific RHR sensor data as a mean of improving on the previous algorithm. The weighting is done as in the work of [21], not using any human direct grasping data (i.e. the heat-maps).

(iv) **SVR-WH (weighted using heat-maps)**. This approach is similar to the SVR-W above, but it weights RHR sensor data according to our human-generated heat-maps (see Secs. IV-A and IV-C).

(v) **SVR-WHT (weighted using both heat-maps and time-series)**. Same as SVR-WH, but this approach takes into account the temporal variation in the finger configuration and sensor pressure during the grasping sequence (see Secs. IV-A, IV-B, and IV-C).

(vi) **Misimi et al. [21] (state-of-the-art)**. This approach is a recent learning policy for autonomous grasping, published in 2018. It uses data derived from human teleoperation and tactile sensors on a robotic hand. It differs from our approach because it uses a simple non-haptic joystick during the teleoperation demonstrations, and it does not

<figure><img src="image_12.png" /><figcaption>Fig. 5. Weighting for each algorithm: How these values relate to each other defines the SVRs work-space. Comparison of SVR weights gives an understanding of how they differ and relate to action inputs. For the abbreviations, see Sec. V-B.</figcaption></figure>

consider any human direct grasping information when weighting the tactile data.

(vii) **Impedance controller (gentle grasp)**. This approach implements a standard impedance controller. The robotic hand increases the grasping force until the average force sensed by the tactile sensors reaches a set threshold. We set this threshold to the average value recorded during trials with haptic teleoperation.

(viii) **Impedance controller (strong grasp)**. This approach is the same as (vii), but the threshold is here set to twice the average value recorded during trials with haptic teleoperation.

Fig. 5 summarizes the internal relative weights applied in these policies, illustrating how the different sensory input is valued.

## C. Autonomous Grasping Task

The task consisted in autonomously grasping seven types of food items using the learning policies cited above. We considered five food items (aubergine, bell pepper, lettuce, navel orange, and beef tomato) coming from the same categories of objects used during training, and two objects (tangerine and plume tomato) significantly different in size.

From the 50 teleoperated grasps per produce (see Sec. III-B), it was possible to tailor our four grasping policies, (ii-v), for each produce used in Sec. III-B, e.g. aubergine-trained SVR-WHT, pepper-trained SVR-WHT, lettuce-trained SVR-WHT, etc. This process yielded a total of 4 policies × 5 (trained) objects = 20 tailored policies. Each (trained) produce was grasped 30 times with its tailored policies. E.g., when grasping the aubergine, we used the four learning policies ii-v trained with data registered during the teleoperation of aubergines. To grasp the two food items whose category was not considered during training (tangerine and plume tomato), we used the four policies trained on navel orange and beef tomato, respectively. This task yielded 30 grasps × 7 produce × 4 policies = 840 grasps. For comparison, we also carried out the same task using the three standard controllers (vi-viii), yielding 30 grasps × 7 produce × 3 policies = 630 additional grasps.

In addition to the above grasping repetitions, we also wanted to study the adaptability of the proposed approach for different objects. To do so, we ran a second set of trials, using tailored
## Successfulness of Prehensile Moves

<figure><img src="image_13.png" /><figcaption>Fig. 6. Success rate of grasping tasks. Results are based on 30 grasp attempts per condition per food item. Autonomous grasping using LfD was implemented for a variety of SVRs. SVR with heat-map and time-series inputs had the best accuracy. We compared our approach to a recent state-of-the-art LfD from 2018, and two impedance controller techniques. Grasping variations and their abbreviations are described in Sec. V-B.</figcaption></figure>

SVR-WHT policies to grasp produce different than the one it was designed for. We only considered SVR-WHT policies because we expect them to be most effective. To avoid repeating grasps already considered in our first set of trials, each tailored policy only grasped the items it was not designed for, e.g., aubergine-trained SVR-WHT was used to autonomously grasp all items but aubergine. Each food item was grasped 30 times per policy, yielding 30 grasps × 6 produce × 5 tailored SVR-WHT = 900 grasps.

In total, we carried out 2370 autonomous grasping actions.

## D. Results and Discussion

Fig. 6 shows the average success rate for autonomous grasping actions for the different food items and policies. On the left-hand side, we can find the results of the first set of trials, which uses the four autonomous policies to grasp the seven objects. In this case, each policy was tailored for each food item, using data coming from grasping trials on the considered object only. For comparison, we also included success rates from (i)–human teleoperation, (vi)–Misimi et al., and (vii,viii)–the two impedance controllers. On the right-hand side, we can find the results of the second set of trials, using SVR-WHT policies trained on objects different than the one they grasped in the testing phase.

The left-hand side of Fig. 6 shows that, among the SVR-based approaches implemented, the SVR-weighted policy using heat-map and time-series data (SVR-WHT) was the one with the highest levels of accuracy. This confirmed our original hypothesis: the addition of real human-interaction elements to the grasping policies increases the overall accuracy, which

peaked at 100% for SVR-WHT with all food items. We recall that this result was validated by 30 trial runs for each approach and food item. Both heat-map-based approaches (iv-v) outperformed Misimi et al. in all cases, performing on average 35% better. For SVR-WHT vs. Misimi et al., a related-samples Wilcoxon-signed-rank-test showed a significant difference in the performance ($z = 1.997$ and $p = 0.046$). Considering [21] uses tactile sensing and a recent LfD approach (work published in 2018), this result strongly supports our hypothesis that combining direct hand interaction and high-dexterity haptic teleoperation is a winning technique (these are the two main differences between our work and [21]). For further comparison, we also included two standard impedance controllers. In this case, the robotic hand grasps until a certain threshold force is reached. We considered as thresholds one and two times the average force registered by the hand sensors during the teleoperation trials. The first approach (gentle grasp) did not damage the food, but it failed to successfully grasp the objects. On the other hand, the second approach (strong grasp) showed very good grasping performance. However, the high force applied severely damaged the considered food (see the red triangles in Fig. 6). Although impedance control could provide acceptable results through fine tuning (i.e. finding a custom force threshold for each item), we believe that LfD provides a more effective (and elegant) solution. Indeed, a simple force threshold approach still requires a significant amount of work from the human user, who needs to program the robot for each different set of objects, defeating the main purpose of our work: efficient non-programmatic automation for food processing.
After these first results, we carried out additional trials to study the adaptability of our approaches for other objects. In these trials, we used SVR-WHT to grasp food items different than the one used for the training, e.g. an SVR-WHT trained on the aubergine was used for grasping all other objects. The large difference between our food items enables us to evaluate the adaptability of our SVR algorithms to different situations. As mentioned above, five objects were in the same category as those used during training (aubergine, bell pepper, lettuce, orange, and beef tomato), while two were much smaller (tangerine and plum tomato). The results on the right-hand side of Fig. 6 suggest that we cannot grasp heavy objects with an SVR policy trained with lighter ones, e.g. a policy trained using lettuce data fails to grasp an aubergine. However, the opposite often works quite well, e.g. a policy trained using aubergine data succeeds at grasping the lettuce. Of course, in general, results show that grasping objects using policies trained on other objects only works to a certain extent. As expected, as the difference between the target object and the training one increases, the success rate decreases. This degradation happens faster when grasping heavy objects with policies trained on lighter ones. However, even though it is possible to grasp lighter items (e.g. lettuce) using policies trained on heavier ones (e.g. aubergine), the higher force applied may risk to damage the object.

Finally, Fig. 6 shows that success rates for tangerines and plum tomatoes are quite low in respect to the others, demonstrating that our approach struggles to handle objects significantly smaller than those used during the training. As we are using both the positions and forces of the robotic fingers to learn our policies (see Fig. 5), this is an expected behaviour. In fact, while the combination of these two prevents the hand from squeezing items too hard, it also limits the capability of grasping objects with internally varying sizes (i.e. fruits or vegetables grown with higher ecological diversity). Results show that, if we had the opportunity to training LfD policies for each and every specific type of produce, this combined approach of finger position and tactile force sensing works well (see the high performance on the first five food objects Fig. 6). On the other hand, if we plan to autonomously grasp objects where size and shape span a wider range (i.e. produces is less uniformed/higher diversity) we need to make further adjustments, such as using finger force only. However, finger positions seem important and leaving this information out might not be viable, in regards to damages. Our grasps have all been made blindly, picking up a produce at position X. Adding further information about the produce through machine vision could be one way to unify finger positioning and variations size and shape.

It is also interesting to notice that the success rate for the teleoperation trials is not as high as one might expect (between 0.46 and 0.68). A grasping task was deemed successful only when the user was able to successfully grasp, lift, and move the object. Often the operator was able to grasp and lift the object, but then it slipped during transport. Upon asking, the operator said that it was not easy to always maintain the right amount of force needed for a robust grasp, as she was afraid of damaging the produce.

## VI. CONCLUSION AND FUTURE WORK

In this paper, we have proposed a novel human-inspired, haptic-enabled learning policy designed to handle compliant food items, originating from the agricultural automation domain. We selected soft food objects because they represent an important example of 3D-compliant object manipulation in robotics, which remains poorly researched due to many manipulation challenges.

The paper presents four LfD learning policies that, after training, were capable of gently and autonomously grasping a representative set of compliant produce such as aubergines, bell peppers, lettuces, oranges, and tomatoes. We compared the four different variants of our LfD approach by carrying out a total of 2370 autonomous grasping actions of compliant food objects. The best proposed approach combines (a) data collected during demonstrations using a haptic-enabled high-dexterity teleoperation interface, and (b) heat-map knowledge acquired from humans directly grasping the items in question. Results indicated that the use of tactile data, weighted according to experience gathered during direct interaction trials, yielded the best performance. We also compared our approach with a recent LfD method for grasping compliant objects and two standard impedance controllers. Results showed that our approach performed better, proving our hypothesis—that to properly learn from humans how to grasp challenging, compliant, and fragile objects, it is beneficial to carry out the demonstrations using an intuitive and natural tool for controlling the slave manipulator. Another key aspect of our approach is the inspiration from direct human grasping actions, represented in the LfD model by the use of heat-maps as an input variable, privileging and activating the “right” sensors on the robotic hand.

The proposed approach has a vast range of potential applications in the agricultural and food sectors, as well as in other industries that deal with compliant and fragile objects and where a knowledge of the applied tactile forces is crucial to achieving successful grasping actions. Our work contributes to a better understanding of the input required during the development of grasping ML policies. Furthermore, our aim was to make a contribution to the topic of *LfD through teleoperation*, by highlighting (a) the necessity of improved operator interaction by means of high-fidelity high-dexterity interfaces, and (b) the importance of combining these teleoperation data with barehanded prehensile movements. These results have the potential to advance the capabilities of robotic interaction, combining human puppetry with inherent intentions. We also believe that in the future, a more intuitive interaction system with strong learning capabilities can even void the need of expert human operators pre-training the robots. If the interface is intuitive enough, anyone able to carry out the target task will be able to also train the robot, allowing for an easier deployment.

In future work, we plan to keep improving our teleoperation system. It is of paramount importance to keep providing human operators with intuitive and natural approaches to control the slave manipulator. The more naturally a human operator carries out a task, the more useful the data will be, enabling effective and true learning of the task in question.
Toward this objective, we plan to design a new multisensory
bi-manual teleoperation system that will provide a wide range
of haptic feedback features (kinesthetic, contact pressure, skin
stretch and vibratory), and to extend the range of tasks that
the robotic system can perform. We will also consider other
ways to map the motion of the human operator into that of
the robotic manipulator, so as to understand how different
mapping techniques can affect the performance of the learning
policies. Another important point to further study is the type and
amount of information we need during the training, especially
to understand what is most important when it comes to grasping
objects with a different size than those used during the training.
For this reason, it will be fundamental to also extend the
evaluation to a wider range of compliant objects. Furthermore,
our technique for heat-mapping, though functional for our
small scale, should get a simpler and faster interface in future
works. Sundaram et al. [51] could be one avenue to explore
with a promising technique for collection of digital heat-maps
and human-object-interaction signatures, when their interface
becomes available for research use. Another point to highlight
is that an advanced master interface such as that used here is
rather expensive (around $110,000). However, it is important to
bear in mind that the master interface is only used in the first
part of the learning phase. Once the LfD policy is trained, only
the slave manipulator is required. For this reason, the effective
cost is significantly lower as it spreads over the total number of
autonomous grasps carried out using this approach. Finally, we
also plan to extend the range of learning techniques considered,
e.g., deep learning, ensemble learning, and the range of objects,
e.g., industrial 3D compliant objects, smaller/larger and more
fragile food items such as raspberries, strawberries, and fish
(whole and fillets). We will also compare all performance
metrics using principled statistical approaches, such as repeated-
measures ANOVA.

ACKNOWLEDGMENTS

We would like to express our appreciation for the financial
support given by Research Council of Norway via iPro-
cess (255596) and GentleMAN (299757). We also wish to
acknowledge the help provided through our two laboratories:
MAROLA at SINTEF Ocean and the RAINBOW team at
IRISA/Inria Rennes. Finally, we are particularly grateful for the
assistance given by Elling Ruud Øye and Martin von Heimburg
(SINTEF Ocean AS) for helping in the vision collection and
learning trials, respectively.

REFERENCES

[1] "SoMa: Soft-bodied intelligence for Manipulation," https://cordis.europa.eu/project/rcn/194335_en.html, 2018, [Online; accessed 23-July-2018].

[2] S. Soper and A. Sherma, "Amazon robots poised to revamp how whole foods runs warehouses," https://goo.gl/15tjvY, 2017, [Online; accessed 23-July-2018].

[3] A. Saxena, J. Driemeyer, and A. Y. Ng, "Robotic grasping of novel objects using vision," *The International Journal of Robotics Research*, vol. 27, no. 2, pp. 157-173, 2008.

[4] H. Zhang, X. Zhou, X. Lan, J. Li, Z. Tian, and N. Zheng, "A real-time robotic grasping approach with oriented anchor box," *IEEE Trans. Systems, Man, and Cybernetics: Systems*, 2019.

[5] C. Liu, B. Fang, F. Sun, X. Li, and W. Huang, "Learning to grasp familiar objects based on experience and objects shape affordance," *IEEE Trans. Systems, Man, and Cybernetics: Systems*, vol. 49, no. 12, pp. 2710-2723, 2019.

[6] E. L. Sauser, B. D. Argall, G. Metta, and A. G. Billard, "Iterative learning of grasp adaptation through human corrections," *Robotics and Autonomous Systems*, vol. 60, no. 1, pp. 55-71, 2012.

[7] A. Namiki and M. Ishikawa, "Optimal grasping using visual and tactile feedback," in *Proc. IEEE/SICE/RSJ International Conf. Multisensor Fusion and Integration for Intelligent Systems*, 1996, pp. 589-596.

[8] C. E. Smith and N. P. Papanikolopoulos, "Vision-guided robotic grasping: Issues and experiments," in *Proc. IEEE International Conf. Robotics and Automation*, vol. 4, 1996, pp. 3203-3208.

[9] R. Johansson and G. Westling, "Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects," *Experimental Brain Research*, vol. 56, no. 3, pp. 550-564, 1984.

[10] G. Westling and R. Johansson, "Factors influencing the force control during precision grip," *Experimental Brain Research*, vol. 53, no. 2, pp. 277-284, 1984.

[11] A. Bolopion and S. Régnier, "A review of haptic feedback teleoperation systems for micromanipulation and microassembly," *IEEE Trans. automation science and engineering*, vol. 10, no. 3, pp. 496-502, 2013.

[12] C. Pacchierotti, S. Scheggi, D. Prattichizzo, and S. Misra, "Haptic feedback for microrobotics applications: a review," *Frontiers in Robotics and AI*, vol. 3, no. 53, 2016.

[13] C. Pacchierotti, F. Ongaro, F. Van den Brink, C. Yoon, D. Prattichizzo, D. H. Gracias, and S. Misra, "Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance," *IEEE Trans. automation science and engineering*, vol. 15, no. 1, pp. 290-306, 2018.

[14] M. Aggravi, F. Pausé, P. R. Giordano, and C. Pacchierotti, "Design and evaluation of a wearable haptic device for skin stretch, pressure, and vibrotactile stimuli," *IEEE Robotics and Automation Letters*, vol. 3, no. 3, pp. 2166-2173, 2018.

[15] S. E. Salcudean, S. Ku, and G. Bell, "Performance measurement in scaled teleoperation for microsurgery," in *Proc. First Joint Conf. Computer Vision, Virtual Reality and Robotics in Medicine and Medial Robotics and Computer-Assisted Surgery*, 1997, pp. 789-798.

[16] L. Meli, C. Pacchierotti, and D. Prattichizzo, "Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation," *International Journal of Medical Robotics and Computer Assisted Surgery*, 2017.

[17] L. Moody, C. Baber, and T. N. Arvanitis, "Objective surgical performance evaluation based on haptic feedback," *Studies in health technology and informatics*, pp. 304-310, 2002.

[18] C. Pacchierotti, D. Prattichizzo, and K. J. Kuchenbecker, "Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery," *IEEE Trans. Biomedical Engineering*, vol. 63, no. 2, pp. 278-287, 2016.

[19] F. Chinello, M. Malvezzi, D. Prattichizzo, and C. Pacchierotti, "A modular wearable finger interface for cutaneous and kinesthetic interaction: control and evaluation," *IEEE Trans. Industrial Electronics*, vol. 67, no. 1, pp. 706-716, 2019.

[20] A. G. Billard, S. Calinon, and R. Dillmann, "Learning from humans," in *Springer handbook of robotics*, 2016, pp. 1995-2014.

[21] E. Misimi, A. Olofsson, A. Eilertsen, E. R. Øye, and J. R. Mathiassen, "Robotic handling of compliant food objects by robust learning from demonstration," in *Proc. IEEE/RSJ International Conf. Intelligent Robots and Systems*, vol. 1, 2018.

[22] P. Kormushev, S. Calinon, and D. G. Caldwell, "Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input," *Advanced Robotics*, vol. 25, no. 5, pp. 581-603, 2011.

[23] K. Kukliński, K. Fischer, I. Marhenke, F. Kirstein, V. Maria, D. Sølvason, N. Krüger, and T. R. Savarimuthu, "Teleoperation for learning by demonstration: Data glove versus object manipulation for intuitive robot control," in *Proc. International Congress on Ultra Modern Telecommunications and Control Systems and Workshops*, 2014, pp. 346-351.

[24] P. Evrard, E. Gribovskaya, S. Calinon, A. Billard, and A. Kheddar, "Teaching physical collaborative tasks: object-lifting case study with a humanoid," in *Proc. IEEE-RAS International Conf. Humanoid Robots*, 2009, pp. 399-404.

[25] M. J. Zeestraten, I. Havourtis, and S. Calinon, "Programming by demonstration for shared control with an application in teleoperation," *IEEE Robotics and Automation Letters*, vol. 3, pp. 1848-1855, 2018.

[26] Y. Lin, S. Ren, M. Clevenger, and Y. Sun, "Learning grasping force from demonstration," in *Proc. IEEE International Conf. Robotics and Automation*, 2012, pp. 1526-1531.

[27] P. Kormushev, D. N. Nenchev, S. Calinon, and D. G. Caldwell, "Upper-body kinesthetic teaching of a free-standing humanoid robot," in *Proc. IEEE International Conf. Robotics and Automation*, 2011, pp. 3970-3975.
[28] C. Schou, J. S. Damgaard, S. Bogh, and O. Madsen, "Human-robot interface for instructing industrial tasks using kinesthetic teaching," in *Proc. International Symposium on Robotics*, 2013, pp. 1-6.

[29] E. Knoop, M. Bächer, V. Wall, R. Deimel, O. Brock, and P. Beardsley, "Handshakiness: Benchmarking for human-robot hand interactions," in *Proc. IEEE/RSJ International Conf. Intelligent Robots and Systems*, 2017, pp. 4982-4989.

[30] E. Hyttinen, D. Kragic, and R. Detry, "Learning the tactile signatures of prototypical object parts for robust part-based grasping of novel objects," in *Proc. IEEE International Conf. Robotics and Automation*, 2015, pp. 4927-4932.

[31] S. Luo, J. Bimbo, R. Dahiya, and H. Liu, "Robotic tactile perception of object properties: A review," *Mechatronics*, vol. 48, pp. 54-67, 2017.

[32] M. Stachowsky, T. Hummel, M. Moussa, and H. A. Abdullah, "A slip detection and correction strategy for precision robot grasping," *IEEE/ASME Trans. Mechatronics*, vol. 21, no. 5, pp. 2214-2226, 2016.

[33] M. C. Gemici and A. Saxena, "Learning haptic representation for manipulating deformable food objects," in *Proc. IEEE/RSJ International Conf. Intelligent Robots and Systems*, 2014, pp. 638-645.

[34] N. Kamakura, M. Matsuo, H. Ishii, F. Mitsuboshi, and Y. Miura, "Patterns of static prehension in normal hands," *American Journal of Occupational Therapy*, vol. 34, no. 7, pp. 437-445, 1980.

[35] P. Menesatti, C. Angelini, F. Pallottino, F. Antonucci, J. Aguzzi, and C. Costa, "Rgb color calibration for quantitative image analysis: the 3d thin-plate spline warping approach," *Sensors*, vol. 12, no. 6, pp. 7063-7079, 2012.

[36] F. Abi-Farraj, C. Pacchierotti, O. Arenz, G. Neumann, and P. R. Giordano, "A haptic shared-control architecture for guided multi-target robotic grasping," *IEEE Trans. Haptics*, vol. 13, no. 2, pp. 270-285, 2019.

[37] R. Rahal, G. Matarese, M. Gabiccini, A. Artoni, D. Prattichizzo, P. Giordano, and C. Pacchierotti, "Caring about the human operator: haptic shared control for enhanced user comfort in robotic telemanipulation," *IEEE Trans. Haptics*, 2020.

[38] Q. Yuan, C.-Y. Weng, F. Suárez-Ruiz, and I.-M. Chen, "Flexible telemanipulation based handy robot teaching on tape masking with complex geometry," *Robotics and Computer-Integrated Manufacturing*, vol. 66, p. 101990, 2020.

[39] J. Bimbo, C. Pacchierotti, M. Aggravi, N. Tsagarakis, and D. Prattichizzo, "Teleoperation in cluttered environments using wearable haptic feedback," in *Proc. IEEE/RSJ International Conf. Intelligent Robots and Systems (IROS)*, 2017, pp. 3401-3408.

[40] M. Selvaggio, F. Abi-Farraj, C. Pacchierotti, P. R. Giordano, and B. Siciliano, "Haptic-based shared-control methods for a dual-arm system," *IEEE Robotics and Automation Letters*, vol. 3, no. 4, pp. 4249-4256, 2018.

[41] J. R. Napier, "The prehensile movements of the human hand," *The Journal of bone and joint surgery. British volume*, vol. 38, no. 4, pp. 902-913, 1956.

[42] A. Pettersson, T. Ohlsson, S. Davis, J. Gray, and T. Dodd, "A hygienically designed force gripper for flexible handling of variable and easily damaged natural food products," *Innovative Food Science & Emerging Technologies*, vol. 12, no. 3, pp. 344-351, 2011.

[43] D. M. Barrett, J. C. Beaulieu, and R. Shewfelt, "Color, flavor, texture, and nutritional quality of fresh-cut fruits and vegetables: desirable levels, instrumental and sensory measurement, and the effects of processing," *Critical reviews in food science and nutrition*, vol. 50, no. 5, pp. 369-389, 2010.

[44] S. Schaal, "Learning from demonstration," in *Advances in neural information processing systems*, 1997, pp. 1040-1046.

[45] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, "A survey of robot learning from demonstration," *Robotics and autonomous systems*, vol. 57, no. 5, pp. 469-483, 2009.

[46] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep reinforcement learning," *Nature*, vol. 518, no. 7540, p. 529, 2015.

[47] H. Van Hoof, T. Hermans, G. Neumann, and J. Peters, "Learning robot in-hand manipulation with tactile features," in *Proc. IEEE-RAS International Conf. Humanoid Robots*, 2015, pp. 121-127.

[48] K.-R. Müller, A. J. Smola, G. Rätsch, B. Schölkopf, J. Kohlmorgen, and V. Vapnik, "Predicting time series with support vector machines," in *Proc. International Conf. Artificial Neural Networks*, 1997, pp. 999-1004.

[49] E. Misimi, E. R. Øye, A. Eilertsen, J. R. Mathiassen, O. B. Åsebø, T. Gjerstad, J. Buljo, and Ø. Skotheim, "Gribbot-robotic 3d vision-guided harvesting of chicken fillets," *Computers and Electronics in Agriculture*, vol. 121, pp. 84-100, 2016.

[50] V. Lippiello, B. Siciliano, and L. Villani, "Position-based visual servoing in industrial multirobot cells using a hybrid camera configuration," *IEEE Trans. Robotics*, vol. 23, no. 1, pp. 73-86, 2007.

[51] S. Sundaram, P. Kellnhofer, Y. Li, J.-Y. Zhu, A. Torralba, and W. Matusik, "Learning the signatures of the human grasp using a scalable tactile glove," *Nature*, vol. 569, no. 7758, pp. 698-702, 2019.

<figure><img src="image_14.png" /></figure>

**Aleksander Lillienskiold** received a MSc in Technical Cybernetics from the Norwegian University of Science and Technology (NTNU) in Trondheim, Norway, in 2012. Currently, he is a researcher at SINTEF Ocean, working in robotics and automation for the food processing industries, and is a PhD candidate at the Norwegian University of Life Sciences (NMBU), working on gentle handling and dexterous grasping of food.

<figure><img src="image_15.png" /></figure>

**Rahaf Rahal** (S'15) is a PhD candidate in the Rainbow team at IRISA/Inria Rennes, France. She received her B.E. in Mechanical Engineering from the American University of Beirut, Lebanon, in 2015, then her M.E. in Mechanical Engineering from the same university in 2017, with focus on robotics and computer vision. Her current research interests lie in design of haptic-based shared control methods, and use of haptic feedback for safe and intuitive robotic teleoperation.

<figure><img src="image_16.png" /></figure>

**Paolo Robuffo Giordano** (M'08 - SM'16) received his MSc in Computer Science Engineering in 2001, and his PhD in Systems Engineering in 2008, both from the University of Rome "La Sapienza". In 2007 and 2008 he spent one year as a Postdoc at the Institute of Robotics and Mechatronics of the German Aerospace Center (DLR), and from 2008 to 2012 he was Senior Research Scientist at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany. He is currently a senior CNRS researcher head of the Rainbow group at IRISA and Inria in Rennes, France. He received the 2018 IEEE Robotics and Automation Letters best paper award, and he is Editor of the IEEE Transactions on Robotics.

<figure><img src="image_17.png" /></figure>

**Claudio Pacchierotti** (S'12, M'15, SM'20) is a tenured researcher at CNRS in Rennes, France. He was previously a postdoctoral researcher at Dept. of Advanced Robotics at the Italian Institute of Technology, Genova, Italy. Pacchierotti earned his PhD at the University of Siena in 2014. He visited: the Penn Haptics Group at University of Pennsylvania in 2014, Dept. of Innovation in Mechanics and Management at University of Padua in 2013, and the Institute for Biomedical Technology and Technical Medicine (MIRA) at University of Twente in 2014. Pacchierotti received the 2014 EuroHaptics Best PhD Thesis Award for the best doctoral thesis in the field of haptics. He is Chair of the IEEE Technical Committee on Haptics and Secretary of the Eurohaptics Society.

<figure><img src="image_18.png" /></figure>

**Ekrem Misimi** received his PhD in Engineering Cybernetics at Norwegian University of Science and Technology (NTNU) in Trondheim, Norway, in 2007—since then he has been at SINTEF Ocean, and as a Senior Research Scientist he conducts research in computer vision, machine learning and robot learning targeting robotic manipulation of 3D compliant objects. He leads several relevant research-driven projects on robotic manipulation of 3D compliant objects, such as iProcess (www.iprocessproject.com) and GentleMAN. The focus is on developing advanced robotic perception, control, and learning frameworks for manipulation of compliant objects. He is currently co-leading GEMINI center ROBOTICS in Trondheim, a joint SINTEF-NTNU collaboration center, focusing on robotic grasping and manipulation.