# Haptic-enabled decentralized control of a heterogeneous human-robot team for search and rescue in partially-known environments

Marco Aggravi, Ahmed Alaaeldin Said Elsherif, Paolo Robuffo Giordano,
Claudio Pacchierotti

► To cite this version:

Marco Aggravi, Ahmed Alaaeldin Said Elsherif, Paolo Robuffo Giordano, Claudio Pacchierotti. Haptic-enabled decentralized control of a heterogeneous human-robot team for search and rescue in partially-known environments. IEEE Robotics and Automation Letters, 2021, 6 (3), pp.4843-4850. 10.1109/LRA.2021.3067859. hal-03161639

HAL Id: hal-03161639

https://inria.hal.science/hal-03161639

Submitted on 7 Mar 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Haptic-Enabled Decentralized Control of a Heterogeneous Human-Robot Team for Search and Rescue in Partially-known Environments

Marco Aggravi<fnref n="1" />, Ahmed Alaaeldin Said Elsherif<fnref n="1" />, Paolo Robuffo Giordano<fnref n="1" />, and Claudio Pacchierotti<fnref n="1" />

***Abstract***—Teams of coordinated robots have been proven useful in several high-impact applications, including urban search and rescue (USAR) and disaster response. In this context, we present a decentralized haptic-enabled connectivity-maintenance control framework for heterogeneous human-robot teams. The proposed framework controls the coordinated motion of a team consisting of mobile robots and one human, for collaboratively achieving various exploration and SAR tasks. The human user physically becomes part of the team, moving in the same environment of the robots, while receiving rich haptic feedback about the team connectivity and the direction toward a safe path. We carried out two human subjects studies, both in simulated and real environments. Results show that the proposed approach is effective and viable in a wide range of SAR scenarios. Moreover, providing haptic feedback showed increased performance w.r.t. providing visual information only. Finally, conveying distinct feedback regarding the team connectivity and the path to follow performed better than providing the same information combined together.

***Index Terms***—Human-Robot Teaming; Haptics and Haptic Interfaces; Multi-Robot Systems.

## I. INTRODUCTION

GROUND Urban Search and Rescue (USAR) robots are already widely used in disaster scenarios [1], [2], while aerial solutions are only recently raising interest [3], [4]. As disaster environments are generally highly dynamic and unstructured, it is crucial to take advantage of the experience of human rescuers, enabling them to control the robotic USAR systems in a reactive and intuitive manner [1], [5].

This work presents an innovative approach to deal with USAR situations, taking advantage of the promising capabilities of coordinated multi-robot systems and wearable haptic interfaces. We propose a haptic-enabled connectivity-maintenance framework able to manage the coordinated motion of a team composed of mobile robots and one human. The human user physically becomes part of the team, exploring/navigating the environment together with the robots. At the same time, she/he receives wearable haptic feedback about the team connectivity level *and/or* the direction towards a safe path. The connectivity information is generated by a decentralized connectivity-maintenance algorithm that controls the motion of the robots for keeping the team connected at all times [6].

The navigation information is generated from an a priori map of the environment. The user is always free to decide whether and to what extent to follow the provided navigation cues, as to account for the possible changes in the actual map of the environment.

## II. RELATED WORKS

***Heterogeneous human-robot teams:*** We can identify two main interaction paradigms for achieving an effective interaction between humans and machines: *one-way*, where the human provides the robots with some local information (e.g., his or her position) [7], [8], and *two-way*, where the human shares again some local information with the robots but also receives back information regarding their states. These interactions can be carried out either through physical [9] or non-physical [10], [11] contacts between robots and operator.

***Multi-robot control:*** A recent survey presents an ample overview on target detection and tracking for multi-robot systems [12], while other surveys on multi-robot organization/coordination are available at [13], [14]. More recently, Hashemipour et al. [15] described how to optimize first-response robotic teams in terms of their size, teamwork skills, and robot reliability. Results show that clustering microtasks could improve operation performance. The opportunity of using multi-robot teams in SAR operations has also captured the attention of our Institutions, e.g., the European Union funded the collaborative projects ICARUS, SHERPA, and AirBorne to develop various robotic tools able to assist human-led intervention teams.

***Path planning:*** This field is very active and has evolved to consider increasingly complex scenarios, cluttered environments, and fast-changing maps [16], [17], [18]. For example, Kulich et al. [16] extended the Watchman Route Problem (i.e., find the minimal closed route such that every point of the environment can be seen from at least one point along the route) for fully-automatic mission planning and control of heterogeneous teams. More recently, Solana et al. [18] used an T-RRT algorithm for pre-planning the connected paths of a formation of robots using the generalized connectivity-maintenance theory of [6].

***Human guidance:*** Of course, as we want to leave users free to make their own decisions, it is important to provide them with timely and effective information, regarding the status of the team and/or a safe direction to follow. Audio cues have often been used to provide guidance cues, especially in those situations where vision is compromised [19]. However, as sensory limitations might affect both visual and auditory capabilities, haptic feedback is very promising in our context. Indeed, the haptic sense is spread throughout the body and

Manuscript received: October 15, 2020; Revised: January 4, 2021; Accepted: February 18, 2021.

This paper was recommended for publication by Editor Jingang Yi upon evaluation of the Associate Editor and Reviewers' comments.

This work was supported by the ANR, Project ANR-20-CHIA-0017-01 "MULTISHARED".

Digital Object Identifier (DOI): see top of this page.

<footnote n="1">¹M. Aggravi, A. Alaaeldin Said Elsherif, P. Robuffo Giordano, and C. Pacchierotti are with CNRS, Univ Rennes, Inria, IRISA, Rennes, France. E-mail: email: {marco.aggravi, claudio.pacchierotti, prg}@irisa.fr</footnote>
<figure><img src="image_1.png" /><figcaption>Fig. 1. Block diagram. Given the environment map $\mathcal{M}$ and the initial/goal positions for the human agent ($\mathbf{x}_{start}$ and $\mathbf{x}_{goal}$), an offline planner (see Sec. III-B) computes the list $\Gamma$ of all possible paths from $\mathbf{x}_{start}$ to $\mathbf{x}_{goal}$. During the task, from all the current agents positions $\mathbf{x}$, our algorithm computes the best connectivity-maintenance control action $\mathbf{F}_{i}^{\lambda}$ (see (2)), the best path to the goal $\gamma^{*}$, and the best next tile in $\gamma^{*}$ for the human agent to reach, $\mathcal{M}_{ij}^{*}$, resulting in $\mathbf{F}_{p}$ (see (4)). Both forces are then sent to the feedback devices (see Sec. IV-A4), informing the human agent about the formation connectivity level and/or the direction of the path to follow.</figcaption></figure>

can process a large amount of information, making it a truly reliable and sensible choice for USAR scenarios. A variety of wearable devices have been developed for providing guidance feedback to humans, including bracelets and armbands [20], belts [21], and shoes [22].

With respect to this existing literature, the contributions of this work are the following: while there has been some work combining mobile robots and humans in USAR operations [3], this is the first time haptic feedback is employed in this scenario, which we believe can significantly improve the understanding of the robotic actions by the human operator. Moreover, we also use wearable haptics in this context for the first time, which is a paramount aspect to guarantee the mobility of the human user within the team. Finally, with respect to [10], [11], [20], here we provide multiple pieces of information through the same haptic sensory channel, which is a first for mobile robotics applications.

### III. METHODOLOGY

Our control methodology consists of three main parts: connectivity-maintenance control for the team of robots (Sec. III-A), path planning for the human agent (Sec. III-B), and information feedback on the team connectivity and go-to- goal directions (Sec. III-C). The general architecture of the system is summarized in Fig. 1.

#### A. Connectivity-maintenance control

We consider a group of $N$ robots modeled as 3D-point masses with positions $\mathbf{x}_i \in \mathbb{R}^3$, $i \in \{1, \dots, N\}$, and double integrator dynamics

$$ \ddot{\mathbf{x}}_i = \mathbf{u}_i, \quad i \in \{1, \dots, N\}, \qquad (1) $$

with $\mathbf{u}_i \in \mathbb{R}^3$ being the force (control action) applied to robot $i$, and $\mathbf{v}_i = \dot{\mathbf{x}}_i$ its velocity. Each robot is subject to

the connectivity-maintenance control described in [6]. This control action steers each robot using a gradient-descent strategy that guarantees connectivity maintenance of the sensing/communication graph $\mathcal{G}$ representing the formation. This connectivity maintenance strategy can account for several constraints, e.g., avoiding obstacles and collisions, keeping a non-occluded line-of-sight visibility, and keeping a desired distance w.r.t. neighbors. These constraints are encoded, via suitable weights, into the adjacency matrix $\mathbf{A} \in \mathbb{R}^{N \times N}$ associated to the graph $\mathcal{G}$ [6], from which one can compute the so-called Laplacian matrix and its second smallest eigenvalue $\lambda_2$ that acts as a connectivity metric: $\lambda_2 > 0$ for a connected graph and $\lambda_2 = 0$ otherwise [23].

The connectivity-maintenance control consists then in a decentralized gradient descent of a potential function $V^{\lambda}(\lambda_2(\mathbf{x}(t))) \ge 0$ of the connectivity eigenvalue $\lambda_2(\mathbf{x}(t))$. In what follows, w.l.o.g. we denote $\lambda_2(\mathbf{x}(t))$ simply as $\lambda_2(t)$. The potential $V^{\lambda}(\lambda_2(t))$ is designed so as to have a vertical asymptote at $\lambda_2^{\min} > 0$, to smoothly decrease from $\lambda_2^{\min}$ to $\lambda_2^{\max} = \lambda_2^{\min} + \Delta$, and to vanish for $\lambda_2(t) \ge \lambda_2^{\max}$. The connectivity maintenance control is then designed as the gradient descent

$$ \mathbf{F}_{i}^{\lambda} = - \frac{\partial V^{\lambda}(\lambda_{2}(t))}{\partial \mathbf{x}_{i}}, \qquad (2) $$

whose evaluation by robot $i$ can be made decentralized by exploiting a decentralized estimation $\hat{\lambda}_2^i(t)$ of the true value $\lambda_2(t)$, see [6]. The control input for robot $i$ in (1) is then finally designed as

$$ \mathbf{u}_i = \mathbf{F}_i^{\lambda} - \mathbf{B}_i \mathbf{v}_i + \mathbf{F}_i^e, \qquad (3) $$

where $\mathbf{B}_i \in \mathbb{R}^{3 \times 3}$ is a damping term (for stability purposes), and $\mathbf{F}_i^e \in \mathbb{R}^3$ is an additional exogenous force that may act on robot $i$. This latter exogenous input is not used in this work ($\mathbf{F}_i^e = 0$ for all $i$) but it can be used to enable, e.g., the external control of one or more robots by a remote human operator [24]. In presence of bounded external forces $\mathbf{F}_i^e$, the connectivity-maintenance control (3) is guaranteed to maintain $\lambda_2(t) > \lambda_2^{\min} > 0, \forall t$. The human agent, referred to with $i = h$, is considered as a ground mobile robot with elevation 0 m and subject to no control inputs (see Sec. III-B), i.e., his or her motion cannot be directly controlled by the algorithm.

#### B. Grid covering and path planning for the human agent

With the connectivity of the team guaranteed by the above control technique, we want to guide the human agent towards the goal. Recent studies show that the kinematics of the human locomotion resembles that of a unicycle robot [25], and that it is possible to provide the human user with guidance along a predefined path using tactile feedback [26]. Of course, although we can guide human agents with high accuracy [26], deviations from the commanded path will inevitably occur. These deviations can be due to users not being able to fully understand and/or react to the haptic guidance, or due to users willingly moving away from the path because of unforeseen circumstances (e.g., unexpected obstacles blocking the indicated path). Our planning technique aims at always providing the best guidance to the human users, from their current position
<figure><img src="image_2.png" /><figcaption>Fig. 2. Grid covering and path planning. The map is discretized in a grid of resolution $r_{grid}$. Tiles occupied by obstacles (black) are discarded from the planning (gray). The path planning algorithm generates paths from each tile to the goal. The best path $\gamma^*$ will always be the shortest one from the current position of the human agent to the goal, in terms of number of covered tiles. Enlargement: From the current position of the user $\mathbf{x}_h$, the algorithm identifies the center of the next tile to be reached, $M_{ij}^*$, on the current optimal path $\gamma^*$. The difference between these two positions $\mathbf{F}_p$ (see (4)) provides the user with information about the path to follow, while the connectivity force $\mathbf{F}_i^\lambda$ (see (2)) informs the user about the connectivity level of the formation.</figcaption></figure>

to the goal, while however leaving them free to deviate from the planned path if needed.

We designed our planning inspired from the Ford Fulkerson flow network method [27], which retrieves the maximal steady state flow from an origin/start point to a terminal/goal one (maxflow on a network problem). Below we include all the details of our path planning algorithm.

Let us consider a known map of the environment $\mathcal{M}$, which describes the area surrounding the team, and let us divide $\mathcal{M}$ in a grid of resolution $r_{grid} \in \mathbb{R}$. Every tile on the map $\mathcal{M}_{ij}$, $i \in \{1, \dots, nr_{grid}\}$, $j \in \{1, \dots, mr_{grid}\}$, $n, m \in \mathbb{N}$, can be free or occupied (e.g., by obstacles), as shown in Fig. 2. W.l.o.g., we consider squared maps ($n = m$). Using this notation, we can define the starting point for the algorithm $\mathbf{x}_{start} \in \mathcal{M}$ as the tile occupied by the starting position of the human agent and the goal location as the tile occupied by the goal position $\mathbf{x}_{goal} \in \mathcal{M}$.

To account for any deviation of the human user w.r.t. the planned path, we compute valid paths from each tile of the map to the goal. To do so in an efficient fashion, we start by dividing $\mathcal{M}$ in large regions of resolution $r_{reg}$, such that every region on the map includes a certain number of tiles, i.e., $r_{reg} = qr_{grid}$, $q \in \mathbb{N}$. Then, we use a Breadth First Search (BFS) algorithm to compute the shortest path $\gamma_1$, going from the initial location of the user $\mathbf{x}_{start}$ to the goal $\mathbf{x}_{goal}$. We store $\gamma_1$ in a list of paths $\Gamma$ and we mark each tile that it covered. After that, we randomly select a new starting tile in the region that includes the least number of covered tiles, and we compute the new shortest path $\gamma_2$ using the BFS algorithm. Again, we store $\gamma_2$ in $\Gamma$ and we mark each tile that it covered. We proceed iteratively in this way until all the tiles in $\mathcal{M}$ are covered. This process has to be run only once per map.

With the map fully covered and all elements (paths) of $\Gamma$ ending at the goal, the best path will always be the shortest one from the current position of the human to the goal, in

terms of number of covered tiles. Of course, our framework is compatible with any other similar path planning technique.

As the human only relies on local information to compute where the best path is headed (i.e., its own position and the list of paths $\Gamma$), the overall framework is still decentralized.

### C. Feedback to the local/remote rescuer

Finally, we need to provide the human rescuer with meaningful information about (i) the current path to follow (Sec. III-B) and (ii) the status of the team (Sec. III-A), which are both important pieces of information. The former enables rescuers to successfully reach the final goal, while the latter informs them about the connectivity level of the team (e.g., so that they can slow down if the team is not able to keep up with the current moving pace). Being informed about the connectivity of the team has been already proven useful when teleoperating multiple drones [28]. The way of providing this information can be tailored to the experience of the users (e.g., expert rescuers or civilians), as detailed in our experimental Section.

To provide navigation information, let us define $M_{ij}^* \in \mathbb{R}^2$ as the center of the next tile along the current selected path<fnref n="1" /> $\gamma^*$, which passes from the human current position $\mathbf{x}_h$ (see enlargement in Fig. 2). The elevation of the human is set to zero and thus, for simplicity, the third dimension of its position vector is not considered here, i.e., $\mathbf{x}_h \in \mathbb{R}^2$. The resulting difference force

$$ \mathbf{F}_p = \kappa_p (M_{ij}^* - \mathbf{x}_h) \quad (4) $$

comprises information about where the human agent should go, $\theta_p = atan((M_{ij,y}^* - \mathbf{x}_{h,y})/(M_{ij,x}^* - \mathbf{x}_{h,x}))$, and how close he or she is to the center of the next tile. $\kappa_p \in \mathbb{R}^+$ is a positive gain; in our application, $\kappa_p = 1$ N/m, but it can be adjusted to reflect how much we let the operator deviate from the target path. Quantities $\theta_p$ and $d_p$ will be used in Sec. IV-A4 for providing visual or haptic feedback on the direction to follow.

For what concerns the formation status, we want to provide the rescuer with information about the team connectivity. Let us recall from Sec. III-A that every robotic agent in the team is controlled following (3), where $\mathbf{F}_i^\lambda$ is the connectivity force steering each agent $i$ to a more connected state. Of course, we cannot directly control the motion of the human as we do for the robots. Nonetheless, we can still suggest $\mathbf{F}_h^\lambda$ through visual or haptic feedback (we use index $i = h$ for the human agent). This force will be used in Sec. IV-A4 for designing visual or haptic feedback techniques, so that human agents are informed about the effect of their movements on the connectivity of the formation.

## IV. EXPERIMENTAL EVALUATION

### A. Human subjects experiment in a simulated environment

1) Context: Imagine a post-disaster situation in which a human civilian survivor is found by the robotic team and escorted out of the area. As soon as the robots locate the survivor, they release two vibrating bands that the human agent wears on the right and left arms. Then, an expert rescuer

<footnote n="1">¹Here we assume that the computing unit worn/brought by the human agent is able to localize the agent and sort the path in $\Gamma$ for obtaining $M_{ij}^*$.</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 3. (top) In the simulated environment, subjects play the remote expert rescuer; they move the virtual civilian survivor toward the goal, acting on its linear and angular velocities ($v_h$ and $\omega_h$) and following the feedback provided by the system. (bottom) In the real environment, the subject plays the in-field expert rescuer; they move toward the goal location following the feedback information provided by the system.</figcaption></figure>

remotely guides the survivor along a safe path, providing him or her with navigation information by manually activating the armbands. In turn, the rescuer is provided with feedback information about the best path to follow and the connectivity of the team (see Fig. 3 top). Navigation information to the remote rescuer is generated from a priori map of the environment, as described in Sec. III-A, while the connectivity metrics is generated from the connectivity-maintenance algorithm, as summarized in Sec. III-B. Finally, how this feedback information is conveyed to the rescuer is described in Sec. III-C. In this experiment, we assume that the survivor can closely follow the indications of the remote expert rescuer using two vibrotactile armbands, as already demonstrated in [26], [29].

The non-expert human agent in the field only needs to follow simple navigation cues, while the expert remote agent is provided with rich information enabling him or her to control the situation. We believe this point to be rather relevant for the considered application, as frightened non-trained civilians cannot be asked to complete or understand complex commands and tasks. The role of the robotic team is also quite important. Ground mobile robots can carry medical and food supplies, while aerial robots can provide the rescuer with a view of the environment and look for other survivors.

2) Virtual environment and task: We consider two different virtual scenes, S1 and S2, both composed of various urban elements, including buildings and open walls, as shown in the video (also available at https://youtu.be/N73QUFqpSvo). We also consider two variations of each scene. One with no changes with respect to the a priori map known by the system ($S1_{free}$ and $S2_{free}$) and one with unexpected obstacles blocking some routes ($S1_{occ}$ and $S2_{occ}$). These unexpected obstacles are unknown to the system and therefore not accounted for in the path planning of Sec. III-B. We consider these variations to simulate situations in which the planner has only access to an outdated version of the map, which might not perfectly reflect the current status of the post-disaster environment.

Our heterogeneous team is composed of five virtual agents: one human survivor, one ground mobile robot, and three quadro-tors. The virtual scene is displayed to the user through an LCD screen.

<table><caption>TABLE I
LEVELS OF VIBROTACTILE STIMULI DEPENDING ON $d_\theta$</caption><thead><tr><th>Range $d_\theta$ (deg)</th><th>Left armband vibration level [G]</th><th>Range $d_\theta$ (deg)</th><th>Right armband vibration level [G]</th></tr></thead><tbody><tr><td>[0, 20]</td><td>0</td><td>[0, -20]</td><td>0</td></tr><tr><td>[21, 60]</td><td>0.6</td><td>[-21, -60]</td><td>0.6</td></tr><tr><td>[61, 100]</td><td>0.9</td><td>[-61, -100]</td><td>0.9</td></tr><tr><td>[101, 180]</td><td>1.2</td><td>[-101, -180]</td><td>1.2</td></tr></tbody></table>

Subjects play the role of the remote rescuer, and their task consists in moving the simulated human survivor/virtual agent toward the goal, following the feedback information provided by the system.

3) Control interface: Subjects control the motion of the virtual agent either through (i) an off-the-shelf joypad Logitech Gamepad F310, or (ii) a grounded haptic device Novint Falcon. When using the joypad, the right analog thumbstick commands a linear velocity $v_h \in \mathbb{R}$ to the virtual agent, whereas the left thumbstick rotates it with angular velocity $\omega_h \in S$. When using the haptic device, the Falcon is used to control the velocity of the virtual agent. A displacement of the end-effector along the negative z-axis (w.r.t. its initial position, see Fig. 4) commands a linear velocity $v_h \in \mathbb{R}$, whereas a similar displacement along the negative x-axis rotates it with angular velocity $\omega_h \in S$.

As discussed in Sec. IV-A1, we assume that the virtual agent perfectly follows the navigation cues conveyed by the remote expert operator. For this reason, as to simplify this interaction and w.l.o.g., we have enabled the operator to directly control its velocities, as detailed above.

4) Experimental modalities: We consider four experimental conditions (see Fig. 4):

(VC) the joypad controls the human agent motion, and one arrow provides combined information about the connectivity force and the path direction;

(VS) the joypad controls the human agent motion, and two separate arrows provide information about the connectivity force and the path direction, respectively;

(HC) the Novint Falcon controls the human agent motion, and kinesthetic feedback provides combined information about the connectivity force and the path direction;

(HS) the Novint Falcon controls the human agent motion, kinesthetic feedback provides information about the connectivity force, and wearable vibrotactile feedback provides information about the path direction.

Let us recall from Sec. III-C that $F_h^\lambda$ is the connectivity force applied to the user and $F_p$ is the guidance force steering the user along the current best path.

In condition VC, the user receives feedback about the weighted sum of the two forces

$$F_t = 3 \left( \frac{F_p}{\|F_p\|} + \frac{F_h^\lambda}{\|F_h^\lambda\|} \right), \quad (5)$$

represented as a single cyan arrow, starting from the position of the human figure, with a max intensity of 6 N. In condition VS, the remote expert receives feedback about the two forces separately, represented as two distinct arrows, both starting from the position of the virtual agent: red for $F_h^\lambda$ and white for $F_p$ (see also Fig. 2). In condition HC, the user receives feedback about the weighted sum of the two forces (as per (5)),
<figure><img src="image_4.png" /><figcaption>Fig. 4. Feedback modalities. We provide feedback about the path to follow, $\mathbf{F}_p$, and the connectivity of the team, $\mathbf{F}_h^\lambda$, in these ways: (VC) the weighted sum of the two forces, $\mathbf{F}_p$ and $\mathbf{F}_h^\lambda$, is shown with a single arrow, (VS) the two forces are shown as a two distinct arrows, (HC) the weighted sum of the two forces is provided by the Falcon device as force feedback; (HS) $\mathbf{F}_p$ is conveyed by the wearable armbands as vibrotactile feedback while $\mathbf{F}_h^\lambda$ is provided by the Falcon device as force feedback.</figcaption></figure>

represented as a force feedback produced by the Falcon device. In condition HS, the remote expert receives feedback about the two forces separately, i.e., $\mathbf{F}_h^\lambda$ conveyed by the Falcon device as kinesthetic feedback and $\mathbf{F}_p$ conveyed by wearable vibrotactile feedback. These latter stimuli are provided via a pair of custom haptic armbands, worn on the right and left upper arms. Each armband comprises four 307-100 Pico Vibe vibrating motors, positioned evenly around the arm at 90 degrees from each other. Each armband vibrates with frequency [0 – 255] Hz and amplitude [0.2 – 1.2] G in square-wave-like vibration patterns of $\nu \in \mathbb{R}^+$, i.e., it vibrates for a period of $2\nu$ ms with duty cycle 50%. The four motors of one armband always vibrate together. W.r.t. Sec. III-C, let us recall the direction of the next point-to-be-reached on the path $\theta_p$ (see (4) and the following definitions) and let us call the human orientation w.r.t. a world frame $\theta_h$. To efficiently convey direction information, we define four different stimulus ranges on $d_\theta = \theta_p - \theta_h$ (see Table I and Fig. 2): no feedback (when $|d_\theta| \in [0-20]$ deg), weak feedback (21-60 deg), medium feedback (61-100 deg), and strong feedback (101-180 deg). The sign of $d_\theta$ (positive for left turning, negative for right turning) tells us which armband to activate. A similar guidance approach has been already successfully employed in [11], [26].

**Subjects:** Twelve participants took part to our experiment, including one woman and eleven men (age 23–32 years old). Users performed one repetition of the task per feedback condition per scenario, yielding a total of 4 experimental modalities × 4 scenes = 16 trials per user and 192 trials in total. Operators were asked to complete the task as fast as

<table><caption>TABLE II
HUMAN SUBJECTS EXPERIMENT</caption><tbody><tr><td><strong>Conditions</strong></td><td colspan="3">Interface = V (joypad/visual), H (haptic)<br>Feedback type = C (combined), S (separated)<br>Variation of the scene =<br>free (map fully known), occ (map partially unknown)</td></tr><tr><td colspan="4"><strong>Statistical analysis (repeated-measures ANOVA, a = 0.05)</strong></td></tr><tr><td><strong>Completion time</strong> (three-way)</td><td>V vs. H<br>free vs. occ</td><td>p = 0.005<br>p = 0.004</td><td>C vs. S<br>p = 0.001</td></tr><tr><td><strong>Path length</strong> (three-way)</td><td>V vs. H<br>free vs. occ</td><td>p = 0.001<br>p &lt; 0.001</td><td>C vs. S<br>p = 0.024</td></tr><tr><td><strong>Velocity of the virtual human agent</strong> (three-way)</td><td>V vs. H</td><td>p &lt; 0.001</td><td>C vs. S<br>p = 0.003</td></tr><tr><td><strong>Force feedback to the operator</strong> (three-way)</td><td>V vs. H</td><td>p &lt; 0.001</td><td></td></tr><tr><td><strong>Connectivity level of the team</strong> (three-way)</td><td colspan="3">No statistically significant differences were found.</td></tr><tr><td><strong>Perceived effectiveness</strong> (two-way)</td><td>V vs. H</td><td>p &lt; 0.001</td><td>C vs. S<br>p = 0.001</td></tr></tbody></table>

possible, taking however into account the feedback received.

**Results:** We registered (i) the average task completion time, (ii) the average length and (iii) linear velocity of the simulated human agent, (iv) the average connectivity force commanded to the human operator, and (v) the average connectivity level of the whole formation. To compare the different metrics, we ran three-way repeated-measures ANOVA tests (significance level $a = 0.05$) [30]. The interface (joypad/visual or haptic, V vs. H), feedback type (combined or separated, C vs. S), and the presence of unexpected obstacles (yes or not, free vs. occ) were the within-subject factors. Data from the two scenes were normalized and averaged. All the data passed the Shapiro-Wilk normality test. Table II summarizes the results.

Fig. 5a shows the average normalized task completion time. It was calculated as the time it took the virtual agent to move from the starting position to the goal, divided by the best registered performance (≈ 137 s). The ANOVA test revealed a statistically significant change in the task completion time for the interface [F-test statistic $F(1,11) = 12.509$, p-value for the F statistics $p = 0.005$], feedback modality [F(1,11) = 20.595, $p = 0.001$] and obstacles [F(1,11) = 13.315, $p = 0.004$] variables. Fig. 5b shows the average normalized path traveled by the simulated agent. It was calculated as the length of its linear trajectory, divided by the best registered performance (≈ 34 m). The ANOVA test revealed a statistically significant change in the task traveled path for the interface [F(1,11) = 18.387, $p = 0.001$], feedback modality [F(1,11) = 6.870, $p = 0.024$] and obstacles [F(1,11) = 43.620, $p < 0.001$] variables. Fig. 5c shows the average velocity of the simulated agent. It was calculated as the average norm of $v_h$. The ANOVA test revealed a statistically significant change in the agent's velocity for the interface [F(1,11) = 27.258, $p < 0.001$] and feedback modality [F(1,11) = 14.886, $p = 0.003$] variables. Fig. 5d shows the average force applied to the operator. It was calculated as the average norm of $\mathbf{F}_t$. The ANOVA test revealed a statistically significant change in the applied force for the interface [F(1,11) = 611.703, $p < 0.001$] variable. Fig. 5e shows the average connectivity level of the team, i.e., $\lambda_2$. The ANOVA test revealed no statistically significant change in the
<figure><img src="image_5.png" /><figcaption>Fig. 5. Human subjects experiment. Mean and 95% confidence interval of (a) completion time, (b) normalized path, (c) velocity, (d) force applied, (e) connectivity level, and (f) perceived effectiveness for the four conditions in the two scene variations *free* and *occ*.</figcaption></figure>

connectivity level for the considered variables.

At the end of the experiment, we asked the participants to rate the perceived effectiveness of the four experimental conditions (VC vs. VS vs. HC vs. HS). The responses were given using bipolar Likert-type scales that ranged from 0 to 10, where a score of 0 meant “very low” and a score of 10 meant “very high” [31], [32]. To compare this metrics, we ran a two-way repeated-measures ANOVA test ($a = 0.05$). The interface (H vs. V) and feedback modality (C vs. S) were the within-subject factors. Data were transformed using a square-root transformation before running the statistical analysis. The transformed data passed the Shapiro-Wilk normality test. Fig. 5f shows the perceived effectiveness of the four experimental conditions. The ANOVA test revealed a statistically significant change in the perceived effectiveness for both the interface [$F(1,11) = 83.993, p < 0.001$] and feedback modality [$F(1,11) = 19.288, p = 0.001$] variables.

Finally, six subjects out of twelve found condition HS to be the most effective at completing the task, followed by VS (four subjects) and VC (two subjects). Similarly, five subjects out of twelve found condition VS to be the easiest to understand

and use, followed by HS (three subjects), VC (two subjects), and HC (two subjects).

## B. Representative task in a real environment

1) **Context:** Imagine a post-disaster situation in which an expert rescuer is escorted by the robotic team inside a disaster area with the aim of reaching a certain point of interest (e.g., a survivors shelter). He or she is equipped with two vibrating bands on the right and left arms. The rescuer moves in the same environment of the robots, directly receiving information about the best path to follow and the connectivity of the team (see Fig. 3 bottom). Feedback information is generated as in Sec. IV-A. Since the rescuer needs to move freely, here all information is provided through the wearable vibrotactile armbands. No remote operator is needed in this case.

2) **Real environment, task, and subject:** One variation of the scene has no changes with respect to the a priori map known by the system ($S3_{free}$) while the other one has unexpected obstacles blocking some routes ($S3_{occ}$, shown in Fig. 6). These unexpected obstacles are unknown to the system and therefore not accounted for in the path planning of Sec. III-B. As in Sec. IV, we consider these variations to take into account situations in which the planner has only access to an outdated version of the map. Our heterogeneous team is composed of four real agents: one human operator, one mobile robot, and two quadrotors. All agents move in the instrumented space. We use two quadrotors from MikroKopter (Germany) and one Pioneer P3-DX mobile robot from Omron Adept (USA), all controlled at 50 Hz.

The subject plays the role of the expert rescuer, and the task consists in moving toward the goal location, following the feedback information provided by the system. The starting points and goal locations are shown in Fig. 6a and were the same for both scenes.

3) **Experimental modalities:** We provide the operator with feedback information about the path to follow and the connectivity information using a fully-wearable solution. This information is conveyed using the two armbands, in a smart combination of conditions HC and HS. In particular, we provide the user with information on $F_t$, as calculated in (5), using two armbands. The policy to activate the two armbands in the same used for condition HS in Sec. IV-A4 and Table I, i.e., more vibrations on one side indicate that $F_t$ points in that direction (similarly to $F_p$ in Fig. 4-HS).

4) **Results:** Figure 6 shows the experiment in the variation with the unexpected obstacle (circled in orange in Figure 6b). Specifically, Fig. 6a shows the trajectories of the four agents (blue: human, cyan: mobile robot, red and green: quadrotors). Figure 6b shows three moments of the experiment, at $t = 0, T_f/2, T_f$ with $T_f = 129$ s, from the point of view of a videocamera placed at around (5, -3) m.

The operator was able to successfully complete the task, reporting no issues and a high perceived effectiveness and comfort in using the system. We registered the completion time ($S3_{free}$: 103 s; $S3_{occ}$: 129 s), the average distance traveled ($S3_{free}$: 7.8 m; $S3_{occ}$: 12.1 m), the average connectivity level of the whole formation ($S3_{free}$: 1.1; $S3_{occ}$: 1.0), and the activation of the vibrotactile armbands ($S3_{free}$: 47.7%; $S3_{occ}$: 37.9%).
<figure><img src="image_6.png" /><figcaption>Fig. 6. Experiment in a real environment: scene S3<sub>occ</sub>. The blue circle indicates the human agent, red and green circles indicate the two quadrotors, and the cyan circle indicates the grounded mobile robot. (top) from left to right: robot initial positions at time *t* = 0 and grid of obstacles (black), robot trajectories and positions *t* = *T*<sub>final</sub>/2 and then at *t* = *T*<sub>final</sub> = 129 s. In this variation of the scene, the team is forced to move around the obstacle, due to the presence of an additional obstacle unknown to the system (orange). (bottom) video stills of the experiment at the same times as above, from two different points of view. A video of the full experiment is available at https://youtu.be/N73QUFqpSvo?t=127.</figcaption></figure>

V. DISCUSSION AND CONCLUSIONS

This paper presents a decentralized haptic connectivity-
maintenance control framework for heterogeneous human-
robot teams. We control the coordinated motion of a team
composed of an arbitrary number of mobile robots and one
human, for collaboratively achieving exploration and SAR tasks.
The human physically becomes part of the team, moving in
the same environment of the robots and receiving rich haptic
feedback about the team connectivity and direction toward
a safe path. While the human explores the environment, the
robots move autonomously to keep the team connected, using
a decentralized connectivity-maintenance algorithm.

The first experiment considered a situation in which the
robotic team escorts a civilian out of a disaster area. Results
show that using the haptic interface significantly improves the
performance of the task with respect to using the joypad and
the visual feedback in all metrics except the connectivity level.
Similarly, providing the two pieces of information separately
(path to follow and connectivity level) improved the user's
performance in all metrics but the connectivity level and the
total force feedback applied to the user. As expected, differences
between the two variations of the scene (with or without
unknown obstacles) were only registered in the completion
time and path length. Most subjects appreciated being provided
with the two feedback cues separately, as it enabled them
to understand the contribution of each piece of information.
Subjects also appreciated the effectiveness of receiving haptic
feedback, although visual cues were considered easier to
understand. This latter result is not unexpected, as humans are
generally more used to follow visual navigation feedback (e.g.,
turn-by-turn car navigation systems, road signs), while they
are not used to follow haptic navigation cues at all. For this
reason, we expect more training to significantly improve the

performance of the haptic modality in all the considered metrics.
It might also seem strange that no differences were found
regarding the connectivity level of the team. However, since
our virtual robots can apply infinite thrusts, the connectivity-
maintenance algorithm is always able to guarantee a well
connected team, even if the human agent completely disregards
the connectivity force cues. However, of course, a failure in
providing the acceleration requested by the algorithm may
result in a loss of connectivity for the team. This undesired
situation might happen in a real scenario. Finally, we discuss
the difference between using kinesthetic feedback only (HC)
vs. combining kinesthetic and vibrotactile feedback (HS). If
no unexpected obstacles are encountered, being gently guided
by kinesthetic feedback seems the straightforward choice, even
if combining the two forces into one conveys less information.
Nonetheless, in the presence of unexpected obstacles, the
operator needs to temporarily move away from the suggested
trajectory to find new unobstructed paths. To do so, in the case
of HC, the operator needs to act against the kinesthetic feedback,
which might become rather strong depending on the situation.
This problem is solved in condition HS, where navigation
information is provided via wearable vibrotactile feedback.
This type of haptic feedback can convey rich navigation cues
while leaving the operator free to disregard it, if necessary.

The second experiment considered a situation in which
a rescuer is escorted by the robotic team inside a disaster
area. One subject carried out the experiment in two real-world
scenarios, receiving feedback about the path to follow and the
connectivity of the team through two vibrotactile armbands.
This success in a real environment using a fully-wearable
solution is very promising, and it may open new interesting
scenarios in the field of ubiquitous robotics and human-robot
collaboration. The human operator does not need anymore to
stay in a fixed position, but instead can move and share the same environment of the robots.

In addition to the great importance of providing feedback information (either via haptic and visual stimuli), experiments show that the overall framework works well and it is robust to rather different conditions. The team remained well connected in all scenarios, providing the user with meaningful information about the current connectivity level and easy-to-follow navigation information. Similarly to a vehicle turn-by-turn navigation system, our path planning algorithm constantly indicates the shortest path from the current position to the goal. If an unexpected obstacle blocks the indicated path, the human can freely move elsewhere and the path planning will adjust its suggestion accordingly. As we tested our system in two different environments, six scenes, enrolling 13 subjects for more than 12 hours of testing, this work represents a much-needed extensive evaluation on the topic.

In the future, we will extend this framework for considering multiple human agents, so as to be able to support an arbitrary number of mobile robots and humans. Moreover, we plan to include a dynamic damping factor in (4) for smoothing any abrupt change in the force feedback when the target direction/tile changes. Finally, we also want to test more extensively real-world situations where the human civilian survivor does not follow exactly the commands of the remote expert rescuer, e.g., when the communication link does not work as well as expected, the survivor is injured and cannot move properly, or the stimulation results hard to understand. For example, we could adjust the gain on the guidance, $\kappa_p$, according to how well the survivor follows the rescuer commands.

## REFERENCES

[1] J. Casper and R. R. Murphy, "Human-robot interactions during the robot-assisted Urban Search and Rescue response at the World Trade Center," *IEEE Trans. Systems, Man, and Cybernetics, Part B (Cybernetics)*, vol. 33, no. 3, pp. 367-385, 2003.

[2] A. Davids, "Urban search and rescue robots: from tragedy to technology," *IEEE Intelligent systems*, vol. 17, no. 2, pp. 81-83, 2002.

[3] L. Marconi, C. Melchiorri, M. Beetz, D. Pangercic et al., "The SHERPA project: Smart collaboration between humans and ground-aerial robots for improving rescuing activities in alpine environments," in *IEEE Int. Symp. on Safety, Security, and Rescue Robotics (SSRR)*, 2012, pp. 1-4.

[4] M. A. Goodrich, B. S. Morse, D. Gerhardt et al., "Supporting wilderness search and rescue using a camera-equipped mini uav," *Journal of Field Robotics*, vol. 25, no. 1-2, pp. 89-110, 2008.

[5] G.-J. M. Kruijff et al., "Experience in system design for human-robot teaming in urban search and rescue," in *Field and Service Robotics*, 2014, pp. 111-125.

[6] P. Robuffo Giordano, A. Franchi, C. Secchi, and H. H. Bülthoff, "A passivity-based decentralized strategy for generalized connectivity maintenance," *The International Journal of Robotics Research*, vol. 32, no. 3, pp. 299-323, 2013.

[7] J. Saez-Pons, L. Alboul, and J. Penders, "Experiments in cooperative human multi-robot navigation," in *IEEE Int. Conf. on Robotics and Automation (ICRA)*, 2011, pp. 1-4.

[8] T. Iqbal and L. D. Riek, "Human coordination dynamics with heterogeneous robots in a team," in *ACM/IEEE Int. Conf. on Human Robot Interaction (HRI)*, 2016, pp. 619-620.

[9] A. Ranasinghe, P. Dasgupta, K. Althoefer, and T. Nanayakkara, "Identification of haptic based guiding using hard reins," *PloS one*, vol. 10, no. 7, pp. 1-22, 2015.

[10] S. Scheggi, F. Morbidi, and D. Prattichizzo, "Human-robot formation control via visual and vibrotactile haptic feedback," *IEEE Trans. Haptics*, vol. 7, no. 4, pp. 499-511, 2014.

[11] S. Scheggi, M. Aggravi, and D. Prattichizzo, "Cooperative Navigation for Mixed Human-Robot Teams Using Haptic Feedback," *IEEE Trans. Human-Machine Systems*, vol. PP, no. 99, pp. 1-12, 2016.

[12] C. Robin and S. Lacroix, "Multi-robot target detection and tracking: taxonomy and survey," *Autonomous Robots*, vol. 40, no. 4, pp. 729-760, 2016.

[13] M. Juliá, A. Gil, and O. Reinoso, "A comparison of path planning strategies for autonomous exploration and mapping of unknown environments," *Autonomous Robots*, vol. 33, no. 4, pp. 427-444, 2012.

[14] F. Amigoni, J. Banfi, and N. Basilico, "Multirobot Exploration of Communication-Restricted Environments: A Survey," *IEEE Intelligent Systems*, vol. 32, no. 6, pp. 48-57, 2017.

[15] M. Hashemipour, S. Stuban, and J. Dever, "A disaster multiagent coordination simulation system to evaluate the design of a first-response team," *Systems Engineering*, vol. 21, no. 4, pp. 322-344, 2018.

[16] M. Kulich, J. Faigl, and L. Preucil, "Cooperative planning for heterogeneous teams in rescue operations," in *IEEE Int. Workshop on Safety, Security and Rescue Robotics*, 2005, pp. 230-235.

[17] Z. Beck, L. Teacy, A. Rogers, and N. R. Jennings, "Online planning for collaborative search and rescue by heterogeneous robot teams," in *Int. Conf. Autonomous Agents & Multiagent Systems*, 2016, pp. 1024-1033.

[18] Y. Solana, M. Furci, J. Cortés, and A. Franchi, "Multi-robot path planning with maintenance of generalized connectivity," in *Int. Symp. on Multi-Robot and Multi-Agent Systems*, 2017, pp. 63-70.

[19] L. C. Boer and D. J. Withington, "Auditory guidance in a smoke-filled tunnel," *Ergonomics*, vol. 47, no. 10, pp. 1131-1140, 2004.

[20] M. Aggravi, G. Salvietti, and D. Prattichizzo, "Haptic wrist guidance using vibrations for Human-Robot teams," in *IEEE Int. Workshop on Robot and Human Interaction (ROMAN)*, 2016, pp. 113-118.

[21] J. B. F. Van Erp, H. A. Van Veen, C. Jansen, and T. Dobbins, "Waypoint navigation with a vibrotactile waist belt," *ACM Trans. Applied Perception*, vol. 2, no. 2, pp. 106-117, 2005.

[22] M. Schirmer, J. Hartmann, S. Bertel, and F. Echtler, "Shoe me the Way: A Shoe-Based Tactile Interface for Eyes-Free Urban Navigation," in *ACM Int. Conf. on Human-Computer Interaction with Mobile Devices & Services*, 2015, pp. 327-336.

[23] M. Mesbahi and M. Egerstedt, *Graph theoretic methods in multiagent networks*. Princeton University Press, 2010.

[24] M. Aggravi, C. Pacchierotti, and P. Robuffo Giordano, "Connectivity-maintenance teleoperation of a uav fleet with wearable haptic feedback," *IEEE Transactions on Automation Science and Engineering*, pp. 1-20, 2020.

[25] G. Arechavaleta, J.-P. Laumond, H. Hicheur, and A. Berthoz, "On the nonholonomic nature of human locomotion," *Autonomous Robots*, vol. 25, no. 1-2, pp. 25-35, 2008.

[26] M. Aggravi, S. Scheggi, and D. Prattichizzo, "Evaluation of a predictive approach in steering the human locomotion via haptic feedback," in *IEEE/RSJ Int. Conf. on Intelligent Robots and Systems*, 2015, pp. 597-602.

[27] L. R. Ford Jr, "Network flow theory," Rand Corp Santa Monica Ca, Tech. Rep., 1956.

[28] M. Aggravi, F. Pausé, P. Robuffo Giordano, and C. Pacchierotti, "Design and evaluation of a wearable haptic device for skin stretch, pressure, and vibrotactile stimuli," in *IEEE Robotics and Automation Letters*, vol. 3, no. 3, 2018, pp. 2166-2173.

[29] T. L. Baldi, S. Scheggi, M. Aggravi, and D. Prattichizzo, "Haptic guidance in dynamic environments using optimal reciprocal collision avoidance," *IEEE Robotics and Automation Letters*, vol. 3, no. 1, pp. 265-272, 2017.

[30] E. R. Girden, ANOVA: *Repeated measures*, 1992, no. 84.

[31] M. Maisto, C. Pacchierotti, F. Chinello, G. Salvietti, A. De Luca, and D. Prattichizzo, "Evaluation of wearable haptic systems for the fingers in augmented reality applications," *IEEE Trans. Haptics*, vol. 10, no. 4, pp. 511-522, 2017.

[32] L. Meli, C. Pacchierotti, G. Salvietti et al., "Combining wearable finger haptics and augmented reality: User evaluation using an external camera and the microsoft hololens," *IEEE Robotics and Automation Letters*, vol. 3, no. 4, pp. 4297-4304, 2018.