# Evaluation of Human-Robot Interfaces based on 2D/3D Visual and Haptic Feedback for Aerial Manipulation

Julien Mellet, Mike Allenspach, Eugenio Cuniato, Claudio Pacchierotti, Roland Siegwart, Marco Tognon

► To cite this version:

Julien Mellet, Mike Allenspach, Eugenio Cuniato, Claudio Pacchierotti, Roland Siegwart, et al. Evaluation of Human-Robot Interfaces based on 2D/3D Visual and Haptic Feedback for Aerial Manipulation. Journal of Intelligent and Robotic Systems, 2025, 111 (3), pp.104. 10.1007/s10846-025-02284-7. hal-05313710

HAL Id: hal-05313710

https://hal.science/hal-05313710v1

Submitted on 14 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluation of Human-Robot Interfaces based on 2D/3D Visual and Haptic Feedback for Aerial Manipulation

Julien Mellet<fnref n="1" />*, Mike Allenspach<fnref n="2" />, Eugenio Cuniato<fnref n="2" />, Claudio Pacchierotti<fnref n="3" />, Roland Siegwart<fnref n="2" />, Marco Tognon<fnref n="3" />

## Abstract

Most telemanipulation systems for aerial robots provide the operator with only 2D screen visual information. The lack of richer information about the robot's status and environment can limit human awareness and, in turn, task performance. While the pilot's experience can often compensate for this reduced flow of information, providing richer feedback is expected to reduce the cognitive workload and offer a more intuitive experience overall. This work aims to understand the significance of providing additional pieces of information during aerial telemanipulation, namely (i) 3D immersive visual feedback about the robot's surroundings through mixed reality (MR) and (ii) 3D haptic feedback about the robot interaction with the environment. To do so, we developed a human-robot interface able to provide this information. First, we demonstrate its potential in a real-world manipulation task requiring sub-centimeter-level accuracy. Then, we evaluate the individual effect of MR vision and haptic feedback on both dexterity and workload through a human subjects study involving a virtual block transportation task. Results show that both 3D MR vision and haptic feedback improve the operator's dexterity in the considered teleoperated aerial interaction tasks. Nevertheless, pilot experience remains the most significant factor.

**Keywords:** Aerial Physical Interaction, Aerial Manipulation, Teleoperation, Haptic, Mixed Reality, Multimodal

<footnote n="1">¹*PRISMA Lab, Department of Electrical Engineering and Information Technology, University of Naples Federico II, Via Claudio 21, 80125, Naples, Italy.</footnote>

<footnote n="2">²Autonomous Systems Lab (ASL), ETH, Zurich, Switzerland.</footnote>

<footnote n="3">³Univ Rennes, CNRS, Inria, IRISA - UMR 6074, Rennes, 35000, France.</footnote>

<footnote>*Corresponding author(s). E-mail(s): julien.mellet@unina.it;</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1:** On the top: Local teleoperation environment with haptic joystick and hologram of the robot in its environment. On the bottom: Remote environment with the real robot.</figcaption></figure>

# 1 Introduction

While small-scale aerial robots have traditionally been employed for tasks in free-flight like surveillance and aerial imaging, there is a growing demand for these platforms to engage in physical interactions [1–3]. This is particularly pertinent in situations where employing human workers might be risky or impractical, such as inspecting remote or hazardous environments, performing maintenance on tall structures, or handling dangerous substances [4–6]. Accordingly, new Omnidirectional Micro Aerial Vehicles (OMAVs) have been presented [7], which are able to generate thrust in all directions and independently control both position and orientation. Their intrinsic capacity to generate thrust in all directions allows them to hover in arbitrary orientations, with independent control of both position and orientation. These vehicles are thus capable of precise motion and interaction force control, while effectively rejecting disturbances [8, 9].

Although progress in the autonomous control of these vehicles is significant [1, 10, 11], for safety concerns, current regulations require a human operator to supervise any mission [12, 13]. Rich sensory feedback is crucial for robotic teleoperation as it enables the human operator to perceive and interact with the remote environment effectively, thereby improving situational awareness and the capacity to perform complex tasks. Achieving a convincing sense of telepresence, creating the illusion of being physically present at a remote site, depends on advanced technology. Dedicated teleoperation systems facilitate the transfer of human expertise, decision-making, and cognitive abilities to remote environments.
A carefully designed teleoperation framework must ensure human situational awareness, by providing visual and/or haptic feedback, and an intuitive way to send reference commands to the remote robot. The enhancement of perceptual awareness of the surroundings contributes to the concept of *bilateral teleoperation* [14]. A typical installation is shown in Fig. 1.

While state-of-the-art bilateral teleoperation proved the potential of haptic and vision devices for free-flight applications, no prior research has investigated the influence of these technologies on operator performance in the aerial manipulation context [15, 16].

## 1.1 Related Work

<figure><img src="image_3.png" /><figcaption>**Fig. 2:** Detailed interactions between the different components of the teleoperation framework.</figcaption></figure>

Most state-of-the-art aerial teleoperation relies on 2D visual displays, providing limited information regarding the status of the robot and the surrounding environment. To enhance situational awareness, researchers proposed the use of immersive Virtual Reality (VR) [17–19] and Mixed Reality (MR) technology. The latter is especially promising as it enables to combine virtual and real content in a seamless way [6, 20, 21]. Differently from VR, in which the user is fully immersed in a virtual environment, MR enriches the real world with virtual content, combining real environment awareness with rich feedback information [22]. However, MR technology has mainly been used in scenarios where operators maintain direct sight on the remote environment [23, 24]. Unlike setups with 2D displays for remote observation, MR integration in the local context lacks exploration.

Human perception, inherently multimodal, involves vision and proprioception, which is known to enhance performance [25]. Thus, haptic feedback represents another relevant source of feedback shown to be effective in providing information in various teleoperation scenarios, including aerial robots [26]. Haptics technology provides touch-related information about the interaction of the robotic system with the remote environment or its virtual reconstruction [18]. It was mostly used in contact-free flight [27, 28] or simulated physical interaction [29]. In aerial manipulation tasks, interaction
forces play a crucial role in enhancing user control and manipulation capabilities [30–32]. Haptic feedback would allow users teleoperating an aerial manipulator to better perceive object properties, exert appropriate forces during manipulation, and enhance precision. To obtain force feedback, various haptic devices, such as wearables [17, 33], robot arms [30], or force-feedback joysticks [31, 32] have been used.

Given the expected benefits of providing MR and haptic feedback in the teleopera-
tion of aerial manipulators, it is relevant to explore and evaluate which type of feedback
is best suited for conveying what type of information. Indeed, most works on the topic
either do not focus on evaluating Human-Robot Interaction (HRI) performance [15],
or only consider free-flight application [16, 27, 28].

In summary, evaluating the effect of haptic and/or MR feedback on operator per-
formance during aerial manipulation tasks remains an open research question. The
importance or interconnection of operator multimodal sensing for aerial manipulation
remains unexplored.

## 1.2 Contributions

This work aims at investigating the impact of sensory feedback, notably visual and
haptic feedback, on user’s experience and task performance during teleoperated aerial
physical interaction. The main contributions of our work can be summarized as: i) the
design of a sensory interface coupling MR and haptic feedback for 6-degrees of free-
dom (DoF) teleoperation, ii) qualitative evaluation of the proposed framework in
real-world benchmark test with sub-centimeter-level-accuracy interaction tasks, and
iii) quantitative evaluation in a user study within a simulated-world interaction task.

The rest of the paper is organized as it follows. Section 2 presents the proposed
system and each of its components. Then, Section 3 shows experiments where the entire
pipeline runs in a real-world application. Description of the quantitative evaluation
scenario is done in Section 4 with the user study. Finally, the results are discussed in
Section 5.

## 2 System

The bilateral teleoperation system under study comprises four main elements: 1. *human operator*, 2. *haptic device*, 3. *MR device*, and 4. *aerial robot*. The human operator, the MR device and the haptic device are in the local environment, while the robot is located in the remote environment and communicates through a link with the local environment. We assume to know the environment. The full system is shown in Fig. 2 and its components are discussed in detail in the following sections.

2.1 Aerial Robot

The aerial robot utilized in this work is the tiltrotor OMAV proposed in [1].

The remote configuration is defined with the inertial world frame $\mathcal{F}_W = \{O_W, x_W, y_W, z_W\}$ with origin $O_W$ and unit axis $\{x_W, y_W, z_W\}$ located at an arbitrary fixed position such that $z_W$ is opposite to gravity. The state of the aerial robot is described by the body frame $\mathcal{F}_B = \{O_B, x_B, y_B, z_B\}$ with origin $O_B$ coinciding with
robot's Center of Mass (CoM) and $x_B$ pointing toward the end-effector. To define the system's state, let $p_B \in \mathbb{R}^3$ express the position of $O_B$ with respect to $\mathcal{F}_W$. The attitude of $\mathcal{F}_B$ with respect to $\mathcal{F}_W$ is described by the rotation matrix $R_B^W \in \mathrm{SO}(3)$. Finally, we denote the body linear and angular velocities with respect to $\mathcal{F}_W$ as $v_B$, $\omega_B \in \mathbb{R}^3$, respectively. For compliance at interaction, the robot is controlled by an impedance controller [1] together with a momentum-based external wrench estimator, which makes the closed-loop dynamics as

$$M_v \begin{bmatrix} \dot{v}_B^B \\ \dot{\omega}_B^B \end{bmatrix} + D_v \begin{bmatrix} e_v \\ e_\omega \end{bmatrix} + K_v \begin{bmatrix} e_p \\ e_R \end{bmatrix} = \tau_{\mathrm{ext}}, \quad (1)$$

with $M_v, D_v, K_v \in \mathbb{R}^{6 \times 6}$ the virtual inertia, the damping and the stiffness, respectively, which are tuning parameters. External wrench acting on the platform is described with $\tau_{\mathrm{ext}} \in \mathbb{R}^6$. Finally, we consider the position $e_p$, orientation $e_R$, velocity $e_v$ and angular rate $e_\omega$ errors as,

$$ \begin{align} e_p &= R_B^{W^T} (\mathbf{p}_B - \mathbf{p}_{B,\mathrm{ref}}), \nonumber \\ e_R &= \frac{1}{2} \left( R_{B,\mathrm{ref}}^{W^T} R_B^W - R_B^{W^T} R_{B,\mathrm{ref}}^W \right)^{\mathrm{V}}, \tag{2a} \\ e_v &= R_B^{W^T} (\mathbf{v}_B - \mathbf{v}_{B,\mathrm{ref}}), \nonumber \\ e_\omega &= \omega_B - R_B^{W^T} R_{B,\mathrm{ref}}^W \omega_{B,\mathrm{ref}}. \nonumber \end{align} $$

with $(\cdot)^{\mathrm{V}}$ the Vee operator to extract a vector from a skew-symmetric matrix. The reference position and orientation, $\mathbf{p}_{B,\mathrm{ref}}$ and $R_{B,\mathrm{ref}}^W$, respectively, are computed as in Sec. 2.2. This allows us to steer the aerial robot and safely interact with the remote environment.

## 2.2 Haptic System and Control

Through the haptic interface, the operator controls the vehicle's velocity. The inertial frame $\mathcal{F}_M = \{O_M, \mathbf{x}_M, \mathbf{y}_M, \mathbf{z}_M\}$ with origin $O_M$ corresponds to the idle position and orientation of the joystick's handle (see Fig. 1). Its current pose is described by the frame $\mathcal{F}_H = \{O_H, \mathbf{x}_H, \mathbf{y}_H, \mathbf{z}_H\}$ with origin $O_H$ fixed with respect to the handle. We denote position and velocity of the haptic device handle with $\mathbf{p}_H \in \mathbb{R}^3$ and $\mathbf{v}_H \in \mathbb{R}^3$, both expressed with respect to $\mathcal{F}_M$. Attitude and angular rate of $\mathcal{F}_H$ with respect to $\mathcal{F}_M$ are defined as $R_H^M \in \mathrm{SO}(3)$ and $\omega_B \in \mathbb{R}^3$, respectively, the latter expressed in $\mathcal{F}_H$.

### 2.2.1 Reference Generation

Similarly to [30, 33], to create an intuitive correspondence between the limited input workspace and the potentially boundless robot workspace, we generate velocity commands through the haptic interface, as it represents a widely accepted approach for teleoperating aerial vehicles [34]. Translational and rotational references are then
computed such that,

$$v_{B,ref} = v_{max}p_{H},$$

$$p_{B,ref} = \int_{0}^{t} v_{B,ref}(b) db,$$

$$\omega_{B,ref}^{B} = \frac{\omega_{max}}{2} (R_{H}^{M} - R_{H}^{M^{T}})^{\vee}, \qquad (3a)$$

$$R_{B,ref}^{W} = \int_{0}^{t} R_{B,ref}^{W}(b) [\omega_{B,ref}(b)]_{\times} db.$$

where $(\cdot)_{\times}: \mathbb{R}^{3} \to \text{SO}(3)$ is the skew-symmetric operator, $v_{max}$ and $\omega_{max}$ are respectively velocity and angular rate gains set by the operator preference.

### 2.2.2 Haptic Feedback Generation

To enhance operator perception during teleoperation, we generate real-time haptic feedback by rendering a virtual wrench at the operator interface. As proposed in [30], we define the total feedback wrench as:

$$\tau_{\text{fb,total}} = \tau_{\text{fb,rec}} + \tau_{\text{fb,ext}}, \qquad (4)$$

where $\tau_{\text{fb,rec}} \in \mathbb{R}^{6}$ is the recentering wrench and $\tau_{\text{fb,ext}} \in \mathbb{R}^{6}$ is the interaction wrench. The former serves to guide the operator back toward a neutral pose or rest configuration, ensuring ergonomic control and stability, while the latter encodes contact forces and environmental interactions perceived by the robot end-effector. This formulation enables the system to convey both proprioceptive cues and task-specific force feedback in a unified and intuitive manner. In details,

$$\tau_{\text{fb,rec}} = -\mathbf{K}_{\text{rec}} \left[ \frac{1}{2} (\mathbf{R}_{H}^{M} - \mathbf{R}_{H}^{M^{T}})^{\vee} \right], \qquad (5)$$

with $\mathbf{K}_{\text{rec}} = \text{diag}(\mathbf{K}_{\text{rec},t}, \mathbf{K}_{\text{rec},r}) \in \mathbb{R}^{6 \times 6}$ a tuning parameter. When the operator release the joystick, thanks to the recentering action, it returns to the idle position, corresponding to a zero robot velocity. Moreover, the wrench feedback at interaction is defined as,

$$\tau_{\text{fb,ext}} = \mathbf{K}_{\text{ext}} \hat{\tau}_{\text{ext}}, \qquad (6)$$

with $\mathbf{K}_{\text{ext}} \in \mathbb{R}^{6}$ a tuning parameter to adjust perceived effort to user preferences and $\hat{\tau}_{\text{ext}} \in \mathbb{R}^{6}$ is the estimated wrench sent to the haptic interface. Since the described haptic feedback only provides information at contact, the visual feedback is added to improve spatial and depth perception.

## 2.3 Mixed Reality Vision

The MR headset creates holograms with respect to a virtual frame $\mathcal{F}_{V} = \{O_{V}, x_{V}, y_{V}, z_{V}\}$ that can be set arbitrarily with origin $O_{V}$.
<figure><img src="image_4.png" /><figcaption>**Fig. 3:** Experimental setup for force exertion 3a and insertion task 3b, with corresponding virtual MR holograms 3c and 3d.</figcaption></figure>

The operator sees the robot in its environment through the MR goggles. The virtual frame $\mathcal{F}_V$ is constant with respect to the inertial local frame $\mathcal{F}_M$, thanks to the localization features implemented on the goggles. Holographic visualizations are designed based on the task at hand. Essential information includes the position and orientation of the robot and interaction objects. Examples for peg-in-hole, pushing, and pick-and-place can be found in Figs. 3c, 3d and 6b, respectively.

# 3 System Demonstration

First, we show the viability of using the proposed approach in a teleoperation benchmark for the aerial robot, teleoperated in physical interaction with a remote environment. The goal is to show the feasibility of controlling the aerial robot in a real physical setup, with data streamlines through a network, while the operator only gets information through MR along with the haptic joystick.

## 3.1 System Setup

We propose two scenarios with physical contact, in which force feedback information for the operator is essential to ensure safe interaction: 1. pushing a wheeled object (in paragraph 3.1), and 2. inserting a peg into a hole (in paragraph 3.1). The potential
<figure><img src="image_5.png" /><figcaption>**Fig. 4:** Experiment 1 - Measured position in $\mathcal{F}_W$ and estimated forces in $\mathcal{F}_W$ while pushing the wheeled object. From left to right the highlighted moments are: before the interaction, the first physical contact, a slight discontinuity of contact, and the final contact disengagement.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 5:** Experiment 2 - Measured position in the world frame $\mathcal{F}_W$ and estimated forces in the body frame $\mathcal{F}_B$ during the peg-in-hole task. From left to right the highlighted moments are: before the interaction, the shocks before the insertion, the reach of the hole's end, and the exit from the hole.</figcaption></figure>

of haptic feedback for exerting force on movable objects, along with the potential of 3D visual feedback for high-accuracy insertion tasks, are both significant in terms of achieving task objectives. The proposed scenarios have been chosen as they require rich visual and force feedback information to be well executed. They also show high flexibility of the framework for different tasks.

Preliminary flight tests have been used for gain tuning. For safety reasons, we limit the maximum velocity of the robot to $v_{max} = 15$ cm/s. Moreover, the force feedback
was scaled to one third of the measured force, to account for the maximum force output of the device (12 N) and avoid any saturation of the force feedback. These adjustments allow the operator to perceive external forces while reducing the physical effort he exerts. The robot is teleoperated by the same experienced operator across the experiments, and has no direct view of the remote environment.

For all these experiments, the interaction surfaces (i.e., movable object, wall plan) and the aerial robot poses are tracked with a motion capture system. The real environment is known a priori and replicated in the virtual space through the Microsoft HoloLens 2 goggles [35].

## Experiment 1 - Pushing a Wheeled Object

This task is relevant because it combines flight accuracy, force exertion regulation, and 3D localization awareness. In particular, visual feedback is required to make the contact with the box in the center, and haptic feedback enables to apply the right amount of force for keeping the box in motion without damaging it. In this scenario, a box is mounted on a wheeled platform and the aerial robot needs to apply varying forces to overcome static friction, to then keep contact. Figures 3a and 3c present the aerial platform pushing the box.

## Experiment 2 - Inserting a Peg Into a Hole

The second task consists of fitting a cylindrical peg into a corresponding hole attached on a vertical interaction surface (see Fig. 3b). This task is relevant because it combines flight accuracy and fine manipulation. Visual feedback is important to provide a close-up view of the hole, while haptic feedback helps during insertion (e.g., [36]). The tip radius measures 20 mm while the hole is 25 mm. The task thus requires a 5 mm accuracy from the operator to be completed.

## 3.2 Results and Analysis

The experiments demonstrate the system's capability to provide intuitive and precise control in complex tasks. Below, we present the detailed outcomes and analyses of these experiments.

### Experiment 1

The outcome of the first experiment, which involved pushing the object, demonstrates that the operator managed to regulate the robot's force using the haptic device, as shown in Fig. 4. The aerial robot applied a force of roughly 5 N to overcome the static friction and pushed the box approximately 1.7 meters. During the interaction, the system experienced a maximum estimated force of up to 5.2 N while maintaining stable contact with an average force of 1.0 N to overcome the kinetic friction.

### Experiment 2

The insertion task was successfully accomplished by fitting the peg into the narrow hole. The operator required less than 5 mm lateral accuracy in operating the robot. With the 3D virtual representation, the user was able to ensure the alignment of the
robot with the target by positioning his head around the scene. At the moment of insertion (see Fig. 5), the force along the manipulator increased up to 5.5 N. The task was completed rapidly, taking a total of 2.5 s inside the hole, avoiding any major interactions between the rigid bodies. The lateral force in the normal plane of the hole was regulated by the operator and anticipated such that no peak was noticeable in the force measurement. This allowed the system to achieve sub-centimetric accuracy in insertion tasks.

In the virtual space, one can notice surface collision of the end-effector mesh with the hole of the whiteboard in Fig. 3d. This is due to some compliance of the board at contact. This was neglected, assuming a solid board. This visual artifact did not interfere with the operator's task, and found the overall system pleasant and natural to use. The haptic feedback has also been felt informative by the operator to understand task completion.

In the end, the experiments showed in Figs. 3 proved the feasibility of using the system for Aerial Physical Interaction (APHI) tasks. Videos of these experiments are in the supplementary material. We demonstrated the capabilities afforded by combining haptic and visual feedback, specifically achieving precise control over force exertion with haptic feedback and accurate positioning and alignment with visual feedback. These experiments served to demonstrate system capabilities in realistic conditions. For rigorous assessment of operator performance, more specialized and well-established dexterity benchmark have to be considered. The question raises if the operator would perform similarly on a physical task without haptics or MR, and if the level of the pilot would influence the result.

# 4 User Study

After demonstrating the potential of combining rich haptic and visual feedback for teleoperating aerial manipulation tasks, we conduct a quantitative study to assess their impact on performance and perceived workload. As the role of sensory feedback also depends on the experience of the operator [37], we also investigate the influence of experience in this context. In conventional statistical design, optimal conditions are determined based on measured values. In contrast, the Taguchi design identifies optimal conditions by selecting parameters that exhibit the lowest variability. Here, the optimal refers to the most stable and consistent combination in terms of task efficiency, robustness through repeatability (i.e., predictable outcomes even with variations in external factors) while reducing the workload.

## 4.1 Design of Experiment

We considered three factors in our analysis, evaluating the role of the *visual feedback*, *haptic feedback*, and *user's experience* in the teleoperation of an aerial manipulator.

The *visual feedback* provides visual information to the operator about the status of the OMAV (see Sec. 2.3 and Fig. 6). We considered two different types of visual feedback:
<table><caption>**Table 1:** Taguchi Design $L_4$ (2³)</caption><thead><tr><th rowspan="2">Experimental Conditions</th><th colspan="3">Column (Factors)</th></tr><tr><th>Display Technology</th><th>Haptics</th><th>Operator Expertise</th></tr></thead><tbody><tr><td>1</td><td>SC</td><td>H</td><td>B</td></tr><tr><td>2</td><td>SC</td><td>H</td><td>E</td></tr><tr><td>3</td><td>MR</td><td>H</td><td>E</td></tr><tr><td>4</td><td>MR</td><td>H</td><td>B</td></tr></tbody></table>

<table><caption>**Table 2:** Complementary Experiments</caption><tr><td>5</td><td>SC</td><td>H</td><td>E</td></tr><tr><td>6</td><td>MR</td><td>H</td><td>E</td></tr></table>

* *SC*: a representation of the robot in a virtual environment using a standard 2-dimensional liquid crystal display (LCD) computer screen (see Fig. 6a),

* *MR*: a representation of the robot in a mixed-reality environment using the 3-dimensional MR vision headset HoloLens 2 (see Fig. 6b).

The *haptic feedback* provides information to the operator about the interaction forces acting on the OMAV (see Sec. 2.2). We considered two haptic feedback conditions:

* *H*: providing force information at physical contact (i.e. $\tau_{fb,ext}$) through an Omega.6 grounded haptic interface,

* *H*: no haptic feedback, i.e., set $\tau_{fb,ext} = 0$ in eq. (4).

Finally, we also considered two levels of *operator expertise*:

* *B*: beginner users, i.e., users who discovered the setup and task on the day of the experiment,

* *E*: experienced users, i.e., users who followed a training before the experiment (see Sec. 4.3).

*Visual feedback* and *haptic feedback* are considered as within-subjects factors, while the *operator expertise* is considered as a between-subjects factor.

As testing all the combinations would lead to an experiment that is very long (>1 hour), we minimize the number of tasks tested to prevent human fatigue and to maintain a full-rank task feature space by using the $L_4$ (2³) Taguchi table [38, 39]. This is a version of fractional factorial design of experiment. Taguchi design lowers the number of tasks to 2 per participant and is presented in Tab. 1. The proposed table ensures that the beginner group performs on what we might call the *standard teleoperation* interface (i.e., *H-SC*) and on the proposed augmented setup (i.e., H-MR). Experiments 2 and 3 are subsequent to the data defined above and complete the Taguchi table.

In addition to the conditions considered by the Taguchi design, we also tested a few additional combinations (see Tab. 2), as to better understand the relationship between the considered variables. Analyse of Variance (ANOVA) is a popular collection of statistical models and estimation tools used to analyze differences among output means of a sample [40]. By increasing the number of data, we aim to cover the factor
levels more thoroughly, validate our initial findings, and reduce the risk of statistical errors. Ultimately, this will lead to more reliable insights for analysing our study. We apply these tools in Secs. 4.5 and 5.

To reduce learning effect, we shuffled randomly and uniformly the experimentation order of the various experimental conditions per participants.

## 4.2 Setup and Task Definition

Like in Sec. 3, the virtual environment appears to the operator thanks to the HoloLens 2 MR headset or the LCD screen display (see Fig. 6), depending on the considered condition, while the remote robot is controlled through the Omega.6 haptic device in all cases.

Even if the proposed system enables full 6-DoF control of the robot pose (see Sec. 2.2), for the purposes of the considered tasks, we decided to simplify the control of the robot keeping a horizontal attitude. The orientation of the vehicle is kept constant along **x**<sub>B</sub> and **y**<sub>B</sub> (see Fig. 1). This means that the operator controls the translational velocities and the orientation around **z**<sub>B</sub> only. Such an approach statistically improves the significance of the results, as rotational commands tend to decrease user performance [31]. The operator sits in a posture similar to the one shown in Fig. 2. The remote environment is simulated in Gazebo with a physics engine [41]. The propellers' drag, lift coefficients and inertial unit characteristics are simulated through the internal plugin.

The Box and Block Test (BBT) is one of the most used and recommended test to evaluate human manual gross dexterity [42]. The participant is required to move blocks from one side of a partitioned box to the other within a specified time frame. It has been shown effective in real, virtual and mixes reality environments [43–46]. Because it has been successfully applied in [47] we adapted the BBT for APhI in teleoperation, calling it Aerial Box and Block Test (ABBT). The original clinical setup (Box and Blocks) has been scaled up of a factor 16, to account for the larger dimensions of the aerial manipulator. ABBT, like BBT, consists of pick and place blocks from one side of the box to another while avoiding the middle partition (see Fig. 6). The virtual setup seen through the MR goggles replicates the official test (see Fig. 6b), so that we keep the essence of the test using only a haptic device instead of the direct hand. Nevertheless, since the OMAV spatial displacement is slower than the hand displacement, we increased the task duration to 80s.

## 4.3 Participants

Twenty-eight participants (6 females, 22 males, aged 18-32) took part in the study, divided in two groups (Beginners and Experienced) with equal numbers, as discussed in Sec. 4.1.

Experienced participants for this user study were selected based on their availability for a two-month training period, facilitating consistent and intensive training sessions. Before training, we evaluated their prior knowledge: most were new to aerial robots, and all considered themselves fast learners. None had haptic device experience,
<figure><img src="image_7.png" /><figcaption>(a) Visual feedback under the 2D-LCD condition</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Visual feedback under the 3D MR condition</figcaption></figure>

**Fig. 6:** User study display technologies showing block transported by the aerial robot in the ABBT environment.

a few had used VR headsets, and none had prior MR combined with haptics for APhI exposure.

All training prior to the experimentation was conducted using our system, which combines haptics and MR. We devised training activities to familiarize with the proposed multimodal framework. The activities included i) a “*Race Track*” for improving aerial trajectories by slalomming between two poles and touching the ground between them (see Fig. 7a), ii) “*Catch a Ball*” to enhance dexterity at catching a randomly jumping ball in mid-air (see Fig. 7b), and iii) “*Push a Ball*” for practicing force control, in an activity similar to golf, hitting a ball into a hole (see Fig. 7c). Videos of the activities are in the supplementary material. Over an 8-weeks period, participants underwent six 15-minutes sessions, with periodic evaluations using the ABBT (see Figs. 7d and 8).

In the first evaluation session, the distribution is centered on the median at 0.5, and has an interquantile range of 0.0. By the second session, the distribution widens, showing a more important interquartile range of 0.125, suggesting slight decrease in consistency but increase of the median to 0.625. In the third session, the distribution narrows to an interquartile range of 0.0 and median of 0.75, indicating that users have continued to improve and stabilized their performance. The difference between session 1 and 3 indicates a significant variation in user performance. Overall, the data shows that user performance becomes more consistent and improves significantly over time with additional training. Notably, all participants exhibited improved telemanipulation skills, and were confident in the task by the end of the training period.

All participants reported no issues with visual or haptic perception. The experimenter provided detailed procedure explanations, taking about 5-minutes to ensure correct setup before the experiment for everyone. For the set of beginner users, subjects had another 5-minutes practice session to familiarize themselves with the telemanipulation system. No sensitive or personally identifiable data was collected, and protocols were approved by an ethical committee (No. EK-2023-N-81).
<figure><img src="image_9.png" /><figcaption>**Fig. 7:** Illustrative overview of the training activities (a)-(c) and the evaluation task (d).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 8:** Evaluation of the experienced operator group during the training period. The violin plots illustrate the distribution of users performance with ABBT evaluation session between training sessions.</figcaption></figure>

## 4.4 Evaluation Metrics

We considered three metrics:

* The number of blocks transferred, $N \in N$, in the given 80 s. It evaluates the gross manual dexterity of the operator. Higher is better.
* The energy spent on block displacement $\mathcal{E} \in \mathbb{R}$

$$ \mathcal{E} = \frac{\int_{0}^{\mathcal{T}} \frac{1}{2} m \|v\|^2 dt}{\mathcal{N}}, \qquad (7) $$

where $\mathcal{T} \in \mathbb{R}^+$ is the task duration, $m = 4.82 \text{ kg}$ is the OMAV mass and $\|v\|$ is its velocity norm. It evaluates the operator efficiency consumption at using the aerial vehicle. Lower is better.

* The perceived workload $\mathcal{W}$ for each experimental condition. It is measured using the NASA-TLX questionnaire [48], assessing the workload experienced by participants during the task and includes six dimensions: 1. mental demand (MD) is the perceived cognitive load, 2. physical demand (PD) is the perceived physical effort, 3. temporal demand (TD) is the perceived time pressure, 4. effort (EF) is the perceived level of exertion required, 5. performance (PE) is the perceived success in achieving the task goal, and 6. frustration (FR) is the perceived level of stress during the task.

## 4.5 Data Collection and Analysis

Taguchi design allows to identify the optimal combination levels (see Sec. 4.1) for each factor that maximize/minimize the selected metrics. Factor rank indicates its relative importance, with Rank = 1 signifying the highest impact on the result. The main effects plots of Taguchi are presented in Fig. 9. In our case, we aim to maximize the mean number of blocks transferred ($\mathcal{N}$) among the experiments, while minimizing the mean energy spent ($\mathcal{E}$). We also want to minimize the standard deviation (StDev) to reduce variability in results. The signal-to-noise ratio (SNR) evaluates the system's quality by comparing the desired signal with the unwanted variation in the output. A higher SNR indicating better performance by minimizing the influence of noise on the experimental output.

To apply ANOVA, data have to comply with the Shapiro-Wilk normality test [49]. The mixed ANOVA has been conducted with a significance level set at 0.05, unless stated otherwise. All data passed Shapiro-Wilk normality test, except for SC-H-B in metric $\mathcal{E}$ with p-values < 0.010. Post hoc analysis yielded significant p-values, which are presented in Tab. 3. Additionally, significant p-values from the consideration of simple main effects are also reported in Fig. 10. Post hoc tests are used to perform pairwise comparison between the different setups. For the comparison of all the metrics, we conducted mixed ANOVA tests.

If a statistically significant interaction between variables is observed, conducting an analysis of the simple main effects becomes necessary [49]. Tukey's method is used to identify confidence intervals for pairwise differences between factor level means. By using the confidence intervals of the differences we determine whether the differences are practically significant, and examine their confidence intervals. Using the grouping table in Tab. 3, we can quickly identify statistically significant mean differences in groups not sharing a common letter. Results are presented in Sec. 5.

After the experiment, participants were asked to complete a NASA Task Load Index (NASA-TLX) questionnaire. The aim is to understand how workload subscales
<figure><img src="image_11.png" /><figcaption>**Fig. 9:** Main effects plots showing the individual influence of factors on response characteristics (i.e., Means, Standard Deviations and SNR). The main effect shows the direction and magnitude of the effect that each factor has on the response, highlighting the identification of the optimal levels for those factors.</figcaption></figure>

are perceived with the different setups. User filled first a pairwise comparison of the different subscales (MD, PD, TD, EF, PE, FR) to associate a rate between 0 and 5. Then, participants were asked to rate subscales from 0 to 100. A score of 0 represents "no workload", and a score of 100 represents "extreme workload". The adjusted workload coefficient is the product of the pairwise with the subscales. Because we found non-normal data from NASA-TLX, a Mood's Median nonparametric test has been run [50].

# 5 Result and Discussion

Main effects plots from the Taguchi design are shown in Fig. 9. Data distributions of the study are plotted in Fig. 10, while distribution from the NASA-TLX is in Fig. 11. The explanation and interpretation of the results follow.

## 5.1 Haptics and 3D Vision Improve Dexterity

The use of haptics and 3D MR vision were independently beneficial to the user. From Fig. 9 we can notice means of MR visual and haptic feedback improving user dexterity with respectively ($\text{Delta} = 0.658$, $\text{Rank} = 2$) and ($\text{Delta} = 0.230$, $\text{Rank} = 3$). Visual feedback reduced standard deviation and increased SNR ($\text{Delta} = 2.41$, $\text{Rank} = 2$), with more blocks $\mathcal{N}$ moved during the ABBT. Haptic feedback improved the overall user dexterity, but contrary to visual feedback, it comes with a slight dispersion, and therefore a smaller performance improvement than with MR. A possible explanation is that users interpret force feedback differently and, if its meaning is not well understood,
<figure><img src="image_12.png" /><figcaption>(a) Quantity of blocks transferred</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) Robot energy consumed per block transfered</figcaption></figure>

**Fig. 10:** Human-subjects study results from ANOVA. Boxplot distribution of metrics for the different setup configurations. For (10a) the quantity of block transferred *N*, and (10b) the energy consumed *E* per block transferred.

it can be perceived as a perturbation instead of a clear force information, especially for some beginners that reported it.

From the grouping information (Tab. 3) we confirm the previous results and add precision. The second column (referred to with letter B) shows significant difference with the groups containing MR and/or H, and the group using SC-И. Thus, the use of haptics and/or MR vision robustifies the operator's dexterity. A beginner with the setup using MR-H, compared to an experienced operator with SC-И (MR-H-B vs. SC-И-Е) are not significantly different in dexterous aerial manipulation. Furthermore, with the number of block displaced (Fig. 10b), contrary to energy efficiency (Fig. 10a)
<figure><img src="image_14.png" /><figcaption>Fig. 11: Distributions of the adjusted workload subscales</figcaption></figure>

where MR vision seems to be more important, a power-saving flight looks to take more
advantage of haptic feedback. In other words, haptics allows longer flight time.

5.2 Training and Transfer Learning

The most important factor at improving blocks quantity transferred N in ABBT is
the operator experience. This can be seen from the means (Delta = 2.413, Rank = 1)
in Fig. 9. User experience also improves repeatability of the result as one can see from
the SNR (Delta = 4.442, Rank = 1). Experienced operators look to better exploit
<table><caption>**Table 3:** Experimental evaluation summary</caption><tbody><tr><td><strong>Subjects</strong></td><td colspan="3">28 (14 beginners, 14 experienced)</td></tr><tr><td><strong>Task</strong></td><td colspan="3">Teleoperation of the aerial manipulator to move blocks from an initial position to a target area.</td></tr><tr><td rowspan="4"><strong>Factors</strong></td><td><u>Operator Expertise</u></td><td colspan="2"></td></tr><tr><td>B (beginner), E (experienced operator)</td><td colspan="2"></td></tr><tr><td><u>Display</u></td><td colspan="2"></td></tr><tr><td>SC (standard 2D screen), MR (MR 3D display)</td><td colspan="2"></td></tr><tr><td><u>Feedback</u></td><td colspan="3"></td></tr><tr><td></td><td>H (haptic feedback), μ̂ (no haptic feedback)</td><td colspan="2"></td></tr><tr><td colspan="4"><strong>Statistical analysis</strong> (only significant <em>p</em> values are shown)</td></tr><tr><td colspan="4"><u>Blocks quantity transferred, N</u></td></tr><tr><td colspan="4"><u>Simple main effect</u></td></tr><tr><td>SC-μ̂-B vs SC-H-E</td><td><em>p</em> &lt; 0.001</td><td>MR-H-B vs SC-H-E</td><td><em>p</em> &lt; 0.001</td></tr><tr><td>SC-μ̂-E vs SC-μ̂-B</td><td><em>p</em> &lt; 0.001</td><td>MR-H-E vs SC-μ̂-B</td><td><em>p</em> &lt; 0.001</td></tr><tr><td>MR-μ̂-E vs SC-μ̂-B</td><td><em>p</em> &lt; 0.001</td><td>MR-H-B vs SC-μ̂-E</td><td><em>p</em> = 0.021</td></tr><tr><td>MR-H-E vs MR-H-B</td><td><em>p</em> &lt; 0.001</td><td>MR-μ̂-E vs MR-H-B</td><td><em>p</em> &lt; 0.001</td></tr><tr><td colspan="4"><u>Grouping information</u></td></tr><tr><td>MR-H-E</td><td>A</td><td></td><td></td></tr><tr><td>MR-μ̂-E</td><td>A</td><td></td><td></td></tr><tr><td>SC-H-E</td><td>A</td><td></td><td></td></tr><tr><td>SC-μ̂-E</td><td>A</td><td>B</td><td></td></tr><tr><td>MR-H-B</td><td></td><td>B</td><td>C</td></tr><tr><td>SC-μ̂-B</td><td></td><td></td><td>C</td></tr><tr><td colspan="4"><u>Energy consumed per move, E</u></td></tr><tr><td colspan="4"><u>Simple main effect of feedback</u></td></tr><tr><td>MR-μ̂-E vs MR-H-B</td><td><em>p</em> = 0.010</td><td>MR-H-E vs MR-H-B</td><td><em>p</em> = 0.012</td></tr><tr><td colspan="4"><u>Workload subscales, W</u></td></tr><tr><td colspan="4"><u>Nonparametric test for main effect</u></td></tr><tr><td>TD</td><td><em>p</em> = 0.013</td><td></td><td></td></tr></tbody></table>

the system, which is confirmed by looking at the energy consumption (Delta = 2892, Rank = 1). This result can be explained by the fact that beginners fly more smoothly, as they don't know the limits of the system. On the other hand, experienced pilots are able to accelerate faster.

By running two-way ANOVA, Fig. 10a reveals a significant difference for the operator expertise with *p*-value < 0.001 (*F* = 46.70). Post hoc Tukey test shows that simple main effects of SC-μ̂-B with each E configuration has a statistical significant difference. Likewise, the MR-H-B configuration has statistically significant difference with MR-H-E, 3D-μ̂-E and SC-H-E. This confirms result coming from Taguchi, but specify that only operator experience is significant. Thus, an experienced operator takes advantage of the 3D MR vision and haptics offered by the setup. Then, we want to minimize the energy consumption *E*. Figure 9 shows that operator expertise (Delta = 3.48, Rank = 1) has still the largest effect. Running the two-way ANOVA
(see Fig. 10b) reveals a significant difference for the levels of expertise of the operator with a *p*-value = 0.006 (*F* = 7.84). Post hoc Tukey Method shows statistically significant difference of MR-H-E vs. MR-H-B (*F* = 7.25, *p* = 0.012) and MR-*H*-E vs. MR-H-B (*F* = 7.74, *p* = 0.010). As a result, an experienced operator will perform better at the task, but will consume more energy with the robot, that in turn results in a reduction in flight time.

We can also notice from the ABBT a transfer learning, from the training with haptics and 3D MR vision, to the setup without haptics and 2D screen. Post hoc Tukey Method shows statistically significant difference with SC-*H*-B vs. SC-*H*-E (*F* = 24.13, *p* < 0.001). This means that the experienced group transferred their capacity obtained with our augmented setup, to the one without haptics and a conventional screen.

## 5.3 Limited Influence on Workload

For the NASA-TLX, because none of the subscales workload data passed Shapiro-Wilk normality test, we ran Mood's Median nonparametric test. The results are plotted in Fig. 11. Only TD accepted the null hypothesis, *population medians are all equal*, with (χ² = 14.53, *p* = 0.013). The different setups seem to not influence the workload perception of the user. A significant difference has been noticed for TD. Experienced operators are more involved in carrying out the task, and are therefore more impacted by time spent on the task, than beginners. On the other hand, when using the 2D screen (Fig. 6a), participants used the shadows to get the relative position of the robot with respect to the payload. Participants also reported that using a standard 2D screen requires more attention to fly. This tendency can be seen on MD of Fig. 11 without being significant. We observed several interesting tendencies that, however, are not statistically significant due to data dispersion. The tendency for PD to be more felt by the beginners than experienced operators can be attributed to the latter's habit of using the device. Additionally, the observation that H-E for PD is greater than *H*-E can be explained by the additional force provided by the joystick. The perceived PE of beginners is lower than that of experienced, who was more aware of their capabilities due to the training. The EF was slightly less for MR-H-E, which was the training factor configuration. Furthermore, some participants verbalized that using haptics reduced FR. Conversely, we observed a tendency for the standard setup with SC-*H* to increase FR for both beginners and experienced.

## 5.4 Limitations

To evaluate human dexterity we chose to adapt the BBT for aerial manipulation, because it has already been used successfully in virtual environments [46]. However there exist other reliable tests such as the Minnesota Manual Dexterity Test (MMDT) and the Purdue Pegboard (PP). These two tasks, which involve peg-in-hole operation, could also benefit from haptic bilateral feedback. Nevertheless, there is a significant correlation of MMDT and PP with the BBT [51]. Thus, evaluating the operator's dexterity in accurately picking up the box during dynamic movement is a reasonable approach.
From a virtual rendering point of view, an interesting point to highlight is that literature has shown a difference in how humans perceive objects in VR vs. MR [52, 53]. Such a difference, although limited to a few sensory features such as stiffness and shape, might have played a role in the outcome of our experiment. Another point to highlight is the reliance on a motion capture system for localization. However, this was not a central focus of our research and does not affect the validity of our findings. The generalization of the system to work in other environments with on board rendering [54] can be addressed with widely available and effective method, like visual dynamic object-aware SLAM system [55].

# 6 Conclusion

This work studies the effect of 3D MR vision and haptics on operator performance in the context of aerial tele-manipulation. A multimodal bilateral teleoperation system based on MR vision and haptic force feedback has been developed for aerial manipulation tasks. Experimental APhI tasks demonstrate sub-centimeter-level accuracy and precise force application when teleoperating aerial manipulators using the proposed system. Results of a user study enrolling 28 participants confirm quantitatively that both interaction feedback and MR-based visualization indeed improve user dexterity and flight efficiency. Nevertheless, the experience of the operator is still the most important factor.

Since the training of the experienced group only happened with 3D MR vision and haptics, future direction would question the learning rate at using or not the system. In robot assisted surgery, haptic feedback has been proved crucial in early psychomotor skill acquisition [56]. Its potential generalisation to aerial manipulation has to be explored. The combination with the transfer learning we observed would reduce the training duration of operators.

# Declarations

* Funding - The research leading to these results has been supported by the AERO-TRAIN project, European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 953454. The authors are solely responsible for its content. This work was supported by the project ANR-23-CE33-0013 "FlyHandyBot".

* Conflicts of interest/Competing interests - The authors declare that they have no competing interests.

* Ethics approval - This work involved human subjects in its research. This study was conducted in accordance with the ethical principles outlined by the ETH Zurich Ethics Committee. The research protocol and procedures were reviewed and approved as proposal No. EK-2023-N-81.

* Consent to participate - All participants provided informed consent before participating in the study.

* Consent for publication - All participants provided written informed consent for the publication of the study results.
* Data availability statement - The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

* Author Contributions – Julien Mellet (corresponding author) led the research, developing the framework, and coordinating its real-world deployment. Mike Allenspach proposed the foundational haptic telemanipulation framework for the OMAV and contributed to its integration. Eugenio Cuniato managed the experimental deployment. Claudio Pacchierotti provided the data processing methodology for the user study. Roland Siegwart, as head of the laboratory, facilitated the system deployment and supported the research. Marco Tognon supervised the study, ensuring its methodological development and direction.

## References

[1] Bodie, K., Brunner, M., Pantic, M., Walser, S., Pfändler, P., Angst, U., Siegwart, R., Nieto, J.: Active interaction force control for contact-based inspection with a fully actuated aerial vehicle. IEEE Transactions on Robotics 37(3), 709–722 (2021) https://doi.org/10.1109/TRO.2020.3036623

[2] Ollero, A., Tognon, M., Suarez, A., Lee, D., Franchi, A.: Past, present, and future of aerial robotic manipulators. IEEE Transactions on Robotics 38(1), 626–645 (2022) https://doi.org/10.1109/TRO.2021.3084395

[3] Tognon, M., Chávez, H.A.T., Gasparin, E., Sablé, Q., Bicego, D., Mallet, A., Lany, M., Santi, G., Revaz, B., Cortés, J., Franchi, A.: A truly-redundant aerial manipulator system with application to push-and-slide inspection in industrial plants. IEEE Robotics and Automation Letters 4(2), 1846–1851 (2019) https://doi.org/10.1109/LRA.2019.2895880

[4] Ruggiero, F., Lippiello, V., Ollero, A.: Aerial manipulation: A literature review. IEEE Robotics and Automation Letters 3(3), 1957–1964 (2018) https://doi.org/10.1109/LRA.2018.2808541

[5] Meng, X., He, Y., Han, J.: Survey on aerial manipulator: System, modeling, and control. Robotica 38(7), 1288–1317 (2020) https://doi.org/10.1017/S0263574719001450

[6] Aleotti, J., Micconi, G., Caselli, S., Benassi, G., Zambelli, N., Bettelli, M., Zappettini, A.: Detection of nuclear sources by uav teleoperation using a visuo-haptic augmented reality interface. Sensors 17(10) (2017) https://doi.org/10.3390/s17102234

[7] Hamandi, M., Usai, F., Sablé, Q., Staub, N., Tognon, M., Franchi, A.: Design of multirotor aerial vehicles: A taxonomy based on input allocation. The International Journal of Robotics Research 40(8-9), 1015–1044 (2021) https://doi.org/10.1177/02783649211025998
[8] Park, S., Lee, J., Ahn, J., Kim, M., Her, J., Yang, G.-H., Lee, D.: Odar: Aerial manipulation platform enabling omnidirectional wrench generation. IEEE/ASME Transactions on Mechatronics **23**(4), 1907–1918 (2018) https://doi.org/10.1109/TMECH.2018.2848255

[9] Brunner, M., Rizzi, G., Studiger, M., Siegwart, R., Tognon, M.: A planning-and-control framework for aerial manipulation of articulated objects. IEEE Robotics and Automation Letters **7**(4), 10689–10696 (2022) https://doi.org/10.1109/LRA.2022.3191178

[10] Ryll, M., Muscio, G., Pierri, F., Cataldi, E., Antonelli, G., Caccavale, F., Bicego, D., Franchi, A.: 6d interaction control with aerial robots: The flying end-effector paradigm. The International Journal of Robotics Research **38**(9), 1045–1062 (2019) https://doi.org/10.1177/0278364919856694

[11] Trujillo, M.A., Martinez-de Dios, J.R., Martin, C., Viguria, A., Ollero, A.: Novel aerial manipulator for accurate and robust industrial ndt contact inspection: A new tool for the oil and gas inspection industry. Sensors **19**(6) (2019) https://doi.org/10.3390/s19061305

[12] Ollero, A., Heredia, G., Franchi, A., Antonelli, G., Kondak, K., Sanfeliu, A., Viguria, A., Martinez-de Dios, J.R., Pierri, F., Cortes, J., Santamaría-Navarro, A., Trujillo Soto, M.A., Balachandran, R., Andrade-Cetto, J., Rodriguez, A.: The aeroarms project: Aerial robots with advanced manipulation capabilities for inspection and maintenance. IEEE Robotics & Automation Magazine **25**(4), 12–23 (2018) https://doi.org/10.1109/MRA.2018.2852789

[13] Elbanhawi, M., Mohamed, A., Clothier, R., Palmer, J., Simic, M., Watkins, S.: Enabling technologies for autonomous mav operations. Progress in Aerospace Sciences **91** (2017) https://doi.org/10.1016/j.paerosci.2017.03.002

[14] Mersha, A.Y., Stramigioli, S., Carloni, R.: On bilateral teleoperation of aerial robots. IEEE Transactions on Robotics **30**(1), 258–274 (2014) https://doi.org/10.1109/TRO.2013.2281563

[15] Lee, J., Balachandran, R., Kondak, K., Coelho, A., Stefano, M.D., Humt, M., Feng, J., Asfour, T., Triebel, R.: Virtual Reality via Object Pose Estimation and Active Learning: Realizing Telepresence Robots with Aerial Manipulation Capabilities (2023). https://doi.org/10.48550/arXiv.2210.09678

[16] Islam, S., Gan, D., Ashour, R., Dario, P., Dias, J., Seneviratne, L.D.: Haptics and virtual reality based bilateral telemanipulation of miniature aerial vehicle over open communication network. In: 2017 18th International Conference on Advanced Robotics (ICAR), pp. 334–339 (2017). https://doi.org/10.1109/ICAR.2017.8023629

[17] Yashin, G.A., Trinitatova, D., Agishev, R.T., Ibrahimov, R., Tsetserukou, D.:
AeroVR: Virtual Reality-based Teleoperation with Tactile Feedback for Aerial Manipulation (2019). https://doi.org/10.1109/ICAR46387.2019.8981574

[18] Lee, J., Balachandran, R., Sarkisov, Y.S., De Stefano, M., Coelho, A., Shinde, K., Kim, M.J., Triebel, R., Kondak, K.: Visual-inertial telepresence for aerial manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1222–1229 (2020). https://doi.org/10.1109/ICRA40945.2020.9197394

[19] Ibrahimov, R., Tsykunov, E., Shirokun, V., Somov, A., Tsetserukou, D.: Dronepick: Object picking and delivery teleoperation with the drone controlled by a wearable tactile display. In: 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1–6 (2019). https://doi.org/10.1109/RO-MAN46459.2019.8956344

[20] Ai, Z., Livingston, M.A., Moskowitz, I.S.: Real-time unmanned aerial vehicle 3d environment exploration in a mixed reality environment. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 664–670 (2016). https://doi.org/10.1109/ICUAS.2016.7502588

[21] Morando, L., Loianno, G.: Spatial assisted human-drone collaborative navigation and interaction through immersive mixed reality. In: 2024 International Conference on Robotics and Automation (ICRA) (2024). https://doi.org/10.48550/arXiv.2402.04070

[22] Hong, S., Kazuya, S., Yuta, G., Shogo, O.: Drone rider: Effects of translucent actual own body parts on embodiment in virtual reality space. International Symposium on Affective Science and Engineering ISASE2024, 1–4 (2024) https://doi.org/10.5057/isase.2024-C000040

[23] Walker, M.E., Hedayati, H., Szafir, D.: Robot teleoperation with augmented reality virtual surrogates. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 202–210 (2019). https://doi.org/10.1109/HRI.2019.8673306

[24] Allenspach, M., Laasch, S., Lawrance, N., Tognon, M., Siegwart, R.: Mixed reality human-robot interface to generate and visualize 6dof trajectories: Application to omnidirectional aerial vehicles. In: 2023 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 395–400 (2023). https://doi.org/10.1109/ICUAS57906.2023.10156523

[25] Beers, R., Wolpert, D., Haggard, P.: When feeling is more important than seeing in sensorimotor adaptation. Current biology : CB 12, 834–7 (2002) https://doi.org/10.1016/S0960-9822(02)00836-9

[26] Pacchierotti, C., Prattichizzo, D.: Cutaneous/tactile haptic feedback in robotic teleoperation: Motivation, survey, and perspectives. IEEE Transactions on
Robotics **40**, 978–998 (2024) https://doi.org/10.1109/TRO.2023.3344027

[27] Schill, F., Hou, X., Mahony, R.: Admittance mode framework for haptic teleoperation of hovering vehicles with unlimited workspace. (2010). https://doi.org/10.13140/2.1.4734.4005

[28] Rifai, H., Hua, M.-D., Hamel, T., Morin, P.: Haptic-based bilateral teleoperation of underactuated unmanned aerial vehicles. IFAC Proceedings Volumes **44**(1), 13782–13788 (2011) https://doi.org/10.3182/20110828-6-IT-1002.02989. 18th IFAC World Congress

[29] Gioioso, G., Mohammadi, M., Franchi, A., Prattichizzo, D.: A force-based bilateral teleoperation framework for aerial robots in contact with the environment. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 318–324 (2015). https://doi.org/10.1109/ICRA.2015.7139018

[30] Allenspach, M., Lawrance, N., Tognon, M., Siegwart, R.: Towards 6dof bilateral teleoperation of an omnidirectional aerial vehicle for aerial physical interaction. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 9302–9308 (2022). https://doi.org/10.1109/ICRA46639.2022.9812346

[31] Young, M., Miller, C., Bi, Y., Chen, W., Argall, B.D.: Formalized task characterization for human-robot autonomy allocation. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6044–6050 (2019). https://doi.org/10.1109/ICRA.2019.8793475

[32] Mellet, J., Berra, A., Marcellini, S., Trujillo Soto, M.A., Heredia, G., Ruggiero, F., Lippiello, V.: Design and control of an omnidirectional aerial robot with a miniaturized haptic joystick for physical interaction. In: 2025 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 340–346 (2025). https://doi.org/10.1109/ICUAS65942.2025.11007780

[33] Aggravi, M., Pacchierotti, C., Giordano, P.R.: Connectivity-maintenance teleoperation of a uav fleet with wearable haptic feedback. IEEE Transactions on Automation Science and Engineering **18**(3), 1243–1262 (2021) https://doi.org/10.1109/TASE.2020.3000060

[34] Conti, F., Khatib, O.: Spanning large workspaces using small haptic devices. In: First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference, pp. 183–188 (2005). https://doi.org/10.1109/WHC.2005.118

[35] Palumbo, A.: Microsoft hololens 2 in medical and healthcare context: State of the art and future prospects. Sensors **22**(20) (2022) https://doi.org/10.3390/s22207709
[36] Pacchierotti, C., Meli, L., Chinello, F., Malvezzi, M., Prattichizzo, D.: Cuta- neous haptic feedback to ensure the stability of robotic teleoperation systems. The International Journal of Robotics Research **34**(14), 1773–1787 (2015) https://doi.org/10.1177/0278364915603135

[37] Meccariello, G., Faedi, F., AlGhamdi, S., Montevecchi, F., Firinu, E., Zan- otti, C., Cavaliere, D., Gunelli, R., Taurchini, M., Amadori, A., Vicini, C.: An experimental study about haptic feedback in robotic surgery: may visual feed- back substitute tactile feedback? Journal of Robotic Surgery **10**, 57–61 (2016) https://doi.org/10.1007/s11701-015-0541-0

[38] Taguchi, G.: Introduction to Quality Engineering: Designing Quality Into Prod- ucts and Processes. Asian Productivity Organization, Tokyo (1986). https://doi. org/10.1002/qre.4680040216

[39] Antony, J.: 6 - full factorial designs. In: Antony, J. (ed.) Design of Experiments for Engineers and Scientists (Second Edition), Second edition edn., pp. 63–85. Elsevier, Oxford (2014). https://doi.org/10.1016/C2012-0-03558-2

[40] Girden, E.R.: ANOVA: Repeated Measures. SAGE Publications (1992). https://doi.org/10.4135/9781412983419.n2

[41] Smith, R.: Open Dynamics Engine. http://www.ode.org/ (2008)

[42] Mathiowetz, V., Volland, G., Kashman, N., Weber, K.: Adult norms for the box and block test of manual dexterity. The American journal of occupational therapy **39**(6), 386–391 (1985) https://doi.org/10.5014/ajot.39.6.386

[43] Cho, S., Kim, W.-S., Paik, N.-J., Bang, H.: Upper-limb function assessment using vbbts for stroke patients. IEEE Computer Graphics and Applications **36**(1), 70– 78 (2016) https://doi.org/10.1109/MCG.2015.2

[44] Gieser, S., Gentry, C., LePage, J., Makedon, F.: Comparing objective and sub- jective metrics between physical and virtual tasks. In: Lackey, S., Shumaker, R. (eds.) Virtual, Augmented and Mixed Reality, pp. 3–13. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39907-2_1

[45] Meli, L., Pacchierotti, C., Salvietti, G., Chinello, F., Maisto, M., De Luca, A., Prattichizzo, D.: Combining wearable finger haptics and augmented reality: User evaluation using an external camera and the microsoft hololens. IEEE Robotics and Automation Letters **3**(4), 4297–4304 (2018) https://doi.org/10.1109/LRA. 2018.2864354

[46] Dong, Y., Liu, X., Tang, M., Huo, H., Chen, D., Wu, Z., An, R., Fan, Y.: A haptic-feedback virtual reality system to improve the box and block test (bbt) for upper extremity motor function assessment. Virtual Reality **27**, 1199–1219 (2023) https://doi.org/10.1007/s10055-022-00727-2
[47] Morgan, A.S., Hang, K., Bircher, W.G., Alladkani, F.M., Gandhi, A., Calli, B., Dollar, A.M.: Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test. IEEE Robotics and Automation Letters 5(2), 454–461 (2020) https://doi.org/10.1109/LRA.2019.2961053

[48] Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload. Advances in Psychology, vol. 52, pp. 139–183. North-Holland, United-Stata (1988). https://doi.org/10.1016/S0166-4115(08)62386-9

[49] Draper, N.R., Smith, H.: Applied regression analysis. John Wiley and Sons (1998). https://doi.org/10.1002/9781118625590

[50] Siegel, S., Castellan, N.J.: Nonparametric Statistics for the Behavioral Sciences, 2nd edn. McGraw–Hill, Inc., New-York (1988). https://doi.org/10.1126/science.126.3267.266.a

[51] Desrosiers, J., Rochette, A., Hébert, R., Bravo, G.: The minnesota manual dexterity test: Reliability, validity and reference values studies with healthy elderly people. Canadian Journal of Occupational Therapy 64(5), 270–276 (1997) https://doi.org/10.1177/000841749706400504

[52] Gaffary, Y., Le Gouis, B., Marchal, M., Argelaguet, F., Arnaldi, B., Lécuyer, A.: Ar feels “softer” than vr: Haptic perception of stiffness in augmented versus virtual reality. IEEE Transactions on Visualization and Computer Graphics 23(11), 2372–2377 (2017) https://doi.org/10.1109/TVCG.2017.2735078

[53] Ahn, J.-g., Ahn, E., Min, S., Choi, H., Kim, H., Kim, G.J.: Size perception of augmented objects by different ar displays. In: HCI International 2019-Posters: 21st International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part II 21, pp. 337–344 (2019). https://doi.org/10.1007/978-3-030-23528-4_46 . Springer

[54] Kelly, A., Capstick, E., Huber, D., Herman, H., Rander, P., Warner, R.: Real-time photorealistic virtualized reality interface for remote mobile robot control. In: Pradalier, C., Siegwart, R., Hirzinger, G. (eds.) Robotics Research, pp. 211–226. Springer, Berlin, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19457-3_13

[55] Zhang, J., Henein, M., Mahony, R., Ila, V.: VDO-SLAM: A Visual Dynamic Object-aware SLAM System (2021). https://arxiv.org/abs/2005.11052

[56] Meijden O.A.J, Schijven, M.P.: The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surgical Endoscopy 23 (2009) https://doi.org/10.1007/s00464-008-0298-x