<figure><img src="image_1.png" /></figure>

HAL
open science

# Haptic Shared Control of a Pair of Microrobots for Telemanipulation using Constrained Optimization

Leon Raphalen, Marco Ferro, Sarthak Misra, Paolo Robuffo Giordano,
Claudio Pacchierotti

► To cite this version:

Leon Raphalen, Marco Ferro, Sarthak Misra, Paolo Robuffo Giordano, Claudio Pacchierotti. Haptic Shared Control of a Pair of Microrobots for Telemanipulation using Constrained Optimization. IROS 2025, Oct 2025, Hangzhou, China. hal-05157735

HAL Id: hal-05157735

https://hal.science/hal-05157735v1

Submitted on 11 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Haptic Shared Control of a Pair of Microrobots for Telemanipulation using Constrained Optimization

Leon Raphalen, Marco Ferro, Sarthak Misra, Paolo Robuffo Giordano, Claudio Pacchierotti

*Abstract*—Microrobotics implies actuation-related con-
straints that make safe telemanipulation particularly chal-
lenging. We present a haptic shared control system for
electromagnetic-based telemanipulation of a pair of microrobots
using a constrained optimization framework. Our contribu-
tions include: (1) a Quadratic Programming formulation with
Control Lyapunov Functions and Control Barrier Functions,
for safe and stable navigation in cluttered environments; (2) a
shared control architecture, combining a haptic interface and
simulation environment, to teleoperate the microrobots and
enable micromanipulation capabilities; and (3) haptic shared
control strategies offering visuo-haptic cues for task execution.
The approach is validated through a user study, highlighting
better navigation accuracy, control stability and task efficiency.

# I. INTRODUCTION

Controlling robotic systems at the microscale is crucial
for a wide range of applications, from targeted drug delivery
and biomedical procedures to microassembly [1]–[4]. In par-
ticular, micromanipulation, involving controlled interactions
with small-scale objects in constrained environments, is a
delicate task that requires high dexterity, accuracy, and real-
time responsiveness [5]. Some proposed approaches include
schemes with single-robot manipulators [6], [7], while recent
advances highlight the potential of multi-robot systems for
this typology of tasks [8].

Optimization-based control techniques also offer promis-
ing approaches [9], [10] to meet these requirements. At the
macroscale, Quadratic Programming (QP) frameworks com-
bined with Control Lyapunov Functions (CLFs) and Control
Barrier Functions (CBFs) have proven effective in handling
multi-objective control tasks while providing formal safety
guarantees [11]–[13]. Model-Predictive Control (MPC) [14]
has also been widely used to predict and control system
behavior over time, particularly under constraints [15], [16].
While these techniques are well-established in macroscale
robotics, their adaptation to the microscale remains limited,
with most studies focusing on autonomous control or simple
teleoperation approaches [17]–[19].

On the other hand, *shared control* strategies also provide
a valuable and complementary approach, by combining the
precision of autonomous systems with the cognitive and
perceptual skills of human operators [20], [21]. This is
particularly advantageous for managing robotic systems with

high degrees of freedom (DoFs), as it reduces cognitive load
and execution time [22], [23]. Including haptic feedback
and delivering real-time force information to the user further
enhances the effectiveness of shared control paradigms, thus
enabling safer and more intuitive interactions [24], [25].

Haptic shared control paradigms have been effectively
applied at the macroscale and, to a lesser extent, at the
microscale [26]–[29]. In [29], we proposed a haptic shared
control strategy for navigating electromagnetic microspheres
in cluttered environments using potential fields. While effec-
tive for obstacle avoidance, this method lacks formal safety
guarantees and does not address micromanipulation tasks.

To the best of our knowledge, integrating haptic shared
control with advanced optimization-based control techniques
for microscale telemanipulation remains unexplored. There-
fore, we present a novel approach to enable haptic shared
control of a pair of microrobots, targeting telemanipulation
tasks through optimization techniques. The main contribu-
tions of our work consist in:

* A constrained optimization framework, leveraging QP, CLFs and CBFs, to enable safe and stable navigation of a pair of microrobots in cluttered environments;

* A haptic shared control architecture, consisting of a haptic interface, a simulation environment and different haptic assistance modalities, to teleoperate the micro-robots during a representative micromanipulation task;

* A set of haptic shared control strategies, enabling the user to accomplish the task by receiving different combinations of visuo-haptic cues, validated through a user study.

<figure><img src="image_3.png" /><figcaption>Fig. 1: A user manipulates a 6-DoF haptic device to control a pair of magnetically actuated microrobots, while haptic feedback provides obstacle avoidance and navigation information.</figcaption></figure>

L. Raphalen, M. Ferro, P. Robuffo Giordano and C. Pacchierotti are with CNRS, Univ Rennes, Inria, IRISA (Rennes, FR).

S. Misra is with the Surgical Robotics Laboratory, Dept. of Biomechanical Engineering, University of Twente (Enschede, NL). S. Misra is also with the Surgical Robotics Lab., Dept. of Biomaterials and Biomedical Technology, Unive. Groningen and Univ. Medical Centre Groningen (Groningen, NL).

This project has received funding from the EU Horizon Europe R&I programme under grant agreement No 101070066 (project RÈGO), and by the French ANR for project MULTISHARED No ANR-20-CHIA-0017.
## II. OPTIMAL MAGNETIC CONTROL OF PAIRS OF MICROROBOTS

### A. Problem formulation

Consider a pair of magnetized microspheres (from now, we refer to them as “microrobots”) inside a small-scale workspace, denoted by the reference frame $\mathcal{F}_W$, and subjected to a controlled magnetic field. This field is generated by an electromagnetic actuation system that regulates currents through $N \ge 6$ coils strategically placed around the workspace, as, e.g., the BatMag system presented in [30] (shown in the bottom left corner of Fig. 1). Hence, the resulting magnetic field and gradient allow to generate arbitrary forces in any arbitrary pair of positions within the workspace.

We want to control the microrobots’ motion through the input forces resulting from the magnetic field, enabling them to navigate within the given workspace. We assume microrobots’ position and velocity to be continuously measured (e.g., through external vision-based tracking systems). The workspace is bounded and contains known obstacles to be avoided. Furthermore, we account for potential dipole-dipole magnetic interactions when the microrobots come within a critical distance, which could lead to unwanted collisions.

The task and requirements described above can be translated in the following list of control objectives:

(O1) drive the pair of microrobots to a desired position;

(O2) avoid collisions with workspace boundaries, surrounding obstacles, and among the two microrobots;

(O3) minimize the generated input control forces.

We show in the next Section how to formulate these control objectives as an optimal control problem.

### B. System modelling

The magnetic field exerts forces $f_{\mathcal{R},1}, f_{\mathcal{R},2} \in \mathbb{R}^3$ on the microrobots at their positions $p_1, p_2 \in \mathbb{R}^3$, respectively, resulting in the following second-order motion model

$$m_i(\ddot{p}_i - \mathbf{g}) + c\mathbf{v}_i = f_{\mathcal{R},i}, \quad i = 1, 2 \quad (1)$$

where $\mathbf{v}_i = \dot{p}_i$ is the velocity of the i-th microrobot, $m_i$ its mass, c the viscous coefficient of the medium filling the workspace, and $\mathbf{g}$ the gravity vector with respect to $\mathcal{F}_W$. Note that the state $p_i$ has a relative degree of $\rho = 2$ w.r.t. the force inputs $f_{\mathcal{R},i}$ — this will be of importance in the following derivations. By choosing $\mathbf{x} = (\mathbf{p}_1, \mathbf{p}_2, \mathbf{v}_1, \mathbf{v}_2)^T = (\mathbf{p}, \mathbf{v})^T \in \mathbb{R}^n$ and $\mathbf{u} = (f_{\mathcal{R},1}, f_{\mathcal{R},2})^T \in \mathbb{R}^m$, with $n = 12, m = 6$, one can reformulate (1) as the *affine system*

$$\dot{\mathbf{x}} = \mathbf{f}(\mathbf{x}) + \mathbf{h}(\mathbf{x})\mathbf{u}, \quad (2)$$

with $\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^n$ and $\mathbf{h}: \mathbb{R}^n \to \mathbb{R}^{n \times m}$ locally Lipschitz continuous functions. Hence, the control problem can be cast as a Quadratic Programming (QP) problem subject to linear inequality constraints w.r.t. the input vector $\mathbf{u}$, i.e., we aim at finding the optimal control input $\mathbf{u}^*$ such that

$$\begin{array}{ll}
\mathbf{u}^* = \arg \min_{\mathbf{u} \in \mathbb{R}^m} & \mathcal{C}(\mathbf{u}) \\
\text{s.t.} & \mathbf{A}\mathbf{u} \le \mathbf{b}, \quad (3) \\
& \mathbf{u}_{lb} \le \mathbf{u} \le \mathbf{u}_{ub}
\end{array}$$

where $\mathcal{C}(\mathbf{u})$ is a quadratic cost function of $\mathbf{u}$, $\mathbf{A}$ and $\mathbf{b}$ define appropriate affine inequality constraints, and $\mathbf{u}_{lb}$ and $\mathbf{u}_{ub}$ represent the input lower and upper saturation limits. This formulation allows to design a safety-critical controller that unifies both stability (O1) and safety (O2) constraints, while minimizing the overall control effort (O3).

### C. Control objectives formalization

#### 1) Objective O1 (Steering to the desired position):

Moving the pair of microrobots to an assigned pair of reference positions $\mathbf{p}_r = (\mathbf{p}_{r,1}, \mathbf{p}_{r,2})^T$, with velocity $\mathbf{v}_r = (\mathbf{v}_{r,1}, \mathbf{v}_{r,2})^T$, is achieved by regulating the system state (2) to the reference $\mathbf{x}_r = (\mathbf{p}_r, \mathbf{v}_r)^T$, i.e.

$$\mathbf{x}_r - \mathbf{x} \to \mathbf{0}. \quad (4)$$

To accomplish this stability objective, we employ a Control Lyapunov Function (CLF) formulation [13]. In particular, a continuously differentiable, positive definite function $V: \mathbb{R}^n \to \mathbb{R}$ is a CLF for the system (2) if it satisfies the following affine inequality in the control inputs $\mathbf{u}$:

$$\inf_{\mathbf{u} \in \mathbb{R}^m} \{L_f V(\mathbf{x}) + L_h V(\mathbf{x})\mathbf{u}\} \le -\gamma(V(\mathbf{x})), \quad (5)$$

where $L_f$ and $L_h$ are the Lie derivative operators with respect to the system trajectories $\mathbf{f}(\mathbf{x})$ and $\mathbf{h}(\mathbf{x})$, and $\gamma$ is a class-$\mathcal{K}$ function. For our control Objective (4), we choose

$$\begin{aligned}
V(\mathbf{x}) &= \frac{1}{2}(\mathbf{x}_d - \mathbf{x})^T(\mathbf{x}_d - \mathbf{x}), \quad k_\gamma > 0 \\
\gamma(V(\mathbf{x})) &= k_\gamma V(\mathbf{x})
\end{aligned} \quad (6)$$

where $\mathbf{x}_d$ is defined by following a backstepping technique [31] to account for the system relative degree $\rho = 2$:

$$\mathbf{x}_d = \begin{bmatrix} \mathbf{p}_r \\ k_p(\mathbf{p}_r - \mathbf{p}) + k_v \mathbf{v}_r \end{bmatrix}, \quad k_p, k_v > 0 \quad (7)$$

#### 2) Objective O2 (Avoid collisions): To address this objective, we consider position-dependent collision avoidance constraints, which are of relative degree 2 w.r.t. the control inputs. As such, we formalize these constraints through a higher-order generalization of the Control Barrier Function (CBF) formulation, known as a High-Order Control Barrier Function (HOCBF) [11], [12].

Specifically, a function $B(\mathbf{x})$ is a HOCBF of order 2 if there exist two extended class-$\mathcal{K}$ functions, $\alpha_1$ and $\alpha_2$, s.t.:

$$\sup_{\mathbf{u} \in \mathbb{R}^m} \{L_f^2 B(\mathbf{x}) + L_h L_f B(\mathbf{x})\mathbf{u} + L_f \alpha_1(B(\mathbf{x})) \\ + \alpha_2(\psi(\mathbf{x}))\} \ge 0, \quad \forall \mathbf{x} \in \mathbb{R}^n \quad (8)$$

where $\psi(\mathbf{x}) = \dot{B}(\mathbf{x}) + \alpha_1(B(\mathbf{x}))$. This formulation, to which we will refer as 2OCBF, is then used to formalize different position-based collision avoidance constraints, through the definition of a suitable barrier candidate $B(\mathbf{x})$. For each constraint, we choose $\alpha_1(B(\mathbf{x})) = k_{\alpha_1}B(\mathbf{x})$ and $\alpha_2(\psi(\mathbf{x})) = k_{\alpha_2}\psi(\mathbf{x})^3$, with $k_{\alpha_1}, k_{\alpha_2} > 0$.

**Workspace boundaries (wb).** We assume the environment to be modeled as the workspace of the BatMag system [30], i.e., as a rectangular cuboid. Thus, we require the i-th microrobot position $\mathbf{p}_i \in [\mathbf{p}_{\min}, \mathbf{p}_{\max}]$, for $i = 1, 2$, where
$p_{min}$ and $p_{max}$ define the limits of the environment. This is ensured through the following barrier candidate:

$$B_{wb}(\mathbf{x}) = \begin{bmatrix} \mathbf{p} - p_{min} \\ p_{max} - \mathbf{p} \end{bmatrix} \qquad (9)$$

Obstacle avoidance (obs). Assuming $N_O$ obstacles in the environment, for each i-th microrobot and for each j-th obstacle in position $p_j$, we consider a spherical collision box with radius $r_j$ and define the following barrier candidate:

$$B_{obs_{ij}} = \|\mathbf{p}_i - \mathbf{p}_j\| - (r_i + r_j) \qquad \begin{cases} i = 1, 2 \\ \forall j = 1, \dots, N_O \end{cases} \qquad (10)$$

where $r_i$ is the i-th microrobot radius.

Mutual collisions (mc). To prevent collisions between microrobots, which may occur due to dipole-dipole magnetic interactions when their relative distance falls below a minimum threshold $d_{min}$, we define the following constraint:

$$B_{mc}(\mathbf{x}) = \|\mathbf{p}_1 - \mathbf{p}_2\| - \delta_{mc} \qquad (11)$$

where $\delta_{mc} > d_{min}$ represents the minimal distance we wish to maintain between the microrobots.

3) **Objective O3 (Minimize input control force):** To minimize input control forces in (2), we consider the norm of the microrobots' acceleration in the cost function $C(\mathbf{u})$, i.e.,

$$C(\mathbf{u}) = \frac{1}{2} \mathbf{u}^T H \mathbf{u} + c^T \mathbf{u} = \sum_{i=1}^{2} \ddot{p}_i^T \ddot{p}_i = \sum_{i=1}^{2} \frac{1}{m_i^2} \mathbf{u}_i^T \mathbf{u}_i + \frac{2}{m_i} (g - \frac{c}{m_i} v_i)^T \mathbf{u}_i + \|g - \frac{c}{m_i} v_i\|^2 \qquad (12)$$

We also enforce input saturation constraints in (3) by scaling the input magnetic force element-wise, setting $\mathbf{u}_{ub} = -\mathbf{u}_{lb} = \frac{1}{\sqrt{3}} \cdot f_{max}$.

### D. Optimal control formulation

The control Objectives formulated above are finally collected to build the QP-based optimal control problem (3), which can be solved using a standard quadratic programming solver. In particular, to handle possible feasibility issues that could arise between the stability and safety constraints, a slack variable $\delta$ is introduced as an additional optimization variable, to relax the stability Objective O1 in the CLF formulation with respect to the safety Objective O2:

$$\begin{array}{l} \mathbf{u}^*, \delta^* = \arg \min \frac{1}{2} \mathbf{u}^T H \mathbf{u} + c^T \mathbf{u} + \mu \delta^2 \\ \text{s.t.} \quad \left\{ \begin{array}{l} L_f V(\mathbf{x}) + L_h V(\mathbf{x}) \mathbf{u} + \gamma(V(\mathbf{x})) \leq \delta \\ L_f^2 B_\circ(\mathbf{x}) + L_h L_f B_\circ(\mathbf{x}) \mathbf{u} \\ + L_f \alpha_{1,\circ} (B_\circ(\mathbf{x})) + \alpha_{2,\circ} (\psi(\mathbf{x})) \geq 0 \end{array} \right. \\ \mathbf{u}_{lb} \leq \mathbf{u} \leq \mathbf{u}_{ub} \\ i = 1, 2 \quad j = 1, \dots, N_O \quad \circ = \{wb, obs_{ij}, mc\} \end{array} \qquad (13)$$

where $\mu > 0$ is a penalizing factor; it is easy to observe that the inequality constraints in (13) preserves the affine expression from (3) w.r.t. the optimization variables $(\mathbf{u}, \delta)^T$.

<figure><img src="image_4.png" /><figcaption>Fig. 2: Left: the designed micromanipulation task for the microrobots' pair, decomposed in the four reported Steps: (1) approach, (2) grasping, (3) transport, and (4) releasing. Right: the user exploits the pinchable claw on the handler of the Virtuose 6D haptic interface, to mimic the grasping by varying the distance between the microrobots (orange arrows).</figcaption></figure>

### III. HAPTIC SHARED CONTROL FOR MICROMANIPULATION

The QP-based control framework enables the motion of the microrobots while ensuring safe navigation in cluttered environments. We aim to apply this control framework to achieve micromanipulation tasks in a shared control fashion, where the controlled motion of the microrobots is achieved by combining human-generated reference commands, provided via a haptic device, with autonomous commands resulting from the QP problem solution (13). In the following, we describe the employed shared control architecture and the micromanipulation task designed for the microrobot pair. Then, we show how the QP formulation can be adapted to integrate both human and autonomous inputs within a shared control paradigm. In this regard, we provide the user with visuo-haptic cues during task execution, including: i) haptic repulsive forces to avoid obstacles; ii) haptic guiding forces to reach a desired microrobot configuration; and iii) 2D and 3D visualizations of the working environment.

#### A. Shared control architecture

1) **BatMag system simulator:** The microrobots are assumed to navigate within the workspace of the BatMag electromagnetic actuation system [30], consisting in a 22 mm cube filled with silicone oil and surrounded by nine electromagnets (see the bottom left corner of Fig. 1). These electromagnets modulate a magnetic field in the cube-shaped workspace, enabling independent actuation of two magnetized spheres ranging from 250 µm to 1 mm in diameter. Two cameras, positioned on the top and side, provide vision-based tracking and visual feedback on the spheres' positions.

To validate our control methodology, we built a virtual replica of the BatMag environment in the CoppeliaSim simulation software [32]. The simulator offers an interactive and customizable scene, supports kinematic design and robotic platform prototyping, and provides a versatile interface for algorithm implementation, validation and debugging. The software also includes a physics engine to simulate rigid body dynamics, that we used to simulate the microrobots as spheres subject to arbitrary bounded forces. This abstracts the electromagnetic actuation system and the current-to-force mapping, allowing to directly simulate the magnetic forces driving the microrobots' motion. The simulated environment
<figure><img src="image_5.png" /><figcaption>Fig. 3: System workflow. The user provides the reference state $\mathbf{x}_r$ through the Virtuose 6D haptic interface, corresponding to a reference microrobots formation $F_d = \langle \mathbf{p}_c, \phi, \theta, L \rangle$ in the simulated BatMag system; the simulated environment returns the current state $\mathbf{x}$. Our control framework computes, through the QP<sub>A</sub> and QP<sub>H</sub> solvers: i) the optimal forces $f_{R,1,R,2}$, moving the microrobots from positions $\mathbf{p}_{1-2}$ to the references $\mathbf{p}_{r,1-r,2}$; ii) the force and torque vectors, $f_{NG-OA}$ and $\tau_{NG-OA}$, that are provided to the user as haptic feedback.</figcaption></figure>

is used to offer enhanced visual feedback within the proposed shared control framework, as described in Sec. III-D.3.

2) *Virtuose 6D haptic device:* In our shared control architecture, the user teleoperates the pair of microrobot in the virtual environment through a Virtuose 6D (Haption, France) device (see the bottom right corner of Fig. 1). The system is a 6-DoF kinesthetic haptic interface, providing real-time measurements of the end-effector position and orientation, held and manipulated by the user. The handle is also equipped with a pinchable claw at the fingertip (see the top right corner of Fig. 1), in order to mimic grasping movements during maneuvers. A three-button pad is attached to the interface, enabling further trigger-based input selections for the user. The Virtuose 6D haptic interface features full 6D force feedback capabilities (force and torque), thus enabling to implement a large set of haptic feedback strategies, that can increase the overall user's sensing capabilities. In our setup, we let the user manipulate the handle of the haptic interface to change the position, orientation and distance of the microrobots' pair, as detailed in Sec. III-C.

### B. Micromanipulation task

The micromanipulation task involves controlling the motion of the microrobots to collaboratively grasp, transport, manipulate and assemble small-scale components.

An example of collaborative grasping achieved through the microrobots' pair is shown in Fig. 2 (see also [8]), where we consider the building of a tower using multiple cubes. The microrobots navigate the workspace under the influence of the magnetic forces, generated by the electromagnetic actuation system, towards the object to grasp (Step 1, *approach*). Then, they are commanded to position around opposite sides of the target object and move towards each other to grasp it (Step 2, *grasping*). Once grasped, the microrobots move the object while keeping it secured (Step 3, *transport*). Finally, they release the object at the target location by moving away from each other (Step 4, *releasing*).

### C. Teleoperation for micromanipulation

We leverage the presented architecture to adapt the QP-based control framework (13) in a shared control fashion.

The user steers the motion of the two microrobots in the simulated environment through the Virtuose 6D haptic interface, while receiving sensory feedback about the action of the autonomous part of the controller.

With respect to the defined control objectives, and with reference to Fig. 3, this is achieved by setting the reference state $\mathbf{x}_r = (\mathbf{p}_r, \mathbf{v}_r)^T$ in the *stability* objective O1 (4). The reference state $\mathbf{x}_r$ is determined by mapping the haptic device's motion to a reference formation $F_d = \langle \mathbf{p}_c, \phi, \theta, L \rangle$ of the microrobots denoting the moving reference frame $\mathcal{F}_R$, where $\mathbf{p}_c = \frac{1}{2}(\mathbf{p}_1 + \mathbf{p}_2)$ is the centroid between the microrobots, $\theta$ and $\phi$ are the polar and azimuthal angles of the segment $(\mathbf{p}_2 - \mathbf{p}_1)$, and $L = \|\mathbf{p}_2 - \mathbf{p}_1\|$ is their distance.

The haptic device's three translational DoFs are mapped to the centroid position $\mathbf{p}_c$, with scaling to account for size differences between the workspace and the interface. The roll and yaw DoFs control the orientation angles $\phi$ and $\theta$. Inspired from [21], only one rotation at a time is enabled, with the switching between the two rotations done through the end-effector button of the haptic device. Finally, the claw of the haptic device is used to regulate the desired distance $L$ between the two microrobots, thus mimicking fingertip grasping and releasing motions (see also Fig. 2).

From the desired formation commanded by the user, the reference positions are computed as $\mathbf{p}_{r,i} = \mathbf{p}_c \pm L/2(\cos \phi \cos \theta, \cos \phi \sin \theta, \sin \phi)^T$, $i = 1, 2$. The regulation of the microrobots' position to this reference is then guaranteed by the CLF constraint (5).

### D. Visuo-haptic sensory feedback

Solving the general QP problem (13) generates input forces that satisfy all constraints, assuming a feasible solution exists. However, when providing haptic cues to the user in shared control, we may need to isolate the contribution from a specific constraint or communicate how close we are to violating it. To achieve this, we consider two parallel instances of the formulation (13), namely QP<sub>A</sub> and QP<sub>H</sub>, to separate autonomous (A) and haptic (H) behaviours, with different dedicated objectives. In this way, QP<sub>A</sub> directly generates microrobots input commands, while QP<sub>H</sub> provides them as haptic feedback to the user (see Fig. 3). This decoupled behavior enables the select which part of the QP problem is provided directly to the microrobots and which is provided as haptic feedback to the user, enabling effective 6D navigation guidance and obstacle avoidance during the task. Additionally, we provide visual feedback through a full 3D visualization of the environment in a virtual simulation.

1) *Haptic navigation guidance:* Navigation guidance consists in providing the user with suitable haptic feedback to reach an assigned goal formation $\mathcal{F}_g$ for the microrobots' pair. In our micromanipulation task, this goal formation depends on the evolution of the task. With reference to Fig. 2, during the *approach* phase (Step 1), guidance brings the user to steer the microrobots towards a good grasping configuration around the cube; during the *transport* phase (Step 3), guidance brings the user to steer the microrobots towards the target releasing area, i.e., above the tower.
To provide such haptic navigation guidance, we set Objective O1 in QP<sub>H</sub> with respect to a goal system state x<sub>g</sub>, that is computed from F<sub>g</sub> as described in Sec. III-C. The solution of QP<sub>H</sub> provides the forces f<sub>NG,1</sub> and f<sub>NG,2</sub>. By exploiting the full 6D feedback rendering of the considered target haptic device, we generate the following haptic force and torque to be sent to the user (see Fig. 3):

$$f_{NG} = s_f \sum_{i=1}^{2} f_{NG,i}, \quad \tau_{NG} = s_{\tau} \sum_{i=1}^{2} R^{i} p_{i} \times f_{NG,i} \quad (14)$$

where R<sub>i</sub>p<sub>i</sub> is the position of the i-th microrobot in F<sub>R</sub>, while s<sub>f</sub>, s<sub>τ</sub> > 0 are appropriate scaling factors.

2) **Haptic collision avoidance:** Haptic collision avoidance provides the user with feedback to steer away from workspace boundaries and obstacles during micromanipulation. To decouple collision avoidance from the control inputs guiding the microrobots to target positions, we differently set the QP<sub>A</sub> and the QP<sub>H</sub> instances. QP<sub>A</sub> handles teleoperation only, enforcing the stability Objective O1 via the CLF constraint based on the user's reference commands. QP<sub>H</sub> includes the 2OCBF constraints related to workspace boundaries and obstacles, to enforce the safety Objective O2. The Objective O3 apply to both QP<sub>A</sub> instances.

In this setup, we isolate the collision avoidance forces to be rendered to the user, by solving QP<sub>H</sub> and computing the corresponding haptic force (f<sub>OA</sub>, τ<sub>OA</sub>) similarly to (14) (see Fig. 3); meanwhile, the microrobots follow the teleoperated reference commands, as a result of solving QP<sub>A</sub>.

3) **Enhanced 3D visualization:** In addition to haptic feedback, we provide the user with enhanced visual feedback using the simulation capabilities of the CoppeliaSim system.

On one hand, we offer a visualization of the simulated environment via two orthogonal 2D views, which mimic the sensing capabilities of the real system. On the other hand, we take advantage of the full visualization capabilities of the simulation software to present the user with a complete, navigable 3D view of the virtual environment. This simulated scene can also display additional virtual objects to improve user visualization and highlight key locations or components (e.g., which object to manipulate, how to grasp it efficiently, where to drop it accurately), thus extending the user's overall sensing capabilities beyond those of the real system.

Combining these capabilities enables to define different visuo-haptic shared control strategies with different level of autonomy, according to, e.g., the needs of the task or experience of user. The evaluation of the effectiveness of these strategies is addressed in the next Section.

# IV. USER EXPERIMENT

To validate the effectiveness of the proposed system, we conducted a human subjects experiment.

## A. Experimental setup

The experimental setup is composed of the simulated BatMag environment, within which we implemented the described micromanipulation task, and of the Virtuose 6D

<figure><img src="image_6.png" /><figcaption>Fig. 4: The simulated scene used for the micromanipulation task. The cube-shaped workspace of the BatMag simulation is populated with cuboid objects; users are required to grasp and move the cuboids in a given order to assemble a tower.</figcaption></figure>

haptic interface, manipulated by the user to teleoperate the microrobots and accomplish the task.

As shown in Fig. 4, the simulated scene contains five cube-shaped blocks available for manipulation, one of which is located at the center and represent the base of the tower to be built. The remaining blocks are spread with various poses around the scene, intentionally close to each other and the boundaries of the narrow environment. This layout requires users to pay particular attention to their trajectory, consider occlusions, and carefully steer the microrobots to grasp and transport the blocks (see Sec. III-B and Fig. 2) while avoiding collisions with the walls, other objects and the to-be-built tower. The goal of the task is to assemble a tower by stacking the blocks in a predetermined order. The user is given 3 minutes to build the highest tower possible.

In this experiment, we set m<sub>i</sub> = 4 · 10<sup>-4</sup> [kg], c = 0.1, p<sub>min,max</sub> = ±7.5e-3 [m], r<sub>i</sub> = 2.5 · 10<sup>-4</sup> [m], r<sub>j</sub> = 10<sup>-3</sup> [m], f<sub>max</sub> = 10<sup>-6</sup> [N], s<sub>f</sub> = 10<sup>2</sup>, s<sub>τ</sub> = 10<sup>5</sup>. For the CLF and CBF constraints: k<sub>γ</sub> = 1.0, k<sub>p</sub> = 1.0, k<sub>v</sub> = 0.1, k<sub>α1</sub> = 1.0, k<sub>α2,mc</sub> = 10<sup>8</sup>, k<sub>α2,obs</sub> = 10<sup>6</sup>, k<sub>α2,wb</sub> = 10<sup>12</sup>.

## B. Experimental conditions, protocol, and participants

We consider the following experimental conditions:

* **Navigation guidance:** i) NG: no navigation guidance is provided to the user, i.e., f<sub>NG</sub> = τ<sub>NG</sub> = 0; ii) HNG: haptic feedback guides the user towards the goal formation of the microrobots through eq. (14) in QP<sub>H</sub>;

* **Obstacle avoidance:** i) OA: no support is provided to the user to avoid collisions, i.e., Objective O2 is not addressed in either QP instance and f<sub>OA</sub> = τ<sub>OA</sub> = 0; ii) HOA: haptic feedback pushes the user away from collisions, i.e., Objective O2 is addressed in QP<sub>H</sub>, that generates the generalized forces (f<sub>OA</sub>, τ<sub>OA</sub>) similarly to (14); iii) AOA: collisions are automatically avoided, i.e., Objective O2 is addressed in QP<sub>A</sub>;

* **Visualization mode:** i) 2D: the environment is observed by two orthogonal views (front and top); ii) 2+3D: the
<figure><img src="image_7.png" /><figcaption>Fig. 5: Human-subjects study results. Mean and standard error of the mean of the (a) task completion and (b) assembly accuracy; (c) user preference for the different experimental conditions. The dotted and plain patterns represent the NG and HNG conditions, the blue, red and green colors the OA, HOA and AOA ones, and the dark and light tones the 2+3D and 2D ones, respectively.</figcaption></figure>

environment is observed by combining two orthogonal views and a complete 3D view (as in Fig. 4).

These conditions result in a total of 12 combinations to test.

The order of the conditions, as well as the order of blocks to be grasped, were generated using a balanced Latin square, so as to minimize learning effects across participants.

24 subjects (6 females, 18 males, average age of 26) participated in the study: 8 of them claimed to be highly familiar with haptic devices, 5 of them claimed to be highly familiar with shared control systems. Users were introduced to the experimental setup and control architecture, and attended a tutorial session to learn how to control the haptic interface in the target environment. Each participant carried the task 12 times, once per experimental condition.

### C. Metrics

To evaluate the effectiveness of the 12 conditions under test, we considered the tower building task completion and assembly accuracy metrics. The task completion corresponds to the height of the tower, counting how many blocks were stacked on each other out of the requested amount. The accuracy is measured in terms of alignment of the assembled tower with respect to the tower blocks' center of masses, i.e.,

$$ accuracy = 1 - \frac{2}{l} \sum_{i=0}^{n} (p_i) - p_0, \quad (15) $$

where $p_i$ is the position of the center of any of the n blocks and l denotes their side length.

Moreover, at the end of each condition, participants were asked to fill out a questionnaire evaluating the perceived effectiveness of each condition, using bipolar Likert-type seven-point scales, and ordered all experimental conditions based on personal preference.

### D. Results and Discussion

Results are shown in Fig. 5, reporting the (a) the assembled tower height in terms of number of stacked blocks, (b) the accuracy in placing the blocks to build the tower, (c) the best performing perceived mode for each experimental variable. To compare the different metrics, we ran three-way repeated-

<table><caption>TABLE I: Experimental evaluation</caption><thead><tr><th colspan="4">Statistical analysis (the better condition is highlighted in bold)</th></tr><tr><th></th><th>Height</th><th>Accuracy</th><th>User rating</th></tr></thead><tbody><tr><td>Navigation guidance</td><td>NG v HNG<br>(p &lt; 0.001)</td><td>NG v HNG<br>(p = 0.010)</td><td>NG v HNG<br>(p &lt; 0.001)</td></tr><tr><td rowspan="3">Obstacle avoidance</td><td>OA v HOA<br>(p &lt; 0.001)</td><td></td><td>AOA v HOA<br>(p &lt; 0.001)</td></tr><tr><td>OA v AOA<br>(p &lt; 0.001)</td><td></td><td>OA v HOA<br>(p = 0.003)</td></tr><tr><td>AOA v HOA<br>(p = 0.028)</td><td></td><td></td></tr><tr><td>Visualization mode</td><td>2D v 2+3D<br>(p = 0.071)</td><td>2D v 2+3D<br>(p = 0.003)</td><td>2+3D v 2D<br>(p = 0.051)</td></tr></tbody></table>

measures ANOVA tests (significance level $\alpha = 0.05$). The navigation guidance (NG vs. HNG), obstacle avoidance (OA vs. HOA vs. AOA), and visualization modes (2D vs. 2+3D) were the within-subject factors. All data passed the Shapiro-Wilk normality test. Data were transformed using the arcsin transformation when necessary to achieve normality. A Greenhouse-Geisser correction was used when the assumption of sphericity was violated. Sphericity was assumed for variables with only two levels of repeated measures (the navigation and visualization modes). Results of post hoc analysis with Bonferroni adjustments are reported in Table I (only significant $p$ values are shown).

These results highlight the efficiency of the haptic feedback strategies. Comparing any HNG- vs NG-based conditions show that haptic navigation guidance helps the user build tower higher and faster. Moreover, except for {HOA, 2D}, haptic navigation guidance systematically improves also the stability of the assembled tower. Obstacle avoidance assistance, regardless if it is autonomous (AOA) or delivered through haptic feedback (HOA), also positively affects the height of the assembled tower, but did not affect accuracy. Additionally, the 2D condition always leads to a more stable tower: direct user feedbacks identified the poor depth information in the 3D view as responsible for the lower performance. The performance rating highlights that HNG was preferred to NG about 95.8% of the time;
HOA and AOA to D' about 91.6%; and 2D to 2+3D about 62.5%. Predominantly, these results display the effectiveness of the shared control strategies proposed in the paper.

## V. CONCLUSIONS

This paper presented a haptic shared control approach to achieve telemanipulation of multiple microrobots, through a constrained optimization formulation. We presented the control architecture and demonstrated it for a representative microassembly task. The optimization framework was adapted to provide 6D haptic cues for navigation and obstacle avoidance. We also introduced visual cues from a simulated environment to enhance user's overall sensing capabilities.

We validated the methodology with a human subject user study. Results show that haptic assistance improved the task effectiveness compared to non-haptic conditions, with users preferring the haptic feedback in 95.8% of cases. Also, the environment visualization through orthogonal views resulted in more accurate maneuvers.

Future work will extend the optimization formalism to include sensing constraints, broader task sets, and more challenging environments. We will also conduct experiments towards real-world scenarios, to tackle challenges from diverse application fields, such as clinical (e.g., endovascular operations and targeted drug delivery) or environmental remediation (e.g., water purification).

## REFERENCES

[1] V. Iacovacci, E. Diller, D. Ahmed, and A. Menciassi, "Medical microrobots," *Annual Review of Biomedical Engineering*, vol. 26, 2024.

[2] D. Jang, J. Jeong, H. Song, and S. K. Chung, "Targeted drug delivery technology using untethered microrobots: A review," *Journal of Micromechanics and Microengineering*, vol. 29, no. 5, p. 053002, 2019.

[3] F. Ullrich, C. Bergeles, J. Pokki, O. Ergeneman, S. Erni, G. Chatzipirpiridis, S. Pané, C. Framme, and B. J. Nelson, "Mobility experiments with microrobots for minimally invasive intraocular surgery," *Investigative ophthalmology & visual science*, vol. 54, no. 4, pp. 2853-2863, 2013.

[4] D. Zhang, T. E. Gorochowski, L. Marucci, H.-T. Lee, B. Gil, B. Li, S. Hauert, and E. Yeatman, "Advanced medical micro-robotics for early diagnosis and therapeutic interventions," *Frontiers in Robotics and AI*, vol. 9, p. 1086043, 2023.

[5] M. A. Rahman and A. T. Ohta, "Micromanipulation with microrobots," *IEEE Open Journal of Nanotechnology*, vol. 2, pp. 8-15, 2021.

[6] F. Ongaro, C. Yoon, F. Van Den Brink, M. Abayazid, S. H. Oh, D. H. Gracias, and S. Misra, "Control of untethered soft grippers for pick-and-place tasks," in *2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob)*. IEEE, 2016, pp. 299-304.

[7] E. Diller and M. Sitti, "Three-dimensional programmable assembly by untethered magnetic robotic micro-grippers," *Advanced Functional Materials*, vol. 24, no. 28, pp. 4397-4404, 2014.

[8] F. N. Piñan Basualdo and S. Misra, "Collaborative magnetic agents for 3d microrobotic grasping," *Advanced Intelligent Systems*, vol. 5, no. 12, p. 2300365, 2023.

[9] R. Pieters, S. Lombriser, A. Alvarez-Aguirre, and B. J. Nelson, "Model predictive control of a magnetically guided rolling microrobot," *IEEE Robotics and Automation Letters*, vol. 1, no. 1, pp. 455-460, 2016.

[10] T. Xu, J. Liu, C. Huang, T. Sun, and X. Wu, "Discrete-time optimal control of miniature helical swimmers in horizontal plane," *IEEE Transactions on Automation Science and Engineering*, vol. 19, no. 3, pp. 2267-2277, 2022.

[11] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, "Control barrier functions: Theory and applications," in *2019 18th European control conference (ECC)*. IEEE, 2019, pp. 3420-3431.

[12] X. Tan, W. S. Cortez, and D. V. Dimarogonas, "High-order barrier functions: Robustness, safety, and performance-critical control," *IEEE Transactions on Automatic Control*, vol. 67, no. 6, pp. 3021-3028, 2022.

[13] E. D. Sontag, "A lyapunov-like characterization of asymptotic controllability," *SIAM journal on control and optimization*, vol. 21, no. 3, pp. 462-471, 1983.

[14] J. B. Rawlings, D. Q. Mayne, M. Diehl et al., *Model predictive control: theory, computation, and design*. Nob Hill Publishing Madison, WI, 2017, vol. 2.

[15] W. Xiao, C. G. Cassandras, and C. Delta, *Safe autonomy with control barrier functions: Theory and applications*. Springer, 2023.

[16] S.-C. Hsu, X. Xu, and A. D. Ames, "Control barrier function based quadratic programs with application to bipedal robotic walking," in *2015 American Control Conference (ACC)*. IEEE, 2015, pp. 4542-4548.

[17] A. Mohammadi and M. W. Spong, "Integral line-of-sight path following control of magnetic helical microswimmers subject to step-out frequencies," *Automatica*, vol. 128, p. 109554, 2021.

[18] W. Amokrane, K. Belharet, and A. Ferreira, "Design and modeling of a two-magnet actuator for robotic micromanipulation," *Sensors and Actuators A: Physical*, vol. 316, p. 112391, 2020.

[19] C. Pacchierotti and D. Prattichizzo, "Cutaneous/tactile haptic feedback in robotic teleoperation: Motivation, survey, and perspectives," *IEEE Trans. Robotics*, 2023.

[20] A. Franchi, C. Secchi, M. Ryll, H. H. Bulthoff, and P. R. Giordano, "Shared control: Balancing autonomy and human assistance with a group of quadrotor uavs," *IEEE Rob. Aut. Mag.*, vol. 19, no. 3, pp. 57-68, 2012.

[21] R. Rahal, A. M. Ghalamzan-E, F. Abi-Farraj, C. Pacchierotti, and P. R. Giordano, "Haptic-guided grasping to minimise torque effort during robotic telemanipulation," *Autonomous Robots*, pp. 1-19, 2023.

[22] M. Selvaggio, F. Abi-Farraj, C. Pacchierotti, P. Robuffo Giordano, and B. Siciliano, "Haptic-based shared-control methods for a dual-arm system," *IEEE Robot. Autom. Lett.*, vol. 3, no. 4, pp. 4249-4256, 2018.

[23] M. Selvaggio, J. Cacace, C. Pacchierotti, F. Ruggiero, and P. R. Giordano, "A shared-control teleoperation architecture for nonprehensile object transportation," *IEEE Trans. Robot.*, vol. 38, no. 1, pp. 569-583, 2021.

[24] K. T. Ly, M. Poozhiyil, H. Pandya, G. Neumann, and A. Kucukyilmaz, "Intent-aware predictive haptic guidance and its application to shared control teleoperation," in *Proc. RO-MAN*, 2021, pp. 565-572.

[25] M. Abayazid, C. Pacchierotti, P. Moreira, R. Alterovitz, D. Prattichizzo, and S. Misra, "Experimental evaluation of co-manipulated ultrasound-guided flexible needle steering," *Int. J. Med. Robot. Comput. Assist. Surg.*, vol. 12, no. 2, pp. 219-230, 2016.

[26] C. Pacchierotti, S. Scheggi, D. Prattichizzo, and S. Misra, "Haptic feedback for microrobotics applications: A review," *Frontiers in Robotics and AI*, vol. 3, p. 53, 2016.

[27] A. Bolopion and S. Régnier, "A review of haptic feedback teleoperation systems for micromanipulation and microassembly," *IEEE Trans. automation science and engineering*, vol. 10, no. 3, pp. 496-502, 2013.

[28] S. Martel, "Magnetic navigation control of microagents in the vascular network: Challenges and strategies for endovascular magnetic navigation control of microscale drug delivery carriers," *IEEE Control Systems Magazine*, vol. 33, no. 6, pp. 119-134, 2013.

[29] M. Ferro, F. N. P. Basualdo, P. R. Giordano, S. Misra, and C. Pacchierotti, "Experimental evaluation of haptic shared control for multiple electromagnetic untethered microrobots," *IEEE Transactions on Automation Science and Engineering*, pp. 1-12, 2024.

[30] F. Ongaro, S. Pane, S. Scheggi, and S. Misra, "Design of an electromagnetic setup for independent three-dimensional control of pairs of identical and nonidentical microrobots," *IEEE Trans. robotics*, vol. 35, no. 1, pp. 174-183, 2018.

[31] H. Khalil, "Nonlinear systems," *3rd edition*, 2002.

[32] E. Rohmer, S. P. N. Singh, and M. Freese, "Coppeliasim (formerly v-rep): a versatile and scalable robot simulation framework," in *Proc. of The International Conference on Intelligent Robots and Systems (IROS)*, 2013.