<figure><img src="image_1.png" /></figure>

HAL
open science

# 6-DoF Optical-driven Micro-robots with Force Feedback Capabilities for Interactive Bio-manipulation

Edison Gerena

► To cite this version:

Edison Gerena. 6-DoF Optical-driven Micro-robots with Force Feedback Capabilities for Interactive Bio-manipulation. Robotics [cs.RO]. Sorbonne Université, 2020. English. NNT: tel-03970598

HAL Id: tel-03970598

https://hal.sorbonne-universite.fr/tel-03970598

Submitted on 2 Feb 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
SORBONNE UNIVERSITÉ

INSTITUT DES SYSTÈMES INTELLIGENTS ET DE
ROBOTIQUE

PHD THESIS

Specialty : Robotics

Defended by

Edison GERENA

6-DoF Optical-driven Micro-robots
with Force Feedback Capabilities for
Interactive Bio-manipulation

defense scheduled on March 23rd, 2020

JURY :

<table><tr><td>Miles PADGETT</td><td>Professor at Glasgow University, UK</td><td>Reviewer</td></tr><tr><td>Michaël GAUTHIER</td><td>CNRS Senior Scientist at FEMTO-ST, FR</td><td>Reviewer</td></tr><tr><td>Mariana MEDINA SANCHEZ</td><td>Research fellow at IFW Dresden, GE</td><td>Examiner</td></tr><tr><td>Cecile PACORET</td><td>Research fellow at ECAM Lyon, FR</td><td>Examiner</td></tr><tr><td>Régis MARCHIANO</td><td>Professor at Sorbonne Université, FR</td><td>Examiner</td></tr><tr><td>Stéphane RÉGNIER</td><td>Thesis director<br>Professor at Sorbonne Université, FR</td><td>Examiner</td></tr><tr><td>Sinan HALIYO</td><td>Co-Supervisor<br>Associate Professor, HDR at Sorbonne Université</td><td>Examiner</td></tr></table>
<figure><img src="image_2.png" /></figure>
To my beloved wife and daughters, Alejandra, Zoé and Willow,
<figure><img src="image_3.png" /></figure>
6-DoF Optical-driven Micro-robots with Force Feedback
Capabilities for Interactive Bio-manipulation

**Abstract:** Optical tweezers (OT) allow to probe and manipulate micro-metric samples in a liquid environment. OT have successfully been applied in a large range of in-vivo and in-vitro bio-manipulation experiments such as the trapping of red blood cells in living animals and the immobilization of bacterial cells for nanoscopy. Furthermore, the linearity of the restoring optical forces on spherical objects, has led to the use of optical trapping for quantitative force measurements, such as the strength of inter-molecular bonds or the stiffness of a cell membrane.

The ambition of this thesis is to provide a complete robotic optical tweezer sys-
tem designed from scratch, that gives to an operator without engineering skills direct
physical access to biophysical interactions at the microscale in a 3D workspace, with
a flexible and intuitive user interface. The proposed innovative robotic platform for
dexterous cell manipulation and force measurement through optical tweezers has
the following major contributions:

* The generation of multiple optical traps in a three-dimensional working space with nanometrical resolution and high bandwidth.

* A 3D real-time force sensing of optical trap with sub-picoNewton (pN) resolution suitable for closed-loop control.

* A tele-robotic system, providing a straightforward human/machine interaction, and intuitive control of biological and synthetic micro objects in six degrees of freedom (three translations and three rotations).

* The so-called “Optobots”, 3D swimming micro-structures actioned by OT with 6-DoF and with a built-in force sensor.

Those capabilities are beyond the state-of-the-art, among commercial systems
and academic literature. Such an interactive robotic instrument is particularly rel-
evant to use-cases in experimental biology and constitutes a unique platform for
probing the micro-world.

**Keywords:** Optical Tweezers; Single-Cell manipulation; Mobile micro-robots; Multi-trap actuation; Force sensor; Bilateral Teleoperation; Micro-manipulation.
Micro-robots optiques à 6 ddl avec retour de force pour une
bio-manipulation interactive

**Résumé** : Les pinces optiques (PO) permettent de sonder et de manipuler des échantillons micro-metriques dans un environnement liquide. Les PO ont été appliquées avec succès dans une large nombre d'expériences in vivo et in vitro, comme le piégeage des globules rouges chez les animaux vivants et l'immobilisation de cellules bactériennes pour la nanoscopie. En outre, la linéarité des forces optiques sur des objets sphériques a conduit à l'utilisation du piégeage optique pour des mesures quantitatives de la force, comme la force des liaisons intermoléculaires ou la rigidité d'une membrane cellulaire.

L'ambition de cette thèse est de fournir un système complet de pince optique robotisée, qui donne à un opérateur un accès direct aux interactions biophysiques à l'échelle micro dans un espace de travail 3D, avec une interface utilisateur flexible et intuitive. La plateforme robotique innovante pour la manipulation des cellules et la mesure des forces présente les contributions majeures suivantes :

* La génération de multiples pièges optiques dans un espace de travail 3D avec une résolution nanométrique et une grande largeur de bande.

* Un capteur de force 3D en temps réel du piège optique avec une résolution sub-picoNewton adaptée au contrôle en boucle fermée.

* Un système télérobotique, offrant une interaction homme/machine simple et un contrôle intuitif de micro-objets biologiques et synthétiques en 6 ddl.

* Les "Optobots", des microstructures actionnées à 6 ddl par des pièges optiques et avec un capteur de force intégré.

Ces capacités dépassent l'état de l'art, parmi les systèmes commerciaux et de
la littérature. Un tel instrument robotique interactif est particulièrement perti-
nent pour les cas d'utilisation en biologie expérimentale et constitue une plateforme
unique pour sonder le micro-monde.

**Mots-clés** : Pinces optiques ; Manipulation unicellulaire ; Microrobots mobiles ; Actionnement multi pièges ; Capteur de force ; Téléopération bilatérale ; Micromanipulation.
# Contents

General Introduction 1

1 Robotic Micro-manipulation for Bioengineering Applications 5

1.1 Robotic Bio-Manipulation 6

1.1.1 Contact Methods for Robotic Bio-manipulation 7

1.1.2 External Energy-fields for Robotic Bio-manipulation 9

1.2 Optical Micromanipulation 11

1.2.1 Principle and Development of Optical Tweezers 12

1.2.2 Optical Manipulation Setups 13

1.2.3 Optical Micro-robots 17

1.3 Objectives 18

2 3D Real-time Force Sensing in Optical manipulation 21

2.1 Introduction 22

2.2 Development of a 3D High-Speed Force Sensing Method 23

2.2.1 Background and Related Work 23

2.2.2 Asynchronous Time-based Image Sensor 25

2.2.3 System Description 26

2.2.4 3D Tracking 28

2.2.5 Evaluation of tracking 29

2.2.6 3D Haptic Feedback Optical Tweezers 32

2.3 Improving Optical Micromanipulation with force-feedback Bilateral Teleoperation 38

2.3.1 Background and Related Work 38

2.3.2 Haptic Optical Tweezers Platform 40

2.3.3 Micro-chip Test bench 41

2.3.4 Experimental Protocol 41

2.3.5 Results and Discussion 44

2.4 Conclusion 48

3 3D High-Bandwidth Multi-Trap Actuation for Optical Tweezers 51

3.1 Introduction 52

3.2 Background and Related Work 53

3.3 Development of a 3D High-Bandwidth Multi-Trap Actuation Method 55

3.3.1 Calibration of the system 57

3.3.2 Evaluation of the Actuation Performances 60

3.4 6-DoF Control of Optical Robots 61

3.4.1 Fabrication of Robots 62

3.4.2 Robot control 64

3.5 Conclusion 65
<table><tr><td>4 Tele-robotic Platform for Dexterous Single-Cell Manipulation and Optical Robots with Force-feedback</td><td>69</td></tr><tr><td>4.1 Introduction</td><td>70</td></tr><tr><td>4.2 Tele-Robotic Platform for Optical Dexterous Single-Cell Manipulation</td><td>71</td></tr><tr><td>4.2.1 Background and Related Work</td><td>71</td></tr><tr><td>4.2.2 Tele-robotic System Design</td><td>72</td></tr><tr><td>4.2.3 Evaluation of the Teleoperation Platform</td><td>75</td></tr><tr><td>4.2.4 Experimental micro-manipulations through the platform</td><td>78</td></tr><tr><td>4.3 Optobots: Optical Robots with Force-feedback</td><td>87</td></tr><tr><td>4.3.1 Background and Related Work</td><td>88</td></tr><tr><td>4.3.2 Optobots</td><td>90</td></tr><tr><td>4.3.3 Bilateral Teleoperation of Optical-robot with Force Feedback</td><td>92</td></tr><tr><td>4.4 Conclusions</td><td>100</td></tr><tr><td>Conclusions and Future Works</td><td>101</td></tr><tr><td>Publications</td><td>107</td></tr><tr><td>Appendices</td><td>109</td></tr><tr><td>A Optical Robots Collection</td><td>111</td></tr><tr><td>A.1 Setup description</td><td>112</td></tr><tr><td>A.2 Robot collection</td><td>113</td></tr><tr><td>B Robot Research Solution</td><td>115</td></tr><tr><td>B.1 Software description</td><td>116</td></tr><tr><td>B.1.1 Exploration Path</td><td>116</td></tr><tr><td>B.1.2 Focus tracking</td><td>117</td></tr><tr><td>B.1.3 Image processing</td><td>118</td></tr><tr><td>B.1.4 Robot Types</td><td>118</td></tr><tr><td>B.1.5 Global map</td><td>118</td></tr><tr><td>B.2 Robot research</td><td>119</td></tr><tr><td>Bibliography</td><td>121</td></tr></table>
List of Figures

<table>
  <tbody>
    <tr>
      <td>1.1</td>
      <td>Micro-robotic aims to manipulate micro-objects such as organelles (&lt;1 µm), single cells (1–100 µm), and small organisms (&lt; 1 mm).</td>
      <td>6</td>
    </tr>
    <tr>
      <td>1.2</td>
      <td>Examples of biological manipulations through motor-driven micro-manipulator and differects end-effectors.</td>
      <td>7</td>
    </tr>
    <tr>
      <td>1.3</td>
      <td>Examples of biological manipulations through untethered physical fields and mobile micro-robots.</td>
      <td>10</td>
    </tr>
    <tr>
      <td>1.4</td>
      <td>Optical Tweezers working principle</td>
      <td>12</td>
    </tr>
    <tr>
      <td>1.5</td>
      <td>In-vivo and in-vitro biological aplications of optical manipulation.</td>
      <td>15</td>
    </tr>
    <tr>
      <td>1.6</td>
      <td>Optical robots examples.</td>
      <td>17</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>Event-based camera principle.</td>
      <td>25</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Homemade optical system scheme.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>CMOS and ATIS image of a microbead under the system.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Working principle of the event-based ring tracking algorithm.</td>
      <td>28</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>The 3D detection range of the system.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Robustness under 30% partial occlusion situation.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Robustness in obstacles disturbance situation.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>The schematic diagram of the bilateral coupling of the position and force in the haptic OTs system.</td>
      <td>32</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>The schematic ilustration with dimensions of RBC's and the SEM image of in vitro RBC's</td>
      <td>34</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>Pressing on a Red Blood Cell from above.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>2.11</td>
      <td>Touch the 3D contour of a RBC using the probe.</td>
      <td>36</td>
    </tr>
    <tr>
      <td>2.12</td>
      <td>Haptic exploration of two connected transparent RBCs.</td>
      <td>37</td>
    </tr>
    <tr>
      <td>2.13</td>
      <td>The trajectories of the probe during the experiment of RBC surface exploration.</td>
      <td>38</td>
    </tr>
    <tr>
      <td>2.14</td>
      <td>Haptic optical tweezers system</td>
      <td>40</td>
    </tr>
    <tr>
      <td>2.15</td>
      <td>The test bench microchip for haptic explorations.</td>
      <td>42</td>
    </tr>
    <tr>
      <td>2.16</td>
      <td>Example of bilateral teleoperation in a triangle shape exploration task.</td>
      <td>43</td>
    </tr>
    <tr>
      <td>2.17</td>
      <td>Examples of different shapes explorations under Vision and Vision+Haptic conditions.</td>
      <td>44</td>
    </tr>
    <tr>
      <td>2.18</td>
      <td>Statistical results of the experiment per subject.</td>
      <td>46</td>
    </tr>
    <tr>
      <td>2.19</td>
      <td>Statistical results of the experiment for each shape.</td>
      <td>47</td>
    </tr>
    <tr>
      <td>2.20</td>
      <td>Statistical global results of the experiment.</td>
      <td>48</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>Schematic representation of the planar and axial motion of the laser beam.</td>
      <td>56</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Optical path of the time-shared multi-trap actuation.</td>
      <td>57</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>The axial position range of the system.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>Cross-talk between planar and axial motions</td>
      <td>59</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>3.5 Controlled movements of thirteen micro-beads arranged in a cube of 16×16×8 µm.</td>
      <td>59</td>
    </tr>
    <tr>
      <td>3.6 Static precision tasks.</td>
      <td>60</td>
    </tr>
    <tr>
      <td>3.7 Velocity control of one micro-bead at different Z positions.</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3.8 Velocity control of 4 micro-beads at different Z positions.</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3.9 Schematic depictions and SEM images of two different optical robots.</td>
      <td>63</td>
    </tr>
    <tr>
      <td>3.10 Representation of robots kinematics and frames used in the 3D control.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.11 Experimental validation of 3D translation control of Optical Robots.</td>
      <td>65</td>
    </tr>
    <tr>
      <td>3.12 Experimental validation of rotation control of Optical Robots.</td>
      <td>66</td>
    </tr>
    <tr>
      <td>4.1 Optical-micromanipulation platform for dexterous single-cell handling.</td>
      <td>73</td>
    </tr>
    <tr>
      <td>4.2 Example of 3D teleoperation using 3 groups of traps.</td>
      <td>74</td>
    </tr>
    <tr>
      <td>4.3 Example of teleoperated rotation in position mode of a single trapped micro-bead.</td>
      <td>76</td>
    </tr>
    <tr>
      <td>4.4 Example of teleoperated rotation in position mode of a group of four micro-beads.</td>
      <td>77</td>
    </tr>
    <tr>
      <td>4.5 3D rotation control of erythrocyte driven by 8 optical traps.</td>
      <td>79</td>
    </tr>
    <tr>
      <td>4.6 Robot with a shovel-shaped end-effector for cell transport.</td>
      <td>81</td>
    </tr>
    <tr>
      <td>4.7 Robot equipped with a fork end-effector for single-cell manipulation.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>4.8 Collaborative tasks demonstration.</td>
      <td>84</td>
    </tr>
    <tr>
      <td>4.9 Micro-clamp components for microassembling.</td>
      <td>85</td>
    </tr>
    <tr>
      <td>4.10 Micro-clamp assembling demonstration.</td>
      <td>86</td>
    </tr>
    <tr>
      <td>4.11 Second micro-assembly technique.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>4.12 Working principle of optobots.</td>
      <td>91</td>
    </tr>
    <tr>
      <td>4.13 A micro-sphere imitating the behaviour of a bacteria.</td>
      <td>92</td>
    </tr>
    <tr>
      <td>4.14 First prototype of an optobot.</td>
      <td>94</td>
    </tr>
    <tr>
      <td>4.15 Optobot prototypes iterations.</td>
      <td>95</td>
    </tr>
    <tr>
      <td>4.16 Multi-trap force calibration with 3µm beads.</td>
      <td>97</td>
    </tr>
    <tr>
      <td>4.17 Optobots force calibration.</td>
      <td>98</td>
    </tr>
    <tr>
      <td>4.18 Bio-compatible Optobot with force-feedbac for topography scanning.</td>
      <td>99</td>
    </tr>
    <tr>
      <td>A.1 Micro-robots collection set-up.</td>
      <td>112</td>
    </tr>
    <tr>
      <td>A.2 Micro-robots collection example.</td>
      <td>113</td>
    </tr>
    <tr>
      <td>B.1 Actuators window, allowing to configure the "Focus tracking" function.</td>
      <td>116</td>
    </tr>
    <tr>
      <td>B.2 Image processing window, controlling the Hough Circle function parameters.</td>
      <td>117</td>
    </tr>
    <tr>
      <td>B.3 Robot type window, allowing the creation of new robot types.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>B.4 Example of global map resulting of a robot search.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>B.5 Main Window, providing the exploration control and the list of found robots.</td>
      <td>120</td>
    </tr>
  </tbody>
</table>
# General Introduction

Dexterous single-cell manipulation and characterisation offers many possible applications in cellular surgery, mechanobiology, tissue engineering, and biophysics. Recent breakthroughs in biotechnology are rising the demand for complex single-cell operation techniques such as cell isolation, 3D orientation and cell-injection. Significant challenges remain for applications related to single cell manipulation mainly due to the physics involved (volumetric forces dominated by the surfaces forces) and the size limitation imposed by the environment. The resolution and precision required at those sizes have a cost in term of Degrees-of-Freedom, workspace, grasping strategies and control schemes. Furthermore, there is an increasing demand to manipulate objects in confined environments like micro-fluidic devices, in order to decrease flow disturbances, contamination or evaporation of the culture medium, rendering external actuators unusable. All these constraints call to replace current techniques by non-contact manipulation methods.

Accordingly, a great effort has been made in the search of solutions for the actuation of mobile microrobots (i.e. untethered robots where the entire body is micrometer-sized) to serve as remote manipulators. Variety of methods have been developed employing chemical reactions, physical fields or bio-hybrid approaches [1]. Remote actuation using different external energy-fields like magnetic, acoustic or optical has appeared as a very promising solution in applications where a high spatial maneuverability and precision are required [2].

Among a variety of micromanipulation techniques, optical tweezers (invented by Arthur Ashkin, awarded with the Nobel Prize in physics 2018) offers several advantages in the characterization and manipulation of small biological samples. Optical manipulation exploits the light radiation pressure to noninvasively trap and position suspended micro-objects and cells with a nanometer resolution; resulting in a contamination-free, contact-free, and label-free method for cell manipulation in their original culture medium. The generated force can go up to few hundreds of pico-Newton. Although seemingly low, this is well in the range of micro-biological interactions and Optical Tweezers have successfully applied in a large range of experiments such as the trapping of red blood cells in living animals [3] and cell rotation for tomographic imaging [4], among others.

Furthermore, optical forces on trapped spherical objects can be modelled linearly, as the restoring optical force is proportional to the distance from its equilibrium position [5]. This particularity has led to the use of optical trapping for quantitative force measurements [6], such as the strength of inter-molecular bonds [7] or the stiffness of a cell membrane [8]. In addition, it is possible to simultaneously trap several objects using a single laser source by spatial (with active diffractive optical elements) or temporal (with rapid laser-deflectors) methods with their specific strengths and limitations. With respect to control and usability, although some recent examples of automated control of biological objects have been proposed,
most of optical manipulation platforms remain with manual **human/machine in-**
**terface** as automatising a specific task is a time-consuming operation, often beyond
the skills of the end-user. Existing commercial interfaces for optical manipulation
allow the user to control the optical traps using 2D mouse position. In order to
enhance the user control, some attempts to incorporate more efficient master de-
vices have been made, such as gesture recognition or multi-touch tablets. Despite
the improvement brought by those attempts in terms of ergonomy and efficiency,
completing complex real-world tasks still remains a challenge.

Recent advances in laser microfabrication by two-photon polymerization (2pp) processes, allows to arbitrary design 3D structures with nanometric resolution. 3D printing micro-tools, so-called **optical robots**, have been designed with spherical handles and actuated by a multi-trap optical system with six Degrees of Freedom (DoF) for specific microbiological tasks. Optical robots can be used to indirectly handle the cells in applications where their viability is an important issue. Physical and chemical treatments on optical-robots allows functionalization for more specific tasks such as pH or temperature sensing. Thereby, optical micro-robots will create new fields of applications for optical manipulation and extend capabilities of micro- robots in fascinating new ways.

This thesis subscribes to the field of micro-robotics and provides a complete
robotic optical tweezers solution designed from scratch. Four distinct contributions
to enhance the capability of the robotic cell manipulation can be enumerated:

* Real-time 3D force sensing: a new low-latency 3D force measurement method using an event-based camera will be presented in Chapter 2. It provides sub-picoNewton (pN) resolution and its bandwidth reaches 10 kHz. 3D high-bandwidth force measurement empowers the OTs as a versatile on-line force sensor. This method has a wide field of applications that includes the investigation of single molecules and motor proteins, intramolecular elasticity, intracellular forces, cell mechanics and cell motility and micro-rheology The method remains compatible with fluorescence, interferential and confocal microscopy and can be also extended to magnetic tweezers and other manipulation techniques.

* High speed 3D multi-trap actuation system: A new approach to generate more than 15 time-shared traps in 3D with low latency and high bandwidth will be proposed in chapter 3. The 3D motion of the focal spot is obtained by the synchronization of the orientation of a galvanometer mirror and the focusing or defocusing of a deformable mirror. This actuation technique is useful in applications where the 3D orientation of microscopic objects is needed, such as cell surgery, 3D tomographic imaging of living samples, or micro-assembling in microfluidic devices. It also allows the simultaneously stimulation and characterization of cells at different out-of-plane locations. This technique can also be used in any applications requiring a 3D high-speed control of multiple focus laser beams such as optogenetics, two-photon microscopy and microfabrication by two-photon polymerization.
* Teleoperation interface for interactive bio-manipulation: a teleoperated optical-micromanipulation platform will be proposed in chapter 4. The system provides a straightforward human/machine interaction through a tele-robotic solution allowing dexterous manipulation of synthetic and biological objects in an efficient and intuitive way. Teleoperation control is implemented with an Omega 7, haptic device with 7-DoF (including an active two-finger grip) and force feedback, and is ensured by a hard real-time system. Traps can be grouped and controlled in a variety of ways for specific purposes. A flexible software allows to create traps on the fly by mouse clicks, or by directly entering the desired 3D coordinates in for each trap.

* Force controlled micro-tools “Optobots”: Thanks to the previous achievements, we introduce force-controlled optical-robots (Optobots) in chapter 4. Those 3D printed micro-structures are actuated by optical tweezers with 6-DoF, they include a built-in real-time high-bandwidth 3D force sensor and are force-controlled via bilateral teleoperation. They have the potential to regulate contact forces between the robot and uncertain environment. This ability to sense mechanical stress or forces is crucial to every living being, from the simplest of micro-organisms – cells– to more complex organisms like – humans or animals –. In synthetic robots, this capacity could be expected to increase the ability to interact in a controlled manner with rigid and biological objects during explorations and manipulations tasks.
<figure><img src="image_4.png" /></figure>
# Robotic Micro-manipulation for Bioengineering Applications

The manipulation and characterization of single biological objects has become a major scientific challenge. Different techniques have been proposed, either by so-called non-contact approaches (use of optical, magnetic, electric force fields, etc.) or by contact approaches (micro-clamps, tips, AFM, etc.), that allow working at the single-cell level. This chapter surveys state-of-the-art works that are at the interface between biology, instrumentation and micro-robotics fields. Optical manipulation is one of the most widespread technique in the biology field. Its main advantages, characteristics and applications are also discussed, and optical robotics is introduced.

To perform robotic manipulation at the small-scales, the tools need to be positioned with nano-metric accuracy, and the end-effectors interactions with the objects need to be sensed and controlled. Then, enhancing the capability of robotic optical manipulation requires to tackle feedback, actuation and control issues that have not been solved in the literature and that represents the heart of the thesis: developing innovative approaches for the optical manipulation of biological objects using a robotic approach.
Contents

<table><tbody><tr><td>1.1</td><td><strong>Robotic Bio-Manipulation</strong></td><td>6</td></tr><tr><td>1.1.1</td><td>Contact Methods for Robotic Bio-manipulation</td><td>7</td></tr><tr><td>1.1.2</td><td>External Energy-fields for Robotic Bio-manipulation</td><td>9</td></tr><tr><td>1.2</td><td><strong>Optical Micromanipulation</strong></td><td>11</td></tr><tr><td>1.2.1</td><td>Principle and Development of Optical Tweezers</td><td>12</td></tr><tr><td>1.2.2</td><td>Optical Manipulation Setups</td><td>13</td></tr><tr><td>1.2.3</td><td>Optical Micro-robots</td><td>17</td></tr><tr><td>1.3</td><td><strong>Objectives</strong></td><td>18</td></tr></tbody></table>

## 1.1 Robotic Bio-Manipulation

The individuality of cells has been recognized since they first have been observed through optical microscopy. However, this fact has been underestimated by the bulk cell culture experiments, which interpret the phenomena through an "average" cell [9, 10]. It is only recently that the heterogeneity of cells of the same type started to be considered relevant to biological phenomena [11, 12], for instance in targeted therapeutics [13] and drug resistance studies [14].

In this context, the ability to manipulate and characterize individual biological objects is become a major scientific challenge. The need to perform tasks such as single-cell deformation, stimulation, rotation, or transportation, has called for new emerging techniques that allow working at the unicellular scale.

Extending the interaction and manipulation capabilities at the sub-millimeter scale, microrobotics have become a key tool in micro/nanoscale science and technology for both industry and academic research, in a wide range of fields such as healthcare, biotechnology, and manufacturing [1]

<figure><img src="image_5.png" /><figcaption>Figure 1.1: Micro-robotic aims to manipulate micro-objects such as organelles (<1 µm), single cells (1–100 µm), and small organisms (< 1 mm).</figcaption></figure>
Operating at the microscopic scale is highly demanding, mainly due to the specific physical effects governing the micro-world and the size limitations. Specific methods for working in the micro-world in terms of fabrication, actuation and sensing have been proposed in the last two decades. The scaling of physical laws, robotic micro-manipulation fundamentals and bioengineering application have been extensively reviewed in the last years [2, 15, 16, 17, 18]. The next section introduced robotic micro-manipulators, with an emphasis on the manipulation and stimulation of single cells, either through motor-driven manipulators or through untethered physical fields. Then, the optical manipulation will be discussed in further detail.

### 1.1.1 Contact Methods for Robotic Bio-manipulation

Single-cell manipulation tasks are usually performed using 3-axis cartesian robots consisting of motor-driven micromanipulators with prismatic joints and equipped with different end-effectors.

Those micromanipulators are not necessary themselves micron-sized, however as the robots performs task in the micro-world their end-effector must be neces-

<figure><img src="image_6.png" /><figcaption>Figure 1.2: Examples of biological manipulations through motor-driven micromanipulator. **A.** Micromanipulator with micro-pipette end-effector. (left) Standard set-up as used in clinics and biomedical labs, with two glass pipettes mounted on two robotic manipulators. (right) Robotic in-plane and out-of-plane orientation of an ellipsoidal oocyte. Adapted from [19]. **B.** Micromanipulator with micro-gripper end-effector. (left) Robotic system for force-controlled micro-grasping. (right) Manipulation and alignment of three cells of different sizes with force-controlled micro-grasping. Adapted from [20]. **C.** Micromanipulator with AFM-probe. (left) Schematic representation and SEM image of fluidic force microscopy (FluidFM) system. (right) Current functionalities of FluidFM system for single-cell manipulation. Adapted from [21].</figcaption></figure>
sary micron-sized. These robots must have a submicron positioning resolution and a range of motion of a few millimeters. Micro-manipulators are generally driven with DC motors and controlled in close-loop with a typical accuracy of 0.2 µm. Piezoelectric manipulators may have a higher resolution, but due to the resolution limit of optical microscopes (light diffraction limit ~ 0.25 nm) they are slightly less used in biological manipulation. End-effectors are tools mounted on a micromanipulator to interact with micro-objects for tasks such as pick and place. The usual end-effectors are micro-pipettes, micro-grippers or atomic force microscopy (AFM) cantilevers.

### 1.1.1.1 Micro-pipettes Based Manipulation

Micro-pipettes have become a widespread standard in the life sciences and biological manipulation. These glass micropipettes, with a small diameter tip that varies from a hundred nanometers to a hundred micrometers, are generally connected to a pneumatic or hydraulic pump to control the pressure at the tip of the micropipette. Typical tasks for these effectors range from patch clamp experiments, microinjection to cell translocation. More complex tasks can be performed using multiple collaborative pipettes, for instance out-of-plane cell rotation [22, 19].

### 1.1.1.2 Micro-grippers Based Manipulation

Micro-grippers have fingers which can open from a few micrometers to a few millimeters, generally actuated by electrostatic or piezoelectric techniques [23]. Although micro-grippers offer more flexibility for micro-manipulation, releasing tasks are more complex than using single-ended probes. They require additional strategies to facilitate object release, such as mechanical vibration [24], or the reduction of adhesion forces by varying humidity or temperature [25]. Exploiting adhesion forces during tasks execution has also been proposed [26]. For these reasons, their use in biological experiments is not widespread. One possible application is the cellular force sensing, that has been performed with a using a microelectromechanical systems (MEMS)-based microgripper with force control [20].

### 1.1.1.3 AFM-probes Based Manipulation

The AFM tip can be used to directly manipulate objects by adhesion forces or to probe the surfaces of an object. The force that the tip applies to the manipulated object can be used to obtain various types of information, such as adhesion, elasticity, viscosity and topography. AFM probes have also been modified with microchanneled cantilevers with nano-sized apertures for local liquid dispensing and stimulation of single living cells (FluidFM) [27, 21], for instance to perform a nano-fountain tip for intracellular delivery [28] or a force-controlled patch clamp of beating cardiac cells [29].
1.1.2 External Energy-fields for Robotic Bio-manipulation

There is an increasing demand to manipulate objects in confined environments like micro-fluidic devices in order to decrease flow disturbances, contamination or evap-oration of the culture medium, rendering external actuators unusable. All these constraints call to replace current techniques by non-contact manipulation meth-ods. Remote actuation using different external energy-fields like magnetic, acous-tic or optical has appeared as a very promising solution in applications where a high spatial manoeuvrability and precision are required. These contact-free solu-tions eliminates adhesive effects, which have a significant impact on handling tasks. However, these processes are often limited to a restricted types of materials.

Accordingly, a great effort has been made in the search of solutions for the remote actuation and power supply of untethered microrobots (i.e. untethered and mobile robots where the entire body is micrometer-sized) to serve as remote manipulators. These remote manipulators are able to handle different types of objects, in terms of shape and physical properties, and more advanced capabilities can be included by design such as sensing, or targeted drug delivery [2]. The new capabilities of these untethered micro-manipulators promise immense potential for a variety of in-vivo and in-vitro biomedical applications.

1.1.2.1 Magnetic Micromanipulation

By applying a magnetic force and/or a magnetic torque on an object, it is possible
to remotely actuate it. This is rendered easier when the object integrates a part
highly sensitive to magnetic fields, often ferromagnetic parts or permanent magnets.
Thus, the magnetic object can be moved and oriented in the workspace.

Direct manipulation and sorting of cells with native magnetic properties, such as red blood cells, is possible by magnetophoretic separation [36]. For other cells that are only very little sensitive to the magnetic field, the magnetic force directly applicable to the cells is weak and can not be sufficient to cause displacements. A common solution is to attach fine paramagnetic particles to the target cells and then manipulate them by magnetic energy.

The remote actuation by magnetic field is the most studied method in the robotic
community due to its suitability for in-vivo experiments, and has made significant
advances in the past decade. Magnetic micro-swimmers (helical or flagela-like)
rotate or oscillate in synchronization with the applied magnetic field and advance
thanks to the forces generated by the interaction with the fluid. They have been
applied in applications such as targeted gene delivery [31] and transporting sperm
cells with motion deficiencies [30].

1.1.2.2 Electric Micromanipulation

The manipulation of individual cells can also be achieved by electric fields. When
a dielectric object (e.g. cell) is subjected to a non-uniform electric field, it is po-
larized and experiences a force from the electric field. This phenomenon is known
as dielectrophoresis (DEP) [37]. The DEP forces applied to the cells depend on the differences between the permittivity and the conductivity of the particle and its surrounding medium, which in turn depends on the frequency of the applied field. Cells can be attracted to the maximum of the electric field (positive dielectrophoresis) or repulsed from it (negative dielectrophoresis). The resulting electrical forces

<figure><img src="image_7.png" /><figcaption>Figure 1.3: Examples of biological manipulations through untethered physical fields and mobile micro-robots. **A.** Magnetic helical spermbot for cellular cargo delivery. Adapted from [30]. **B.** Magnetic helical microswimmers for targeted gene delivery. Adapted from [31]. **C.** Micro-device for 3D single cell electro-rotation. Adapted from [32]. **D.** Optoelectronic mobile microrobot for single-cell transportation. Adapted from [33]. **E.** 3D acoustic manipulation of single-cells. Adapted from [34]. **F.** A flagellum-like flexible tail robot propelled by acoustic waves. Adapted from [35].</figcaption></figure>
have been used for isolation, positioning [38], 3D cell rotation [32], or cell fusion at single-cell level [39].

Optically induced dielectrophoresis, known as optoelectronic manipulation uses a photosensitive surface to allow an optical pattern to control the electric field [40], and has been used for instance for single-cell trapping and cell separation [41]. Recently, an optoelectronic mobile microrobot for single-cell isolation and delivery was proposed [33].

### 1.1.2.3 Acoustic Micromanipulation

Acoustic micromanipulation, as the name implies, manipulates objects using radiation force of sound waves. Acoustic radiation force can be used to separate particles based on their size and density in a micro-channel. [42].

Solid particles can also be trapped and manipulated using standing acoustic waves, usually generated by an ultrasonic transducer and a reflector. The objects are trapped in the pressure nodes (minimum-pressure regions) or pressure antinodes (maximum-pressure regions) of the acoustic field. Thus, by varying the emission frequency, the pressure nodes move with the trapped particles. A two-dimensional (2D) acoustic force field with an inter-nodal spacing of the same order as the patterned cell dimensions has been proposed [43]. Three-dimensional manipulation of single cells using surface acoustic waves has been also demonstrated [34].

It is also possible to power mobile micro-robots through acoustic waves. A flagellum-like flexible tail robot is propelled by the small-amplitude oscillation of travelling acoustic waves [35]. 3D-microprinted microrobot that contains a trapped spherical air bubble has also been acoustically powered [44].

## 1.2 Optical Micromanipulation

Among the variety of non-contact methods, optical trapping offers several advantages in the handling and the mechanical characterisations of small biological samples, ranging from a hundred of nanometres to tenth of millimeters, in a confined environment such as microfluidic devices.

Optical manipulation exploits the light radiation pressure to noninvasively trap and position suspended micro-objects and cells with a nanometer resolution; resulting in a contamination-free, contact-free, and label-free method for cell manipulation in their original culture medium. Its compatibility with other optical techniques, especially microscopy implies that it is highly appropriate for lab-on-chip systems and micro-fluidic devices. In addition, controlling multiple focal spots enables to simultaneously trap and manipulate several objects in 3D [45].

Optical manipulation has become a popular tool for manipulating single biological samples, successfully demonstrated in a large range of in-vivo [46] and in-vitro [6] experiments such as the trapping of red blood cells in living animals [3], the immobilization of bacterial cells for nanoscopy [47] and cell rotation for tomographic imaging [4], among others.
In the next section, principle of Optical Tweezers and design considerations will be discussed. For further details, readers are directed to two excellent reviews, written by Neuman and Block [48], and Bowman and Padgett [49].

1.2.1 Principle and Development of Optical Tweezers

Optical trapping uses the optical force generated by the energy and momentum exchange between light and particles to drive the mechanical motions of micro objects with nanometer resolution. When an individual light ray is refracted by the surface of an object, its path is altered resulting in a light momentum change. This change of momentum generates a pressure upon the surface, called radiation pressure.

In his paper, "history of optical trapping and manipulation" [51], Ashkin relates his first experiment to look for particle motion by laser radiation pressure using transparent latex spheres and a Gaussian laser beam. The speeds corresponds to Ashkin's estimation, demonstrating that radiation pressure is pushing them. However, an additional unexpected force was appears which strongly pulled particles located in the fringes of the beam into the high intensity region on the beam center.

From the ray optics theory and the intensity profile of the Gaussian beams,

<figure><img src="image_8.png" /></figure>

Figure 1.4: An optical tweezers uses forces exerted by a strongly focused beam of light to trap small objects. By controlling the position of the focal point, the position of the trapped object can be controlled. (a) The variation in the momentum of photons produces a force in the direction of the intensity gradient (gradient force) and a force in the direction of the beam propagation (scattering force). Adapted from [50]. (b) In unfocused laser, the gradient is directed towards the laser center and the scattering force pushes the object in the laser direction. (c) In a focused laser, near the focal point, the intensity gradient is directed towards the laser focus. Thus the gradient force is able to balance out the scattering force, and a particle can be trapped in three dimensions.
Ashkin deduced that the pressure force can be decomposed into two components. An optical force related to the gradient of light intensity, called gradient force, and a force component pointing in the direction of the incident light, called scattering force. Based on this, he conceived the first stable optical trap between two opposing diverging Gaussian beams [52].

Ashkin et al. has reported in 1986 the use of a single strongly focused Gaussian beam to stably trap particles in three dimension, that he originally called single-beam gradient force trap, today known as optical tweezers [53].

Optical tweezers use the pressure of light radiation to trap an object by providing a steep potential well in all axes, generated by the balance between the gradient forces and the scattering force. An equilibrium position is created near the narrowest focal point, known as beam waist, as the gradient force becomes dominant above the scattering force (cf. Figure 1.4). Dielectric particles around this position are attracted and trapped by the optical tweezers. Then, by controlling the position of the beam waist, the motion of the trapped object is controlled.

This method was then extended to manipulate atoms [54], cells [55] and viruses [56] by the same group. Since then, the applications of the optical tweezers have not stopped increasing in diverse fields as molecular biology, cell biology, materials science and quantum physics. Ashkin has been awarded with the Nobel Prize in physics in 2018 for this development and their application to biological systems.

### 1.2.2 Optical Manipulation Setups

The essential elements to implemented an optical trap are the laser and a high numerical aperture (NA) objective. A Gaussian laser beam is traditionally chosen as laser source, but other modes can also been used [57]. The wavelength depends on the size of the trapped objects and the environment. For the manipulation of micro-metrical objects, laser wavelengths that fall into the visible or near-infrared spectral region are usually used. Last consideration, if biological material is manipulated, a wavelength between 800 and 1064 nm is usually used in order to minimize photonic damage due to the laser absorption and the oxidation reaction of organic material. The power of the laser must be sufficient to produce a trap and depends on multiple parameters such as the components that are between the laser output and the microscope objective that may induces energy losses. In general terms when using micron-scale beads, for every 10 mW of power delivered, the maximum forces are in the order of 1 pN [58].

A microscope objective with high NA (typically 1.2–1.4 NA) is required to produce a gradient of sufficient intensity to overcome the dispersion force and produce a stable optical trap for microscopic objects. In this way, the microscope objective plays a fundamental role in the efficiency and stability of the traps. The working distance and the immersion medium of the objective (oil, water or glycerol) must be chosen depending on the application specifications, such as the aqueous capture medium and the depth to which it will be manipulated. Spherical aberrations in the beam waste highly degrade the trap’s performance [48]. The various optical
manipulation setups differ mainly in how the force is measured and the way in which the position of the trap is controlled.

### 1.2.2.1 Force Measurement in Optical Traps

In an optical trap, the tightly focused laser beam provides equilibrium position near the beam waist. When the trapped objects are laterally or axially displaced away from the equilibrium position, the optical forces acts to pull it back towards the equilibrium position. For small displacements, the force applied to the particle is a linear function, and the optical trap can be compared to a simple spring system [5], which follows Hooke's law. In this model the restoring optical force is proportional to the distance from its equilibrium position, and the trap stiffness is proportional to the light intensity.

This particularity has led to the use of optical trapping for quantitative force measurements [6], such as the strength of inter-molecular bonds [7], the stiffness of a cell membrane [8] or intracellular measurements for micro-rheology [59].

Force calibration of arbitrary objects with reasonable accuracy is still a challenge as the optical stiffness depends on several factors such as the material properties, the refractive index of both object and environment, the sizes and shapes of trapped objects etc. For these reasons, silicon or polystyrene microspheres are often used as a force *probe* to indirectly manipulate and sense the force during micromanipulation tasks. By using artificial microbeads which are uniform in material and shape, the optical force applied on the microbead can be obtained. For small displacements, the optical force is described by a spring model [5]:

$$ \mathbf{F}_{opt} = \mathbf{K} \times (\mathbf{P}_{laser} - \mathbf{P}_{probe}) $$

Where $\mathbf{P}_{laser}$ and $\mathbf{P}_{probe}$ represent the laser and the probe position respectively. $\mathbf{F}_{opt}$ is the 3D optical force. $\mathbf{K}$: $[K_x, K_y, K_z]$ is the trap stiffness where $K_x, K_y, K_z$ represent the stiffness in x, y, and z direction respectively. Therefore, the optical force is obtained by measuring the particle's displacement from the equilibrium trap position.

Well-established methods exist for precise stiffness calibration of spherical objects, as power spectrum analysis, equipartition theorem, and Stokes' drag [48, 60, 61]. Hence, force sensing directly stems from the capability to detect the position of the laser and the one of the trapped object. In quantitative measurements with optical tweezers, the optical trap is often fixed, and only the trapped object position information is needed. The determination of this position can be done by image-based position detection where the images of video cameras are used to extract the information, or by interferometric methods where beam displacements after the laser passes through the sample are measured via quadrant photodiode QDP. In section 2.2.1 further discussion in the methods for 3D high-Speed force detection in optical tweezers are presented.
### 1.2.2.2 Position Control in Optical Traps

The incorporation of a motorized stage allows dynamic control of the sample chamber, while the manipulated object remains fixed on the trap. This type of actuation technique, refereed as passive-actuation, facilitates greatly the trap calibration, and not interfere with the stability of the trap.

<figure><img src="image_9.png" /></figure>

**Figure 1.5:** Biological applications of optical manipulation. **A.** Optical tweezers (OTs) are used to trap and manipulate red blood cells within subdermal capillaries in living mice, inducing an artificial clot [3]. **B.** OTs is used to impose local forces on cell contacts in the early Drosophila embryo and measure tension at cell junctions [62]. **C.** OTs in the study of propulsion forces and motility efficiency of the unicellular parasites trypanosomes [63]. **D.** Optical manipulation of injected nanoparticles (fluorescent, red) inside living zebrafish [64]. **E.** Out-of-plane cell rotation through two OTs [65]. **F.** Large deformation of human red blood cells subjected to direct stretching by OTs [66]. **G.** OTs are used to measure the force during kinesin motor stepping and structured RNA molecule unfolding [67]. **H.** OTs-based single-cell sorting device [68]. **I.** Mouse embryonic stem cells patterned into precise complex cellular microenvironments using holographic OTs [69].
The motion of the trapped object can also be done by moving directly the laser beam, refereed as active-actuation, through active optical components that change the laser direction from the light path. In addition, it is possible to multiplex the trapping laser beam in order to simultaneously trap several objects using a unique laser source.

By rapidly deflecting the laser beam between several trap positions, it is possible to trap several objects, in such a way that the objects do not have time to spread between two laser visits. Laser-deflection can be done by actuated mirrors, acousto-optic deflectors (AODs), or electro-optic deflectors (EODs). This methods are referred as Time-shared methods and are commonly limited to two-dimensions.

Using active diffractive optical elements [45] such as Spatial Light modulators (SLM) or Digital Micromirror Device (DMD), several traps can be dynamically controlled creating a number of difraction spots at different 3D positions. This method has the ability to move traps in a three-dimensional space and are referred as Holographic Optical Tweezers.

Further discussion on 3D multi-trap actuation methods for the 3D motion-control of objects are presented in section 2.2.1.

### 1.2.2.3 Commercial Optical Tweezers Systems

Some commercial optical tweezer systems are available in the marked. JPK Instruments<fnref n="1" /> propose a dual optical tweezer system with force measurement. The two traps are created by divided the laser source using polarized beamsplitters. The 3D position of one of the two traps can be controlled by a 2D beam steering and the displacement of a lens. The 3D force detection is implemented via QDP. Thorlabs<fnref n="2" /> propose a 2D optical tweezer with dual path (two independent trap beams) with a maximum of 15 stable traps using galvanometer steering mirrors for each path. An optional force module facilitates the 2D position detection by interferometry of one trapped probe. Ionovation proposes PicoTweezers<fnref n="3" />, a single or dual beam optical trap system with force measurements based on video detection at 400 Hz via particle tracking in stationary optical trap. Elliot Scientific<fnref n="4" /> proposes a wide range of options, which includes a 2D multi-trap optical tweezer (E3500) based in Acousto-optic beam steering, a QDP force measurement option (E4 100 ADP) for a single trap and a force measurement option via camera particle tracking (E4500 CPT) for 2D multiple particle tracking. However, most of the systems proposes only two traps, none of them present multi-trap actuation in a 3D space, and none propose a 3D force measurement for more than 2 traps.

It is common for research groups to build their own optical tweezer setups, in a custom optical system, or more often around a commercial optical microscope. In the latter approach, the construction of the optical trap (laser and actuation

<footnote n="1">¹https://www.jpk.com</footnote>

<footnote n="2">²https://www.thorlabs.com</footnote>

<footnote n="3">³https://www.ionovation.com</footnote>

<footnote n="4">⁴https://www.elliotscientific.com</footnote>
<figure><img src="image_10.png" /><figcaption>Figure 1.6: Optical robots examples. **A.** Automatic cell transportation with 3-beads gripper formation. Adapted from [70]. **B.** An optical robot for temperature sensing. Adapted from [71]. **C.** A surface scanning probe. Adapted from [72]. **D.** Articulated robot for out-of-plane mobility. Adapted from [73]. **E.** An optical robot for cell puncture. Adapted from [74]. **F.** A micro-robot with syringe function for cargo delivery. Adapted from [75]. **G.** An optical screw-wrench for micro-assembly. Adapted from [76].</figcaption></figure>

system) is coupled with the conventional optical microscope according to its optical path before the objective. This facilitates the use of different types of optical images, such as phase contrast or epi-fluorescence.

### 1.2.3 Optical Micro-robots

Usually, optical traps are directly used to manipulate the object of interest. This approach suffers mainly from two issues: the exposure of biological samples to the laser beams can be damaging, and the quality of the trap strongly depends on the shape, material and the refraction index of the target, hence it's difficult to reliably hold organic samples. For these reasons, indirect manipulation methods have been proposed, where optically trapped structures are used as tools. Additionally, implementing sensing, actuation and feedback control leverage these structures as *optical robots*. Optical manipulation allows to apply an action in a precise area (diffraction limited) and not over the entire workspace like magnetic or electric fields. This greater spatial resolution has the potential to allow direct collaborative tasks by several micro-robots, and additional degrees of freedom for each individual robot.

The most basic and common example of such a tool is a polystyrene bead attached to the target sample [46]. Multiple beads arranged into gripper formations for the manipulation of cells are then explored [70]. More complex non-spherical structures have been proposed for specific functions as a pH measurement gel-microtool by connecting micro-beads impregnated with indicators [77] or a force probe formed with cadmium sulphide rods and silica microspheres [78].
Recent advances in laser microfabrication by two-photon polymerization (2pp) processes [79] have led the way to novel examples of micro-mechanisms actuated through optical trapping. These 3D-printed structures are designed with attached spherical parts serving as optically trappable handles, and specifics end-effectors. Eventually using an appropriate mix of surfactants or nano-materials, some interesting functions have been demonstrated: a micro-tool integrating a silicon nano-wire for temperature sensing [71]; a surface scanning probe [72]; an optical robot with carbon nanotube end-effector for cell puncture through photothermal effect [74]; a micro-robot with syringe function using photothermally induced convection [75]; an optical screw-wrench for micro-assembly [76].

## 1.3 Objectives

As these examples depict, optical robots promise significant impact on cell manipulation and characterisation. However, exploring the full potential of optical robots requires robust and intuitive control in demanding micro-world scenarios. Enhancing the capability of optical robotic micro-manipulation requires feedback, actuation and control improvements. Proposed methods must be robust for manipulation and demonstrate high spatial and temporal resolution. High level control should be intuitive, flexible and efficient. The main reason for actual limitations is technological because of the system itself and its design. A second source of limitation is the complexity of the system's operation, resulting in complicated user interfaces and reduced capabilities.

Robotics has made tremendous progress in the last decade, in the design and operation of accurate, repeatable and fast moving systems. This led to the application of those approaches to an optical tweezer setup. Instead of applying robotic control techniques to existing optical tweezers systems, a dedicated novel robotic optical manipulation system is created from scratch with an emphasis on the need for real-time applications.

Contributions on the two fundamental components of an optical tweezers setup, the force measurement system and the actuation system, is proposed. They should have sufficient performances to be used in closed-loop control, namely high bandwidth, low latency and robustness during manipulation. Their suitability for real-time applications is demonstrated in bilateral teleoperation scenarios. This in turn has evolved into a completely interactive platform capable of manipulating several samples simultaneously while sensing mechanical interactions in real time.

Each contribution is accompanied by a section called *Background and Related Work*, where the need for a solution is clarified and the contribution is put into perspective with the other propositions of the state-of-the-art. This thesis is organized as follows:

**Chapter 2** is divided in two main sections. The first part presents a new 3D force sensing method for Optical Tweezers with high bandwidth and low latency
based on pattern tracking with an event-based camera. Its real-time capabilities are
demonstrated in a bilateral teleoperation control scenario. The second part explores
the impact of haptic feedback in optical manipulation regarding user dexterity on
tactile exploration tasks.

**Chapter 3** presents a new approach to generate and control multiple traps in a 3D workspace with low latency, high bandwidth and nanometric resolution using only mirrors. It is suitable for real-time trajectory control, as demonstrated by the actuation of optical robots with six Degrees-of-Freedom (DoF).

**Chapter 4** is divided in two main sections. The first part proposes a new tele-
robotic solution for dexterous cell manipulation through optical tweezers based on
the combination of the proposed multi-trap actuation technique, motorized stages,
and a robotic master-device. Different micro-manipulation experiments with cells
and synthetic objects are presented to illustrate the capability to perform complex
tasks in efficient and intuitive ways. In the second part of this chapter, the design
and the implementation of a force sensing optical micro-robot is presented. The
3D printed micro-structures, actuated in 6-DoF by optical tweezers with a built-in
high-bandwidth 3D force sensor and controlled by bilateral teleoperation, are based
on previous contributions. Optobots allow to control the interaction and mechan-
ical stress during manipulation and indirectly handle cells in applications where
their viability is a major consideration. Target applications are bio-compatible
scanning force microscopy, as well as haptic exploration and force-controlled cell
manipulation.
<figure><img src="image_11.png" /></figure>
# 3D Real-time Force Sensing in Optical manipulation

The first part of this chapter presents a new 3D force sensing method for Opti-cal Tweezers with high bandwidth (up to 10Khz) and low latency. The proposed technique uses high speed image tracking in event-based camera with nano-metric resolution in 3 directions. This method is advantageous for the implementation of closed-loop robotic approaches such as automatic force control or bilateral teleoper-ation. Its capabilities are demonstrated in a teleoperated 3D manipulation scenario with a haptic user interface, where naive users performed direct in vitro haptic ex-ploration of isolated Red Blood Cells inside a Petri dish.

The second part of this chapter explores the case of an optical tweezer platform coupled to an haptic device providing transparent force feedback. The impact of haptic feedback regarding user dexterity on tactile exploration tasks is studied using 3 µm micro-beads and a test bench with micro sized shapes. The results reveal a consistent improvement in both users' trajectory tracking and their control of the contact forces. This also validates the experimental setup which performed reliably on 140 different trials during the evaluation.
Contents

<table><tr><td>2.1 Introduction</td><td>22</td></tr><tr><td>2.2 Development of a 3D High-Speed Force Sensing Method</td><td>23</td></tr><tr><td>2.2.1 Background and Related Work</td><td>23</td></tr><tr><td>2.2.2 Asynchronous Time-based Image Sensor</td><td>25</td></tr><tr><td>2.2.3 System Description</td><td>26</td></tr><tr><td>2.2.4 3D Tracking</td><td>28</td></tr><tr><td>2.2.5 Evaluation of tracking</td><td>29</td></tr><tr><td>2.2.6 3D Haptic Feedback Optical Tweezers</td><td>32</td></tr><tr><td>2.3 Improving Optical Micromanipulation with force-feedback Bilateral Teleoperation</td><td>38</td></tr><tr><td>2.3.1 Background and Related Work</td><td>38</td></tr><tr><td>2.3.2 Haptic Optical Tweezers Platform</td><td>40</td></tr><tr><td>2.3.3 Micro-chip Test bench</td><td>41</td></tr><tr><td>2.3.4 Experimental Protocol</td><td>41</td></tr><tr><td>2.3.5 Results and Discussion</td><td>44</td></tr><tr><td>2.4 Conclusion</td><td>48</td></tr></table>

## 2.1 Introduction

Optical instruments, from the microscope of van Leeuwenhoek to super-resolved fluorescence microscopy, have always been precursors to great advances, especially in life sciences. Optical tweezers offer a similar potential, by augmenting visual feedback by mechanical interactions allowing to probe and characterise biological samples in a liquid environment. In an optical trap, the restoring optical force is related to the distance from his equilibrium position. This particularity has led to the use of Optical Tweezers for quantitative force measurements, such as the strength of inter-molecular bonds [7] or the stiffness of a cell membrane [8].

In most of these experiments, an inert bio-compatible polystyrene bead attached to the molecule or the cell, is used to indirectly handle the samples. This indirect approach is applied for two main reasons. Because precise force calibration for non-spherical objects is still a complex challenge [80] and to avoid direct laser exposure in biological samples that can negatively influence their behaviour or cause photo-damages. In the case of a spherical object, the effect of the trap is akin to a linear stiffness around the focal point of the beam. Indeed, the optical trap acts as a spring of stiffness proportional to the light intensity [48] and the force acting on the object can be obtained by measuring its motion [5].

Current robotics research on optical manipulation are focused in the implementation of control techniques based on position. In existing techniques, direct
trapping and manipulation of objects are performed [81, 82, 83]. Some recent works introduce controls and planning approaches for indirect manipulation of cells using silica beads arranged into gripper formations [70, 84]. However, to our best knowledge, none of current robotics techniques exploit the inner capability of force sensing of optical trapping. Possibly due to the high dynamics effects in the micro-world (e.g., quasi instantaneous acceleration) and the limitations in the current detection methods.

Considering the trapped probe as the end-effector, the external forces acting on the probe can be used in the feedback path to close the control loop. Those forces can be provided by tracking the motion of the probe under the optical microscope. Considering the force range and the trap stiffness, pico-Newton resolution is reachable. This kind of performance would make Optical Tweezers a formidable apparatus for micromanipulation in general and for biology and biochemistry in particular. In the first part of this chapter we will present the development of a 3D high-speed force sensing method for optical tweezers. Then in the second part of the chapter, the proposed sensor is used in a bilateral teleoperation coupling scheme, and the impact of haptic feedback regarding user dexterity on tactile exploration tasks is studied.

## 2.2 Development of a 3D High-Speed Force Sensing Method

### 2.2.1 Background and Related Work

As mentioned above, the precise force detection is directly relied to the capability to track the position of the trapped object. Nevertheless, high dynamics effects at the microscale and the limitations of the optical microscope render most classical methods useless. Hence, the lack of robust 3D tracking reduces most applications to simple planar tasks. Also, the latency and low bandwidth hurts the system stability and its real-time capabilities [85]. Improving the automation requires indubitably to improve the force sensing performances. Moreover, making this feedback available with low latency at high bandwidth will open the road to complex applications requiring closed-loop control.

Several methods to improve the tracking has been developed. Most commonly, a quadrant photodiode (QDP) is used to sample the position of the target at tens of kHz with nanometric precision [86]. Nonetheless, this method is vulnerable to occlusions and disturbances and works reliably only on isolated objects. Hence, it is not suitable for micro-manipulation tasks.

An alternative is image processing through video cameras integrated into the microscope. visual tracking algorithms using CMOS cameras offer straightforward implementations, but their bandwidth is limited by the amount of data that should be transmitted and processed. Real-time force information is hardly available. State-of-art of real-time visual tracking algorithms on commercial CMOS cameras can
rarely exceed 60 Hz [87]. Limiting the imaging to a smaller region of interest (ROI) can accelerate processing, at the detriment of resolution and precision [88]. To obtain more than 1 kHz sampling rate, the ROI have to be decreased to 60 pixels × 60 pixels which compromises the working scale and resolution. By combining a high-speed CMOS with tracking implemented on GPUs (Graphics Processing Unit), 3D tracking at several of kHz is reported [89]. This approach is however fairly complex as it requires special knowledge and hardware.

As an alternative to classical CMOS cameras, event-based cameras were proposed [90]. These bio-inspired sensors are frame-free and eliminate data redundancy by design. They are shown to allow the 2D tracking at a speed in the order of tens of kilo-Hertz [91, 90]. Because of the particularity of the image-data provided, well-known processing techniques and algorithms cannot be used. Further investigations are especially needed for real-time 3D robust tracking.

In this section, a 3D motion tracking technique using an event-based algorithm taking advantage of an event-based sensor is presented. It provides pico-Newton resolution and its bandwidth reaches 10 kHz. Its capabilities are demonstrated in a teleoperated 3D manipulation scenario with a haptic user interface. This kind of control scheme is very demanding and requires indeed a feedback loop at 1 kHz for stability. Reliable and reproducible 3D exploration of biological surface for non experts users is demonstrated for the first time.

<table><caption>Table 2.1: Comparison of tracking techniques under optical microscopy</caption><thead><tr><th>Method</th><th>Acquisition rate</th><th>Processing rate</th><th>Advantage</th><th>Limitation</th></tr></thead><tbody><tr><td>QDP</td><td>MHz</td><td>MHz</td><td>Fast, high precision</td><td>Small working space, only isolated objects</td></tr><tr><td>H-CMOS</td><td>Khz</td><td>&lt; 60 Hz</td><td>High precision</td><td>Not for real-time use</td></tr><tr><td>H-CMOS + GPU</td><td>kHz</td><td>kHz</td><td>Fast, High precision</td><td>Not for long time continuous use</td></tr><tr><td>H-CMOS + ROI</td><td>kHz</td><td>1 kHz</td><td>Fast, high precision</td><td>Small working space</td></tr><tr><td>EV-C</td><td>&gt;100 kHz</td><td>&gt;10 kHz</td><td>Fast, low consumption</td><td>No 3D applications</td></tr></tbody></table>

QDP: Quadrant photo-diode, H-CMOS: High-speed CMOS camera, GPU: Graphics Processing Unit, ROI: Region of interest, EV-C: Event-based camera.
### 2.2.2 Asynchronous Time-based Image Sensor

Asynchronous Time-based Image Sensor (ATIS) is the latest generation prototype of silicon retinas [93]. In this sensors, every pixel combines a brightness change detector and an additional exposure measurement circuit, which provide illuminance polarity. The most distinctive difference between the conventional CCD/CMOS camera and the silicon retina is the image capture mechanism. For the digital camera, the brighter image at a given pixel produces a larger intensity value at that pixel. The image sequence is collected frame by frame with a fixed sampling interval. While for silicon retina, for each pixel, as long as the outside stimuli or the input light intensity surpasses a threshold, an “event” is triggered independently and asynchronously without any global clock. See Fig 2.1.

Each event contains the information of position, polarity and time stamp. Thus by conveying the time-stamped events instead of the whole frames, ATIS shows the advantages of having low latency of 15 µs for each coming event and being free of redundancy. Also ATIS achieves larger dynamic range of 143 dB compared to conventional cameras, which means it is more capable of capturing light in different environments without under- or over-saturation. The temporal resolution is 100 kfps under 100 lux scene illumination, and the pixel bandwidth may be decreased by low illumination conditions in microscopy.

The absence of events when there is no changes in scene luminance implies the suppression of data redundancy, that is a common problem in high-speed CMOS

<figure><img src="image_12.png" /><figcaption>Figure 2.1: Event-based camera principle. (a) Picture of a commercial event-based camera (Gen3 VGA-CD, Prophesee, https://www.prophesee.ai/). (b) Events generation principle of event camera single pixel. Events with +1 or -1 polarity are emitted when the change in log intensity exceeds a predefined threshold. (c) Data generation comparison between standard cameras and event-based cameras. In contrast to standard video frames shown in the upper graph, a stream of events from an event camera, plotted in the lower graph, offers no redundant data output, only informative pixels or no events at all. Red and blue dots represent positive and negative events respectively. Adapted from [92].</figcaption></figure>
tracking. ATIS has the particularity to be adaptable to high-speed on-line tracking
due to its efficient encoding of movements. Thereby, the system captures only the
dynamic information and the amount of data to be processed is therefore consider-
ably reduced, which makes it possible to process at speeds up to 10k fps [94]. More
details of this technology are discussed in [93, 95].

The earliest application of silicon retina is the vehicle detection in traffic moni-
toring [96]. Delbruck introduced several event-based vision methods on background
noise filter, orientation feature extraction, and cluster tracking [97]. The event-
based optical flow is presented by Benosman's group, which benefit to significantly
reduce running time compare to the conventional method [98]. Event-based blob
tracking algorithms are proposed by Lagorce, which allow the tracking of multiple
shapes with the processing rate of several hundred kilohertz on a standard desk-
top PC [99]. By combining the epipolar geometric with the temporal constraints,
real-time 3D reconstruction was proposed [100].

Although many applications of event-based vision are presented in macro-scale, its usage in micro-scale is an almost unexploited area. Regnier's group proposed for the first time to use an event-based camera for tracking of objects under an optical microscope [91] and same group has proposed the 2D particle tracking under optical tweezers [90]. In the next section we demonstrated the first 3D high-speed tracking system of micro-particle under Optical tweezers with an event-based camera.

2.2.3 System Description

An Optical Tweezers set-up have been built. The optical scheme of the system is shown in Fig.2.2. The singularity of this set-up is to inclusion of an ATIS, “Asyn- chronous Time-based Image Sensor”. It is based on an inverted microscope where the same objective is used for both imaging and producing the optical trap. The laser source is a 1070 nm wavelength with a maximum output of 10 Watts. This laser beam is expanded firstly and directed into the oil immersion objective (Olym- pus UPlanFLN 40x, NA 1.3) to produces a fixed optical trap. Two microstages are used for x-y coarse positioning and a 3D nanostage is applied for x-y-z fine position- ing of a Petri-dish. The illumination(LED, 3W) pass through a lens to create the collimated beam. After passing through the sample, all the outcome visible light is reflected by a long pass dichroic mirror to cameras. Then the beam is divided by an unpolarized beam splitter cube (9:1) into the event-based camera (ATIS, 240×304 fully autonomous pixels) and the CMOS camera (Basler, 659×494 pixels), where 90% of the light is led to ATIS for better brightness. The CMOS camera provides the environment information and visual perception to operators, while ATIS is used to calculate the relative position between probe and OTs laser spot.

In a scene with the stable light environment, only the dynamic information
stimulated by moving object is recorded. In the case of the presented system, the
trap center has a fixed position in the ATIS image. With an appropriate threshold,
image variations are mostly generated on the contour of the trapped bead as shown
in Fig.2.3. This information is exploited to infer the 3D motion of the probe.
<figure><img src="image_13.png" /><figcaption>Figure 2.2: Home-made optical system scheme. The source is a 1070 nm laser with a maximum output of 10 Watts. An oil immersion objective (Olympus UPlanFLN 40x, NA 1.3) produces a fixed optical trap. Two microstages (x - y) and a 3D nanostage provide respectively coarse and fine positioning of samples. The illumination(LED, 3W) is reflected by a long pass dichroic mirror (900nm cut-off) then is divided by an unpolarized beam splitter (R9:T1) into the silicon retina camera (ATIS, 240×304 pixels) and the CMOS camera (Basler, 659×494 pixels), where 90% of the light is led to ATIS for better brightness. $f_1 = 30, f_2 = 125, f_3 = 100, f_4 = -50, f_5 = 45$.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 2.3: Image of microbead (3µm polystyrene) under the system. First column: the CMOS image at different z-displacements respect to the focus plane (z=0). Second column: the corresponding ATIS image with 33 ms accumulation time. Different colors indicate positive or negative polarity of the events. The displacements are in micro-meters.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Figure 2.4: Working principle of the event-based ring tracking algorithm.</figcaption></figure>

### 2.2.4 3D Tracking

Fig.2.3 shows the conventional and event-based images of the probe. In ATIS image, accumulated events can be grouped in two concentric circles. Their center position is related to the planar motion of the probe while the radius is linked to the depth position. An event-based ring tracking algorithm is developed. It's used to recognize both circles, then to selectively extract center position and radius parameters from the inner one. Indeed, when the probe is in contact with other samples, the inner one is much more robust and stable. However, it's necessary to track both circles to be able to set them apart. The principle of ring tracking is to minimize the distance of the events' spatial coordinates and the ring model [101] as presented in Fig.2.4.

Denoting $E(\mathbf{p}, t)$ as an event occurs at time $t$ with spatial location $\mathbf{p} = (x, y)$ in ATIS coordinate. $U(t)$ is defined as the set of useful events' locations at time $t$:

$$U(t) = \{E(\mathbf{p}, t)|\mathbf{p} \in ROI(t)\} \qquad (2.1)$$

where $ROI(t)$ is the region of interest (ROI) of one circle model at time $t$ [102]. Since the fitting methods are vulnerable to noise, the ROI is used as solution to filter the outliers.

Suppose that the unknown circle model's parameters at time $t$ is $C_t(\mathbf{P}_t, R_t)$, where $\mathbf{P}_t = (X_t, Y_t)$ is the circle center's position and $R_t$ is the radius. Then, a fast non-iterative algebraic fit [103] minimizes the cost function:

$$\min_{\mathbf{P}_t \in \mathbb{R}^2, R_t \in \mathbb{R}} \sum_{\mathbf{p}_k \in U(t)} \|d(\mathbf{p}_k, \mathbf{P}_t)^2 - R_t^2\|^2 \qquad (2.2)$$
where $\mathbf{p}_k$ is the kth event's location in $U(t)$. $d(\mathbf{p}_k, \mathbf{P}_t)$ is Euclidean distance between $\mathbf{p}_k$ and the circle center $\mathbf{P}_t$. Introducing parameters $A = -2X_t$, $B = -2Y_t$, and $C = X_t^2 + Y_t^2 - R_t^2$, (2.2) can be written as a linear least square problem:

$$ \min_{\mathbf{P}_t \in \mathbb{R}^2, R_t \in \mathbb{R}} \sum_{\mathbf{p}_k \in U(t)} \|Ax_k + By_k + C + x_k^2 + y_k^2\|^2 \quad (2.3) $$

Solving A, B, and C, gives the parameters of circle ($\mathbf{P}_t, R_t$). This single circle tracking method is used to parametrize the two concentric circles of the ring (inner $C_t^1$ and outer $C_t^2$). All incoming event occurring in a considered time interval are assigned to the closest circle model. Events at the intersection of two ROIs are discarded to reduce the ambiguity. This condition makes sure that the inner and outer circle will not become into one. The ROIs are updated accordingly. The algorithm is given below.

<caption>**Algorithm 1 Event-based Robust Ring Fitting**</caption>

**Require**: Events $\mathbf{E}(\mathbf{p}, t)$

1: **for** every step **do**

2: Update the content of $\mathbf{U}^1(t)$ and $\mathbf{U}^2(t)$ according to (1)

3: Estimate $C_t^1$ and $C_t^2$ parameters according to (3).

4: Update output: $[\mathbf{X}_t^1, \mathbf{Y}_t^1, \mathbf{R}_t^1]$

5: Update $\mathbf{ROI}_{t+1}^1$ and $\mathbf{ROI}_{t+1}^2$

6: **end for**

### 2.2.5 Evaluation of tracking

#### 2.2.5.1 Range and resolution

A 3 µm silica microsphere fixed to the sample-holder is used to evaluate the tracking. Fig.2.5 gives the real displacement, as reported by the nanostage vs tracked motion in pixels for all directions as well as the calculated pixel/µm transform.

The detectable motion range is 6µm and 7µm for x and y respectively with less than 3% standard deviation (SD). On z-axis, the relative radii variance is shown in Fig.2.5 (c). The linear detection range is about ±2 µm around the focus plane with 5% SD. This 3D detection range (6 × 7 × 4µm) is sufficient since the linearity of the trap stiffness is valid around one diameter of the trapped bead [5], here 3 × 3 × 3µm³.

With 204×304 pixels, the theoretical resolution is 23.8 nm/pixel in x and y and 166.6 nm/pixel in z. Practically, the bead center position and radius were estimated with sub-pixel accuracy using the circle tracking algorithm. This sensitivity varies in different illumination conditions and working environment.
### 2.2.5.2 Robustness

Robustness here is defined as the ability to extract the 3D position of the target from noisy data, or even a small subset of data. It will be tested under two most com-

<figure><img src="image_16.png" /><figcaption>Figure 2.5: The 3D detection range of the system. (a), (b) The inner center position x and y (in pixels) with the microsphere's position (in µm). (c) The relative inner radii change (in pixels) with the microsphere's axial position (in µm). At focus plane, the inner radius is 64 pixels. The linear regression coefficients show the conversion between pixels in ATIS coordinates and micrometers in the world coordinates.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 2.6: Robustness under 30% partial occlusion situation. (a) Image recorded by the CMOS camera. (b) Corresponding image recorded by ATIS with accumulation time of 33 ms. (c) 3D tracking error under 30% occlusion at different depths. The tracking error is determined as the difference between the detection and the ground truth, divided by the detection range in corresponding axis.</figcaption></figure>
<figure><img src="image_18.png" /><figcaption>Figure 2.7: Robustness in obstacles disturbance situation. (a) Image recorded by the CMOS camera. (b) Corresponding image recorded by ATIS with accumulation time of 33 ms. (c) The mean and SD of the 3D tracking error at different depth.</figcaption></figure>

monly encountered situations partial occlusion i.e. only part of the target image is captured, and obstacles disturbances i.e. to track the target among many obstacles.

In the occlusion test, the microsphere's image is partially out of the view of ATIS. The tracking errors for 30% occlusion of the inner circle are shown in Fig.2.6(c). As can be seen, for less than 30% occlusion, the tracking error and the SD are less than 5% for both lateral and axial detection.

In obstacles disturbance test, the target bead (in the center) is surrounded by two other similar objects as shown in Fig.2.6 (d). They are fixed on Petri-dish and animated with a sinusoidal movement. The tracking result is shown in Fig.2.6 (f). The lateral tracking errors are less than ±5% with 2% SD in within ±2.5 µm. The axial errors are less than ± 20 % around the focus plane with less than 10 % SD. As the image plane move far away from focus plane, the radius error increases up to 30 %.

### 2.2.5.3 Computational Load

The algorithm is implemented in C++ on a hard real-time framework. The testing relies on a 2.9 GHz Dual core desktop PC, with a total CPU load about 50 % of its power and a memory consumption of about 4 MB. The average running time for each iteration is less than 60 µs, with less than 2 µs deviation. Then the system is successfully pushed to 10 kHz real-time sampling rate.
To sum up, a cutting-edge solution for obtaining 3D robust high-speed force during the micromanipulation tasks has been developed. It is the first time to our knowledge to allow this performance in OTs system. This development paves the way for achieving 3D stable close-loop OTs manipulation systems. It also brings the great potential to various applications as a powerful 3D high-speed force sensor.

### 2.2.6 3D Haptic Feedback Optical Tweezers

The proposed force sensor methods discussed in this section could be used in fully automated systems or teleoperated systems. In order to show its benefits and validate the proposed force sensing, a teleoperation scenario with haptic feedback is implemented. This is a demanding application from the control point of view as the stability and transparency of haptic feedback coupling requires a sampling of 1 kHz [104, 85]. Previous works presented haptic feedback on Optical Tweezers but were limited to 2D [105, 106]. The 3D tracking allows for a real spatial coupling in this case.

#### 2.2.6.1 Haptic Coupling

Haptic feedback requires a bilateral control scheme, as presented in Fig.2.8. In this scheme, the user handles the haptic interface (Omega.3, ForceDimension). Its position is used to control the motion of the sample through the nano-stage. The force measured on the probe is feedback to the user with an appropriate gain.

Motion of the master device is scaled down by $6 \times 10^{-4}$ to drive the trap position relatively to sample holder. Actually, the trap position is fixed, and the mobile part is the nanostage holding the samples. The measured forces are magnified by $1 \times 10^{12}$ and sent back to the user. A single PC (Intel Xeon core, 2.93 GHz) operating under a real-time co-kernel Linux and RTOS APIs Xenomai is used to control the system. The control-loop runs with a force refresh rate of 1 kHz.

<figure><img src="image_19.png" /><figcaption>Figure 2.8: The schematic diagram of the bilateral coupling of the position and force in the haptic OTs system.</figcaption></figure>
### 2.2.6.2 Calibration

The force on the trap is calculated using the optical force model [5]:

$$F_{opt} = K \times (P_{laser} - P_{probe}) \qquad (2.4)$$

where $P_{laser} - P_{probe}$ represent the displacement between the laser and the probe position as obtained from the tracking method. $K$ is the stiffness of the trap.

Practically, to obtain the laser position, the positions of trapped probe before touching anything are recorded for a period of time and the average position is considered as the position of OTs, which is around (120,150) in the ATIS coordinate. The stiffness is calculated experimentally using the Equipartition method [48]. For an object in a harmonic potential:

$$\frac{1}{2}k_B T = \frac{1}{2}K\langle d \rangle^2 \qquad (2.5)$$

where $k_B$ is Boltzmann's constant, $T$ is absolute temperature, and $\langle d \rangle^2$ is the displacement variance (brownian motion) of the particle from its trapped equilibrium position. Considering 300 mW laser power it is estimated in x-axis, y-axis and z-axis as $K_x=12.3$ pN/µm, $K_y=12.6$ pN/µm s and $K_z=1.5$ pN/µm respectively under room temperature of 25.5 °C. The axial stiffness is less than the lateral stiffness which is normal according to [107]. This indicates that during manipulation, the loss of the trapped object is more likely caused by the reacting force in the axial direction. The same laser power will be used for the following experiments, under the same condition, i.e. temperature, laser power, medium, etc. Stiffness adjustments may be required depending on the specific applications, then the calibration process will be re-conducted for obtaining the OTs' stiffness.

<table><caption>Table 2.2: Summary of the force sensor parameters</caption><thead><tr><th>Parameter</th><th></th><th>Value</th><th></th></tr></thead><tbody><tr><td>OTs stiffness (pN/µm)*</td><td>$x=12.3$</td><td>$y=12.6$</td><td>$z=1.5$</td></tr><tr><td>Force detection range (pN)*</td><td>$x=36.9$</td><td>$y=37.8$</td><td>$z=4.5$</td></tr><tr><td>Force resolution (pN)*</td><td>$x=0.3$</td><td>$y=0.3$</td><td>$z=0.25$</td></tr><tr><td>Haptic loop</td><td></td><td>1 kHz hard real-time</td><td></td></tr></tbody></table>

### 2.2.6.3 3D Haptic Experiments on biologic samples

Biologic samples are chosen to illustrate the use of the system in a real world scenario. Red Blood Cells (RBC) are easy to acquire and have an 3D irregular dumbbell-shaped profile, and hence are well suited for this illustration. They are fixed in 4% formaldehyde for biological stability. Probes are 3µm polystyrene beads incubated in PBS and Ethylenediaminetetraacetic acid (EDTA) solution to prevent

<footnote>*Depends on the laser power. Values for 300 mW laser power</footnote>
<figure><img src="image_20.png" /><figcaption>Figure 2.9: (a) Schematic illustration with dimensions of RBC's and probe. (b) Scanning Electron Microscope (SEM) image of in vitro RBC's.</figcaption></figure>

the surface sticking. First we squeeze the cell from the top verify the axial force sensation, after that we contour the shape of a RBC from the sides in a 3D haptic exploration, and then, we will explore two transparent RBCs attached together by following along a ∞ shaped path. Finally, preliminary user experiments will be implemented.

### 2.2.6.3.1 Z-axis Haptic Feedback

The first experiment will validate the force feedback on z axis. The RBC sample is fixed on the bottom of a Petri Dish. The user first traps a bead to serve as probe, then moves the sample-holder as to position the probe above a fixed RBC. The planar motion of the nanostage is then artificially blocked and the user controls only the z motion until the probe touches the cell and he feels an obvious counter-force. The experimental results are shown in Figure 2.10. Small fluctuations are caused by the 3D Brownian motion of probe which are largely present at the considered scale. The contact point is shown in region II in (a), (b), (c). At contact, a sudden reaction force in the z-axis of about 0.3 pN is detected, and the Brownian motion is attenuated compared to the region I. Then the probe is pushed deeper into the cell until 1.6 pN in the z-direction, as shown in III. At this time, user felt about 1.6 N force in the z-axis. The cell is contacted twice during the presented experiment. Similar results are obtained during the two passes which also proves the repeatability of the axial force detection.

Notice that this experiment aims to validate the axial haptic feedback during biological manipulations. In addition, the stiffness of the RBCs can be roughly obtained from the result. By using the Hertz model, which considers the cell a homogeneous smooth semi-sphere, the elastic modulus of the RBC is the order of magnitude of thirty KPa [108]. This result is similar to literature, with the measure of the elastic modulus of adherent living cells [109]. This is an approximated result; for a proper mechanical characterization, the trap stiffness should be calibrated in the neighborhood of the cell before [110]. The system would also allow to automatize the experiment and to repeat it accurately over a large number of trials.
<figure><img src="image_21.png" /><figcaption>Figure 2.10: Pressing on a Red Blood Cell from above. (a) The 3D force applied on the probe during the experiment. The x-, y-, z- coordinates are the position of nanostage in world coordinates. inset picture I: the optically trapped probe is not in contact with the cell; II: by increasing the position of the nanostage, the probe is coming into contact with the RBC; III: the probe pushes deeper into the RBC. (b) Pictures of RBC and the probe corresponding to the three stages I, II, III. (c) The optical force and the position of nanostage in z-direction during the cell pressing process. The scale bars are 3 µm.</figcaption></figure>

### 2.2.6.3.2 3D Haptic Exploration

This experiment is dedicated to touch the 3D contour of cells and explore their shapes. The difficulty of these tasks comes from the uncertainty contour of the biological objects. Since the visual information may be blurry or lost at some parts, the haptic feedback will help users to maintain the contact and decrease the possibility of losing the trap.

First, haptic exploration of an isolated RBC is conducted (see Fig. 2.11). Then two flat and transparent RBCs, damaged after the hemoglobin leakage, stuck together forming a ∞ shape is explored fig. 2.12. The 3D contact force is successfully perceived and maintained by operators. Haptic feedback allows to keep the contact even when the probe is occluded and users were able to achieve a surface exploration with consistent force feedback.
<figure><img src="image_22.png" /><figcaption>Figure 2.11: Touch the contour of a RBC using the probe. (a) The 3D path of the probe and the 3D contact forces during the RBC contour exploration. (b) Pictures of this process under microscopy. (c) 3D optical force during this process. The scale bars are 3 µm.</figcaption></figure>

From the figures, it is observed that the 3D pico-Newton contact forces are maintained along the exploration. Besides, the 3D force direction indicates a counterforce from the cell's surface. For the two-cell exploration, when the probe pass through the connected part of two cells, a burst of axial force of 1 pN is detected as shown in Fig. 2.12 (a) and (c) pointed by the red arrow. Since the connected part of the two cells is higher in z-direction than the regular path of the probe, when the probe crossing over this part, a force in z-axis is detected demonstrated that there are obstacles below the probe. Biological objects are often transparent, so their manipulation is even more difficult. This experiment is a good illustration of scenarios where haptic force will be useful, as the cells are really hard to see. The haptic feedback help user to maintain the contact even if they cannot see the object.
<figure><img src="image_23.png" /><figcaption>Figure 2.12: Haptic exploration of two connected transparent RBCs. (a) The 3D path of probe and 3D contact forces during the contour exploration. (b) Corresponding pictures under microscope during this process. (c) 3D optical force. When the probe pass through the connected part of two cells, a burst of axial force of 1 pN is detected (red arrow). The scale bars are 3 µm.</figcaption></figure>

### 2.2.6.3.3 Preliminary user evaluations

To further prove the repeatability and effectiveness of the 3D haptic feedback during biological manipulations, a preliminary user study is conducted. 6 volunteers explored the shape of one RBC. Each were subject to those three different conditions: with only vision feedback, both vision and haptic feedback, and only haptic feedback respectively. The probe trajectory during three users' experiments are depicted in Fig. 2.13 (a), (b) and (c).

During this experiment, it is observed that exploration with only haptic information is time-consuming. Users are required more to concentrate on the force feedback hence move slowly and the users can hardly estimate the path to follow. With only vision feedback, users frequently lose the trapped object because of large
<figure><img src="image_24.png" /><figcaption>Figure 2.13: The trajectories of the probe during the experiment of RBC surface exploration. (a), (b), (c) The result of three participants under different experiment conditions: touching with only visual feedback, only haptic feedback, and both haptic and visual feedback are shown in green, blue, and yellow respectively. The red cross indicates that the user has lost the trapped probe during the experiment. The scale bar is 3 µm</figcaption></figure>

contact forces difficult to infer from vision alone. This is also probably caused by
some shadows and invisible features that are barely noticed. Combining the hap-
tic feedback and vision overcomes the above shortages. Vision provides general
overview information of the scene and haptic feedback allows for fine control skills.
Note however that this is a preliminary qualitative analysis. A formal user study
exploring the impact of haptic feedback in optical micromanipulation tasks is car-
ried out in the next section.

## 2.3 Improving Optical Micromanipulation with force-feedback Bilateral Teleoperation

2.3.1 Background and Related Work

Interactive micromanipulation has gained much attention due to a wide range novel of applications such as biological cell manipulation, drug delivery and microassem- bly. Interactive approaches with only visual feedback are limited to the 2D image of the microscope, and have forcibly lower bandwidth. Recently, haptic feedback teleoperation systems have been developed to try to overcome those difficulties.

To perform dexterous manipulation, humans mainly combine the information
coming from touch and vision senses. Touching an object give the possibility to
apprehend subtle mechanical properties like friction forces, torques, or contact po-
sition with the object. From those information can be estimated characteristics of
the object such as geometry, stiffness, and surface condition. Robotic research at
macro-scale has since several years integrate force feedback in the control-loop for
teleoperated manipulation and grasping tasks.
Furthermore, handling physical contact and interactions in terms of force is a key capability in robot control [111, 112], as pure motion control can lead to unstable behaviour during an interaction, especially in the presence of rigid or in-motion objects. Also, in optical tweezers, objects are stably trapped up to a force threshold, depending on the distance to the focal point. This threshold is easily reached in most cases, for example by drag forces occurring during manipulation, ejecting the trapped object.

Hence, force control approach implies immediate benefits in robotic optical tweezers. However, due to the difficulty to access the force sensing in real-time, there are only few works implementing a closed force control loop for optical manipulation. Most approaches are open-loop techniques [113, 114, 115], or automated one based on position [116, 84, 70].

Moreover, forces in the environment are important sources of information to complement the visual feedback and allow estimating physical characteristics such as stiffness or surface condition. As in macroscale, it is expected that user dexterity and precision would improve by enriching their perception. Consequently, several haptic-feedback teleoperated systems dedicated to micromanipulation have been proposed [117, 118]. Most of these works are based on tethered micromanipulators (i.e. manipulators where only the end-effector is micrometer-sized) as Atomic Force Microscopy (AFM) systems [119] or a piezo-electric gripper [91]. Some non-contact manipulators with force-feedback has been proposed with magnetic manipulation techniques [120, 121, 122].

Concerning optical manipulation, Arai et al. [123] have proposed the first haptic coupling for optical tweezers. 2D haptic explorations of microsurfaces have been demonstrated [124, 90]. 2D Multi-trap bilateral teleoperation of optical tweezers has also been proposed [106, 125]. In the previous section, a proof of concept of the first 3D haptic optical tweezers system was presented. These prototypes of haptic optical tweezers allow conducting micromanipulation tasks in a more flexible and intuitive way. However, in these works experiments are generally done by their creators, albeit potential end-users are not roboticians, but lab technicians who expect a turn-key system. Some works studied the usefulness of haptic feedbacks combined with optical tweezers. Lee et al. [126] proposed a model to simulate force interactions between a particle and a laser beam for haptic manipulation of microparticles, but their work is purely theoretical. Artificial guides and force-fields are explored to help the operator steer the trapped particle, avoiding collisions and contacts with other particles [127, 128, 129]. However, these experiments use virtual forces to improve the manipulation efficiency of optical tweezers systems. No study has evaluated a direct bilateral coupling with naive users.

We report here a systematic study on a bilaterally coupled haptic teleoperation on optical tweezers, providing real-time force sensing and straightforward human/machine interaction. We conduct haptic exploration experiments of microsynthetic objects with 14 subjects, resulting in 280 micromanipulation trials, comparing performances with and without force-feedback couplings. The use of synthetic objects helps the repeatability of the experiments and provides a ground
truth for the shapes explored. The results indicate that haptic feedback improves
the operator's dexterity in practical micromanipulation tasks, and improves by 35%
the success rate in exploration tasks.

2.3.2 Haptic Optical Tweezers Platform

A versatile optical manipulation system (see Fig. 2.14) with 3D haptic feedback was designed to offer robust and intuitive control of suspended microbeads. Note that human sense of touch functions over a significant band of frequencies reaching 500 Hz. Double this value is considered a requirement for the control bandwidth to maintaining stability and a quality perception. The latency should not exceed 2 or 3 sampling periods [85]. This particular set-up is based in the previous developed 3D high-speed and high resolution force sensor and the home-made optical tweezers system. Also, the workspace is significantly extended by the combined use of redundant micrometric and nanometric motion stages. The resulting platform provide a user friendly and robust micro-manipulation platform where non expert users can perform exploration task without any intensive training.

A direct bilateral coupling has been implemented, where the master position
drives the trap position, and sensed forces are sent to the haptic device with linear
scaling factors [85]. Contrary to the experience with the RBC, this new experiment
is more demanding because this time we touch rigid and fixed objects. This adds
a degree of difficulty in the stability of the system. Random vibrations introduced

<figure><img src="image_25.png" /><figcaption>Figure 2.14: (a) Schematic representation of the haptic optical tweezers platform. (b) Master device and user interface. (c) Slave device and optical path.</figcaption></figure>
into the system by Brownian motion and fast acceleration when they are a collision
between two rigid body's are then amplified by inertia at the macroscopic level.
In order to improve the user perception, a dissipation block is hence inserted by
virtually damping the force signal sent to the user. This is equivalent to a simple
low-pass filter.

Users control the 3D position of the haptic interface (Omega.3, ForceDimen-
sion). Then, these translations are scaled down by $G_{p,xy} = 1.5 \times 10^{-4}$ and $G_{pz} =
0.9 \times 10^{-4}$ and sent to the nanostage to control the trap position relatively to
sample holder. Finally, estimated forces are scaled up by $G_{f,xy} = 0.3 \times 10^{12}$ and
$G_{fz} = 1.2 \times 10^{12}$ and send to the master device. The scaling factors in position is
chosen as to cover $40 \times 40 \times 20 \ \mu m^3$ in the optical tweezers workspace. Scaling factors
in force are empirically chosen for a subjectively optimum tactile perception.

The entire solution is deployed on a hard real-time framework (co-kernel, Xeno-
mai) to guarantee 1 kHz control-loop rate. The proposed direct coupling provides
very good transparency as the force signal sent to the haptic device is a linear func-
tion of the sensor output. Fig. 2.16 depicts both these signals during a manipula-
tion. A slight difference appears due to the virtual dissipation and the mechanical
response of the haptic device.

2.3.3 Micro-chip Test bench

To evaluate the utility of haptic feedback in micromanipulation experiments and to
ensure the repeatability of the task, a test bench has been manufactured. Fig. 2.15
shows this chip containing different basic shapes ranging in size from 7 µm to 14 µm.

The chip was printed using two-photon lithography with a Nanoscribe Photonic
professional GT (Nanoscribe GmbH, Germany). The 63x objective and IP-Dip
photoresist were used for printing to achieve the highest resolution (<1 µm) when
writing 3D microstructure. After the completion of the printing, the wafer was
submerged in PGMEA for 40 minutes and then in Isopropanol (IPA) for 30 minutes.
After that, the IPA was removed by simply letting it evaporate in a clean room.

The entire chip body has been designed to the millimeter scale, so that it can be
simply manipulated with forceps. During an exploration task, the chip is placed in
a Petri dish filled with a solution composed of 90% distilled water, 5% ethanol and
5% *Tween20* to prevent surface adhesions. It is oriented in a way that the columns
with the micrometer size shapes come into contact with the cover-slip. Then, the
microbeads are added from a solution using a micropipette. After each trial, the
chip is washed with distilled water and IPA until evaporation.

2.3.4 Experimental Protocol

The goal of the experiment is to analyse the performance of the subjects in an
exploration task with and without haptic feedback.
<figure><img src="image_26.png" /><figcaption>Figure 2.15: (a) CAO model and dimensions of the test bench microchip. (b) Scanning electron microscopy (SEM) image of the microchip.</figcaption></figure>

### 2.3.4.1 Participant

The set of 14 volunteers were from Sorbonne Université. The age range were from 20 to 30 years old. The accepted volunteers were all right handed and did not present any physical abnormality.

### 2.3.4.2 Tasks performed

The task is to explore edges of different shapes using a optically trapped microbead of 3 µm diameter. The participant needed to first establish contact between the microbead and the explored shape, then follow the edges of the shape while maintaining contact until a complete tour around the shape is done. If for any reason the microbead is ejected from the optical trap, the task ends and is logged as 'failed'. The bead ejection can typically be caused by an abrupt movement or an excessive pressure on the explored shape.

The complete set of tasks contains two exploration attempts of five different shapes located on the microfluidic chip. The five shapes are a square, a trapeze, a triangle, a half-circle and a cross. Participants were divided in two groups of equal number. A first group of 7 participants performs first the set of tasks with only visual feedback (V), then the set of tasks with visual feedback and haptic
<figure><img src="image_27.png" /><figcaption>Figure 2.16: Example of bilateral teleoperation in a triangle shape exploration task. (a) The 3D optical force applied on the microbead and the 3D haptic force feedback during the manipulation. (b) The 2D trajectory of the microbead during the triangle surface exploration.</figcaption></figure>

feedback (V+H). A second group of 7 participants performs the set of tasks with visual feedback and haptic feedback (V+H) first, then the set of tasks with only visual feedback (V). Consequently, each participant performs 10 exploration tasks with visual feedback (V) and 10 with dual feedback (V+H), therefore a total of 140 trials were performed for each condition.

### 2.3.4.3 Conduct of the experiment

The experiments were conducted one participant at a time and each trial lasted around 30 minutes, depending on the participant's performance.

Before the formal evaluation, the participants were trained to use the system for 10 minutes and shown how to manipulate microbeads (3 µm, polystyrene) with the help of the haptic interface. An expert user traps a microbead, approaches the different shapes to explore, and constraints the z axis motion, then hands the control to the participant. This way the experiment is dissociated from the trapping phase and focuses on the feedback during execution.
### 2.3.5 Results and Discussion

In order to evaluate the performances of the subjects, the position of the nanostage and the estimated force applied on the microbead are recorded at a rate of 500Hz during the complete duration of each task.

<figure><img src="image_28.png" /><figcaption>Figure 2.17: Examples of an exploration of trapeze (A), half-circle (B) and cross (C) shapes. (a) The 2D path of the probe and the 3D contact forces during the experiment under vision condition (V). (b) The 2D path of the probe and the 3D contact forces during the experiment under vision + haptic condition (V+H). (c) The trajectories of the probe during the shapes surface exploration under the two conditions (V and V+H). (d) The histograms of the forces during the exploration task. Results are normalized to facilitate the comparison.</figcaption></figure>
After the experiments, the maximum, the mean and the standard deviation are extracted from the norms of the recorded forces for every task. Those force results are then used to compute the desired average maximum, average mean and average standard deviation displayed in Fig. 2.18, Fig. 2.19 and Fig. 2.20. In addition, the task is written as failed if the participant loses the trapped microbead, and its data is used for the calculation of success rates shown in Fig. 2.18.(d), Fig. 2.19.(b) and Fig. 2.20.(d). Finally, the duration of each task is stored to evaluate the timing aspect of the participants performances.

### 2.3.5.1 Results per subject

Fig. 2.17 shows examples of exploration results in both condition (V) and (V+H) for the trapeze, the half-circle and the cross. Fig3.(a) and (b) show the path followed and the contact forces applied on the microbead during the contour exploration. Note that the forces are never completely null as a consequence of the Brownian motion applying constantly an erratic force on the bead, which is one of the important differences between macro-manipulation and micro-manipulation.

In A, the participant did not lose the microbead and complete the trapeze exploration task in both conditions (V) and (V+H). However, the representation of the forces shows a far more irregular pattern in condition (V) than in (V+H). The trajectory around the shape shown in Fig. 2.17.(c) reveals a more accurate tracking of the outline of the shapes when the haptic feedback is enabled.

With vision only, the participant tends to move away from the shape or to apply excessive forces risking the loss of the microbead. In B and C, the participant loses the microbead in condition (V), but not in (V+H). A force spike can be seen near the end point when the bead was lost in (a) of B and C.

The distribution of the forces displayed in Fig. 2.17.(d) present a Gaussian shape in the condition (V+H), in contrast to the (V) case centered at 0pN with a decreasing slope. During the exploration in the condition (V+H), the participants exhibit a relatively constant force, relying obviously more on the haptic feedback than on vision for the fine movements. Without force information in the (V) condition, the participants were trying to interpret the visual feedback to evaluate the force applied, resulting in a more demanding situation hence less accurate movements. In the condition (V+H), the haptic information naturally integrates with the vision to achieve finer motor-control.

Fig. 2.18 depicts the force analysis and success rate for an individual subject. The most obvious result is the low rate of failed trials when haptic is used, as shown in Fig. 2.18.(d). It is effectively quite difficult to visually evaluate when the microbead risks to be ejected. The force feedback marks the difference by providing this critical information. The effect is also visible in the maximum force graph in Fig. 2.18.(a) which shows a decrease of the maximum force exerted, meaning that the participant manages the contact better, releasing pressure on the explored object when needed.
<figure><img src="image_29.png" /><figcaption>Figure 2.18: Statistical results of the experiment per subject.</figcaption></figure>

We also observe for 13 out of 14 participants a decrease in the mean force exerted between (V) and (V+H) (Fig. 2.18), attesting that participants proceed more gently when the haptic feedback is enabled.

The force variability shown in Fig. 2.18.(c) has dropped for every subject between (V) and (V+H), reflecting the previous observation on the force's distribution displayed in Fig. 2.17.(d). The participants naturally attempt to keep a constant force during the exploration in condition (V+H), which diminish the variability of the exerted forces.

No meaningful differences are observed in the individual results between the group starting with (V) and the group starting with (V+H). This absence of dis-similarity tends to demonstrate that no additional training is necessary to handle the haptic feedback. If a period of familiarization were needed, the participant start-ing with (V) would have performed better in (V+H) than the participant starting directly with (V+H) because they would have already completed 10 exploration tasks and substantially progressed in optical-manipulation.

### 2.3.5.2 Results per shape

Fig. 2.19 shows the average duration of the completed tasks and the corresponding success rate per shape. Only successful trials are counted in the computation of the time of exploration.

A small decrease of duration is observed for each shape between (V) and (V+H). Since no instructions were given to go fast, it indicates that the participants tend to naturally proceed slightly better with the haptic feedback in condition (V+H).
<figure><img src="image_30.png" /><figcaption>Figure 2.19: Statistical results of the experiment for each shape.</figcaption></figure>

An expected inverse correlation appears between the duration of the task and the success rate. The most complex shape is the cross by the number of sharp corners. It has a longer duration of exploration and a lower successful trial rate, while a simpler shape like the triangle shows shorter duration of exploration and a higher success rate.

### 2.3.5.3 General results

In Fig. 2.20 are displayed global results representing the average data of all participants over all shapes. Between (V) and (V+H), the average maximum force decreases by 46.0% (6.70pN) and the average force drops by 40.4% (1.45pN). The force variability drops by 55.7% (1.87pN). Finally, the global successful trials rate rises by 35% from 61% to 96%, meaning that the fail rate is divided by almost 10 from 39% in (V) to 4% in (V+H).

To perform optical-micromanipulations, visual feedbacks are suffering from two major issues. First, the broadcasted image is a 2D projection of a 3D workspace. The operator faces difficulties to apprehend the working environment and a mental effort is required to grasp a clear 3D representation of the ongoing manipulation. Secondly, an objective with high numerical aperture is necessary to create an optical tweezer. The focused region of the resulting image is narrow and generates visual artefacts around the observed micro-structures. Those two aspects compromise the user's vision and thus his performances. In addition, the forces generated by an optical tweezer are weak and a trapped micro-object can easily be ejected by a collision or an abrupt movement.

The deployed haptic feedback system complements the visual feedback with 3D forces information. Therefore, it helps in the 3D apprehension of the workspace and in the perception of the forces exerted by the optical tweezer on the trapped microbead.
<figure><img src="image_31.png" /><figcaption>Figure 2.20: Statistical global results of the experiment.</figcaption></figure>

## 2.4 Conclusion

In the first part of this chapter we have presented a new method for 3D force sensing for Optical Tweezers, with high bandwidth (up to 10Khz), in a large three-dimensional workspace that covers the full linear domain of a single trapped bead and pico-newton sensibility with a theoretical resolution of 0,3pN. The force sensing is based on high-speed optical processing implemented through a silicon retina and its dedicated tracking algorithm.

The real-time particle-tracking scheme can be also extended to magnetic tweezers and other micro-robotics techniques. The method remains compatible with fluorescence, interferential and superresolution microscopy. An interesting development would be a Scanning Force Microscopy technique adapted to biological environments. Thanks to the capabilities of force sensing, and because of the stable and simple set-up design, complex tasks have been demonstrated in real biological environment on a haptic teleoperation platform. This is a good illustration of the kind of new robotic application that can benefit from the proposed 3D sensing.

In the second part of the chapter, we have explores the case of an optical tweezers platform coupled to an haptic device providing transparent force feedback. A user study has shown that non-expert users successfully achieve to use the system after preliminary training. It is safe to assume that expert users will be able to perform complex tasks more efficiently.

This study demonstrates that haptic feedback improves the user performance in optical manipulation by a significant margin. Vision provides general overview information on the scene and haptic feedback is used for fine control when contacting or manipulating an object. The operator naturally integrates this additional information channel. Participants apply in average less pressure on the manipulated objects, their handling is smoother and contact forces exhibit far less variance. The decrease in the execution duration lets also assume a better mental efficiency, correlated to a lesser cognitive load, which indicates that the user considers the task easier [130].
Therefore, proposed multimodal stimuli enables users to better apprehend mi-croscale phenomenons and be more effective to achieve desired results during ma-nipulation tasks. Additionally, the 3D force complements the 2D vision information delivered by the microscope and allows users to react faster, as human haptic per-ception bandwidth is one order of magnitude higher than vision. This also gives a far richer information: for example changes in high-frequencies of the Brownian motion which would go unnoticed for the eyes becomes perceivable with haptics. As an additional last remark, the system used in experiments also proved itself repeatable and reliable.

As future works, we plan to further test the method on different test bench with various materials. Specially, soft materials will be implemented on the test benches to mimic biological samples, e.g. cells and tissues, and perform user studies with other tasks, eg. pick-and-place.
<figure><img src="image_32.png" /></figure>
# 3D High-Bandwidth Multi-Trap Actuation for Optical Tweezers

Optical robots are micro-scale structures actuated using laser trapping techniques. However, the lack of robust and real-time 3D actuation techniques reduces most applications to planar space. In this chapter, we present a new approach to generate and simultaneously control more than 15 optical traps in a 3D workspace with low latency, high bandwidth (up to 1 KHz) and nanometric resolution. This time-shared technique only uses mirrors, hence is aberration-free. Precision and efficiency studies with individual beads are carried out with trajectory control tasks. Simultaneous traps are used to actuate optical robots, thus demonstrating true six Degrees-of-Freedom (DoF) control. This strategy do not only allow 3D actuation of optical robots or individual objects but open the road for novel applications in micro-robotics and biology requiring precise, flexible and effective 3D handling during micro-manipulation tasks.
Contents

<table><tr><td>3.1 Introduction</td><td>52</td></tr><tr><td>3.2 Background and Related Work</td><td>53</td></tr><tr><td>3.3 Development of a 3D High-Bandwidth Multi-Trap Actuation Method</td><td>55</td></tr><tr><td>3.3.1 Calibration of the system</td><td>57</td></tr><tr><td>3.3.2 Evaluation of the Actuation Performances</td><td>60</td></tr><tr><td>3.4 6-DoF Control of Optical Robots</td><td>61</td></tr><tr><td>3.4.1 Fabrication of Robots</td><td>62</td></tr><tr><td>3.4.2 Robot control</td><td>64</td></tr><tr><td>3.5 Conclusion</td><td>65</td></tr></table>

## 3.1 Introduction

Mobile micro-robots promising access to intangible scales brings revolutionary potential to a vast amount of biological applications, ranging from minimally invasive unicellular surgery to exploration of fundamental biological phenomena [1]. Optical trapping is a practical mean for actuating these tiny micro-machines in confined environments, such as lab-on-chip devices. Using diffractive optical elements or active components, multiple traps can be generated from a single source, allowing simultaneous manipulation of several independent samples or applying torques on non-spherical objects.

The approach of using intermediate tools widens clearly the scope of optical trapping. However, controlling these so-called optical robots requires to be able to synchronously control each handle. This in turn requires a precise control of the laser exposure and the trajectory of individual traps. Also, the overall stability is dependent on guaranteeing the same amount of power is transmitted at each trap regardless of its position. This is only possible on an efficient, high-bandwidth and low-latency system. Implementing true 6-DoF control of Optical Robots necessitate real-time 3D multi-trap actuation technique. These independent optical traps need to be fast but also stable and efficient as possible (i.e., the same amount of trapping power, regardless of the movement of the beam). Large latency and low bandwidth in the actuation technique will hurt the system stability and its real-time capabilities [85].

These constraints are quite difficult to satisfy, therefore most optical robots are confined in 2D space in the optical plane (two translations and a rotation), limiting the applications where 3D controlled motion is crucial, such as cell orientation for micro-injection, polar-body biopsy, nuclear transplantation or 3D tomographic imaging [131].
The main aim of this chapter is to present a new design of optical actuation to generate and control several simultaneous traps in 3D space. This design is solely based on mirrors, thus maximizing the optical efficiency. Also, the choice of components with very high response times lowers latency and increases considerably the bandwidth for the control. Following a literature survey on multi-trap solutions, the proposed system is described next, and demonstrated controlling simultaneously 13 individual objects in 3D. Then, the performance of the actuation system in terms of static and dynamic precision is evaluated using trajectory control tasks. Finally, its capabilities are depicted controlling true 6D motions of optical robots.

## 3.2 Background and Related Work

A stable trap should provide a steep potential well in all axes. This is generated by the balance between the gradient forces on the optical plane and the scattering force in the normal direction and its efficiency is determined by the intensity distribution near the focal point. A sharp focus produces stronger intensity gradients and, therefore, better traps. This diffraction-limited spot size is very sensitive to wavefront distortions [48].

Controlling the 3D motion of non-spherical objects (like cells, or synthetic robots) where optical handles are rigidly linked is much more demanding than controlling separate and individual beads. Relative deviations in the position of each trap and synchronisation problems in the movements of the traps will affect the stability of the whole structure. Besides, the force on each trap will interfere with the dynamics of the motion, as well as differences in the power distribution between traps.

The generation of several simultaneously trappings points using a single laser source can be achieved using spatial or temporal sharing methods with their specific strengths and limitations. Table 3.1 summarizes the properties of principal methods.

In spatial-sharing, using the light-diffraction, the beam is split on several less powerful beams. This requires a diffractive optical component such as a *Spatial Light Modulator* (SLM). Holographic optical tweezers (HOT) [45] or Generalized Phase Contrast (GPC) techniques [132] are one of the most common techniques for creating multiple traps in 3D, in our days. However, calculating the hologram for a given number of traps in space is quite processor intensive. Liquid crystal components have still relatively low response time. The computational cost and trajectory control are considerably more complex and less suitable for real-time control. Although parallel computing and graphics processing units (GPU) [133] improve the situation, SLM spots present interferences that lead to a substantial reduction in efficiency [134]. Hence, it's currently quite impossible to obtain sufficient bandwidth for real-time operation using spatial sharing. Recently, some studies utilizing HOT systems for rotational motions are reported. 2D actuation of an optical micro-robot is described [135]. In this case, limited by the response time
of the SLM (80 ms) and the stability of the traps, the maximum control frequency of the microrobot is 3 Hz. 3D rotation control of single live mammalian cells was reported [136]. Nonetheless, out-of-plane rotation speed of the cell and the traps are different with an important delay time of 1 to 5 seconds. Multiple degrees of freedom in HOT systems are also possible through wave-front shaping, using vortex [137], Laguerre-Gaussian [138], Airy [139] and other beam shapes [140], [141]. Although these SLM-based techniques represent elegant solutions for the 3D motion or rotations, the extensive computational cost and complexity of trajectory control make their implementation in real-time scenarios a very difficult task.

On the other hand, multiple stable trapping points can be created by rapidly deflecting a laser beam among a set of positions. In these temporal-sharing methods the frequency of the trap scanning should be faster than the Brownian relaxation time of the trapped objects [48]. Usually these methods use a scanning mirror system [142] or an acousto-optic deflector (AOD) [86]. Time-shared methods are suitable for controlled manipulation under real-time automatic/interactive control. The complexity here is synchronizing the focus of the beam with the vertical motion.

Using a motorized objective or a z stage, the axial position of the beam can be controlled [143, 144, 145]. However, the inertia of such a system makes the synchronization of axial and planar motions a very difficult task. Some interesting approaches are reported to overcome this limitation. By combining a deformable mirror and acousto-optic deflector, a 3D steering system for single particle manipulation is presented, but no multi-trap capability has been reported [146]. An oscillating optical trap is used to rotate rod-shaped bacterial cells with respect to the optical axis [147]. The oscillating trap was produced by means of a galvanometric mirror and the angle of rotation is determined by the amplitude of the oscillation. An electrically focus-tunable lens combined with a two-axis steering mirror is proposed [148, 149]. The delayed response and distortion of the electrically focus-tunable lens make it impossible to control 3D rotations accurately. A combination of a mirror galvanometer with a GPC to manipulate a micro-tool was

<table><caption>Table 3.1: Comparison of 3D multi-trap actuation methods</caption><thead><tr><th>Method</th><th>Response time</th><th>Bandwidth</th><th>Trajectory computation</th><th>Trap force</th></tr></thead><tbody><tr><td>GPC</td><td>High</td><td>Low</td><td>Medium*</td><td>Weak</td></tr><tr><td>HOT</td><td>High</td><td>Low</td><td>Complex*</td><td>Weak</td></tr><tr><td>MO + SM</td><td>Very High</td><td>High</td><td>Simple</td><td>Strong*</td></tr><tr><td>GPC + SM</td><td>Low</td><td>High</td><td>Complex*</td><td>Weak</td></tr><tr><td>EFL + SM</td><td>Medium</td><td>High</td><td>Simple</td><td>Medium*</td></tr></tbody></table>

GPC: Generalized Phase Contrast, HOT: Holographic Optical Tweezers, MO: Motorized objective or stage, SM: Scanning mirror, EFL: Electrically focus-tunable lens. * Depends on the number of traps.
proposed [150]. The *GPC* generates fixed laser traps and the galvanometer is used to manipulate the overall configuration. The trapping force induced by *GPC* is weaker than what can be obtained directly by a mirror, hence the motion is quite limited and confined in the optical plane. As a workaround for the axial motion, an articulated robot with an out-of-plane mobility of an appendix using AOD is recently proposed [73]. The precise and stable control of this DoF is a complex endeavor as it controlled through handles on the structure in the same focal plane.

The lack of high bandwidth and low latency control makes precise control impossible in real-time, limiting the potential of optical robots that interacts in a micro-world with high dynamics effects. An alternative design introducing a time-shared scanning technique for 3D multi-trap actuation is presented next. This system is based on the synchronization of a deformable mirror and a steering mirror and allows simultaneous control of focal and planar positions. This approach combines the efficiency of optical components with high reflectance and low latency response times.

## 3.3 Development of a 3D High-Bandwidth Multi-Trap Actuation Method

The location of the trap is determined by the angle of incidence ($x-y$ axes) and the degree of collimation ($z$ axis) of the laser beam and its stability is very dependent on the quality of the beam that reaches the entrance of the microscope objective. However, given the correlation between the beam that reaches the objective and the focal point, it is difficult to change the 3D position of the trap without distorting its shape and introducing aberrations that will degrade its performance [151].

To overcome these constraints in all three dimensions, a mirror galvanometer is used for the in-plane ($x-y$) scanning. A deformable mirror (*DM*) which can focus or defocus the beam is used on $z$. The deformable mirror and the galvanometer are positioned in a conjugate plane on the entrance aperture of the objective (OBJ). Hence, the laser beam will pivot around the entrance aperture of the microscope objective and retain the same degree of overfilling, independently of the angle or degree of collimation of the incident beam, producing equally efficient and stable traps as much as possible [152]. Fig. 3.1 depicts this principle.

The deformable mirror (*PTT111 DM*, *Iris AO*) is a microelectromechanical component with 111 actuators and 37 piston-tip-tilt segments with an update rate of more than 2 kHz. Each segment has 700 µm diameter while the array have an aperture of 3.5 mm and with a maximum dynamic range (Stroke) of 5.8 µm. Electrostatic actuation allows precise positioning of each segment with nanometer and microradian resolution (wavefront resolution <15 nm rms). The mechanical step-response speed is less that 200 µs (10-90%) and has high reflectance (>99.9%).

The segmented *DM* can effectively create smooth shapes, and it is used to control the degree of collimation of the laser beam. By synchronizing the orientation of the mirror galvanometer and the motion of the *DM* it is possible to displace the fo-
cal spot laterally and axially while maintaining the diffraction-limited performance. This axial scan implies a very low mechanical inertia, hence the *z* bandwidth is compatible with the galvanometer's and it's possible to obtain very stable traps without introducing any significant aberrations. The overall switching speed between 3D trap positions is in the same order of magnitude of equivalent 2D systems in the literature, as is shown below.

The optical scheme of the system is shown in Fig.3.2.(a). The Gaussian laser beam (1070 nm) is guided into the inverted microscope through the galvanometer, the *DM* and standard optical elements. Two afocal systems ($f_1:f_2$ and $f_3:f_4$) are used to conjugate the two actuators with the entrance aperture of the objective and to expand the laser beam. It's expanded in order to overfill (20%) the objective entrance to improve the trapping efficiency [48]. A picture of the whole platform is shown in Fig.3.2.(b).

<figure><img src="image_33.png" /><figcaption>Figure 3.1: (a) Schematic representation of how the laser beam will pivot around the objective (OBJ) entrance aperture for small movements of the mirror galvanometer (GVT), creating the motion of the optical trap in the optical plane; (b) Schematic representation of the effect of focusing or defocusing the deformable mirror (DM) for change the *z* position of the traps. The size of the beam at the objective entrance aperture remains the same, regardless of the degree of collimation of the incident beam.</figcaption></figure>
A critical issue in the system is the balanced distribution of power among different traps and the synchronization between the galvanometer and the *DM*. A hard-real time control framework implemented in C++ on a Real-Time kernel is set up to manage the synchronization. A single PC (Intel Xeon core, 2.93 GHz) operating under a real-time Linux co-kernel and RTOS APIs Xenomai [153] gives satisfactory performances, without requiring any special hardware nor GPU. Synchronized parallel threads corresponding to the galvanometer and the *DM* run at hard real-time controlled by real-time co-kernel Xenomai/Linux with fixed sampling rate. Given the very low response time of these components, near perfect synchronization is observed.

### 3.3.1 Calibration of the system

Precise positioning in 3D of each trap requires calibration between the optical path and real world coordinates. The calibration in axial direction *z* is not an obvious task, as one can not rely upon the 2D optical feedback. The proposed tracking position sensor developed in chapter 2 is used in this purpose.

A 3 µm silica micro-sphere is trapped and displaced along the *z* axis, by varying the defocus magnitude of the *DM* from -0.5 µm rms to 0.5 µm rms, while tracing

<figure><img src="image_34.png" /><figcaption>Figure 3.2: (a) Optical path of the time-shared multi-trap actuation. The source is a 1070 nm laser with a maximum output of 10 Watts. The deformable mirror (DM) and the galvanometer (GVT) are positioned in a conjugate plane of the entrance aperture of the objective (OBJ) and used to control the axial and the planar positions respectively. The oil immersion objective (Olympus UPlanFLN 40x, NA 1.3) is used to produce the optical traps and to image the scene. The illumination (LED, 3W) is reflected by a longpass dichroic mirror (LDM, 900 nm cutoff) into the CMOS camera (Basler, 659×494 pixels). $f_1 = f_2 = 50$, $f_3 = 60$, $f_4 = 250$, M:Mirror. (b) Image of the optical trapping system with a close-up photograph of a Deformable Mirror (PTT111 DM, Iris AO) and a triangular optical robot. Two micro-stages (x - y) and a 3D nano-stage provide respectively coarse and fine positioning of samples.</figcaption></figure>
the position of the bead as seen by the above mentioned sensor. As the sensor has a range of ±2 µm around the focus plane with 5 % SD, 70,000 data points are recorded in the 4 µm range of the sensor given an excellent linear fit depicted in Fig.3.3. The working range of the axial displacement is estimated as 9 µm between the maximal defocusing and the minimal defocusing position of the *DM*. This workspace is dependent on the current set-up and can be easily increased using a *DM* with a larger stroke (eg. DMP40, Thorlabs).

On the other hand, using the microscope and CMOS camera, *x* − *y* calibration can be obtained quite straight forwardly, using any known distance. The precision in this case is limited by the optical properties of the microscope, and the quality of the external gauge. The actuation sensitivities in *x* − *y* axes is 37 µm/degree and 35 µm/degree respectively. The lateral workspace is limited by the range of view of the camera and is estimated as 70×50 µm.

The initial calibration showed a cross-talk between planar and axial motions as shown in Fig. 3.4.b. This is due to a small misalignment in the light path between galvo and DMD. Also, in the planar motion, as we used deflection laser system with two-mirror (galvanometer *x*,*y*), a distortion of the x-axis on the y-axis is observed, and vice versa. This distortion arises from the fact that the distance between mirrors and the image field depends on the size of the mechanical scan angles of mirrors. These cross-talks has been corrected in control using linear regression for DMD/galvo and quadratic regression for galvo.x/galvo.y and galvo.y/galvo.x. Hence, all three axes can be controlled independently.

The system allows a sampling rate of 2 kHz for the control of *z* axis for small displacements, below 2 µm. In order to exploit the full range of 9 µm allowed by

<figure><img src="image_35.png" /><figcaption>Figure 3.3: The axial position range of the system. The linear regression coefficients show the conversion between DM coordinates and the real displacement, as reported by the position sensor.</figcaption></figure>
<figure><img src="image_36.png" /><figcaption>Figure 3.4: (a) The y axis error in function of the x value and the quadratic regression. (b) The correction in x – y axes when a motion in z is generated.</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>Figure 3.5: Controlled movements of thirteen micro-beads arranged in a cube of 16×16×8 µm.</figcaption></figure>

the *DM*, the control loop is executed at 200 Hz. This bandwidth is in the same order of magnitude of the 2D mirror galvonometer (GVS002, Thorlabs), with full scale bandwidth of 250 Hz and 1 kHz for small angles (±0.2deg).
<figure><img src="image_38.png" /><figcaption>Figure 3.6: Static precision tasks. (a) Four micro-beads positioned in a square of 40×40 µm. The maximum error between the command and the actual position is 400 nm. (b) Thirteen micro-beads set to cover a 40×40 µm square. The maximum position error is up to 1 µm and is essentially due to Brownian motion. Positions are measured during 15 s.</figcaption></figure>

The absence of external perturbations makes the synchronization between those 2 actuators reliable enough: the system allows the trapping and independent control of more than 15 micro-beads in different axial positions. Fig. 3.5 shows 13 micro-beads simultaneously trapped in 3D space manipulated synchronously. The micro-spheres are 3 µm in diameter, and are arranged within a cube of 16×16×8 µm³.

### 3.3.2 Evaluation of the Actuation Performances

To evaluate the system's performances, several experiments with micro-beads have been carried out. Polystyrene beads have been chosen, as they are the most common tool used in optical tweezers manipulation systems.

Set-points of actuators, and video images at 64 fps from the CMOS camera are recorded during different tasks. Each data record contains the current system time (Xenomai timer) for synchronization purposes. Measured positions of the trapped beads are extracted in an off-line process using the circle hough transform algorithm from OpenCV software. The image has a resolution of 659×494 px and cover a surface of 70X50 µm and the theoretical resolution of the tracking algorithm is 2 px, corresponding to about 200 nm. As the 2D image tracking algorithm does not permit to estimate the depth, only 2D projection of the 3D motions are studied. Same laser power (400 mW), trap irradiation time (5 ms), and polystyrene micro-beads (3 µm diameter, refractive index ~1.59) were used for all tasks.

#### 3.3.2.1 Static Precision

The static precision depends on the actuators' accuracy, thermal noise, and trap stiffness. The deformable mirror *DM* lacks position feedback and is controlled in
open-loop, based on the calibration and has nanometer and microradian resolution (manufacturer data: wavefront resolution <15 nm rms). The galvanometer is controlled in closed-loop with an angular resolution of 15 µrad.

In time-sharing scenarios, when more than one trap is created, the laser switches from trap to trap. In this case, each bead is only held by the laser during a fraction of the cycle, and the rest of the time the micro-bead is subjected to Brownian fluctuations and other environmental forces [53]. In consequence, the effective stiffness of each trap is diminished by the reduced duty cycle and the precision of the bead's position is inversely related to the number of traps created.

Different static tasks have been performed and recorded during 15 s. The static position error value is computed as the maximum error between the trap's position command and the bead's tracked position. For a single trap, no differences between the trap's command position and bead's measured position is detected, as the stiffness of the trap highly reduced the Brownian fluctuations. We conclude that the error position is less than the resolution of the tracking algorithm, i.e. the position's maximum error is less than 200 nm. For four and thirteen micro-beads positioned in a square of 40x40 µm, position error is calculated as 400 nm and 1 µm respectively, and is essentially due to Brownian motion as the effective stiffness of each trap decreases as the number of traps increases. These experimental results for four and thirteen traps are presented in Figure 3.6.

### 3.3.2.2 Dynamic precision

Two different tasks have been performed to evaluate the dynamic performances. The first task consists in trapping a micro-bead at different axial positions and rotates at constant velocity, increasing the speed by steps of 21 µm/s. Positions of micro-beads are extracted off-line from video images using a circle tracking algorithm and the velocity is computed as the discrete derivative of the position. Results shown in Figure 3.7 confirm that the micro-bead accurately follows the velocity reference from 21 µm/s to 462 µm/s with an error of 4 %. Finally, the bead is lost at 483 µm/s.

The same experiment performed with a group of 4 trapped micro-beads with different axial configurations, shown in Figure 3.8, also demonstrates proper following of the references. The maximal reachable velocity without losing the beads are 105 µm/s with an error of 5 %. Since the beads are lost at the same speed in the different configurations, these experiments validate the multi-trap actuation system and corroborate that the system produces equally efficient and stable traps, regardless of the traps' 3D positions.

## 3.4 6-DoF Control of Optical Robots

Optical robots are basically small inert structures, with spherical handles implemented at different locations. These can be custom designed for a particular experiment and different robots can be used also simultaneously. In this section, we
<figure><img src="image_39.png" /><figcaption>Figure 3.7: Velocity control of one micro-bead at different Z positions. (a) Pictures of a trapped bead in different axial configurations. Scale bar is 5 µm long. (b) Velocity reference and measured velocity in 3 different configurations (c) X and Y positions of the trap at a speed of 21 µm/s and 462 µm/s. The maximal reachable velocity without losing the traps is 462 µm/s with an error of 4 %.</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>Figure 3.8: Velocity control of 4 micro-beads at different Z positions. (a) Pictures of 4 trapped beads in different axial configurations. Scale bar is 5 µm long. (b) Velocity reference and measured velocity in 4 different configurations. (c) X and Y positions of the Trap 1 in "Z=0 µm" configuration at a speed of 21 µm/s and 105 µm/s. The maximal reachable velocity without losing the traps is 105 µm/s with an error of 5 %.</figcaption></figure>

will demonstrate real-time control in 6-Dof of Optical robots. The high bandwidth of the proposed actuation system and the lack of any noticeable latency allow such a direct control to be easily implemented.

### 3.4.1 Fabrication of Robots

At least three 'handles' are necessary to induce complete spatial motion. Two types of optical robots have been designed, with three and four optical handles,
<table><caption>Table 3.2: Summary of the actuation system parameters</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Actuation range</td><td>70×50×9 µm³</td></tr><tr><td>Full scale bandwidth</td><td>200 Hz</td></tr><tr><td>Small scale (±2 µm in z) bandwidth</td><td>2 KHz</td></tr><tr><td>Latency</td><td>200 µs</td></tr><tr><td>Static error (1T,4T,13T)</td><td>&lt;200 nm, 400 nm, 1 µm</td></tr><tr><td>Stable independent optical traps</td><td>&gt; 15</td></tr><tr><td>Optical loss</td><td>&lt; 3%</td></tr></tbody></table>

spheres of 3 µm diameter (Fig. 3.9). Different spacers are attached on both sides of the robot to minimize the adhesion effects. They are fabricated through two-photon polymerization (2PP) (Nanoscribe Photonic Professional). Each robot can be printed with various end-effectors appropriate for different applications.

The robots are incubated in distilled water, 5% ethanol and 0.5% Tween20

<figure><img src="image_41.png" /><figcaption>Figure 3.9: (a) Schematic depictions with dimensions of two different optical robots; (b) Scanning electron microscopy (SEM) images of robots.</figcaption></figure>
solution to prevent the surface adhesion. For the experiments, micro-robots are transferred to a sample chamber through an micromanipulated system (MP-28 5, Sutter Instrument CO) and actuated microliter syringe (for further details please see annex A). Then sealed with another cover-slip, thus enclosed in a confined space.

### 3.4.2 Robot control

The robot kinematics was expressed in a local reference frame attached to its body $F_r$. This frame has as origin a virtual point $G$ in the center of rotation of the robot. Each optical handle is represented initially with a 3D vector $T_n = \{X_n, Y_n, Z_n\}$ where $n$ depicts the number of the trap (Fig. 3.10.b and Algorithm 1).

<caption>**Algorithm 1** 6-DoF teleoperated control of optical robots</caption>

Require : Initial traps position $T_n = \{X_n, Y_n, Z_n\}$ and center of rotation $G = \{X, Y, Z\}$; i = 1
0: every 5 ms switch to next trap i+1
1: do for trap i
2: Update desired position $P=\{px,py,pz\}$ and orientation $\Omega=\{rx,ry,rz\}$
3: Calculate $G: G = P \times gain\_pos$.
4: Update Rotation Matrix : $R[3][3] \leftarrow \text{Rot\_Mat}(rx,ry,rz)$.
5: Calculate new trap position of trap i: $N_p = R \times T_i$
6: Transform $N_p$ and $G$ in mirror coordinates
7: Send Output : $M = N_p + G + \text{axes\_corrections}$ to mirrors controllers
8: end do

#### 3.4.2.1 3D translations

Virtual point $G$ is used to translate all the traps, resulting in a uniform motion of the robot as depicted in Fig. 3.11. The 3D translation of the optical robot are

<figure><img src="image_42.png" /><figcaption>Figure 3.10: (a) Optical robot on the screen and global reference frame. (b) Representation of robots kinematics and frames used in the 3D control.</figcaption></figure>
limited to the field of view of the camera in *x* − *y* and by the working space of the axial actuation in *z*-axis.

### 3.4.2.2 3D Rotations

The rotation of the robot is controlled by the desired orientation $\Omega$={rx,ry,rz} with global reference frame $F_g$. The rotation matrix $R$ from $F_g$ to $F_r$ is calculated using the *Tait-Bryan angles* with $z - y - x$ convention (yaw, pitch and roll). Oriented 3D trap positions are calculated as $R \times T_n$. Figure 3.12 shows a succession of different rotations around all-three axes of the two different robots. True 6-DoF motion ability of an optical robot is successfully demonstrated.

The achievable orientation range depends naturally on the maximum distance separating optical handles and the center of rotation, and limited by the working space of the 3D actuation technique. The rotation around *x* (*roll*) is ±35deg for four-handles robot and ±45deg for three-handles robot. The rotation around *y* (*pitch*) is limited to less than ± 90deg in order to avoid the eventual occlusion between two traps in a vertical configuration. Around *z* (*yaw*) the rotation is virtually unlimited. Every DoF can naturally be controlled independently, coupled to another, or fixed.

## 3.5 Conclusion

In this chapter, a new design with a fast and efficient architecture to generate several (up to 15) independent optical traps in arbitrary positions in a volume of approx. 50×45×9 µm³, with a high bandwidth (up to 200 Hz) and nanometric resolution, is presented. The characterization of the system shows a static precision of less than 0.4 µm and 1 µm for the manipulation of 4 and 13 simultaneous trapped micro-

<figure><img src="image_43.png" /><figcaption>Figure 3.11: Experimental validation of 3D translation control of Optical Robots.</figcaption></figure>
beads, respectively. Evaluation of the stability and efficiency in dynamic tasks shows the suitability of the system for real-time control.

The proposed system takes advantage of a combination of a mirror galvanometer and a deformable mirror with high reflectance, hence is aberration-free. The implementation of 3D trajectory control is straightforward and its low computational cost and latency makes it suitable to be exploited for closed-loop or interactive control schemes. This approach can be advantageously used in different applications where the orientation of microscopic objects (biological or synthetic) is needed, such as cell surgery applications, 3D tomographic imaging of living samples, optically driven micromachines or micro-assambling in microfluidic devices.

Thanks to the capabilities of the method, and because of it stable and simple set-up design, 6-DoF control of an optical robot have been demonstrated. This is a significant illustration of new robotic applications that can benefit from the pro-

<figure><img src="image_44.png" /></figure>

Figur
posed 3D optical manipulation. To the best of our knowledge, this is the first report about optical microrobots, which are controlled in true real-time 3D space (6DoF).

Possible applications of actuated 6-DoF optical robots include drug delivery tasks, pushing, 3D orientation of biological samples, 3D sensing, localized mechanical stimulation of single cells and any application requiring dexterous cell handling. More advanced functionalization (chemical, magnetic) of optical robots will take even further their capacity for micro-manipulation. Compared to other micro-robotic actuation approaches as electromagnetic or acoustic techniques, optical trapping has the advantage to generate very localized forces. This greater spatial resolution has the potential to allow direct collaborative tasks by several micro-robots, and additional degrees of freedom in individual robot.

In addition to provide the axial displacement, the *DM* can be also used for shaping the beam structure. Compared to a *SLM*, it has excellent wavelength response and speed but lower spatial resolution. Combined the wave shaping capabilities of the *DM* and the lateral deflection of the galvanometer, it will be possible to create more sophisticate motions and functions of the trapped objects. This technique can also be used in any applications requiring a 3D high-speed control of multiple focus laser beams such as optogenetics, 2-photon microscopy and microfabrication by 2-photon polymerization.

In future work, we plan to study the indirect 3D orientation of cells through optical robots with different end-effectors. We anticipate that the 3D micro-manipulation of biological samples without direct exposing to the laser beam will contribute to many cellular applications, where the cell rotation task is a required step, as nuclear transplantation, embryo micro-injections and polar-body biopsy [154]. We also plan to extend this 6-DoF control to several simultaneous optical robots for collaborative tasks.
<figure><img src="image_45.png" /></figure>
# Tele-robotic Platform for Dexterous Single-Cell Manipulation and Optical Robots with Force-feedback

In this chapter we propose a new tele-robotic solution for dexterous cell manipulation through optical tweezers. A slave-device consists of a combination of robot-assisted stages and a high-speed multi-trap technique. It allows for the manipulation of more than 15 optical traps in a large workspace with nanometric resolution. A master-device (6+1 DoF) is employed to control the 3D position of optical traps in different arrangements for specific purposes. Efficiency studies are carried out with trajectory control tasks. Five state-of-the-art micro-manipulation experiments were performed to verify the efficiency of the proposed platform. Results illustrate the capability to perform complex tasks in efficient and intuitive ways, opening possibilities for new biomedical applications.

In the second part of this chapter, we introduce the force-controlled optical robots (Optobots), 3D printed micro-structures actioned by optical tweezers with 6-DoF including a built-in force sensor. Optobots can be used to indirectly handle the cells in applications where their viability is an important issue. They have the capacity to interact in a controlled manner with rigid and biological objects just like their biological counterparts, -microorganisms-. This ability to regulate the contact force between robots and uncertain environment makes Optobots a unique platform for probing, exploring and manipulating the micro-world.
Contents

<table><tr><td>4.1 Introduction</td><td>70</td></tr><tr><td>4.2 Tele-Robotic Platform for Optical Dexterous Single-Cell Manipulation</td><td>71</td></tr><tr><td>4.2.1 Background and Related Work</td><td>71</td></tr><tr><td>4.2.2 Tele-robotic System Design</td><td>72</td></tr><tr><td>4.2.3 Evaluation of the Teleoperation Platform</td><td>75</td></tr><tr><td>4.2.4 Experimental micro-manipulations through the platform</td><td>78</td></tr><tr><td>4.3 Optobots: Optical Robots with Force-feedback</td><td>87</td></tr><tr><td>4.3.1 Background and Related Work</td><td>88</td></tr><tr><td>4.3.2 Optobots</td><td>90</td></tr><tr><td>4.3.3 Bilateral Teleoperation of Optical-robot with Force Feedback</td><td>92</td></tr><tr><td>4.4 Conclusions</td><td>100</td></tr></table>

## 4.1 Introduction

Nowadays, single-cell manipulations tasks are usually performed using simple 3-axis cartesian robots consisting of motor-driven micromanipulators with prismatic joints and equipped with micro-pipettes or micro-grippers as end-effectors. The operator directly controls a single actuator through buttons or knobs, ignoring the overall kinematics of the robot. Basic tasks, like the rotation of a cell, are proven to be quite time-consuming and challenging due to the lack of dexterity of those micromanipulators and their control interfaces. Consequently, those devices have a steep learning curve.

As established in chapter 1, Optical trapping offers several advantages in the handling and the mechanical characterisations of small biological samples in their own culture medium. However, the lack of intuitive and effective systems makes this technology less accessible. The ambition of this chapter is to provide a complete robotic optical tweezers system based in the achievements of the two previous chapters, that gives to an operator without engineering skills direct physical access to biophysical interactions at the microscale. And, to evolve the instrument into an off-the-shelf product, with a flexible and intuitive user interface. A parallel objective in this chapter is to demonstrate that such an interactive robotic instrument is particularly relevant to use-cases in experimental biology.

The first part of this chapter will introduced a tele-robotic system, providing a straightforward human/machine interaction, and intuitive control of biological and synthetic micro objects in six degrees of freedom (three translations and three rotations). Then, the proposed platform is used to carry out relevant biological experiments. First, the reliable 3D rotation of a cell is demonstrated. Secondly, a 6-DoF teleoperated optical-robot is used to transport a cluster of cells. Thirdly,
a single-cell is dexterously manipulated through an optical-robot with a fork end-effector. Fourth, the collaborative manipulation between two optical robots are demonstrated. Finally, the micro-assembling of two mobile components to form a micro-clamp is demonstrated for the first time.

The second part of this chapter, will introduced the so-called “Optobots”, 3D swimming micro-structures actioned by OT with 6-DoF and with a built-in force sensor. First, the iterative process of Optobots design will be introduced. Then, the stiffness of Optobots will be characterized. And finally, the first bilateral tele-operation of 6-DoF optical robots with haptic feedback will be presented. These micro-tools with force feedback will constitute a unique platform for probing the micro-world.

# 4.2 Tele-Robotic Platform for Optical Dexterous Single-Cell Manipulation

## 4.2.1 Background and Related Work

Although recent few examples of automated direct cell-rotation [65] or automated cell transportation through microtool [116] have been proposed, automation at micro-scales remains a very challenging issue in most cases where working conditions are uncertain and samples unstructured. Automatising a specific task is a time-consuming operation, often beyond the skills of the end-user. As optical-robots can be directly 3D printed according to the needs of a given experiment, each structure will have its own characteristics, and the system must be able to adapt to this new tool. Also, most tasks require some human know-how as the operator can determine the optimal method or protocol depending on the use case. Proposing a fully automated control for each situation can be inefficient, as most of the protocols are not completely defined before the operation.

Similar issues in macro-manipulation robotics have been treated using teleoperated schemes, where a master device is handled by the operator to control a slave robot. This approach integrates human intelligence in the robotic control loop, allowing user's expertise and ability to adapt the manipulation protocol to environmental disturbances and the peculiarities of the task. A very well-known example is the Da Vinci System (Intuitive Surgical, Inc.) for robotic surgery. It is designed to facilitate complex minimally invasive surgery and it is controlled by a surgeon from console. Tele-robotics have also been successfully implemented in a large range of applications where the operator is unable to interact directly with the environment, like underwater vehicles or nuclear robots.

Existing commercial interfaces for optical manipulation allow the user to control the optical traps using 2D mouse position. In order to enhance the user control, some attempts to incorporate more efficient master devices have been made, such as joysticks [155], gesture recognition (e.g. 2D cameras, kinect, leap-motion) [156, 114, 157], multi-touch tablets [158, 113] and 3D robotised interfaces (or haptic
interfaces) [159, 124, 160, 161]. Multi-modal approaches have also been presented with a combination of gesture recognition, eye gaze tracking or speech recognition [115, 129]. Despite the improvement brought by those attempts in terms of ergonomy and efficiency, completing complex real-world tasks still remains a challenge. Mouses, joysticks, and tablets do not permit a three dimensions workspace. Tracking sensors suffer from low temporal and spatial resolution. 3D robotised interfaces have only been used to manipulate one particle at a time, limiting the scenarios in which the number of objects exceeds the abilities of one operator. In addition, multi-trap actuation techniques used on most of these platforms are based in Spatial Light Modulators (SLM), which suffers from high time latency resulting from the reactivity of their hardware and a high computational cost for a given trajectory. Using GPU computation and high speed SLMs, closed-loop control with a bandwidth of 20 Hz has been reported [88].

Robust 3D real-time micro-manipulation require high spatial and temporal resolution, as the end-effector interacts in a micro-world with high dynamics effects. Working-space, Degrees of Freedom (DoF), immersion, and flexibility are also essential characteristics of an efficient micro-manipulator.

Based on all these observations, in this chapter we propose a teleoperated optical-micromanipulation platform for direct and indirect single-cell dexterous manipulation. The system is based on optical actuation, allowing a non-contact manipulation of biological samples or micro-machines. The workspace is optimised by combining a 3D multi-trap time-shared method, a 3D nano-stage, and a 2D micro-stage. To solve the latency issues, the implemented 3D multi-trap technique is based only on high-bandwidth steering mirrors as presented in chapter 3. Teleoperation control is implemented with a master device, Omega.7, providing 6+1 DoF and is ensured by a hard real-time system. Traps can be grouped and controlled in a variety of ways for specific purposes enlarging the DoF of the slave device. Five state-of-the-art experiments have been performed to verify its efficiency. First, the reliable 3D rotation of a single erythrocyte, a suspended red blood cell, has been demonstrated. Second, the transport of a cluster of cells has been performed with an optical-robot. Thirdly, a single-cell has been dexterously manipulated using an optical-robot with a fork end-effector. Fourth, the collaborative manipulation between two optical robots is presented. Finally, the micro-assembling of two mobile components to form a micro-clamp is demonstrate. These results illustrate the kind of complex biomedical applications that can be effectively and intuitively accomplished with the proposed platform.

### 4.2.2 Tele-robotic System Design

The platform is composed of three main parts: the optical set-up (Laser and passive optical components forming the light path), the slave robot (robot-assisted stages and active optical components for 3D multi-trap actuation), and the Master Device (7 DoF Robot Manipulator). The system can be used to manipulate biological samples directly or indirectly (i.e. through trapped inert objects).
<figure><img src="image_46.png" /><figcaption>Figure 4.1: Optical-micromanipulation platform for dexterous single-cell handling.</figcaption></figure>

### 4.2.2.1 Optical System

The system is constructed around a custom made inverted microscope. The microscope objective (Oil immersion, Olympus UPlanFLN 40x, NA 1.3) is used to visualise the sample and to generate the optical-traps. A Near-Infrared laser (1070 nm) has been chosen as source to minimise biological damages. The beam is expanded in order to overfill (20 %) the objective entrance, thus improving the trapping efficiency [48]. The illumination (LED, 3 W) is reflected by a longpass dichroic mirror into a High-Speed CMOS camera (Basler, 659×494 px) to provide visual feedback and environmental information to the operator.

### 4.2.2.2 Slave robot: robot-assisted stages and high-speed 3D multi-trap actuation

Two different types of actuation coexist to control the motion of optical traps. The first one is composed of a 3D nano-stage (PI P-562.3CD) mounted on a 2D micro-stage (PI M-126.CGX). These nano-stage and micro-stage move the sample chamber while trapped objects remain fixed. The micro-stage allows for a large workspace (25×25 mm²), while the nano-stage gives a finer control in 200 µm³ range.

The second actuation uses high-speed laser-deflection generated by a galvanometer (GVS002, Thorlabs) and a deformable mirror (PTT111 DM, Iris AO) as presented in chapter 3. 3D motion of focal spot is obtained by the synchronisation of the orientation of the galvanometer mirror and focusing or defocusing the deformable mirror. Multiple traps are hence created by sequentially moving the focal spot between different positions.
<figure><img src="image_47.png" /><figcaption>Figure 4.2: Example of 3D teleoperation using 3 groups of traps. The operation is a succession of four different tasks demonstrating the system's capabilities. The first task (1, 2 and 3) is a flip around the y-axis of three micro-beads. The second task (4, 5 and 6) shows the 3D control of four micro-beads. The third task (7, 8 and 9) presents the 'Radial' operating mode of the gripper. Finally, the last task (Picture 10, 11 and 12) presents the travel of one micro-bead around all the others. Scale bars are 5 µm long.</figcaption></figure>

The processed sample is placed on the nano-stage. The nano-stage and the micro-stage are directly moving the sample, which creates a translation of the surroundings while optical traps maintain the manipulated object fixed to the field-of-view (passive actuation). Multi-trap actuation dynamically control the 3D position of the different traps (active actuation).

### 4.2.2.3 Master-Slave Coupling

The Master device is an Omega (Omega.7, ForceDimension), which allows 7 Degree of Freedom (DoF). On the 7 DoF, 3 DoF are for translations, 3 DoF are for rotations and a last DoF is given by a gripper under the index finger of the user. The workspace is 160×160×110 mm³ for translation, 240×140×180deg for rotations and 25 mm for grasping.
Active and passive actuation are controlled in a transparent way from the point of view of the users. The master device translation is appropriately transformed (scaling factor of ×10⁻⁴) and sent to the nano-stage. The orientation of the master device is used to compute a rotation matrix which indicates trap positions to the galvanometer and to the deformable mirror. The rotation matrix is calculated using the *Tait-Bryan angles* with z − y − x convention (yaw, pitch and roll). Finally, the gripper position is interpreted to determine new trap positions depending on the configuration. The gripper has two operating modes. The radial mode moves the desired traps toward or away from the rotation center. This type of radial motion allows the user to grasp objects using trapped beads as 'fingers'. The scissor mode rotates the wanted traps towards the Y-axis in order to give a scissor-like movement to a set of traps. It can be used to actuate a tool such a clamp.

The optical traps can be dynamically created via the control interface. Every trap position can be directly edited in the micro-world coordinates. The traps can also be organized by groups. For each group, the 3D rotations, the 3D translations and the gripper functionality can be independently enabled or disabled.

Translations and rotations have two control modes:

1. **Position Control:** This first mode mirrors the master device's position and orientation to the slave robot, with an appropriate scaling factor. This factor can be chosen according to the task's dimensions and the operator's comfort. This method is suitable to execute precise tasks.

2. **Velocity Control:** This second mode enables control of the slave robot's velocity. The motion's direction and amplitude are computed according to the vector made by the center of the master device's workspace and the position of the handle. A scaling factor can also be chosen according to the task's requirements. The velocity control mode can be enabled independently for translations and rotations, and is suitable for long displacements like sample chamber exploration or for continuous rotation of an object like micro-pump [162] or cell rotation for tomographic imaging [163].

### 4.2.3 Evaluation of the Teleoperation Platform

The number of traps is virtually unlimited; however as the stiffness decreases with the number of traps, the amount of stable traps in dynamic and static configuration is around 15 and 30 respectively. Figure 4.2 shows an example of 3D tele-operation using eight micro-beads arranged in three different groups.

#### 4.2.3.1 Trajectory Control

For translations motions, nano-stage (PI P-562.3CD) and piezo-controller (PI E-725.3CD) are used in closed-loop with a resolution of 1 nm, 20 kHz sampling rate and factory calibration. For rotation motions, the 3D multi-trap actuation is used
according to the calibration perform in section 3.3.2.2. The deformable mirror
can handle a 1kHz sampling rate for Z axis displacement below 2µm. In order
to use the full available axial range of 9µm, the control loop frequency is fixed at
200Hz. A hard real-time control framework executed in C++ on a Real-Time kernel
(Xenomai) is implemented.

Two different tasks have been performed to evaluate the teleoperated trajectory control. First, a single micro-bead have been trapped and rotated in position mode. The master device orientation is used to compute the optical trap position. The scaling coefficient is set to 5 in order to achieve complete rotations of the trapped particles. The measured position of the Trap and the master trajectory are shown in Figure 4.3. Then, 4 micro-beads have been trapped and rotated in position mode. The scaling coefficient is set to 5. The measured position of the Trap T1 and the master trajectory are shown in Figure 4.4. The mean error between command and real position is 0.31 µm with standard deviation of 0.23 µm mainly due to Brownian motion.

<figure><img src="image_48.png" /><figcaption>Figure 4.3: Example of teleoperated rotation in position mode of a single trapped micro-bead; (a) The master device orientation is used to compute the optical trap position. The actual angle is estimated from the measured trap position using CMOS camera; (b) Trajectory of the micro-bead. During the experience, the micro-bead rotate around the marked center; (c) Position command sent to the mirrors and measured position computed from the video.</figcaption></figure>
<figure><img src="image_49.png" /><figcaption>Figure 4.4: Example of teleoperated rotation in position mode of a group of four micro-beads. (a) The master device orientation and the command sent to the actuators. Z angle is the angle of rotation around the z-axis. Scaling factor is set to 5 in order to allow a complete rotation. The measured angle is estimated from the measured position of Trap T1. (b) The 2D trajectory followed by the Trap T1. (c) The measured position of Trap T1 computed from the video and the set-point. Estimated error is 0.31 µm with a standard deviation of 0.23 µm.</figcaption></figure>

### 4.2.3.2 Velocity limitation

In order to assist the user and help the trapped objects' retention, a maximal handling velocity is defined according to the number of traps.

As expected, the escape velocity to release the trapped bead is higher for one trap than for four traps. The laser is deflected at a constant frequency of 200 Hz from one trap to another, meaning that the position is moving by steps, and greater velocities imply larger steps. The velocity thresholds estimated in chapter 3 section 3.3.2.2 are 462 µm/s for 1 trap and 105 µm/s for 4 traps. For both experiments, the velocity threshold corresponds to a position step of approximately 2.3 µm. Same results are found with 2, 3, 5 and 9 traps. Hence, this step-size defines the limit before risking to free a trapped object.

The following equation predicts the theoretical highest reachable velocity depending on the number of optical traps generated and the deflection frequency of the laser:

$$V_{max} = \frac{D_{max} * f}{N_{trap}} \qquad (4.1)$$
Where $V_{max}$ is the velocity threshold, $D_{max}$ the position step-size, $N_{trap}$ the number of optical traps, and *f* the deflecting frequency of the laser. As a consequence, if the deflecting frequency of the laser is set to a higher value, the velocity threshold increases.

In regard to the translation actuation, since the whole sample-chamber is moved, if only one trap is generated the velocity threshold is only limited by the drag force, as the trap is always active. The escape velocity is measured at 1500 µm/s, resulting in an escape force of 38 pN. When several traps are generated, the velocity threshold is limited by the deflecting frequency and the viscous drag forces. During the time period when the trap is not active, viscous forces will shift the object out of the trap. If the motion is small enough, the bead will be attracted to equilibrium position when the trap is activated back. However, when the displacement of the bead is large enough during a period (i.e. if the translation velocity is too high) the bead is not in the attractive zone any more when the trap is active again. For 2, 3, 4 and 5 traps, the velocity threshold measured is respectively 300 µm/s, 220 µm/s, 110 µm/s and 80 µm/s. Note that escape forces are defined by optical properties at the very edges of the trap, where the restoring force is no longer a linear function of the displacement [164]. Further investigations on the effective stiffness of traps combining time-sharing actuation and stage-based actuation is required for a higher precision on force measurements.

<table><caption>Table 4.1: Summary of the tele-robotic platform</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td>Teleoperation loop</td><td>200 Hz</td></tr><tr><td>Simultaneous optical traps</td><td>&gt; 15</td></tr><tr><td>Slave Translation range</td><td>200×200×200 µm³</td></tr><tr><td>Slave Rotation range</td><td>70×50×9 µm³</td></tr><tr><td>Trans Max velocity (1T,3T,4T)*</td><td>1500,220, 110 µm/s**</td></tr><tr><td>Rot Max velocity (1T,3T,4T)*</td><td>462,153, 105 µm/s**</td></tr><tr><td>*Depends on the trapped object</td><td>**For 3 µm polystyrene beads</td></tr></tbody></table>

### 4.2.4 Experimental micro-manipulations through the platform

To illustrate the kind of biological applications that can be accomplished with the proposed platform in a real world scenario, direct and indirect manipulations of mouse erythrocytes, suspended Red-Blood Cells (RBCs), are presented next. Then, collaborative task between robots and micro-assembling is demonstrated.
### 4.2.4.1 Direct manipulation: 3D rotation of a cell

Presently, 3D orientation of biological samples have gained much attention due to their involvement in various single cell surgeries and cell imaging techniques. Accordingly, significant efforts have been made toward achieving 3D cell orientation control using holographic optical tweezers (HOT) [136, 4, 141, 65]. Although these SLM-based techniques represent elegant solutions for the 3D rotations, the intrinsically slow response of liquid crystal and the complexity of trajectory computation induce important delays making their implementation in real-time scenarios a difficult task even when only reduced to two traps. In time sharing techniques, one of the main advantages is that the control of traps is straightforward, requiring only the 3D transformation that produces the desired movement and its direct conversion into actuator coordinates.

<figure><img src="image_50.png" /><figcaption>Figure 4.5: 3D rotation control of erythrocyte driven by 8 optical traps. (a) Schematic 3D depiction of erythrocyte and the reference frame used in the 3D control. (b) Schematic 2D depiction of the erythrocyte with dimensions in y-x and z-x plane. (c) Time-lapse images demonstrating multiple degrees-of-freedom cell rotational control. Scale bar is 2 µm long.</figcaption></figure>
The first experiment is dedicated to the 3D orientation of a suspended erythrocyte. A group of 8 traps arranged along the perimeter of the cell have been created through the user interface. Initially, the erythrocyte was sedimented on a cover slip with face-on orientation to the optical axis. The eight traps enable the rotation of the cell with respect to the x, y, and z axes, and the simultaneous translation in all three axes. Thanks to the high bandwidth and the efficiency of the proposed system, the user can control the 3D motion of the cell without any noticeable latency.

Figure 4.5 show the 6-DoF control of an individual cell. Every DoF can be controlled independently or coupled to another. Rotation's scaling factor is set to 3 in order to achieve full 360° rotation. The maximum velocity before observing minor overshooting is 188.6 deg/s or 9.92 µm/s for a cell with 6 µm diameter. The difference between the theoretical maximum velocity for eight micro-beads (57.5 µm/s) and the measured velocity for RBCs may be due to the lower cell's refractive index (RI~1.38), the higher viscous drag torque due to larger surfaces, and the shape differences. Factors as symmetry, size and non-homogeneity of manipulated objects will impact the performance of the direct manipulation.

Rotation can also be controlled in speed mode, in order to allow constant displacement of a cell. True 360° rotation around all-axes of a single RBC is successfully demonstrated. The achievable orientation range depends on the maximum distance separating the traps and the center of rotation, and will be limited by the workspace for bigger cells.

### 4.2.4.2 Indirect manipulation: 6-DoF Teleoperated Optical Robot for cell manipulation

Direct optical manipulation is the simplest and most used manipulation technique; however several studies have shown that direct laser exposure can cause considerable photo-damage [165, 166]. Furthermore, it is difficult to reliably hold different kinds of biological samples as the stability of the traps depends on the shape, material, and the refractive index of the target. Therefore, indirect manipulation through beads formations [167] or more complex micro-tools [168] have been proposed.

This section demonstrates the capability to use the proposed platform for the indirect manipulation of cells through optical robots. This type of manipulation is a good illustration of the capabilities and versatility of the platform as indirect manipulation through beads is more complex and time consuming than direct manipulation [169].

Two optical robots with three and four spherical handles have been designed. These micro structures are manufactured following dimensional specifications shown in Figures 4.6.(b) and 4.7.(b), by two-photon polymerisation (Nanoscribe) using IP-Dip resin (refractive index ~1.52). The first robot has a shovel-shaped end-effector, in order to tow and move several cells at the same time. The second robot has a fork end-effector in order to dexterously manipulate a single-cell. Different spacers are attached on both sides of the robot to minimise the adhesion forces. (See Figures 4.6.(a) and 4.7.(a)).
<figure><img src="image_51.png" /><figcaption>Figure 4.6: Robot with a shovel-shaped end-effector for cell transport. (a) Scanning electron microscopy (SEM) image of the robot. (b) Schematic depictions with dimensions of the robot. (c) 2D trajectory of the teleoperated robot with an inset of the 3D trajectory. Note that frames between T: 20 s and T: 30 s show the erythrocytes out-of-focus as the robot is elevated in the Z-direction. Then, a cluster of cells (colored in violet) is transported for 80 µm.</figcaption></figure>

After fabrication, the robots are incubated in a 94.5% distilled water, 5% ethanol and 0.5% *Tween20* solution to prevent the surface adhesion. For experiments, micro-robots are transferred to a sample chamber containing suspended erythrocytes through an actuated microliter syringe (please see Annex A for further information). Then, the sample chamber is sealed with a cover-slip, forming a confined environment.

### 4.2.4.2.1 Teleoperated Optical Robot for cells transportation

The first experiment consists in the indirect transportation of a cluster of cells. A group of three traps in triangle formation is generated using the user interface. Then, teleoperation is performed through the master device.
Thanks to the low response-time and transparency of the system, the operator manages to open a path through a sample, heavily loaded with suspended RBCs. Figure 4.6.(c) shows the trajectory of the teleoperated Robot.

Initially the robot moves in the microscope-slide bottom (T: 0-19 s). Then, the robot is lifted in the Z-direction (10 µm) as the obstacles completely block the path (T: 20 s). Once the target cells are identified, the axial position of the robot is lowered until it hits the bottom of the slide again (T: 30 s). Then, the robot moves the target cells indirectly (T: 35-45 s). The user dexterously moves the robot through the sample for more than 100 µm to finally displace the cluster of cells for 80 µm.

4.2.4.2.2 Teleoperated Optical Robot for dexterous single-cell manipulation

This second experiment is dedicated to dexterously manipulate a single-cell. A robot
with fork end-effector is designed according to a RBC shape (~6 µm in diameter)
in order to allow full mobility of the manipulated cell. Clamp edges touching the

<figure><img src="image_52.png" /><figcaption>Figure 4.7: Robot equipped with a fork end-effector for single-cell manipulation. (a) Scanning electron microscopy (SEM) image of the robot. (b) Schematic depictions with dimensions of optical robot. (c) Two sets of experiments manipulation of a single Red Blood Cell and its 2D trajectory.</figcaption></figure>
cell are jagged, to limit the contact surface. A group of four traps is used to induce 3D motions.

In the experiments, the cell is seized and manipulated without major difficulties. Experimental results are shown in Figure 4.7.(c). In the first task, the optical robot turns around the target, a single RBC located in point A, during an exploration phase (T: 0-20 s). Then, seize the cell and transports it to point B, about 100 µm away between two obstacles (T: 20-39 s). Finally, it deposits the cell to point B moving backwards (T: 39-42 s). The second task consists in manipulating a single cell avoiding a big obstacle.

During all manipulations, the rotations of the robot have been controlled in position mode. On the other hand, translations have been executed with a mixed control in position and velocity according to the situation. In displacements and cells transportation, the speed control has been preferred, while in contact, loading, and delivery of cells, the control in position has been chosen. The maximal velocity for a four handles optical robot is measured as 60 µm/s for rotation and 100 µm/s for translations.

Indirect 2D rotation of the cell can be performed around any center point. To achieve complete indirect 3D rotation of a cell, it is necessary to firmly grasp the cell and move it along Z-direction without losing it and could be accomplished, e.g. by using two separated robots to handle the cell.

### 4.2.4.3 Simultaneous control of several microstructures

The greater spatial resolution of optical trapping allows straightforward implementation of collaborative tasks by several optical robots, and additional degrees of freedom in individual robot, contrary to other remote techniques where the force generating field is applied to the entire working space (e.g. magnetic, electric). We present here, the simultaneous control of several optical robots for collaborative tasks, and the micro-assembling of two mobile components forming a micro-clamp.

#### 4.2.4.3.1 Collaborative tasks in Optical Robot

Thanks to the capabilities of the platform and the groups arrangements feature, 6-DoF control of several simultaneous optical robots for collaborative tasks can be directly implemented.

For this experiment, two optical robots have been simultaneously controlled by creating two different groups with four traps each one (cf. Figure 4.8). In the first task, we move the two robots in a kind of synchronized dance, by directly sending the rotations and translations of the master device to each group. In the second task, the blue robot is used to turn around the red robot and pushing it at different locations of its body. This is performed by locking translations and rotations of the red robot and can be useful to clamp and manipulate an object between the two robots. Finally, by combining the two previous tasks, the two robots are used to push an third object with a strange shape, that could hardly be done with a single robot. Then collaborative task is successfully demonstrated.
<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>
Using biomolecules with high affinities applied as particle coatings, durable linkage between individual particles is realized, then the manufacture of stable planar and 3D microstructures [172]. Employing thermal [173] or photopolymerization [174, 175] effects is also possible to build stable joints. However, in these assembly techniques once the microstructures are joined together, it is not possible to release them without destroying the components.

An optical screw-wrench for microassembly has been recently proposed [76]. Using the screw-wrench, micro-rotor assembly and actuation has been demonstrated as a possible application, and stable and releasable joint form has been demonstrated. In this work, the screw is printed in such a way as to be attached to the cover-slip, and only the nut are freely moveable.

In this section, we presented the micro-assembly of two completely moveable components using the developed platform. Careful design and fabrication enabling the manufacturing of two micro-components that are complementary. The design of this complementaries "male" and "female" components have two main purposes: the validation of a stable and releasable connection in completely untethered structures, and the validations of mobile joint formation for additional degrees of freedom in optical robots.

The sequence of images in Fig 4.10.(a-f) shows the assembling process. This micro-assembly task also introduced the idea of handles specialisation in optical robots, i.e. putting several trapped handles for different functions in the structure. Traps T1, T2 and T4 are used for the out-of-plane rotation and the 3D positioning of the male component (Fig 4.10.a). Traps T5, T6 and T8 are used for the out-of-plane rotation of the female component (Fig 4.10.b). Traps T5, T7 and T8 are used for the 3D positioning of the female component during the steps c, d and e in order to keep the optical traps away from the male component preventing any accidental trapping (Fig 4.10.(c-e)). Once assembled, only T1, T3, T5 and T7 are used to induce the 3D movement of the clamp (Fig 4.10.f) and perform manipulation tasks.

<figure><img src="image_56.png" /><figcaption>Figure 4.9: Micro-clamp components dimensions. The gap $\lambda$ varies from 100 nm to 600 nm by steps of 100 nm. With gap dimension above 300 nm the structure can be assembled.</figcaption></figure>
Figure 4.10.(g-h) shows the manipulation of the two objects with different shapes. First, the manipulation of a micro-sphere of 6 µm and second, the manipulation of a residue found in the bottom of the sample.

Additionally a study of the assembling success rate with respect to the gap between the objects to be assembled has been carried out from 100 nm to 600 nm, which shows that from a gap above 300 nm the structure can be assembled each time.

Although these examples demonstrate the ability to indirectly manipulate different kind of objects, manipulation continues to be challenging. Mainly because in order to grip an object, it is necessary to place the end-effector in the same plane as the object, especially in the case of the sphere. Also, when external forces are applied to the structure, for example during a grasping, the system can be disassembled itself if not enough attention is paid. For this reason, another assembling design reducing the risk of accidental disassembly is proposed. In this new design, two movements are needed to assembled or disassembled will be soon tested (cf. Fig. 4.11).

<figure><img src="image_57.png" /><figcaption>Figure 4.10: Assembling of a micro-clamp. (a) Male component rotating. (b) Female component rotating. (c) Positioning. (d) Lowering. (e) Locking. (f) Manipulating. (g) Handling of a micro-sphere of 6 µm (h) Handling of a residue found in the sample. (i) Assembling handles: T1, T3, T4, T5, T7, T8. Manipulation handles: T1, T3, T5, T7.</figcaption></figure>
<figure><img src="image_58.png" /><figcaption>Figure 4.11: Second micro-assembly technique. (a) CAO model of the components (b) SEM image of male and female components.</figcaption></figure>

These techniques may be valuable to assembling complex microsystems in a bottom-up approach. Besides to additional degrees of freedom and micro-motors or other micro-machines for micro-fluidics, this techniques allow the combination of structures with diverse materials and properties. The utilisation of building blocks with local properties will allow the micro-fabrications of complex hybrid micro-systems. This could enable more complex microrobots allowing for either more functionalities on a single robot or the fabrication of more specialised robots (e.g. ph sensing, heat sensing). In the next section, a specialisation of optical robot for force measurement are presented.

4.3 Optobots: Optical Robots with Force-feedback

Biological micro-organisms have evolved to operate and survive in a world marked by rapid changes, high uncertainty, and limited availability of information ([176]. Cells adapt their growth, motility, and physiological process according to external stimuli in several ways. Even memory, learning and anticipation has been suggested in bacteria [177] and other non-neural unicellular organisms as the amoeba Physarum polycephalum [178, 179, 180]. This wide range of adaptation behaviours in micro-organisms implies the ability to sense, transfer and process external information. In contrast, their synthetic counterpart, micro-robots, have not still yet sensing capacities, then they do not can react and adapt to changes in the environmental conditions.

Micro-robots promise revolutionary innovations in biomedicine and bioengineering applications [1, 181] including intracellular surgery, drug delivery and cell manipulation. However, exploring their full potential will require controlled interaction and locomotion within challenging environments, ranging from lab-on-chip devices to a living organism. Different challenges appear with the scale reduction in particular by the physics involved, where the surface forces (Van der Waals, capillarity, fluid drag, among others) dominating over the inertial forces. For locomotion, this means that the dynamics of the robot will be dominated by the viscous forces (low Reynolds number), and object moving very close to a boundary will experience larger viscous force compared to the one experienced far away [182]. Also the en-
vironment will be influenced by thermal and intrinsic biological noise (Brownian motion). Concerning the interaction with the environment, the adhesion forces between the robot and other surfaces, make a complex issue the prehension and manipulation tasks. Furthermore, their integration to highly confined and con- strained environments that can be perceived only through a microscope and where traditional actuators and sensor are not usable is an additional difficulty.

Compared to actuation systems, mobile micro-robots sensing systems is almost unexplored area, and most of the existing micro-robots have not any such capa- bilities. Imitate the complex behaviours and interactions between micro-biological organism and their environments requires the availability of sensory information in the feedback path of synthetic micro-swimmers. The lack of real-time sensors that can be included in their control scheme hurts their adaptability and make the systems not robust during dynamic interactions.

The main contribution of this section involves the design and the implementation of a force sensitive optical micro-robot that regulate the contact force between him and uncertain environment, which in our best knowledge has never been considered in existing literature. We demonstrate locomotion, and a capacity to interact in a controlled manner with rigid and biological objects.

4.3.1 Background and Related Work

4.3.1.1 Force sensing in the feedback path

In nature, the ability to sense mechanical stress or forces is crucial for simplest
micro-organisms – cells– to more complex organisms like – humans or animals –.
Until the most “primitive” organisms have sensory capacities, often mechanorecep-
tors that allows them to react and to interact with a physical world.

It is now well recognized that mechanical forces acting on a living cell play
a fundamental role in the regulation of cell functions from differentiation to con-
trolled cell death. Paramecium, a unicellular swimming organism widely used in
laboratories, respond actively when touched. If the mechanical stimulus is applied
to the front, it changes the beating direction of all its cilia simultaneously leading
to a quick change of direction in an “avoidance task”. If the stimulus is applied
in back part, it increases their cilia beat frequency, which generates an increase in
velocity of forward locomotion in an “escape task” [183, 184]. Certain plant cells
demonstrate some kind of “avoid-obstacles skill” reacting to a barrier by chang-
ing their growth direction [185] and recent advances have led to the proposal of a
plant-specific mechanosensory network within plant cells [186]. Bacteria cells have
different strategies for responding to fluid flow and to initiate or maintain contact
with solid surfaces. For example, E. coli bacteria adapt the flagellar motor speed
according to drag force, as they are close or not to a boundary [187] and many bacte-
rial species possess a mechanism for disabling flagellar rotation or adapt behaviour
in response to mechanical forces in the transition from a planktonic swimming to
surface attachment state [188].
Very recent work has shown that these rapid changes in behaviour are possible because bacteria have a sense of touch. This tactile sensing makes use flagella motor [189] or bacterial pili [190] as mechano-sensitive device. Sense of touch (or haptic perception) is also essential for the survival and the development of animals and humans. They have specialised mechano-receptors (eg, tactile corpuscles) that guarantee their responses to mechanical stimulus fast enough to ensure the survival. These responses are useful for locomotion, exploration, and for all manipulation and recognition tasks, as grasping, shape and texture recognition.

Design robots that mimic essential functions of biological organisms, such as adaptivity, robustness, and agility require the integration of mechanical information in the feedback path. Robotic research at macro-scale has for several years, integrated force feedback in the control-loop for task requiring robust interaction between the manipulator and its environment. This is because pure motion control can be insufficient for these tasks, especially in the presence of rigid, delicate or in-motion objects. Then, force control allows the simultaneous protection of the manipulator and its environment.

Force information can be used in fully automated control for simple and completely pre-determined tasks as peg in the hole operation, polishing, deburring or painting. For more complex applications where human skills are required, haptic-feedback teleoperation is a promising solution.

### 4.3.1.2 Force sensing in mobile micro-robots

In mobile microrobotics research, many works in the motion control techniques have been proposed, however published research concentrates mostly on the precise positioning of the micro-swimmers [2, 191, 17] and the contact force on the microrobot was not considered. This could result in task failure or damage the structures that the microrobot works with or its own structure, in real-life case scenarios.

Micro-robotics tasks where the control of the contact force will be useful include pushing, twisting, rotating, injecting, applying pressure at a defined position of a cell and all different application where delicate action is needed as biomedical microsurgery. Furthermore, micro-robots are commonly looked through a microscope where the image is a 2D projection of the workspace and depth information is often insufficient or distorted. Thereby, force information can be useful also to identify and better handle unknown biological objects.

Several types of micro-force sensors have been developed based on different physical principles such as capacitive, piezoresistive and strain gauges-based [192]. These systems are often micromanipulators, i.e. it performs tasks in the microworlds, but it is not itself micron-sized. Fully on-board these systems in a mobile microrobot will require still efforts in the miniaturisation and address the challenges of embedded electronics, communication, power supply and compatibility with liquid environments for biomedical applications. Vision-based force measurements it a good option to overcome these drawbacks, they allow the measurement with the absence of contact and the possibility of measurement along several axis. These