# Contributions to utilize a Cobot as intermittent contact haptic interfaces in virtual reality

Vamsi Krishna Guda

<figure><img src="image_1.png" /></figure>

## To cite this version:

Vamsi Krishna Guda. Contributions to utilize a Cobot as intermittent contact haptic interfaces in virtual reality. Robotics [cs.RO]. École Centrale Nantes, 2022. English. NNT: tel-03838173

HAL Id: tel-03838173

https://hal.science/tel-03838173

Submitted on 3 Nov 2022

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# THÈSE DE DOCTORAT DE

## L'ÉCOLE CENTRALE DE NANTES

ÉCOLE DOCTORALE N° 602

Sciences pour l'Ingénieur

Spécialité : Robotique-Mécanique

Par

Vamsi Krishna GUDA

Contributions à l'utilisation de cobots comme interfaces haptiques à contact intermittent en réalité virtuelle.

Thèse présentée et soutenue à l'École Centrale de Nantes, le 16 juin 2022

Unité de recherche : UMR 6004, Laboratoire des Sciences du Numérique de Nantes (LS2N)

### Rapporteurs avant soutenance :

Gérard POISSONI Professeur, Université de Orléans, Orléans

Charles PONTONNIER Maître de Conférences, HDR, Ecole Normale Supérieure de Rennes, Rennes

### Composition du Jury :

Président : Anatole LECUYER

Directeur de Recherche INRIA, Inria Rennes

Examinateur : Fouad BENNIS

Professeur, École Central de Nantes

Gérard POISSONI

Professeur, Université de Orléans, Orléans

Charles PONTONNIER

Maître de Conférences, HDR, Ecole Normale Supérieure de Rennes, Rennes

Dir. de thèse : Damien CHABLAT

Directeur de Recherche CNRS, LS2N, Nantes

Co-dir. de thèse : Christine CHEVALLEREAU

Directrice de Recherche CNRS, LS2N, Nantes
Dedicated to my parents, Venugopal Rao GUDA and Subbaravamma KOKA.
Acknowledgments

I would like to thank my thesis directors, Christine Chevallereau and Damien Chablat for offering me this opportunity, giving me the freedom to work on my own, and for being there whenever I needed help in spite of their super-tight schedule. Thanks to their collaborations across the world, I have had the honor to meet and interact with many clever people and learn from them. My co-supervisor, Christine Chevallereau for wonderful insights and ideas, especially for having patience during our long meetings. They have been a great mentor throughout this journey.

Special thanks to Lionel Dominjon (Scientific Manager, CLARTE), for helping me in establishing the virtual environment setup for the experiments. The mechanic of our lab, Stéphane Jolivet for his ultimate patience to print and design the required mechanisms. All the administrative staff on the third floor of our lab, staff of École doctorale and scolarité who are incredibly nice and friendly, and make it a cakewalk to deal with the otherwise annoying administrative stuff.

My parents have always encouraged me to pursue what I like, despite many hardships they had to face. I am eternally grateful to them for their love, education and support for I would not be here without them.

PhD life would not have been so much fun without my friends. Thanks to my friends Jahnavi Marthala, Valentin Le Mesle and Saketh Panuganti for sharing my happy moments and being there through tough times along the journey. I am also grateful to my friends and colleagues Guillaume, Julian, Daravuth, Saman, Keerthi, Kashyap, Sashi, Manohar, Franco, Quynh, Bin Bin, Lauren, Florence, Vivi and Margarita, my office-mates Samuel and Zhen, for interesting discussions and beautiful memories that I cherish.

Thanks to everyone and to all the personnel who kindly volunteered their valuable time to complete all the proposed user tests.

The research work for this doctoral thesis was conducted at Laboratoire des Sciences du Numérique de Nantes (LS2N, Ecole Central de Nantes), and in collaboration with Institut national de recherche en informatique et en automatique (INRIA, Rennes). The funding for this project was made possible by the Agence Nationale de la Recherche (ANR).
# Contents

<table><tr><td>Acknowledgements</td><td>v</td></tr><tr><td>List of Figures</td><td>xi</td></tr><tr><td>List of Tables</td><td>xv</td></tr><tr><td>List of Algorithms</td><td>xvii</td></tr><tr><td>Glossary</td><td>xix</td></tr><tr><td>Nomenclature</td><td>xxi</td></tr></table>

## Introduction

<table><tr><td>Description of context</td><td>2</td></tr><tr><td>Applications of ICIs</td><td>3</td></tr><tr><td>Challenges</td><td>5</td></tr></table>

## Chapter 1

### State of the art and theoretical background

<table><tr><td>1.1 Introduction</td><td>9</td></tr><tr><td>1.2 What is virtual reality?</td><td>10</td></tr><tr><td>1.3 Human senses</td><td>14</td></tr><tr><td>1.3.1 Sight</td><td>14</td></tr><tr><td>1.3.2 Hearing</td><td>15</td></tr><tr><td>1.3.3 Smell</td><td>15</td></tr><tr><td>1.3.4 Taste</td><td>16</td></tr><tr><td>1.3.5 Touch</td><td>17</td></tr><tr><td>1.4 Haptic technologies for VR</td><td>17</td></tr><tr><td>1.4.1 Human haptic perception</td><td>17</td></tr><tr><td>1.4.2 Tactile feedback interfaces</td><td>18</td></tr><tr><td>1.4.3 Force feedback interfaces</td><td>19</td></tr><tr><td>1.4.4 Haptic bodysuits</td><td>20</td></tr><tr><td>1.4.5 Analysis of haptic technologies in VR</td><td>21</td></tr><tr><td>1.5 Existing haptic interfaces</td><td>22</td></tr><tr><td>1.5.1 Haptic senses</td><td>22</td></tr><tr><td>1.5.2 Haptic feedback and sense of touch</td><td>23</td></tr><tr><td>1.5.3 Examples of force feedback devices</td><td>24</td></tr><tr><td>1.5.4 Commercially available interfaces</td><td>25</td></tr><tr><td>1.5.5 Limitations of existing haptic interfaces</td><td>28</td></tr><tr><td>1.6 Intermittent-contact interface</td><td>30</td></tr><tr><td>1.6.1 Encountered-type haptic interfaces</td><td>30</td></tr><tr><td>1.6.2 Close-tracking-type haptic interfaces</td><td>33</td></tr></table>
<table><tr><td>1.7</td><td>Equipment</td><td>36</td></tr><tr><td>1.8</td><td>Research proposal</td><td>36</td></tr></table>

## Chapter 2

### UR-5 configurations, workspace and placement

<table><tr><td>2.1</td><td>Introduction</td><td>39</td></tr><tr><td>2.1.1</td><td>Safety standards</td><td>43</td></tr><tr><td>2.1.2</td><td>Safety in human robot collaboration</td><td>44</td></tr><tr><td>2.1.3</td><td>Methodology to achieve the safety standards in HRC</td><td>46</td></tr><tr><td>2.2</td><td>Robot configurations</td><td>46</td></tr><tr><td>2.2.1</td><td>Inverse kinematics</td><td>47</td></tr><tr><td>2.3</td><td>Generate workspace</td><td>48</td></tr><tr><td>2.4</td><td>Regions of interest</td><td>53</td></tr><tr><td>2.4.1</td><td>Workspace for given orientation</td><td>55</td></tr><tr><td>2.5</td><td>Base placement of the robot</td><td>55</td></tr><tr><td>2.5.1</td><td>Requirement</td><td>55</td></tr><tr><td>2.5.2</td><td>Methodology</td><td>56</td></tr><tr><td>2.5.3</td><td>Results</td><td>57</td></tr><tr><td>2.6</td><td>Obstacle avoidance and dimensions of base table</td><td>60</td></tr><tr><td>2.7</td><td>Conclusions</td><td>61</td></tr></table>

## Chapter 3

### Design of a user mannequin model for ROS

<table><tr><td>3.1</td><td>Introduction</td><td>65</td></tr><tr><td>3.1.1</td><td>Using kinect to know the environment</td><td>65</td></tr><tr><td>3.2</td><td>Selection of human model</td><td>66</td></tr><tr><td>3.3</td><td>Description of the motion capture system</td><td>68</td></tr><tr><td>3.4</td><td>Construction of human model</td><td>68</td></tr><tr><td>3.4.1</td><td>Calculation of the configuration of the arm</td><td>71</td></tr><tr><td>3.4.2</td><td>Calculation of movement in torso</td><td>74</td></tr><tr><td>3.5</td><td>Analysis of user model</td><td>75</td></tr><tr><td>3.5.1</td><td>Accuracy of arms</td><td>75</td></tr><tr><td>3.5.2</td><td>Accuracy of torso</td><td>76</td></tr><tr><td>3.5.3</td><td>Scalability of the model</td><td>76</td></tr><tr><td>3.6</td><td>Conclusions</td><td>77</td></tr></table>

## Chapter 4

### System architecture and robot trajectory planning

<table><tr><td>4.1</td><td>Introduction</td><td>79</td></tr><tr><td>4.2</td><td>Proposed architecture and data flow</td><td>79</td></tr><tr><td>4.2.1</td><td>MoveIt</td><td>81</td></tr><tr><td>4.3</td><td>Implementing laboratory setup based on proposed architecture</td><td>81</td></tr><tr><td>4.3.1</td><td>Requirement</td><td>81</td></tr><tr><td>4.3.2</td><td>Methodology for calibration of real and virtual environments</td><td>82</td></tr><tr><td>4.4</td><td>Computation and trajectory planning</td><td>86</td></tr><tr><td>4.4.1</td><td>Various path planning algorithms</td><td>86</td></tr></table>
<table><tr><td>4.4.2</td><td>Comparison of various path planning algorithms</td><td>93</td></tr><tr><td>4.4.3</td><td>Selection of planning algorithm</td><td>95</td></tr><tr><td>4.4.4</td><td>Description of Unity's virtual environment</td><td>100</td></tr><tr><td>4.4.5</td><td>Different mobility schemes</td><td>101</td></tr><tr><td>4.5</td><td>Conclusion</td><td>105</td></tr></table>

## Chapter 5

### Prediction of user intention

<table><tr><td>5.1</td><td>Introduction</td><td>107</td></tr><tr><td>5.2</td><td>Human intention prediction</td><td>108</td></tr><tr><td>5.2.1</td><td>Detection of target</td><td>108</td></tr><tr><td>5.2.2</td><td>Proposed model</td><td>109</td></tr><tr><td>5.2.3</td><td>Scene information</td><td>109</td></tr><tr><td>5.3</td><td>Safe and fast motion</td><td>110</td></tr><tr><td>5.3.1</td><td>Cobot motion</td><td>110</td></tr><tr><td>5.3.2</td><td>Velocity zones</td><td>111</td></tr><tr><td>5.3.3</td><td>Velocity profiles</td><td>113</td></tr><tr><td>5.3.4</td><td>Safe-points</td><td>114</td></tr><tr><td>5.3.5</td><td>Comparison of motion with or without safe-points</td><td>115</td></tr><tr><td>5.4</td><td>Proposed strategies</td><td>115</td></tr><tr><td>5.4.1</td><td>Strategy A: Hand position</td><td>116</td></tr><tr><td>5.4.2</td><td>Strategy B: Hand position and gaze direction</td><td>120</td></tr><tr><td>5.4.3</td><td>Strategy C: Addition of safe points</td><td>121</td></tr><tr><td>5.4.4</td><td>Strategy D: Head gaze and safe points</td><td>123</td></tr><tr><td>5.5</td><td>Experiments and analysis</td><td>123</td></tr><tr><td>5.5.1</td><td>Criterion</td><td>123</td></tr><tr><td>5.5.2</td><td>Experimental setup</td><td>125</td></tr><tr><td>5.5.3</td><td>Analysis of one experiment</td><td>126</td></tr><tr><td>5.5.4</td><td>Analysis of all recorded experiments</td><td>129</td></tr><tr><td>5.5.5</td><td>Discussion</td><td>133</td></tr><tr><td>5.5.6</td><td>Analysis on hand threshold</td><td>134</td></tr><tr><td>5.6</td><td>Conclusions</td><td>137</td></tr></table>

### Conclusions and Future Work

<table><tr><td>Synopsis</td><td>139</td></tr><tr><td>Contributions</td><td>140</td></tr><tr><td>Future work</td><td>141</td></tr></table>

### Personal publications

## Chapter A

### UR-5 kinematics

## Chapter B

### Error analysis for user model
x

Contents

Chapter C

What is Movelt ?

References
List of Figures

<table>
  <tbody>
    <tr>
      <td>1</td>
      <td>Conceptual scheme of the experimental platform</td>
      <td>2</td>
    </tr>
    <tr>
      <td>2</td>
      <td>The LobbyBot project setup.</td>
      <td>3</td>
    </tr>
    <tr>
      <td>3</td>
      <td>Application of ICI in industry</td>
      <td>4</td>
    </tr>
    <tr>
      <td>4</td>
      <td>Application of ICI in entertainment industry</td>
      <td>4</td>
    </tr>
    <tr>
      <td>5</td>
      <td>Application of ICI in medical field</td>
      <td>4</td>
    </tr>
    <tr>
      <td>6</td>
      <td>Application of ICI in research</td>
      <td>4</td>
    </tr>
    <tr>
      <td>1.1</td>
      <td>First Head Mounted Display (HMD)</td>
      <td>10</td>
    </tr>
    <tr>
      <td>1.2</td>
      <td>Classic HMD and VR glove environment</td>
      <td>11</td>
    </tr>
    <tr>
      <td>1.3</td>
      <td>First virtual reality system</td>
      <td>11</td>
    </tr>
    <tr>
      <td>1.4</td>
      <td>Haptic display system (GROPE-III)</td>
      <td>12</td>
    </tr>
    <tr>
      <td>1.5</td>
      <td>Example of Cave Automatic Virtual Environment (CAVE)</td>
      <td>12</td>
    </tr>
    <tr>
      <td>1.6</td>
      <td>Commercial HMD.</td>
      <td>13</td>
    </tr>
    <tr>
      <td>1.7</td>
      <td>Principle of avatar.</td>
      <td>13</td>
    </tr>
    <tr>
      <td>1.8</td>
      <td>Virtual reality feedback loop.</td>
      <td>13</td>
    </tr>
    <tr>
      <td>1.9</td>
      <td>Basic five human senses.</td>
      <td>14</td>
    </tr>
    <tr>
      <td>1.10</td>
      <td>Glove for tactile interface.</td>
      <td>18</td>
    </tr>
    <tr>
      <td>1.11</td>
      <td>Glove for force feedback interface.</td>
      <td>20</td>
    </tr>
    <tr>
      <td>1.12</td>
      <td>Haptic body suits.</td>
      <td>21</td>
    </tr>
    <tr>
      <td>1.13</td>
      <td>Virtual reality systems using force feedback.</td>
      <td>24</td>
    </tr>
    <tr>
      <td>1.14</td>
      <td>Applications of force feedback interfaces.</td>
      <td>26</td>
    </tr>
    <tr>
      <td>1.15</td>
      <td>Commercially available force feedback interfaces.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>1.16</td>
      <td>Principle functioning of encountered-type haptic interfaces.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>1.17</td>
      <td>Virtual haptic space representation for ETHI.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>1.18</td>
      <td>Virtual haptic space representation for ETHI.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>1.19</td>
      <td>Multiple finger ETHI.</td>
      <td>32</td>
    </tr>
    <tr>
      <td>1.20</td>
      <td>Path-planning issue in ETHI.</td>
      <td>32</td>
    </tr>
    <tr>
      <td>1.21</td>
      <td>Principle of functioning close-tracking-type haptic interfaces.</td>
      <td>33</td>
    </tr>
    <tr>
      <td>1.22</td>
      <td>Surface display.</td>
      <td>34</td>
    </tr>
    <tr>
      <td>1.23</td>
      <td>2-DoF CTHI system.</td>
      <td>34</td>
    </tr>
    <tr>
      <td>1.24</td>
      <td>2-DoF CTHI system.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>1.25</td>
      <td>3-DoF CTHI system.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>1.26</td>
      <td>Two finger dexterous CTHI.</td>
      <td>36</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>The real UR-5 manipulator.</td>
      <td>40</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Coordinate frames for UR-5 manipulator.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>Conceptual scheme of the experimental platform.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>The UR-5 robot with a prop attach to the end-effector.</td>
      <td>42</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Virtual environment and a set of task to be reached.</td>
      <td>42</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>2.6</td>
      <td>Real system setup for human robot interaction.</td>
      <td>43</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Interaction between robot and human.</td>
      <td>44</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>Configurations of the robot for same end-effector position and orientations.</td>
      <td>49</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>Discretization of the workspace into voxels.</td>
      <td>50</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>Construction of complete workspace of UR-5.</td>
      <td>51</td>
    </tr>
    <tr>
      <td>2.11</td>
      <td>Generated Workspace for Config7.</td>
      <td>52</td>
    </tr>
    <tr>
      <td>2.12</td>
      <td>The prop used in the experiment.</td>
      <td>53</td>
    </tr>
    <tr>
      <td>2.13</td>
      <td>Different regions in the virtual environment.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.14</td>
      <td>Different regions in the virtual environment.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.15</td>
      <td>Workspaces with region orientations.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.16</td>
      <td>Workspace including all region orientations.</td>
      <td>56</td>
    </tr>
    <tr>
      <td>2.17</td>
      <td>S<sub>1</sub> and S<sub>2</sub> region task points and base location.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>2.18</td>
      <td>S<sub>2</sub> and S<sub>3</sub> region task points and base location.</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.19</td>
      <td>S<sub>1</sub>, S<sub>2</sub> and S<sub>3</sub> region task points and base locations.</td>
      <td>60</td>
    </tr>
    <tr>
      <td>2.20</td>
      <td>Introduction of User model into environment.</td>
      <td>62</td>
    </tr>
    <tr>
      <td>2.21</td>
      <td>Analysis on the dimension of the base table of the robot.</td>
      <td>63</td>
    </tr>
    <tr>
      <td>2.22</td>
      <td>Placement analysis of the base table of the robot.</td>
      <td>63</td>
    </tr>
    <tr>
      <td>2.23</td>
      <td>New table design.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>Environment scene setup.</td>
      <td>66</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>The environment scanned using Kinect.</td>
      <td>67</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Models referenced in this experiment.</td>
      <td>67</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>The model visualized in Rviz.</td>
      <td>69</td>
    </tr>
    <tr>
      <td>3.5</td>
      <td>Tracking system used.</td>
      <td>69</td>
    </tr>
    <tr>
      <td>3.6</td>
      <td>Front and side view of the subject attached with the trackers.</td>
      <td>70</td>
    </tr>
    <tr>
      <td>3.7</td>
      <td>Model for right arm.</td>
      <td>71</td>
    </tr>
    <tr>
      <td>3.8</td>
      <td>Structure for both arms.</td>
      <td>73</td>
    </tr>
    <tr>
      <td>3.9</td>
      <td>Model for left arm.</td>
      <td>73</td>
    </tr>
    <tr>
      <td>3.10</td>
      <td>D-H parameters for chest.</td>
      <td>75</td>
    </tr>
    <tr>
      <td>3.11</td>
      <td>Transform frames for estimation of error between shoulder frames.</td>
      <td>76</td>
    </tr>
    <tr>
      <td>3.12</td>
      <td>Scalability of the mannequin model based on height.</td>
      <td>77</td>
    </tr>
    <tr>
      <td>3.13</td>
      <td>User with the HTC vive trackers</td>
      <td>78</td>
    </tr>
    <tr>
      <td>3.14</td>
      <td>Complete mannequin model of the user.</td>
      <td>78</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Proposed system architecture.</td>
      <td>80</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>The system setup in the physical environment.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>Calibration setup for virtual and real environment</td>
      <td>83</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Representation of the user's effective workspace as a plane</td>
      <td>86</td>
    </tr>
    <tr>
      <td>4.5</td>
      <td>Work logic of path planning.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>4.6</td>
      <td>Representation of an Artificial Potential Fields.</td>
      <td>88</td>
    </tr>
    <tr>
      <td>4.7</td>
      <td>Representation of Probabilistic Road-maps.</td>
      <td>89</td>
    </tr>
    <tr>
      <td>4.8</td>
      <td>Representation of Probabilistic Road-maps (shorter path).</td>
      <td>89</td>
    </tr>
    <tr>
      <td>4.9</td>
      <td>Representation of Rapidly Exploring Random Trees.</td>
      <td>90</td>
    </tr>
    <tr>
      <td>4.10</td>
      <td>Representation of Rapidly Exploring Random Trees*</td>
      <td>91</td>
    </tr>
    <tr>
      <td>4.11</td>
      <td>Representation of Cell Decomposition.</td>
      <td>92</td>
    </tr>
    <tr>
      <td>4.12</td>
      <td>Representation of Quadtree and Octree decompostion.</td>
      <td>93</td>
    </tr>
    <tr>
      <td>4.13</td>
      <td>Planning group for the robot system.</td>
      <td>95</td>
    </tr>
    <tr>
      <td>4.14</td>
      <td>Planned paths with out re-planning.</td>
      <td>96</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>4.15 Planned paths with re-planning.</td>
      <td>97</td>
    </tr>
    <tr>
      <td>4.16 Comparison of all planning algorithm.</td>
      <td>98</td>
    </tr>
    <tr>
      <td>4.17 Comparison of best planning algorithm's times</td>
      <td>98</td>
    </tr>
    <tr>
      <td>4.18 Comparison of average execution times of the algorithms.</td>
      <td>99</td>
    </tr>
    <tr>
      <td>4.19 Comparison of average amount of generated waypoints.</td>
      <td>99</td>
    </tr>
    <tr>
      <td>4.20 Unity VR system and representation of interaction points</td>
      <td>101</td>
    </tr>
    <tr>
      <td>4.21 Representation of the user's effective workspace as a sphere</td>
      <td>101</td>
    </tr>
    <tr>
      <td>4.22 Representation of dynamic obstacles using the user model.</td>
      <td>104</td>
    </tr>
    <tr>
      <td>4.23 Example of re-planning with dynamic obstacles.</td>
      <td>104</td>
    </tr>
    <tr>
      <td>5.1 The complete system setup.</td>
      <td>108</td>
    </tr>
    <tr>
      <td>5.2 Proposed schematic diagram.</td>
      <td>110</td>
    </tr>
    <tr>
      <td>5.3 Scene information inside the virtual environment.</td>
      <td>111</td>
    </tr>
    <tr>
      <td>5.4 Representation of user occupancy using a sphere.</td>
      <td>112</td>
    </tr>
    <tr>
      <td>5.5 Interior of car and user workspace.</td>
      <td>112</td>
    </tr>
    <tr>
      <td>5.6 2D representation of interior of car and user workspace.</td>
      <td>113</td>
    </tr>
    <tr>
      <td>5.7 Illustration for motion using safe-points.</td>
      <td>114</td>
    </tr>
    <tr>
      <td>5.8 Comparison of motion with and without safe-points.</td>
      <td>115</td>
    </tr>
    <tr>
      <td>5.9 Pictorial representation of Strategy A.</td>
      <td>116</td>
    </tr>
    <tr>
      <td>5.10 k-d tree decomposition and structure.</td>
      <td>117</td>
    </tr>
    <tr>
      <td>5.11 Motion of hand tracker.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>5.12 Motion of robot TCP without head gaze.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>5.13 Motion of hand tracker.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>5.14 Motion of robot TCP with head gaze.</td>
      <td>121</td>
    </tr>
    <tr>
      <td>5.15 Pictorial representation of Strategy B.</td>
      <td>121</td>
    </tr>
    <tr>
      <td>5.16 Pictorial representation of Strategy C.</td>
      <td>122</td>
    </tr>
    <tr>
      <td>5.17 Pictorial representation of Strategy D.</td>
      <td>123</td>
    </tr>
    <tr>
      <td>5.18 User hand motion trail, from point 2 to 11.</td>
      <td>126</td>
    </tr>
    <tr>
      <td>5.19 Detection of points of interest for different strategies.</td>
      <td>126</td>
    </tr>
    <tr>
      <td>5.20 Robot motion, user distance and time for detection, for strategies A and B.</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5.21 Robot motion, user distance and time for detection, for strategy C.</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5.22 Robot motion, user distance and time for detection, for strategy D.</td>
      <td>129</td>
    </tr>
    <tr>
      <td>5.23 Comparison of time for detection (Q<sub>2norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>131</td>
    </tr>
    <tr>
      <td>5.24 Comparison of robot distance (Q<sub>4norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>131</td>
    </tr>
    <tr>
      <td>5.25 Comparison of time for robot (Q<sub>3norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>132</td>
    </tr>
    <tr>
      <td>5.26 Comparison of time for robot (Q<sub>3norm</sub>) vs time for detection (Q<sub>2norm</sub>).</td>
      <td>132</td>
    </tr>
    <tr>
      <td>5.27 Comparison of robot distance (Q<sub>4norm</sub>) vs time for robot (Q<sub>3norm</sub>).</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.28 Comparison of time for detection (Q<sub>2norm</sub>) vs robot distance (Q<sub>4norm</sub>).</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.29 Comparison of time for detection (Q<sub>2norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>134</td>
    </tr>
    <tr>
      <td>5.30 Comparison of robot distance (Q<sub>4norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>135</td>
    </tr>
    <tr>
      <td>5.31 Comparison of time for robot (Q<sub>3norm</sub>) vs user distance (-Q<sub>5</sub>).</td>
      <td>135</td>
    </tr>
    <tr>
      <td>5.32 Comparison of time for robot (Q<sub>3norm</sub>) vs time for detection (Q<sub>2norm</sub>).</td>
      <td>136</td>
    </tr>
    <tr>
      <td>5.33 Comparison of robot distance (Q<sub>4norm</sub>) vs time for robot (Q<sub>3norm</sub>).</td>
      <td>136</td>
    </tr>
    <tr>
      <td>5.34 Comparison of robot distance (Q<sub>4norm</sub>) vs time for detection (Q<sub>2norm</sub>).</td>
      <td>136</td>
    </tr>
    <tr>
      <td>A.1 Coordinate frames for UR-5 manipulator.</td>
      <td>146</td>
    </tr>
    <tr>
      <td>A.2 Geometric illustration for computing θ<sub>1</sub>.</td>
      <td>148</td>
    </tr>
    <tr>
      <td>A.3 Geometric illustration for computing θ<sub>5</sub>.</td>
      <td>150</td>
    </tr>
  </tbody>
</table>
<table><tr><td>A.4</td><td>Geometric illustration for computing θ<sub>6</sub></td><td>151</td></tr><tr><td>A.5</td><td>Geometric illustration for computing θ<sub>3</sub>, θ<sub>2</sub> and θ<sub>4</sub></td><td>152</td></tr><tr><td>B.1</td><td>Error for position in shoulder sensor in right hand.</td><td>153</td></tr><tr><td>B.2</td><td>Error for position in shoulder sensor in left hand.</td><td>154</td></tr><tr><td>B.3</td><td>Error for position in wrist sensor in right hand.</td><td>155</td></tr><tr><td>B.4</td><td>Error for position in wrist sensor in left hand.</td><td>156</td></tr><tr><td>B.5</td><td>Error between actual and estimated right shoulder frame.</td><td>157</td></tr><tr><td>B.6</td><td>Error between actual and estimated left shoulder frame.</td><td>158</td></tr><tr><td>C.1</td><td>MoveIt system architecture.</td><td>160</td></tr><tr><td>C.2</td><td>Move group architecture.</td><td>161</td></tr><tr><td>C.3</td><td>Planning scene architecture.</td><td>163</td></tr></table>
# List of Tables

<table><tr><td>1</td><td>Research contributions</td><td>7</td></tr><tr><td>2.1</td><td>Denavit-Hartenberg parameters for the UR arms.</td><td>40</td></tr><tr><td>2.2</td><td>Different configurations for UR-5</td><td>48</td></tr><tr><td>2.3</td><td>Color code for reachable sphere in workspace</td><td>51</td></tr><tr><td>2.4</td><td>Reachable spheres in workspace.</td><td>52</td></tr><tr><td>4.1</td><td>Comparative table between path planning algorithms.</td><td>94</td></tr><tr><td>5.1</td><td>Strategy analysis for the trajectory 2-11</td><td>129</td></tr><tr><td>5.2</td><td>Complete analysis for seven trajectories</td><td>130</td></tr><tr><td>5.3</td><td>Analysis of strategies for all trajectories.</td><td>133</td></tr><tr><td>5.4</td><td>Analysis for various thresholds</td><td>134</td></tr><tr><td>A.1</td><td>Modified Denavit-Hartenberg parameters (DH-parameters) of a UR5 robot</td><td>146</td></tr></table>
List of Algorithms

<table><tr><td>1</td><td>Construction of workspace</td><td>50</td></tr><tr><td>2</td><td>Base placement of the robot</td><td>57</td></tr><tr><td>3</td><td>Tracker information in ROS.</td><td>83</td></tr><tr><td>4</td><td>Tracker world reference in ROS.</td><td>84</td></tr><tr><td>5</td><td>Hand tracker information in ROS.</td><td>84</td></tr><tr><td>6</td><td>Motion of robot based in hand position.</td><td>85</td></tr><tr><td>7</td><td>Trajectory computation and storage.</td><td>102</td></tr><tr><td>8</td><td>Trajectory upload and execution.</td><td>103</td></tr><tr><td>9</td><td>Strategy A: Predictions with hand.</td><td>116</td></tr><tr><td>10</td><td>Nearest neighbor search.</td><td>118</td></tr><tr><td>11</td><td>Predictions with head gaze.</td><td>119</td></tr><tr><td>12</td><td>Strategy B: Predictions with head gaze.</td><td>121</td></tr><tr><td>13</td><td>Strategy C: Addition of safe-point.</td><td>122</td></tr><tr><td>14</td><td>Strategy D: Predictions with head gaze and safe-points.</td><td>124</td></tr></table>
# Glossary

The following terminologies are repeatedly used in this doctoral thesis:

**Definition 1.** A joint is a connection between two or more links. It is also known as a kinematic pair.

**Definition 2.** *Degree of freedom (dof) or mobility* is the number of independent parameters required to define the position of a rigid body in space.

**Definition 3.** A *Workspace* of a six-axis robot is the set of all poses attainable by a particular end-effector mounted on the robot

**Definition 4.** The configurations of a manipulator where it loses or gains one or more dof are called singular configurations.

**Definition 5.** A *virtual world* is an imaginary space often manifested through a medium.

**Definition 6.** *Virtual reality* is a medium composed of interactive computer simulations that sense the participant's position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation (a virtual world).

**Definition 7.** *Immersion* is the sensation of being in an environment. It can be a purely mental state or can be accomplished through physical means. Physical immersion is a defining characteristic of VR.

**Definition 8.** *Position tracking* is the computerized sensing of the position (location and/or orientation) of an object in the physical world—usually including one or more parts of the participant's body.

**Definition 9.** *In collaborative environment*, multiple users interacting within a virtual world that enables interaction among participants; not necessarily manifested in virtual reality; a collaborative VR environment can be referred to as multipresence or multiparticipant.

**Definition 10.** An *avatar* is a virtual object used to represent a participant or physical object in a virtual world. The typically visual representation may take any form.

**Definition 11.** A *base* is the link in a mechanism where the first joint(is) is(are) connected and is usually fixed.

**Definition 12.** An *end-effector* is a link in a mechanism where the last joint(s) is(are) connected.

There are other terms used in this doctoral thesis, which are defined directly in the text for the sake of continuity.
# Nomenclature

<table><tr><td>dof</td><td>degree of freedom</td></tr><tr><td>M</td><td>Transformation matrix</td></tr><tr><td>R</td><td>Rotation matrix</td></tr><tr><td>d</td><td>Displacement vector</td></tr><tr><td>VR</td><td>Virtual Reality</td></tr><tr><td>DoF</td><td>Degrees of Freedom</td></tr><tr><td>ICI</td><td>Intermittent Contact Interfaces</td></tr><tr><td>SE</td><td>Special Euclidean</td></tr><tr><td>SO</td><td>Special Orthogonal</td></tr><tr><td>APF</td><td>Artificial Potential Fields</td></tr><tr><td>PRM</td><td>Probabilistic Roadmaps</td></tr><tr><td>ECD</td><td>Exact Cell Decomposition</td></tr><tr><td>ACD</td><td>Approximated Cell Decomposition</td></tr><tr><td>RRT/RRTs</td><td>Rapidly Exploring Random Trees</td></tr><tr><td>BRRT</td><td>Bi-directional RRT</td></tr><tr><td>RT-RRT</td><td>Real-time RRT</td></tr><tr><td>BiTRRT</td><td>Bi-directional transition-based RRT</td></tr><tr><td>ROS</td><td>Robot Operating System</td></tr><tr><td>API</td><td>Application Programming Interface</td></tr><tr><td>URDF</td><td>Unified Robot Description Format</td></tr><tr><td>SRDF</td><td>Semantic Robot Description Format</td></tr><tr><td>OMPL</td><td>Open Motion Planning Library</td></tr><tr><td>EST</td><td>Expansive Spacial Trees</td></tr><tr><td>BiEST</td><td>Bi-directional Expansive Spacial Trees</td></tr><tr><td>FCL</td><td>Flexible Collision Library</td></tr><tr><td>ACM</td><td>Allowed Collision Matrix</td></tr><tr><td>KDL</td><td>Kinematics and Dynamics Library</td></tr><tr><td>FOV</td><td>Field of View</td></tr><tr><td>HVS</td><td>Human Visual System</td></tr><tr><td>VE</td><td>Virtual Environment</td></tr><tr><td>HMD</td><td>Head Mounted Display</td></tr><tr><td>VOs</td><td>Virtual Objects</td></tr><tr><td>CHIs</td><td>Classical-Contact Haptic Interfaces</td></tr><tr><td>FPS</td><td>Frames Per Second</td></tr><tr><td>GUI</td><td>Graphical User Interface</td></tr><tr><td>HCI</td><td>Human Computer Interaction</td></tr><tr><td>LCD</td><td>Liquid Crystal Display</td></tr></table>
<table><tr><td>VPL</td><td>Visual Programming Lab</td></tr><tr><td>×</td><td>Cross product</td></tr><tr><td>.</td><td>Scalar product</td></tr><tr><td>∀</td><td>For all</td></tr><tr><td>∈</td><td>Belongs to</td></tr><tr><td>⊂</td><td>Subset</td></tr><tr><td>∪</td><td>Union</td></tr><tr><td>∩</td><td>Intersection</td></tr></table>
Introduction

Virtual reality technologies allow a user to get immersed in virtual worlds. Haptic technolo-
gies born from robotics have increased the immersion in these virtual worlds by providing the
sensation of touch.

In a Virtual Reality (VR) simulation, haptic interfaces allow a tangible and physical inter-
action with the virtual environment, but they must generally be constantly held in hand by the
user and therefore do not allow objects to be touched in a natural way. At the same time, many
applications require hands-on interaction without intermediaries. This is particularly the case
for simulations that require tactile exploration of the physical properties of virtual objects.

Classical force feedback interfaces, also called here classic contact haptic interfaces (CHIs),
are robotic systems allowing natural motion interactions with virtual or remote environments.
They are used in several domains such as design, manufacturing, assembly, scientific visualiza-
tion, entertainment, education, medicine, space, rehabilitation, micro manipulation and molecu-
lar biology. In all cases, they should provide adequate kinesthetic (force) feedback, contributing
to enhance the sense of presence in the virtual environment.

With CHIs, the user is usually mechanically linked to the device’s end-effector, typically a
handle, whose movements, measured by the robot, are used to know the configuration (position
and orientation) of his/her hand. This information is necessary to provide force feedback which
is consistent with the virtual scene and the mechanical properties of the virtual object (VO)
being touched. The mechanical link that is established when the user manipulates the haptic
device has however a non-negligible influence since he/she experiences the friction, inertia and
vibrations of the mechanical structure, even in free space where he/she is expected to feel
nothing. Such unwanted sensations decrease the realism of the interaction since the user feels
all the time the presence of the robot. In addition, the difference between free space and contact
is less distinctively felt than in the real world.

In order to cope with these issues, several efforts can be made in terms of mechanical design,
e.g. use of very lightweight and very stiff structures (even if an optimal trade off is difficult to
attain) and more efficient transmission systems. Another approach consists in installing a force
sensor at the level of the robot's end-effector in order to measure and compensate any resisting
force in the direction of displacement. However, resisting forces can never be totally canceled
and none of these approaches completely eliminates the feeling of the presence of the robot in
free space.

Years of research on haptics, robotics, and interaction in VR have led to the development
of a new generation of haptic devices called intermittent contact interfaces (ICI) [73],[87],[7].
Intermittent-contact interfaces (ICIs) represent an original and promising approach aiming to
cope with the aforementioned issues. Its principle consists in removing the mechanical link
between the human operator and the force feedback interface during manipulations in free space
and come at his/her contact only when force feedback is required. This solution implies the need
to track and closely follow the user's movements in free space and to prevent him/her to move
in the constraint direction when a VO is being touched. This way, the user doesn't feel any force
in free space (perfect transparency) and the transitions from free space to contact are deemed
to be felt more naturally as the robot really touches the user at the simulated contact moment.
This approach aims to improve the realism of the interactions, however it suffers from several
shortcomings. First, its efficiency has not yet been proven in terms of user safety. Second,
even if IC interfaces are experimentally proven to be stable at low speeds, they tend to become
oscillating at higher speeds. Finally, despite the fact that a lot of tasks are performed by the
mean of tools in the real world, most of existing ICIs focus on bare finger interactions and are
therefore not optimal for simulating tool-mediated tasks.

User safety is central to the implementation of an ICI. To satisfy this constraint, we propose to use a cobot, for which we will develop specific path planning algorithms taking into account its low performance in terms of travel speed in the intrinsically safe mode, as well as all segments of the robot and the user's entire body in interference management. These algorithms will be based on a user activity prediction model that will allow both specifying a final desired location and the constraints to be respected when defining the path to reach this desired location.

<figure><img src="image_3.png" /><figcaption>Figure 1 – Conceptual scheme of the experimental platform</figcaption></figure>

This thesis, entitled "Contributions to utilize a cobot as intermittent contact haptic interfaces in virtual reality" presents a series of research works aiming to leverage the usability of such a system. This research work is created to address the current limitations in such a haptic device system.

## Description of context

This thesis was developed under the frame of the French National Research Agency (ANR) LobbyBot Project. The LobbyBot project consisted in developing a system that could be integrated into an industrial application for automotive interior prototyping. The system is intended to be used to recreate an automotive cockpit for faster prototyping in VR for the Renault Group, a French automotive company. In this fast prototyping process, designers had to explore and evaluate the perceived quality of a virtual car interior with their sense of touch of different materials, shapes, and objects that could be arranged in a virtual automotive interior that could be easily configured in VR. This new paradigm was conceived as a means to save the Renault Group
costs in budget and time for fabricating actual automotive cockpits that are exclusively used for prototyping purposes. The object of the thesis will be the control of the robot to implement this ICI.

<figure><img src="image_4.png" /><figcaption>Figure 2 – The LobbyBot project setup.</figcaption></figure>

The project assembled a consortium consisting of four partners: The Renault Group which was in charge of providing the use-case scenario and problematic. The Laboratory of Digital Sciences of Nantes (LS2N) was in charge of the path-planning algorithm for avoiding collisions with users. The National Institute for Research in Computer Science and Automation (INRIA) was in charge of conceiving new 3D interaction techniques to compensate for the inherent limitations of ICI. CLARTE was in charge of integrating the aforementioned contributions from all the other partners to implement the system.

Interaction techniques will allow managing any delays by the ICI in relation to the user. To increase the extent of sense of the form and materials can be restored by the use of an adapted pseudo-haptic return, and to follow a surface with the finger through appropriate sensory feedback. The integration of all the results into an ICI prototype will make it possible to validate the interest of the solution on an industrial case study. With current technologies, this industrial application (evaluation of the perceived quality of a virtual car interior) cannot be treated in a fast and low cost way.

## Applications of ICIs

ICI applications range from industry (Figure 3), entertainment (Figure 4), medicine (Figure 5), and research purposes (Figure 6). In all these cases, users expect to “encounter” a surface to touch or manipulate in a Virtual Environment. In the case of industrial applications, these devices are considered for virtual prototyping that requires to have haptic feedback in several locations to recreate workspace or expected location for objects to be manipulated. In the case of entertainment, they are used to recreate elements that can come in contact with the users when interacting with a virtual environment [61]. In the case of medicine, it is often used for remote body-palpation and surgery practice [42]. The use in research purposes often looks for leveraging the device's capabilities for rendering more complex surfaces and objects [74].
<figure><img src="image_5.png" /><figcaption>Figure 3 – Application in Industry by Lobbybot [104].</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4 – Haptic-go-round from one direction to another while fishing in the virtual scene. [61].</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5 – The virtual palpation system. [42].</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 6 – Synthesizing the Roughness of Textured Surfaces for an Encountered-type Haptic Display [74].</figcaption></figure>
# Challenges

In order to leverage the capabilities of ICI, several challenges must be addressed, these are addressed in different dimensions such as: **User safety and security** and **User immersion experience**. The challenges are divided into research questions (RQ) that this thesis addresses in the following chapters, namely; (RQ1) Robot placement; (RQ2) User model; (RQ3) Trajectory planning and (RQ4) User prediction techniques. These questions are further presented below.

## Challenge 1. User safety and security

The concern to avoid unexpected collisions with users has been present ever since the early days of Human-Robot Interaction research. Measures for addressing this issue normally consider path-planning algorithms that help the robot to actively avoid the user in cases where both user and device could come into contact involuntarily. However, the use of path-planning often causes the device to move more slowly. This delay affects response time and user's perceived immersion in the virtual environment, and thus, it has been recognized as an issue to be addressed by the research community.

As an alternative to solutions exclusively relying on path-planning for avoiding collisions with users, we propose additional techniques like the placement of robot to minimize the workspace interaction, and creation of multiple zones for robot motion. Research concerning user safety for ICI systems needs to look for strategies that help to avoid any undesired contact that could break users perceived immersion in the virtual environment. The main objective remains to ensure the safety of the operator. With virtual reality, the operator has a modified perception of the robot and of the risk of collision. A visualization in the virtual world of the robot reduces the immersion but increases the safety. In the intersection zones between the space possibly scanned by the robot and the human, the robot (UR-5) will be placed in cobot mode, i.e. it will stop and switch to gravity compensation mode if a contact effort is detected. Thus it cannot injure the operator. However, the speed of the robot is then limited.

## Research Question 1. Robot placement

Robot placement holds the promise of removing the need for a highly dedicated and structured workspace, as well as responding more quickly to environmental changes. Within the systems, dynamic and robust placements are crucial and strategically important, since they are often done in early stages in the process.

A complete set of operations consists in performing a specific task/operation by a robot on a set of task points. Often, after the robot returns to its starting configuration, a user is introduced to the system, and operations are performed. Since these cycles are repeated several times, it is very important that they are executed as fast as possible in order to increase immersion.

Once a set of specific points is assigned to a robot, the layout has limited freedom to optimize the robot workstation:

- robot's base placement (translation and rotation); The orientation of robot TCP at each task points is important as it affects the immersion of the user. So the robot should be able to reach the task points with the required orientations. The study started with no information on the robot base location in the virtual environment.

- visiting order of the work-points; In this approach the user decides the order of interaction with the task points. To improve user safety, the robot has to be closer to the user hands near the task points and as far as possible to the human body (torso, head and especially
neck). Major concerns were to restrict the operation of the robot to one aspect to avoid crossing singularities while performing the tasks.

— robot's home configuration in the station (six joints); initial approach is to achieve one single base location and have single aspect of robot to connect and move between all the task point.

The last three ones may be modified by changing the robot program, whereas the first has to be completely decided before installing the robot in the workspace.

## Research Question 2. User model

There is no fixed technique or standard procedure to use a tracking system to explore user actions in VR. Human movement tracking systems are expected to generate real-time data that dynamically represent the pose changes of a human body (or a part of it), based on well-developed motion sensor technologies. Generic tracking systems employed within these systems adhere to the human body in order to collect movement information. These sensors are commonly categorized as mechanical, inertial, acoustic, radio, microwave, and magnetic-based. In such systems, the user is generally equipped with sensors on his hands, that give the sense of interacting with a virtual object. Additionally, these sensors don't give information about the position and orientation of the user's location in the system. Visual marker-based tracking is a technique where cameras are applied to track human movements, with identifiers placed upon the human body.

The whole idea of ICI is to increase the immersion and remove the sensation of robot presence in free space. Since the robot has a plan to move between task points without colliding with the user. For safety concerns, the position of the user is very important. However, we cannot have complex tracking systems to locate the user. Normally the use of external cameras and markers are used to locate the user's position. But such systems increase the number of sensors attached to the user, which reduces the immersion. So the goal is to use less additional sensors as possible to give the best immersion experiences while getting the accurate location of the user for planning trajectories of the robot.

## Challenge 2. User immersion experience

Haptic feedback for ICIs goal is to improve the immersion experience of users. The major idea is to eliminate the mechanical link so as to have perfect transparency in free space. But even trying to achieve this experience comes with the challenges like planning the robot motion between the VO being touched (without colliding with the user), and predicting which VO object the user intends to touch in the environment.

## Research Question 3. Robot motion planning

In the given scenario the user is immersed in Virtual reality. The user has no information regarding the motion of the robot. The robot's motion has to be safe for the user, experiencing the VR environment. It should avoid collision with the user and also any other obstacles in the environment. The robot motion should take into consideration collision with the user based on the tracking system used. It's not only for the user's safety, but also to have a better immersion of the user, the robot must place a real object at the place where the user want to touch the VO as quickly as possible, and before the user hand reach the contact location.
## Research Question 4. Target object detection.

Since they touch the user only when force feedback is required, intermittent contact interfaces, and in particular close-tracking-type devices, aim to provide more realistic interactions with virtual environments than classical contact haptic interfaces (CHI). User intention prediction in ICI remains a challenge to be properly addressed by the research community.

Previous systems force the user interact with the VO, selected by it. These systems have to use control algorithms and interaction techniques to make the haptic rendering as efficient as possible. For the new system we want to give the user freedom to select the VO to interact and the robot system should adapt to align itself w.r.t to the VO. The goal of such a system is to predict the VO the user wants to interact with as soon as possible. However, for these new systems the two solutions (control algorithms and interaction techniques) are not enough to render the haptic sensation.

## Thesis road map

The main research axis considered as contributions of this thesis is improving user safety and immersive experience for users of an ICI haptic device in human-robot interaction.

The user safety axis is improved by addressing three contributions, the first contribution (C1) consisting of a set of safety techniques based on robot placement, the second (C2) creation of a user model for tracking user information, and the third (C3) techniques for robot trajectory planning.

Then, the second axis immersive experience is improved by making sure the user does not lose the illusion of the environment. This aspect can be addressed by making sure the user does not wait a long time for the robot to reach the desired interactive location, this is achieved by addressing 2 contributions, first contribution (C3) techniques for robot trajectory planning and second contribution (C4) techniques for user goal predictions, designed to optimize the response time in ICI systems.

The relationship between the contributions and the research axis can be seen in Table 1.

<table><caption>Table 1 – Contributions to the research axis</caption><thead><tr><td>Contribution</td><td>User safety</td><td>Immersive experience</td></tr></thead><tbody><tr><td>(C1) Safety techniques based on robot placement.</td><td>x</td><td></td></tr><tr><td>(C2) User model for tracking user information.</td><td>x</td><td></td></tr><tr><td>(C3) Techniques for robot trajectory planning.</td><td>x</td><td>x</td></tr><tr><td>(C4) Techniques for user goal predictions.</td><td></td><td>x</td></tr></tbody></table>

In the following chapters the details about how the contributions addressed the research questions will be presented.

## Contributions

The contributions of the thesis are depicted in chapters which are described hereby:

**Chapter 1** presents a literature review concerning present ICI haptic devices. In the first part, the history of these devices is narrated. The second part presents an analysis of the hardware used for these devices. The third section presents the types of haptic perception used
in literature. The fourth part discusses application scenarios. The literature review concludes
with a discussion of the presented research works.

**Chapter 2** presents a set of safety techniques for users based on the placement of the robot (C1). This chapter introduces a design space for safety techniques using visuals restricting the interactive workspace of the robot and the user, in order to reduce potential unintended collisions. The dimensions of the space focus on where the user wants to interact with the virtual environment. Explanation on how it protects the user. Using this design of robot placement, a set of workspaces was developed to explore variations of the prop orientations. An evaluation focusing on the best solution for robot placement is done. Safety was evaluated and the ideal location of the robot was defined.

**Chapter 3** presents the design and evaluation of the user model (C2). The model was designed to estimate the user location in the ROS environment. The chapter introduces first the need for such a model and different ways of achieving the goal. The model created had to have features such as movement of both arms and also the mobility of the user around the hip. These major three moving parts provided better information about the user's location, improving his safety in using an ICI system. A user study was designed to assess test the performance of these motions to test the accuracy and reliability of creating user model. Results suggested that the model is accurate for the given scenario in locating the user in the environment.

**Chapter 4** presents an approach for the trajectory planning of robots. The goal is to increase immersive experience (C3) by reducing the time taken by the robot to reach the contact area. This approach renders large surfaces and multiple textures through the use of a rotating prop, that couples the prop's rotation and position with the users' hand position when exploring a textured surface in VR. A use-case scenario was designed for contextualising this approach. Later, a user study was conducted to validate the approach haptic rendering performance.

**Chapter 5** presents a novel approach for the prediction of user intention based on head gaze and hand position. Different strategies are presented to predict where the user wants to interact with the virtual environment (C4). The chapter starts by introducing the different approaches and by describing their main features. Each approach presented is evaluated on four key features: (1) time taken to prediction, (2) time taken by the robot to reach the contact area, (3) distance traveled by the robot, and (4) safety of the user. These approaches integrate an interaction technique for contact area selection and release. An evaluation concerning the speed of the system is presented. Finally, the results are presented and discussed.

**Chapter 6** concludes the thesis manuscript summarizing the thesis contributions as well as providing perspectives for future work for ICI.

## Collaborations

A rehabilitation project entitled "Development, engineering, prototyping of a low-cost robotic system for postural rehabilitation with intuitive interface using consumer VR-AR technology and wearable IoT sensorization of the patient". Collaborators: Stanley Mugisha, Matteo Zoppi and Rezia Molfino, University of Genoa, Italy.
# Chapter 1

## State of the art and theoretical
background

### 1.1 Introduction

This chapter forms the basis of this doctoral thesis. It puts forth the state of the art and explains the fundamental concepts used to analyze and understand the behavior of the studied herein.

First, a brief literature review is presented. Then, the human senses are introduced, starting from vision to touch. Furthermore, the usefulness of virtual reality, and how we achieve immersion using the senses in virtual reality is described. While this offers significant immersion, there are some problems to be addressed. To overcome these problem, a new system called ICI is introduced. Different ways to implement such a technique are then presented.

Finally, a system is defined for the given context of study and how to achieve such a dynamic system is explained in short.

There is an increasing demand for simulators aiming to artificially recreate real world environments. These systems are used for various applications such as assembly verification, e.g. virtually checking that a new system can be easily assembled before launching the production, training, e.g. fight simulators, or simply entertainment, for instance immersive video-games.

An ideal simulation environment should stimulate each one of our senses (sight, hearing, smell and touch) to give us the illusion of being present in the recreated environment (immersion) and allow us to interact with it. However, recreating the corresponding sensations requires complex devices while all of them are not always required.

Indeed, it can be observed in practice that visual and touch feedback are sufficient to cover a wide scope of applications, even when fine interactions are required, e.g. dexterous manipulation of virtual objects (VOs). Visual rendering is obtained with systems ranging from simple monoscopic screens to more complex configurations like head mounted displays or multi-screen immersive systems such as CAVEs, with techniques allowing to recreate photo-realistic images. The study of the sense of touch is unfortunately less advanced. It is addressed with haptic interfaces, i.e. complex mechatronic devices able to provide either tactile or force feedback.

Force feedback interfaces are widely employed in virtual reality (VR) environments, e.g. for the training of novice surgeons in medicine, for the simulation of a product assembly, for education purposes or for scientific research. However, despite the wide range of applications of these devices, they still require to be improved in order to provide force sensations that are totally realistic.
<figure><img src="image_9.png" /><figcaption>Figure 1.1 – First HDM by NASA. [91]</figcaption></figure>

Recently, human robot collaboration become popular and interesting in the field of engineer-
ing. For robotics, environments that require both humans and robot to work simultaneously to
improve the performance in industries.

1.2 What is virtual reality ?

Even if virtual reality sounds as a very modern concept, it has been around for more than
half a century.

VR accounts for an interactive computer simulation allowing the user to visualize a virtual environment (VE) which recreates a scene of the real world. Interactive means that such simulation acquires, by means of a tracking system, information on the user's state and is able to provide sensory feedback to one or more senses. The main objective is to make the user feel as being immersed in the VE [93].

Four basic characteristic elements of VR can be identified: the virtual world (or VE), im-
mersion, sensory feedback (as a response to user's actions) and interactivity.

- A computer-based VE contains the description of the simulated objects as well as the rules governing their interactions, e.g. the effect of gravity making virtual objects to fall as it does with those in the real life.

- In the present context, immersion implies the notion of being present in the VE. In simple words, it is the belief of having left the real world to be present inside the VE. Immersion can be achieved if the VR system is able to provide realistic stimuli to all human senses [90].

- Sensory feedback requires proper displays, e.g. a head-mounted display (HMD, see Figure 1.1 and Figure 1.2) able to visually immerse the user in the virtual world, and a force feedback interface, held by the user, that will apply on him/her the forces generated when VOs are being touched.

- Finally, interactivity refers mainly to the fact that the VE is responsive to the user's actions (inputs). It requires a tracking system able to determine the movements of the user to ensure consistency between the real world and the VE. As an example, the visualized scene will adapt so that the user has the impression to move inside the VE.

The first practical implementation of the VR paradigm was developed in 1957 and patented
in 1962 by Morton Heilig, who is considered as the father of VR. In such a system, known as
Sensorama (see Figure 1.3), a single person could perceive, e.g. a motorcycle ride through a city
via sight, smell, vibration and wind [120].
<figure><img src="image_10.png" /><figcaption>Figure 1.2 – Classic HMD and VR glove environment. [54]</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 1.3 – First developed and patented VR system. [120]</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 1.4 – GROPE-III haptic display system. [17]</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 1.5 – Example of a CAVE. [54]</figcaption></figure>

Several other VR systems were developed. The GROPE I to III systems, created in the late 60s and early 70s, allowed the user to move and feel interaction forces between molecules thanks to a haptic interface (see Figure 1.4). Other examples of VR applications using force feedback will be described in future sections. As stated in the above paragraphs, one of the main goals of a VR system is to create the sensation of "presence". A step forward in this direction was achieved during the 90s with the concept of CAVE (Cave Automatic Virtual Environment). It consists of a room whose walls are in fact screens displaying a VE (see Figure 1.5).

This technology is however very expensive and requires an important volume to be installed, e.g. four equally sized walls of $1.8m \times 1.8m$. As a consequence, the current tendency is to use HMDs instead. This technology have made huge progress in the recent years and is now both efficient and affordable (see Figure 1.6).

Another important concept in VR applications is the notion of avatar, which is the virtual representation, for instance, of a human being as well as objects of interest regarding the task or the scene (see Figure 1.7). Finally, Figure 1.8 shows how the aforementioned elements are linked.

The following section will provide an insight on the importance of the sense of touch and, in particular, the role it plays when non-direct manipulation of virtual objects and interaction with VEs takes place.
<figure><img src="image_14.png" /><figcaption>Figure 1.6 – Commercially available HMD. [54]</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 1.7 – Principle of avatar.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 1.8 – Virtual reality feedback loop.</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Figure 1.9 – Ratio of information provided by each of 5 human senses.</figcaption></figure>

## 1.3 Human senses

Our senses are physiological tools to perceive information about the physical world. As defined and classified by Aristotle, human beings have at least five senses. These are sight, hearing, touch, smell and taste. Sensory processing is the capability to acquire, elaborate and incorporate information through our senses. There is a minimal point below which any stimulus does not produce any effect on sensory organs. Above this level, a stimulus' minimum perceptible variation is proportional to the absolute value of intensity (Weber's law). A specific sensory organ such as eyes, ears, skin, nose and tongue receives a stimulus which is above the minimum perceptible level. These sensory organs are the starting point for information transmission through the body through the nerve tracks. The afferent message from the sensory receiver is transferred to the nerve centres that are spinal cord, brain and cerebellum [100]. Once information is integrated and processed in the brain, nerve centers pass the nerve messages to the related sensory organs (skeletal muscles, eye muscles, vocal cord muscles, etc.).

Morton Heilig, in 1952, studied the senses in terms of their capacity to activate human attention [57]. It is essential to know the contribution of each sense while perceiving the world. Figure 1.9 clearly shows that the sight provides most information and captures the majority of our attention. The stimulation of the visual system, therefore, plays a significant role in fooling our the senses. Hearing, which is also often taken into account, is the second most crucial sense. Touch does not in general play an important role comparing to sight and hearing, even when it is essential for specific manipulation tasks. Due to the marginal role and difficulties of implementation, smell and taste are currently under development in most VR systems. However, immersive VR system should produce different signals and stimuli in several ways to replicate the physical world and use various display devices to immerse the user.

### 1.3.1 Sight

Human use sense of sight to assess the position of the objects with respect to them, so they can approach the objects, grasp them or avoid them for survival when needed.

Without eyesight, dangerous situations can be quite challenging to identify. Sight can therefore be used to avoid dangerous situations and materials.
Since sight is a sensory channel that provides most information and captures the majority of our attention, it is essential to achieve visual immersion that enables users to see the virtual world in a realistic way. For the Immersive VR experience, ideally, visual displays are able to produce feedback equal to or above the limitations of the human visual system [57].

Visual perception is the ability to see and interpret an environment using light in the visible spectrum, which is reflected by objects in the surrounding environment. In human visual perception, the two eyes act as a sensor, the neurons function as a connecting cable and the brain acts as a processor. The eyes, as sensory receptors, are engaged in the observation of the environment.

A wide Field of View (FOV) in virtual environments can enhance the sense of immersion and performance. The term FOV is one of the essential features of the Human Visual System (HVS). FOV is an angular measure that represents an open area visible to the human eye. For a healthy eye, two FOV information together describe human vision. The visible area through the eyes is generally indicated as an angle for the horizontal and vertical component of the FOV. It is approximately 160 degrees horizontally and 130 degrees vertically when one eye is used and rises horizontally to 200 degrees when looking straight ahead with two eyes - thus we can see behind us by 10 degrees. When we rotate our eyes to one side or another, we are able to see an additional 50 degrees on each side [64]. Therefore, to cover the entire range of natural human FOV, HDM needs approximately 300 degrees FOV horizontally. Moreover, the human brain uses the horizontal change of picture location reported by both eyes to determine the depth or the distance to the virtual object displayed on the scene from the viewer. This means that FOV is also crucial for depth perception.

### 1.3.2 Hearing

The sound gives us precise details about our surroundings and can often help us to assess the distance and direction of objects very accurately [138]. Sense of hearing is very useful for humans and important for survival in many circumstances. We can detect a sound in the dark when we cannot see. The auditory perception is omnidirectional, as opposed to limitations on the visual field of view, a human can detect sound from any place in 3D. In view of this omnidirectional aspect, hearing directs our visual senses, or “the function of the ears to point the eyes” as Cohen and Wenzel said [26].

In Multimedia, hearing assumes at least equal importance to the user's feeling of immersion [112]. A 3D (Spatial) audio-system enables a user to perceive the position of sound sources emanating of an arbitrary position in 3D space from speakers or pair of headphones. Spatial sound processing goes far beyond conventional stereo sound technologies by allowing virtual sound sources to possess attributes like left-right, up-down and back-forth.

### 1.3.3 Smell

Sense of smell, Olfaction, is essential to everyday life. It is a part of the chemosensory system that enables us to distinguish between a wide range of smells and tastes [24]. Olfaction plays a vital role in making us understand the environment in real world. It enables us to recognize food, drinks and also provides a sense of pleasure and warnings of danger. For instance, it advises us to leave a building with fire or if the food is rotten and poisonous. Furthermore, smell sense is more connected with feelings and emotions than thoughts and decisions [24]. Olfactory information can enhance the user's sense of immersion in the EV by tricking their sense of smell. Visual,
audio and tactile senses are widely included in VR, but the technology to produce olfactory stimulation is currently limited and still under development.

Olfaction has a strong influence over humans. The sense of smell is closely connected with the memory of an emotion. For that reason, it is important to achieve the goal of immersion by tricking the sense of smell. However, Olfactory simulation technologies have not reached that level yet and are still under development. Due to some limitations on the chemically based simulation as discussed, digital smell studies have grown in recent years. Real Feel [40] and Vaqso [133] devices are examples of digital smell devices which are specifically designed for VR headsets. They are able to create limited smell sensations as well as water mist, wind, heat and vibrations. However, a human being can differentiate one trillion different odours, while Feelreal can only simulate nine different, and Vaqso fifteen simultaneously. By comparing the natural olfactory system, it is quite low and not even close. For that reason, Olfactory displays are still under development to achieve the immersion by tricking sense of smell. When technology reaches or at least approaches to human smell perception, it can be used a wide range of VR applications including meditation, learning, movies and game VR applications to increase user's feeling of immersion. Game developers can use custom made smells and effects based on the content of the games. Smells and effects can also be added in VR movies without any programming skills.

### 1.3.4 Taste

Taste plays a major role in human life. There are nearly 10,000 taste buds on the tongue, roof of the mouth, and throat. These tiny bumps power our sense of taste. As a human being, we need food to survive. When we are eating, the taste of the foods directly influenced what we eat and how much we eat. Sense of taste, Gustation, is directly related to foods, and for that reason, it is one of the most important sense for life. Without sight, hearing or smell, people in our societies can still live a normal life. However, people who lack taste generally may not eat and may ultimately die if no medical treatment exists [24]. Research has shown that when human eat their favorite food, it triggers the production of β-Endorphin, which enhances the mood [34]. It explains why we feel happier when we eat chocolate. It is therefore stated that if food is the body, the taste is for the soul [34]. Moreover, taste also serves as a human defense mechanism. For instance, we judge the quality of the food on the basis of certain taste sensations and avoid decomposed food.

Despite the importance of the sense of taste in our daily lives, research and developments for taste stimulation in VR much weaker than for other senses. However, to be able to achieve the goal of immersion through tricking human senses, all of the human senses should be displayed in the VR.

At present taste sensations are difficult to use as a VR output by comparing other human senses, for instance, sight, audio and touch. Two main methods are in use to simulate the taste, which is chemical-based and non-chemical based approaches. Food simulator [24], Taste screen[86] and Virtual cocoon [36] are examples of the chemical-based approaches. However, there are some limitations such as limited availability of virtual tastes, refilling, cleaning, durability of the devices since the chemical content can be harmful to the device. Due to the limitations on the chemical based taste simulations, current research continues strongly on the basis of digital tasting devices. However, since digital tasting simulation technology is quite new, there is no specific method for it. Only three different, which are salty, bitter and sour achieved the simulate in Voctail whereas human tongue can approximately differentiate 10,000 tastes. The mixture of the taste sensation is difficult to predict due to complex interactions between the other senses; therefore, the sense of taste is currently considered the ultimate limit
for creating immersive VR [108].

### 1.3.5 Touch

Touch gives us our most simple tool to access the outside world [47]. It is a critical sense for a human being to connect with the physical world and plays a central role in human communication with the environment. Important details, for instance, warmth, coldness, softness and friction or more nuanced emotional contacts between humans can only be experienced through the sense of touch. Touch has often called “the sense” that helps us to differentiate between what is real and what is not [47]. This ensures that the tactile information given by the skin is considered real by default, and we always come to question how genuine a sensory signal is. The sense of touch leads to reduce the gap between the virtual and physical world [79]. It is important to trick the user’s touch sense through haptic technology to achieve the goal of immersiveness in VR. Further on this will be discussed in Section1.4, we begin by describing the human haptic perception that is related to haptic technologies. We then discuss tactile feedback, force feedback gloves and bodysuits. The chapter ends with a brief assessment of haptic technology for VR.

## 1.4 Haptic technologies for VR

Haptic technologies allow the user to touch and feel the objects that simulated in VE. Haptic word is derived from Greek “haptai” meaning to “touch” that convey important sensory information which helps users to identify virtual objects and move them to perform a task in VE [19]. The study of haptics is a consequence of advances in VR. VR is a type of HCI (Human Computer Interaction) that creates an immersive virtual world that can be experienced by direct interaction with our senses. When added to the visual and 3D audio feedback, haptic feedback dramatically improves the reality of simulation [19]. To be able to increase immersion in the virtual world, haptic feedback must be added. In HCI, Haptic feedback includes both force(kinetic) and touch(tactile) feedback as in the physical world.

### 1.4.1 Human haptic perception

Human sensation comes from the dermis layer of our bodies. The skin has a surface area of 18.000 square centimetres that is about 16-18% of the bodyweight of an adult; it is also the largest of our sense organs [94]. The dermis is lined with several sensory neurons’ nerve endings [23]. When an entity touches or is touched, the sensors of the skin are activated by force. This force is then transmitted to the brain as a piece of information. A healthy human can feel about twenty different types of haptic sense of which hot, cold, pain are the most common ones. Some areas of the body contain more receptors than others, making them more sensitive to sensation. The tongue, for instance, has many nerve endings with pain and heat. For that reason, it hurts more when we bite our tongue and why we can burn our mouths quickly when we drink something hot. Other most sensitive places are palms, ears, fingertips and feet. Human haptic sensations can be classified into two major groups of kinetic(force) feedback and tactile touch feedback.

- Kinetic(force) feedback occurs when muscle, joints and tendons feel the forces. As a result of clinical studies [6] on human manual force exertion capability, it was found that males can exert a maximum force of 400 Newton while female hand strength is 60% to 80% of the males.
- Tactile touch feedback includes feedback through the skin, such as a sense of touch, temperature, texture and skin surface pressure. There are four different types of tactile sensors that are Meissner corpuscles (the majority), Merkel disks, Pacinian and Ruffini corpuscles. When they are excited, they create tiny electrical discharges that are perceived by the brain. Merkel and Ruffini are the most sensitive to lower frequency stimuli (0-10 Hz) whereas Meissner and Pacinian sensors respond to the higher frequency (50-300Hz) stimuli [19].

### 1.4.2 Tactile feedback interfaces

Tactile feedback provides real-time information about contact geometry, surface softness and roughness, friction and temperatures [19]. It does not actively resist contact movement of the user and does not prevent the user from moving through virtual surfaces. In the commercial side, the most common tactile feedback devices are hand controllers serving as navigation, pointing and selection. They also have a hand tracking system to increase interaction in the virtual world. However, when interacting with the virtual world with hand controllers, their tactile feedback is only limited by vibrations. HTC Vive Controller and Oculus Touch are the examples of the commonly used hand controllers.

#### CyberTouch glove

The CyberTouch, in Figure 1.10, glove provides vibrotactile feedback to the user. It has six vibrotactile actuators, one at the back of each finger and one on the palm of the hand. Each actuator has a plastic capsule containing a DE electrical motor. The motor shaft is vibrated by an off-centred mass that produces vibration. The vibration frequency can be changed between 0 and 125Hz. Each actuator has a small force (1.2 Newton) that can be felt through the singer's bones [19]. They can be individually programmed to vary the strength of touch sensation [30]. Whenever fingers or palms of the user interacts with the virtual object, the CyberTouch glove reads the user hand configuration and transmits the data to the host computer. Then the host computer sends the necessary informations for activating the vibrotactile actuators. In this way, the feedback loop is closed, and the user feels the vibrations on the skin. The glove enhances the user freedom of movement and sense of vibration significantly comparing to the hand controllers that require to be kept on user's hands.

<figure><img src="image_18.png" /><figcaption>Figure 1.10 – Tactile interface by CyberTouch glove [30].</figcaption></figure>
Temperature feedback glove

The temperature feedback gloves enable users to perceive thermal properties that can help to recognize the material of the product. These thermal properties are, for instance, temperature, thermal conductivity and diffusiveness. When grasped, materials with high conductivity feels cold and those of low conductivity, such as wood, feels warm. It is because of the heat transfer from the finger to the surface of the objects [19]. C&M Research developed a displaced temperature sensing system (DTSS) which consists of eight thermodes and control interface panel. Each thermode consists of a Peltier heat pump and a thermocouple temperature sensor attached to the skin-contact plates [19]. The fingertip temperature is determined by the thermocouple temperature sensors in real-time and then sent to the control system. The fingertip temperature is compared to the target temperature and sent to the DTSS control interface. The temperature difference between the target temperature and the user's fingertip temperatures is the input for the temperature controller system. The output is then forwarded to current amplifiers, which drive the Peltier heat pump and thermocouple temperature sensors. These thermocouple sensors are located in each fingertip and other locations such as on the palm. The host computer can modify the temperature according to the VR simulation requires.

HaptX

HaptX is another glove device which offers a convincing touch feeling to virtual objects by incorporating mid-air haptic technologies that touch the skin physically in a similar way real objects do. Mid-air haptic technology makes a simulation of texture, shape and motion of virtual objects. There are 130 points of feedback that displace user's skin up to 2mm [52]. User also can feel the size, weight of the virtual objects with the help of exoskeleton mechanism structure of the glove. It also produces force feedback to limit and resist the user's movements. However, it is almost impossible to replicate all data transmitted by touching a virtual object, including its surface and weight with HaptX glove.

1.4.3 Force feedback interfaces

Force feedback conveys information on compliance with the virtual object surface, object weight and inertia in real-time. It actively resists and can stop the user's contact movement with large feedback forces [19].

CyberGrasp system

CyberGrasp is an exoskeleton device. CyberGlove, as seen Figure 1.11, is the most well-known and precursor of all commercial exoskeletons [103]. It has a jointed structure that the user wears over his hand that transmits forces to his fingers. It is used to provide feedback from resistive force power. The system prevents the fingers from moving by applying the force resistance and not allowing the movements of the fingers when the user tries to move them. The system imitates the actual phenomenon when attempting to compress a rigid item in the physical world. The CyberGlove can also make users feel the weight to the virtual objects by using the same concept of force feedback. The interface box of the CyberGlove transmits the finger position data to the Cyber grasp Force Control Unit (FCU). The same FCU collects wrist-position data from the 3D magnetic tracker. Both finger and wrist 3D positions are sent to the host computer during the simulation. The host computer then senses inputs that are resulting from finger touch forces into the FCU. The Cyber Grasp FCU then converts the
contact force targets into analogue currents which are amplified and delivered to one of the five electric actuators in an actuator housing equipment. The actuator torques are conveyed to the user via a cable network and a mechanical exoskeleton that is placed on the top of the glove. The maximum force that can be generated on each finger is 16 N [19]. One of the main disadvantages of CyberGrasp system is their large weight and height. For instance, CyberGlove weight is approximately 0.4 kg [38]. It could be stressful for the user to do long simulations.

<figure><img src="image_19.png" /><figcaption>Figure 1.11 – Force feedback interface by Cyber glove [30].</figcaption></figure>

### 1.4.4 Haptic bodysuits

In recent years, there have been several body haptic suits on the market; for instance, Tesla suit as seen in Figure 1.12, allow the user to touch and feel the sensation in Virtual Reality. The suit can actively stimulate heat, cold, vibration and tactile stimulation through a variety of electrical impulses by using conductive polymers and special add on materials as the primary structure of the suit. It has an ability to generate senses such as touch, pressure and pain by using two functional electrical stimulation: Neuromuscular electrical stimulation (NMES) and Transcutaneous electrical nerve stimulation (TENS) [10]. Touch sensation is mainly produced by the use of Transcutaneous Electrical Nerve Stimulation that is normally used in pain control treatments to block peripheral nerves [77]. Touch sense as tactile feedback can be simulated in bodysuits by varying frequency amplitude and pulse length. Moreover, Neuromuscular Electrical Stimulation is commonly used for muscle training and treatment for physiotherapy patients. It has the capacity to generate a sense of force or pressure on the user by producing muscle contractions, whereas TENS trigger a sense of touch depending on the virtual scenario.
<figure><img src="image_20.png" /><figcaption>Figure 1.12 – Haptic body suits - Tesla Suit [10].</figcaption></figure>

### 1.4.5 Analysis of haptic technologies in VR

Sense of touch is the link between the physical world. It plays a crucial role in human contact with the environment. Since the VEs are designed to simulate physical worlds, it is important to achieve the goal of immersivess by tricking the sense of touch. Haptic technologies which we have discussed are capable of providing sensations not only for hands, wrist, and palms but also for the human body. However, the skin is the largest part of our body including ears and tongue. The Immersiveness is increasing when we involve more parts of the body in the simulation. Current VR haptic devices do not support HCI fully in real-time and new technological approaches needed to achieve the goal of immersion through tricking our haptic senses. However, there are a still number of limitations on all existing haptic devices. They have limited feedback capacity in comparison to the tactile sensory system in humans. The human hand consists of millions of specialist touch sensors all working in tandem, while the latest haptic interfaces typically have fewer than ten tactile feedback engines. Other limitations on existing haptic devices include a high price, large size and weight, bandwidth restrictions, latency between a human operator and force feedback, design for very specific purpose purposes and instability if the update rate is well below 1 kHz [38]. Nevertheless, it will still take some time, until VR haptic technologies achieve the goal of immersiveness through tricking sense of touch.

Immersive haptic technologies can be used in e-commerce. For instance, a human can test products by sensing the warm, cold, soft, hard, light or heavy properties of product's surfaces and textures. Since consumers usually tend to touch items (e.g. clothes) before purchasing. Haptic feedback is important for impaired people who are not able to VE through visual displays. Moreover, there is great benefit in meditation and training immersive VR haptic applications.
1.5 Existing haptic interfaces

Haptic technology, also known as kinesthetic communication or 3D touch, refers to any
technology that can create an experience of touch by applying forces, vibrations, or motions to
the user. These technologies can be used to create virtual objects in a computer simulation, to
control virtual objects, and to enhance remote control of machines and devices (telerobotics).
Haptic devices may incorporate tactile sensors that measure forces exerted by the user on the
interface.

Haptic technology can communicate information to the user that could not be detected by sight or sound alone. Haptic sensation is what one first feels when touching something. Haptic sensation is also what allows us to determine that we are touching an actual object. We can use this human trait to help communicate information on substance and feel that could not be obtained through sight and sound alone. Haptic technology holds great promise as a means of bringing a greater sense of reality to games and virtual reality.

1.5.1 Haptic senses

The haptic senses refer to the perception of the variety of sensations that relate to touch: tactile sensibility, the perception of our limbs in space and the stress that they undergo, but also equilibrium, pain, and temperature [75] Those various signals are all integrated by the sensomato ry system, giving us a global haptic image of our physical state and that of our environment.

Haptic technology facilitates the investigation of how the human sense of touch works by
allowing the creation of controlled haptic virtual objects. Most researchers distinguish three
sensory systems related to the sense of touch in humans: cutaneous, kinaesthetic, and haptic.
All perceptions mediated by cutaneous and kinaesthetic sensibility are referred to as tactual
perception. The sense of touch may be classified as passive and active, and the term "haptic"
is often associated with active touch to communicate or recognize objects.

This section presents an overview of the sensory aspect of the human hand and focuses on
the predominant sensations involved in object manipulation: cutaneous sensibility and proprio-
ception.

Cutaneous sensibility

Cutaneous sensibility enables the perception of small-scale features such as texture and roughness. These sensations stem from mechanoreceptors, nerve endings encapsulated inside corpuscles that deform under stress, located below the superficial layer of the skin [35]. Structural variations in the capsule tissue make the four types of receptors react to different stimuli [75]. For instance, Meissner's corpuscles are sensitive to changes in velocity. Pacinian corpuscles are sensitive to vibration and light touch. Ruffini's corpuscles are sensitive to skin stretch and detect the direction of forces. Merkel's disks are sensible to pressure and smallscale shapes, which helps in feeling the edges of objects. These mechanoreceptors can be found all over the body in varying quantities but the hands are among the most densely populated areas, which grants them a particularly acute sensibility [84].

The types of mechanoreceptors differ in other properties than the stimuli they respond. First, they can be distinguished according to their adaptation rate, which is the speed of the transition from the excited state to the neutral state. Rapidly adapting receptors quickly return to their neutral state. Thus, they are not fit to detect the material properties of a surface from a
static observation; which explains why we slide our fingers over a surface in order to perceive its
texture. On the contrary, slowly adapting mechanoreceptors detect if an event is continuously
occurring. They play an essential role in gauging the weight, balance, and slippage of the objects
that we manipulate in order to ensure secure grasps.

Mechanoreceptors are also characterized by their input frequency, which corresponds to the speed at which separate stimuli can be detected. Rapidly adapting mechanoreceptors can respond to events between 20 and 300 Hz whereas slowly adapting receptors are typically limited to 10 Hz [19]. Additionally, the size of the receptive field, which corresponds to the skin area in which a contact is detectable by a single touch receptor, is larger for Pacinian and Ruffini's corpuscles. However, the larger is the receptive field, the lower the spatial resolution is [132].

Proprioception

Proprioception, or kinesthesia, is the perception of the motion of our limbs in space as well
as the perception of the stress that they undergo; sensations which are essential for interacting
with our hands [11].

At the limb level, proprioceptive information comes from mechanoreceptors in the joint,
the muscles, and the tendons. First, the same Pacinian and Ruffini's corpuscles that can be
found below the superficial layers of the skin are also located in the joints between bones. The
amplitude of the signal that they send to the brain informs about the joint angles between
limbs whereas its frequency informs about their angular velocities [20]. Then, internal and
external forces that our muscles are subjected to are evaluated by the Golgi organs located
at the junction between muscles and tendons. They measure muscle tension, stabilize heavy
grasps, and additionally play an inhibiting role by relaxing muscles when tension is too high.
Other receptors, the muscle spindles, are located inside of the muscles of the arms and hands,
and measure their length [75], which determines the shape and rigidity of the objects that we
manipulate. The various mechanoreceptors of the skin responsible for cutaneous sensibility
also play an indirect role in proprioception since extension/flexion of the limbs may result in
a stretching/bulging of the skin adding extra information about our haptic state. At a higher
level, the global position and orientation of each of our limbs is evaluated from the vestibular
system in the inner ear. Canals filled with fluid subject to gravity inform about the orientation
of the head. The perceived state of the rest of the body then depends on the proprioceptive
links between head, neck, trunk and limbs.

The maximum frequency at which a human finger can apply output forces is between 5 and
10 Hz [16]. Regarding the input bandwidth, variable numbers were proposed: [118] suggested
that it could go as high as 10 kHz and [16] supposed that a finger cannot discriminate two
consecutive force signals above 320 Hz. It is however admitted that a input frequency between
20 and 30 Hz is a minimum for meaningful perception [121].

1.5.2 Haptic feedback and sense of touch

The sense of touch is very important in our daily life. Without it, it would cause serious
difficulties in performing tasks such as holding and manipulating tools or making contact with
surrounding objects [111]. Thanks to this sense, which gives us the ability to perceive the
physical properties of our environment (shape, size, texture) and to control the forces applied
to it, allows us to perform a wide range of tasks with a high level of dexterity.

However, in some situations, for example, the human operator may not have direct contact
with the object being manipulated. Because the environment is hostile (nuclear, underwater,
<figure><img src="image_21.png" /><figcaption>Figure 1.13 – Teleoperated and virtual reality (VR) systems using force feedback.</figcaption></figure>

space) or not usable at the user's scale (micromanipulation, microsurgery). In such cases, tasks can be performed using a remote control system, a remote robot that mimics the movements of an operator-controlled master hand. The ability to control the movement of a remote system is also very important in virtual reality, in which case the user remotely controls the virtual avatar instead of the robot.

The concepts of remote manipulation (controlling a remote robot system), virtual reality (controlling a virtual avatar), and telepresence (makes a user feel like he/she is in a place other than their current location) are either remote (see Figure 1.13a) or Development of a system that allows the operator to feel the force and physical properties applied to an object that is virtually manipulated (see Figure 1.13a) [9], that is, i.e. by means of haptic interfaces.

Haptics was defined as a perceptual system that uses both cutaneous (including thermal) and kinesthetic inputs to derive information about objects, their properties, and their spatial layout [81]. To stimulate these sensations, users need special devices called tactile interfaces, devices that can interact with remote/virtual environments by reproducing touch using kinesthetic (force/position) and skin (tactile) receptors [51].

There are four distinguish methods for artificially creating haptic sensations : vibrotactile devices, force feedback systems, surface displays and distributed tactile displays [56]. The following section will describe their main applications with a focus on VR.

### 1.5.3 Examples of force feedback devices

Force feedback interfaces are used in many domains (most of them using VR) such as design, manufacturing, assembly, scientific visualization, entertainment, education, medicine, space, rehabilitation, micro-manipulation, as well as molecular biology [51]. The interface should then provide adequate kinesthetic information contributing to enhance the sense of presence in the VE [109].

In the fields of design, manufacturing and assembly, haptic feedback can contribute to changing traditional product development approaches by allowing the users to get the feeling of touching objects, perceiving the nature of their surfaces and their dynamics before producing any real prototype [139]. This way, users can gain a comprehensive understanding and accurate evaluation of the design and manufacturing process (see Figure 1.14).

In scientific visualization (see Figure 1.14b), haptic devices can be used to simulate object's physical properties, e.g. texture and/or interaction forces. This allows direct and immediate
control over simulations as well as sensing of the results of the scenario of interest [128].

The entertainment industry has also benefited from haptic developments. Force feedback technology enhances the game experience by providing more realistic sensations while playing a game [37].

Haptics is also of great interest in education. For example, a multimedia system for learning handwriting of alphabet letters/characters in different languages can make use of a haptic device to provide force feedback to guide the user's gesture at following a pre-recorded letter trajectory (see Figure 1.14c).

In medicine also, VR simulation-based training appears as a promising solution for skill transfer. Novice surgeons can acquire and/or improve their skills before operating on a real patient and experienced surgeons can learn new techniques and even rehearse when the patient to operate presents a complicated surgery case (see Figure 1.14d), e.g. a congenital anomaly or a known difficult anatomy [4].

Force feedback interfaces can also be used to intelligently guide/regulate the motion of the user during an operation [15], e.g. preventing the user to move towards a restricted region, contributing that way to improve the security of the patient and postoperative results.

More generally, robot-assisted minimally invasive surgery (RMIS) offers advantages to patients such as less trauma, shorter hospital stay and reduced recovery times [106]. Robots in master-slave configuration can cope with motion constraints of surgical instruments, improving surgeon's dexterity [105]. The surgeon is expected to act and feel as if he/she were holding directly the surgical tools being in contact with the patient. In such scenarios, any strategy to estimate/ sense the applied force is expected to improve the accuracy and dexterity of the surgeon [98] and therefore reduce the tissue trauma and organic damage. The development of force sensors for RMIS is thus still an active research topic as reported in [105].

One can also refer to physical rehabilitation which is a growing field of use for haptic interfaces. Unlike a human therapist, robots can train patients for long periods of time without tiring. Robotic systems coupled with VR simulations also bring additional improvements to today's conventional physical therapy methods, since they introduce objective measures of performance (see Figure 1.14e).

Space is an environment inaccessible to humans where force feedback systems can contribute to train operators and help them performing maintenance, reparation or exploration tasks while preventing and reducing risks. In [113] for instance, a VR platform for telerobotic on-orbit servicing missions helps to train users (providing visual and haptic feedback) for satellite maintenance tasks (see Figure 1.14f.

When it comes to manipulation of tiny objects, haptic systems allowing the user to interact with objects at the micro scale are needed (see Figure 1.14g). In molecular biology for example, the size and complexity of molecular structures make it difficult, if not impossible, to show all of their features in a physical model alone. In combination with augmented reality (AR) and voice commands, haptic feedback provides additional information about these models using a force display [116].

All the above mentioned applications require a device compatible with the targeted task. Several examples of such devices are presented in section 1.5.4, with focus on commercially available interfaces.

### 1.5.4 Commercially available interfaces

Force feedback systems are robotic mechanisms capable to measure the user's movements, this information being used to control the movements of user avatar, and deliver a force signal
<figure><img src="image_22.png" /><figcaption>(a) Virtual aircraft engine maintenance operation</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>(b) GROPE III system used for scientific visualization</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(c) Telemaque platform used for children's handwriting acquisition</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(d) Training platform for maxillofacial surgery</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(e) MIT MANUS for shoulder rehabilitation</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>(f) HUG for VR on-orbit servicing missions</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>(g) Teleoperation for micromanipulation</figcaption></figure>

Figure 1.14 – Applications of force feedback interfaces: (a) [14], (b) [17], (c) [101], (d) [49], (e) [18], (f) [113], (g) [13]
to his/her hand, usually through a stylus or a thimble [21].

The realism of the interaction is strictly related to the capabilities of the device to allow natural movements of the human operator's hands as well as to reproduce adequate stimuli on them. An ideal force feedback system should be transparent: the user should be able to move in free-space without feeling any force, i.e. to make pure free movements, and the device should prevent him/her to move in the constraint direction if a stiff object is being touched, i.e. the device should be able to simulate any type of contacts, even hard ones [29].

It is worth noting that a perfect transparency cannot be attained in practice. Fortunately, most of the time it is not necessary to reproduce the whole bunch of physical interactions experienced everyday, and force feedback interfaces can be tuned to the requirements of the tasks of interest. It is thus important to identify what sources of kinesthetic information are relevant for the targeted tasks, and which degree of fidelity is needed [111].

Several performance metrics are usually employed to qualify a force feedback interface: workspace, position resolution, continuous force, peak force, friction, apparent stiffness, apparent inertia and bandwidth. All these notions, defined below with reference to e.g. [95] contribute to the transparency of the device.

- The workspace is the volume that the device can cover, considering both translations (ranges usually expressed in the Cartesian space in mm, cm or m) and rotations (ranges usually expressed in degrees in yaw, pitch and roll).

- The position resolution is the smallest amount of movement which can be detected by the arm sensors. It is usually expressed at the level of the handle (i.e. in the Cartesian space) in m or mm in translation and in degrees or radians for the rotations.

- The continuous force is the amount of force/torque that the device can exert for an extended period of time. It is usually expressed in the Cartesian space in N for the translations and in N.m for the rotations.

- The peak force is the maximum force that the actuators can exert, e.g. over a small period of time. The peak force can typically be between 2 and 10 times higher than the continuous force for typical actuators usually implemented in force feedback interfaces, allowing to momentarily stop the user against the environment, e.g. when taping or touching a hard object.

- The friction is a measure of the resistance of the device to movement. It is expressed in N for translations and N:m for rotations. It should be as low as possible.

- The apparent stiffness is the maximum stiffness that can be simulated by the device. It informs on the different types of objects that can be simulated. Indeed many objects in VR environments behave like a spring, i.e. they deform linearly in response to an external force. The apparent stiffness is expressed in N/m for translations and N.m/rad for rotations. The higher it is, the better solid virtual walls can be simulated.

- The apparent inertia is a measure of the mass (in kg) and inertia (in kg.cm²) that is sensed by the user when he/she tries to move the handle of the robot. It should be as low as possible in order to give user the impression that he/she moves freely in free space.

- The bandwidth is the frequency (in Hz) up to which the device can be properly controlled. It should not be confused with the update frequency of the controller nor with the bandwidth of the human somatosensory system, i.e. about 300 - 1000Hz for tactile sensing and 20 - 30Hz for kinesthetic and proprioceptive sensing.

It is worth noting that, even if the afore-mentioned data are usually given in the Cartesian space, this information can be given instead in the joint space.

Six main manufacturers were identified in the market: 3D Systems, Force Dimension, Hap-
tion S.A., Quanser, MOOG Inc. and MPB Technologies Inc. Representative examples of their products will be briefly described below.

The Touch device (see Figure 1.15a, left) is a desktop interface providing 3DoF force feedback. This low power device can be used in several applications, including 3D modeling, training, skill evaluation, virtual assembly and robotic control [1]. The premium series provide 6 active DoF and cover a vast range of research and commercial applications, e.g. virtual prototyping, maintenance path planning and molecular modeling applications. Within this category, the Phantom Premium 3.0 provides a range of motion compatible with full arm movement pivoting at the shoulder (see Figure 1.15a, right).

The omega.6 (see Figure 1.15b left) provides 3DoF force feedback. It is considered by Force Dimension as the most advanced pen-shaped force feedback device available. Focused on ergonomics, this device enables the rendering of high contact forces and high stiffness thanks to an update rate of 4kHz. The sigma.7 (see Figure 1.15b, right) is a high sensitivity 7 active DoF device from Force Dimension that can be used in applications including medical and space robotics, micro and nano manipulation, bi-manual teleoperation, virtual simulations, training systems and research [43].

Haption S.A. provides a wide range of high performance force feedback interfaces. It proposes 3 or 6 DoF devices like the Virtuose 3D Desktop (see Figure 1.15c, left) and the Virtuose 6D TAO that can be used for VR and teleoperation applications (see Figure 1.15c, right). It is particularly well suited for scale 1 virtual/remote manipulations. It is also used for comanipulation in laparoscopic surgery as well as in rehabilitation [134].

The High Definition Haptic Device (HD²) from Quanser (see Figure 1.15d) is a high-fidelity 6 DoF haptic interface for advanced research in haptics and robotics. It is particularly suitable for the development of test beds for various emerging applications such as medical simulators and teleoperation. This haptic interface can track the operator's motion in 6 DoF and apply force to the user in 5 DoF [107].

The Desktop Haptic Interface from MOOG, Inc. (see Figure 1.15e) is a general purpose haptic interface, providing force feedback in 3 DoF. Its admittance control allows to provide realistic crisp touch and feel [31].

Finally, the Freedom 6S (see Figure 1.15f) from MPB Technologies Inc. is a high-fidelity force feedback device operating in 6 DoF, providing the user with the sense of touch in both virtual and real-world applications. It is ideally suited for medical and master /slave robotics [95].

It is worth noting that the development of haptic devices is part of an increasing market. More and more applications are being imagined every day, generating the continuous need for novel devices.

### 1.5.5 Limitations of existing haptic interfaces

As mentioned earlier, the user should not feel the presence of the robot in free space. All existing devices lack the design to answer this need. As a consequence, a perfect transparency is not attained. Transparency in free space could be improved by implementing accompanying control strategies.

Part of the problem is related to the fact that tactile feedback interfaces usually require the user to be mechanically linked to them in order to know the user's position and provide feedback that is consistent with the virtual scene. This link has a non negligible influence as the user experiences mechanical structure even when moving in free space. In this case the difference
<figure><img src="image_29.png" /><figcaption>Figure 1.15 – Examples of common commercial force feedback interfaces.</figcaption></figure>
between free space and contact is less distinctively felt and muscular fatigue is expected to
appear in case of long manipulations.

A relatively novel paradigm known as intermittent-contact aims to cope with the limitations of conventional devices in order to improve the realism of the haptic interaction and relief the user when long manipulations take place. Its principle and the advances in this field of research will be described in the next section 1.6.

1.6 Intermittent-contact interface

The intermittent-contact (IC) model proposes to remove the mechanical link between the
interface and the user in free space, obtaining a perfect transparency and letting the user touch
the haptic device only when a contact occurs in the VE. This way, intermittent-contact interfaces
(ICIs) aim to improve the realism of the haptic interaction.

Two categories of devices implementing the IC model can be identified. The first one is known as Encountered-Type haptic interfaces (ETHI) and the second one as Close-Tracking-type haptic interfaces (CTHIs). In the former, a robot manipulator, which end-effector has the shape of the simulated object, is controlled to make it be encountered the user's hand at the position where the VO is found. In the later, the device follows the user at a short and constant distance, without contact, and touches user only when a VO is reached.

These devices are particularly interesting to reduce the fatigue during long manipulations
and to perceive small interaction forces as it is the case in medicine, e.g. the force applied on an
organ. Their principle and relevance will be discussed in sections 1.6.1 and 1.6.2 respectively.

1.6.1 Encountered-type haptic interfaces

The ETHI principle consists in imagining that the virtual space is superimposed with the
real environment. The robot is then controlled to be positioned along its surface at the nearest
position of the user's hand and simulate the VO properties (see Figure 1.16). This way, the
user remains away from the robot in free space and he/she touches its end-effector only when
and where he/she is supposed to touch the VO, a perfect transparency is therefore achieved.
To do this, the user's position must be tracked, e.g. using video cameras or body-mounted
position tracking sensors, and the robot must be controlled so that it can rapidly and accurately
anticipate the user's movement intentions.

The ETHI approach was first proposed in [87]: a robot manipulator, which end-effector has the shape of the simulated object, is controlled to make it encounter the user's hand at the position where the VO is expected to be found. This way the user is totally free in free space and he/she can feel the corresponding interaction forces.

The ETHI approach was also used in [127] for the construction of a haptic space: the user's arm motion is measured by a passive master arm, allowing to compute the position and orientation of the user's fingertip. This information is used by a computer to calculate the nearest object to the user's hand in the virtual haptic space (see Figure 1.17). The local geometry of the simulated object near the contact point and its mechanical impedance are displayed by a 6DoF impedance controlled manipulator and a shape approximation device (SAD). The SAD possesses curved surfaces as well as convex and concave edges and its positioning and orientation are modified in order to simulate the desired object configuration.
<figure><img src="image_30.png" /><figcaption>Figure 1.16 – Principle functioning of encountered-type haptic interfaces.</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 1.17 – Virtual haptic space representation (adapted from [127]).</figcaption></figure>

Another ETHI system was proposed in [140][141], called WYSIWYF (What You can See Is What You can Feel), this concept allows the user to feel the object precisely where the user sees it. Employing vision-based tracking, the system can blend live video with the virtual scene, i.e. a portion of the user's hand is extracted from the captured image and is superimposed on the virtual scene (see Figure 1.18).

<figure><img src="image_32.png" /><figcaption>Figure 1.18 – Virtual haptic space representation [141].</figcaption></figure>
In [143], an ETHI device allowing the simulation of precision grasping tasks using three fingers (thumb, index and middle fingers) was presented (see Figure 1.19). The system consists of two modules, a base module composed of a robot manipulator (see Figure 1.19a) and a specifically designed contact module for finger interaction (see Figure 1.19b), together accounting for the necessary number of DoF to place a local surface patch any where in space. The positioning and orientation of the system are predicted based on the measure of the wrist speed profile and aperture distance between the fingers.

<figure><img src="image_33.png" /><figcaption>Figure 1.19 – Multiple finger ETHI [143]</figcaption></figure>

When working with ETHI devices, the user's hand is a kind of moving obstacle (see Fig.1.16a) that should be avoided (see Fig.1.16b). Miscalculations or system slowness can lead to painful collisions. In order to cope with this issue, path planning techniques are necessary.

In [142], a path planning algorithm for ET-HI devices that renders multiple VOs in 3D space was proposed. Here a convex polyhedron is constructed from the reference points of the virtual objects (see Fig.1.16c). As long as the user's hand is outside the polyhedron, the device stays on its surface. On the other hand, if the user's hand penetrates in it, the device avoids the user's hand and goes inside the polyhedron if necessary. Such algorithm not only considered the security of the user but also the efficiency of the device movement.

<figure><img src="image_34.png" /><figcaption>Figure 1.20 – Path-planning issue in ETHIs [142]</figcaption></figure>

As previously explained, ETHIs have the advantage over classical type haptic interfaces to
be perfectly transparent in free space as they are not at all in contact with the user's hand. With this approach however, the robot may reach very high speeds, which is dangerous for the user, e.g. if bilateral contacts with concave surfaces have to be simulated. Another issue is that the amount of objects that can be simulated is limited to a restricted set of pre-manufactured objects, which often display only very simple mathematical shapes.

### 1.6.2 Close-tracking-type haptic interfaces

The CTHI approach is another way to implement the IC model. Contrary to the ETHI principle where the device follows the surface of the VO, it proposes to let the device follow the user at a short and constant distance, without contact, and touch user only when a VO is reached (see Figure 1.21).

The CTHI principle requires to measure the distance between the device and the user's body (typically user finger), using e.g. sensors installed on the end-effector. The device will then closely follow the user's movements without any path planning as long as the user remains in the reachable volume (workspace) of the interface. Despite this limitation, the security of the user is improved. Several examples of such devices will be given below.

<figure><img src="image_35.png" /><figcaption>Figure 1.21 – Principle of functioning close-tracking-type haptic interfaces.</figcaption></figure>

In [59], the existence of an object is simulated thanks to a mechanism for surface displaying (see Figure 1.22). 3-DoF tracking of the index finger is accomplished by means of magnetic sensors integrated in a moving head and magnets mounted on the finger (see Figure 1.22a). When the finger is being displaced in free space (i.e. no VO is being touched), the moving force feedback head follows it (tracking mode).

When a VO is about to be touched, the display mode is activated, the force feedback head stops at the VO position and waits for the finger to contact its inner surface composed of a tube (see Figure 1.22b).
<figure><img src="image_36.png" /><figcaption>Figure 1.22 – Surface display [59]</figcaption></figure>

In [144], a 2-DoF arm (see Figure 1.23a) allows finger interaction with virtual worlds thanks to a ring-like end-effector equipped with a set of eight lightweight optical on-off sensors used to roughly estimate the fingertip position in the ring (see Figure 1.23b). When the finger is far away from a VO, the ring's position is controlled to keep the fingertip in its center. When the fingertip is near a VO, the ring moves closer to it. Finally, when the fingertip is in touch with a VO the display mode is activated (see Figure 1.23c). A force sensor placed between the ring and the tip of the arm allows then to measure the force applied by the arm on the fingertip.

<figure><img src="image_37.png" /><figcaption>Figure 1.23 – 2-DoF CTHI systems, adapted from [144]</figcaption></figure>

A similar approach was used in [48] for the development of a 2-DoF CTHI (see Figure 1.24a). The tracking is however improved to get better performances: the end-effector is instrumented with infrared proximity sensors allowing to reconstruct the finger shape and to precisely estimate the position of its center using distance measurements (see Figure 1.24b). The control of the device is more precise, especially regarding the transitions between free space and contact (see Figure 1.24c).
<figure><img src="image_38.png" /><figcaption>(a) 2-DoF touch and force display system</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>(b) Sensor principle</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>(c) Control</figcaption></figure>

<caption>Figure 1.24 – 2-DoF CTHI system, adapted from [48]</caption>

In [144], a three-dimensional haptic display composed of two arms and a cap-like end-effector attached to their tips was presented (see Figure 1.25a). The user can insert user finger in the cap to interact with the virtual world. The device's sensor system is composed of optical glass-fiber on-off sensors, allowing to roughly estimate the 6-DoF configuration (position and orientation) of the finger in the cap (see Figure 1.25b). In free space the device tracks and follows the fingertip without contact. At contact (when a virtual object is touched), the cap is controlled so its inner surface touches the finger (see Figure 1.25c).

<figure><img src="image_41.png" /><figcaption>Figure 1.25 – 3DoF CTHI system, adapted from [144]</figcaption></figure>

Finally, a recently developed CTHI presented in [22] allows two finger dexterous interactions (index and thumb fingertips) with digital mock-ups in VR (see Figure 1.26a). This interface should be able to precisely measure and follow both fingertips remotely without equipping the user with any marker on skin or nail, and serve as a contact surface when force feedback is required. To perform this, two 6-DoF end effectors were specially designed and tested. Each end-effector integrates 9 proximity sensors, 8 placed around the finger in two planes, the last one being in front of it (see Figure 1.26c). A 6-DoF (position and orientation) robot is associated with each finger and connected to a base plate fixed on the hand palm. All these elements are
associated with a positioning arm (see Figure 1.26b) in order to allow for as free as possible
hand movements.

<figure><img src="image_42.png" /><figcaption>(a) Two fingers close-tracking-type haptic interface.</figcaption></figure>

<figure><img src="image_43.png" /><figcaption>(c) Prototype end-effector with 9 proximity sensors.</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>(b) Interface main elements.</figcaption></figure>

<caption>Figure 1.26 – Two finger dexterous CTHI, adapted from [22].</caption>

1.7 Equipment

In this thesis, it is expected that specific hardware and software will be used. Some basic knowledge of this software is also required and challenges regarding installation are also expected. As explained in "Description of Context", this thesis is a part of Renault Group project. The selection of hardware components like robot, VR systems and software platforms are already established. Since the consortium had selected Universal Robots UR-5 as the collaborative robot and the use of HTC Vive for Virtual Reality display. Similarly the car model used in this thesis and definition of iteration points are also established by Renault Group.

The use of Unity3D software for virtual reality scene development has also been established. Since the final goal is to collaborate and combine the thesis with Renault Group project these established hardware and software have not been challenged or changed. For robot planning in this thesis, ROS has been selected because of its versatile features. This poses a challenge in cross-platform between Windows and Linux which is also addressed in Chapter 4.

1.8 Research proposal

Haptic feedback interfaces are intended to allow a user to interact naturally with a virtu-
al/remote environment with haptic feedback. Therefore, they should be both light and safe
enough so that they do not interfere with the user's movement in free space (transparency) and
mechanically stiff to provide realistic feedback. Several efforts can be made to achieve these
goals, e.g. by optimizing the mechanical design or by using force sensors (in case of force feed-
back systems) to measure and compensate the resistance of the device to the user's movements
in the direction of displacement. These solutions are however not sufficient to achieve a perfect
transparency and the performances of current feedback devices still require to be improved, in particular for tasks lasting long hours.

The ICI model, presented in section 1.6, aims to solve this problem by physically disconnecting the user for the device in free space and letting user touch the robot only when it is required, this way providing only the necessary amount of interaction. This principle can theoretically render feedback interfaces perfectly transparent, allowing to increase the realism of the interaction. It has been implemented in two ways: ETHI and CTHI devices.

In the former approach, the system anticipates the user's movements, positions the end-effector of the robot at the place where he/she is expected to touch a VO and waits for the user to encounter it. User tracking is commonly achieved by means of a motion capture system or a passive mechanical master arm attached to the user's arm. Such approach is however dangerous, as without meticulous path planning, the robot may follow trajectories near the user. Furthermore, if bilateral contacts are simulated the robot may attain very high speeds, compromising the security of the user.

Therefore intermittent contact interfaces, should be safe enough so that they do not interfere with the user's movement in free space (transparency). Several efforts can be made to achieve these goals, e.g. by optimizing the mechanical design (robot base placement wrt to the task points, planning and control of the robot trajectories)or by using sensors to measure and predict the user's movements in the direction of displacement. These solutions are however are a step forward to achieve a perfect transparency and the performances of current haptic feedback devices.

The later approach proposes to closely track (advantageously with a system integrated in the end-effector of the robot) and follow the user's movements (in position and orientation) without contact in free space and stop the robot when a VO is being touched to make the user feel the interaction forces. With this approach, the security of the user is improved, even if user movements are restricted to the workspace of the robot. However the system restricts the interaction of user with only maximum of 2 fingers. For this reason, in the present work we will focus on Encountered-type haptic interfaces.

It is worth noting that, despite their potential advantages, ETHI devices still suffer from some limitations. The objective of the doctoral thesis presented here is to implement a system for the texture perception. It is organized around three key questions that, according to the explored literature, have not yet been addressed for this type of devices. Each of these issues will be addressed in a specific chapter of this thesis, as explained in "Introduction".
# Chapter 2

## UR-5 configurations, workspace and placement

### 2.1 Introduction

Serial manipulators are used in many robotic systems like manufacturing, handling material, and teleoperation. There is an increasing number of these robots worldwide with some big names in that domain such as ABB and Kuka.

In recent years, Universal Robots have developed a series of robotic manipulators that are now widely used by many universities and industries. These are a special kind of serial robots which have a cobot mode. The ability for the robot to have a protective stop when violating a set defined boundary. Additionally this robot is claimed to be fast, easy to program, flexible, safe, and offers low-level programming access to the robot controller with a high cycle time [1]. Among the UR products, the family of UR-3, UR-5, and UR-10 have received great attention within the robotics community and industries specifically by the robotic research community.

UR-5 robot, Figure 2.1, is a well-known 6-degree-of-freedom (DoF) robotic manipulator manufactured by Universal Robots Company [130]. The most renowned feature of this robot is its agility due to its lightweight, speed, ease of program, flexibility, and safety. One of the main characteristics of the UR-5 is that the last three joints on it do not act as a coincidental wrist. Therefore, all its six joints contribute to the transformational and rotational movements of its end-effector.

In Figure 2.2 the schematic of the robotic arm and the allocation of each joint's frame are illustrated. It is mentioned that the DH parameters that are used in the kinematic model are based on these frames. The DH parameters of the UR-5 for the specified joint frames in Figure 2.2, are presented in Table 2.1.
<table><caption>Table 2.1 – Denavit-Hartenberg parameters for the UR arms [130].</caption><thead><tr><th>i</th><th>a<sub>i</sub></th><th>&alpha;<sub>i</sub></th><th>d<sub>i</sub></th><th>&theta;<sub>i</sub></th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>&mdash;</td><td>&mdash;</td></tr><tr><td>1</td><td>0</td><td>&pi;/2</td><td>d<sub>1</sub></td><td>&theta;<sub>1</sub></td></tr><tr><td>2</td><td>a<sub>2</sub></td><td>0</td><td>0</td><td>&theta;<sub>2</sub></td></tr><tr><td>3</td><td>a<sub>3</sub></td><td>0</td><td>0</td><td>&theta;<sub>3</sub></td></tr><tr><td>4</td><td>0</td><td>&pi;/2</td><td>d<sub>4</sub></td><td>&theta;<sub>4</sub></td></tr><tr><td>5</td><td>0</td><td>&minus;&pi;/2</td><td>d<sub>5</sub></td><td>&theta;<sub>5</sub></td></tr><tr><td>6</td><td>&mdash;</td><td>&mdash;</td><td>d<sub>6</sub></td><td>&theta;<sub>6</sub></td></tr></tbody></table>

<figure><img src="image_45.png" /><figcaption>Figure 2.1 – The real UR-5 manipulator [130].</figcaption></figure>

The context of study is the evaluation of perceived quality of a virtual car interior during the first design phases.

In a given scenario, the user sits in the real world with a visual virtual reality experience inside the car. The user wears a helmet and cannot see the robot, which explains the safety problem (Figure 2.3). While the user is trying to interact with the virtual object in the environment, the robot must come and position a sample of the material associated with the local surface, to provide a tactical sense of touching the object. Currently, the prop can carry six different materials (Figure 2.4). There is a cushion surface, a glossy glass surface, two different textures of rubber, a simple plastic surface and a smooth leather surface.
<figure><img src="image_46.png" /><figcaption>Figure 2.2 – Coordinate frames for UR-5 manipulator [130].</figcaption></figure>

<figure><img src="image_47.png" /><figcaption>Figure 2.3 – Conceptual scheme of the experimental platform.</figcaption></figure>
<figure><img src="image_48.png" /><figcaption>Figure 2.4 – (a) The UR-5 robot with a prop attach to the end-effector and (b) The prop used to carry six sample textures</figcaption></figure>

So in this scenario, we want the robot workspace to cover all the virtual environments (in this case the interior of the car) the user could interact. Figure 2.5 depicts inside a Dacia Duster and three areas to touch.

<figure><img src="image_49.png" /><figcaption>Figure 2.5 – Example of virtual environment and a set of task to be reached.</figcaption></figure>
<figure><img src="image_50.png" /><figcaption>Figure 2.6 – The complete system setup for human robot interaction.</figcaption></figure>

### 2.1.1 Safety standards

Possible collaboration of humans and robots in shared workplaces without perimeter guarding opens up new possibilities and concepts. The standardization and legal situation allows for human-robot collaboration (HRC) within defined limits.

#### Basics of Human-Robot Collaboration (HRC)

A collaborative robot is designed for direct interaction with a human within a collaborative workspace, meaning a common workspace where the robot and a human work simultaneously. In comparison to a traditional robot system, operators can work in close proximity to the robot system while power to the robot's actuators is available, and physical contact can occur within a collaborative workspace. The term collaboration refers to cooperation between humans and robots. This cooperation is limited to a precisely defined collaborative workspace.

Multiple definitions for levels of interaction between humans and robots exist. In one of the first definitions Helms et al. identified four types of interaction between a human and an industrial robot [58]:

- **Independent**: The human and the robot work on different workspace as in a traditional robotic cell.

- **Synchronized**: The human and the robot work consecutively on the same workspace in separate areas.

- **Simultaneous**: The human and the robot work on the same workspace at the same time without having physical contact.

- **Supportive**: The human and the robot work cooperatively together on the same workspace. Aaltonen et al. proposed a new framework based on a literature review and case study examples [2]. They suggested four levels of HRC which are further divided into sublevels through the factors: workspace sharing, robot's activity when the human is present, type of joint effort, and physical contact. This framework includes a wider variety of possible forms of HRC but encompasses a high level in detail and complexity. Wang et al. classified the relationship between human and robot based on five characteristics [137]:

- **Shared workspace**: working in the same workspace without fences.
<figure><img src="image_51.png" /><figcaption>Figure 2.7 – Interaction between robot and human.</figcaption></figure>

— Direct contact: direct physical contact between both entities exists.

— Shared work task: working on the same task but not simultaneously.

— Simultaneous process: working at the same time on the same or a different task.

— Sequential process: working one after another, meaning in sequential order.

In this thesis work, the term HRC refers to combination of supportive interaction with simulta-
neous process having direct contacts, shown in Figure 2.7.

2.1.2 Safety in human robot collaboration

The four modes of collaborative operations were first introduced in ISO 10218 and have been
further detailed in ISO/TS 15066. Collaborative operations should include one or more of the
following four safety methods: 1) Safety-rated monitored stop, 2) Hand-guiding, 3) Speed and
separation monitoring (SSM) and 4) Power- and force-limiting (PFL). ISO/TS 15066 defines
them in detail as follows [62]:

— Safety-rated monitored stop; robot stops when operator enters the collaborative workspace and continues when the operator has left the collaborative workspace.

— Hand guiding robot; movements of robot are controlled by the user.

— Speed and separation monitoring; contact between operator and moving robot is pre-
vented by the robot.

— Power and force limiting; contact forces between operator and robot are technically lim-
ited to a safe level.

The industrial HRC installations are limited by the current safety legislation. The existing stan-
dards provide rather guidance than clear definitions and do not clarify what sufficient safety
for the operator in collaborative operations is. They are loosely defined and complex. A high
inter dependency exists, meaning the standards build up on each other so that it takes longer to
look up the related definitions from previous standards. Further, different use case scenarios are
not specified, which makes the current standards too general and allows for multiple interpreta-
tion possibilities. As a result, companies cannot follow defined guidelines and depending on the
company’s internal safety level they interpret the recommendations in ISO/TS 15066 differently.

Safety schemes in HRC can be defined as pre-collision and post-collision systems [82][76]. These terms are directly related to the safety strategies collision avoidance and impact force limitation.

— Pre-collision systems: Collisions are prevented by stopping the robot motion, altering
its trajectory, alarming the operator with an audio output and moving back from the operator. Workspace monitoring systems detect the position of the human with external safety sensors.

- Post-collision systems: In case of an unexpected or unavoidable collision the harm to the human is minimized. This can be achieved with integrated joint torque sensors in the robot's internal safety control systems for collision detection which initiate a robot stop and mechanical compliant systems such as lightweight robot structures and robot skins which limit the maximum transmittable energy to the operator during an impact.

The disadvantages of post-collision systems are 1) incorrect defined limits might result in serious accidents, especially in combination with sharp tools or parts 2) frequent production stops in case of collision occurrence. The advantages of a pre-collision system are 1) visual or acoustic signals can warn the operator of a coming robot stop 2) a speed reduction can delay the time point of a motion stop. However, external sensors require more space around the robot because of the sensor detection time.

## Contacts between robots and humans

In this thesis, for HRC we want to have contact with the human. So the robot and human have to work in the same workspace with complex fence boundaries (as the interactive surface is complex). From the classification discussed in previous section we can say the goal of this new system is to have a hybrid of the existing classification. We establish a new safety constrains taking into the ISO/TS 15066, requirements of having at least one or more safety methods. In this process the robot should avoid unwanted collisions with the user. In this context we have two types of contacts: (i) task contacts and (ii) unwanted contacts. The goal is to have a the set safety methods in these 2 scenarios. The implemented safety methods are as follows:

1. Unwanted contacts are collisions that happen when the robot end-effector or robot body comes in contact with the user while it is not expected in the task description. Multiple safety standards are implemented, explained as below:

- The idea is to have more free space (space outside the reach of user workspace), possibly to move the robot at higher speeds in these non-contact space;

- To have limited robot speed depending on user workspace boundary, robot, tool and workspace;

- To have a fast and reliable collision recognition.

2. Task contacts are contacts that are related with the actual task of the robot and therefore they are expected and mainly initiated by the robot. They are located on the frontiers of the virtual environment and occurs between the end-effector and the human's hand.

- To have low robot speeds in areas where contact is possible;

- To limited contact force depending on region and shape of robot, tool and workspace;

- Making sure the velocity and force limits do not exceed.

Speed and force limiting must be implemented. Without risk assessment, human-robot collaboration cannot take place. The overall application must always be considered, i.e. not only the robot. Safety functions must be implemented in accordance to determined requirements. The initial step towards safety is based on the location of the robot. That has to be far during the no contact phase and be able to reach the human during the static contact. The process on how we achieve this discussed in this chapter.

To respect safety constraints and limit the risk of injury when the robot effector is in contact with the operator, the positioning of the robot must be optimized. This placement should also
take into account the size of the environment in which the operator has to interact in immersion.
Constraints will also come from the desired haptic rendering. Indeed, during interactions, the
properties (crossing singularity, self collision and precision of movement) of the robot must be
adapted to the interaction modes required. The interface placement will be fixed for a given
experiment if the size of the environment to be simulated is smaller than the cobot workspace.
So the initial objective is to find a placement of the robot that allows to reach all the desired
zone of interactions. If this is not possible, then to constrain the robot to limited workspace of
the robot, it is possible to consider several placements adapted to the task.

2.1.3 Methodology to achieve the safety standards in HRC

Before beginning any robot task, users must position the robot’s base, a task that was
performed on user intuition (in the beginning of Lobbybot Project). While slight perturbation
is tolerable for robots with movable bases, correcting the problem is imperative for fixed-base
robots, especially if some essential task sections are out of reach. For non-mobile manipulation
robots, it is necessary to decide on a specific base position before beginning manipulation tasks.
The manual work of identifying potential base location is defined effectively, so that, where the
position of the robot and workspace ensure the maximum amount of work performed, where the
maximum amount of workable area can be reached. Since complicated robotics tasks require
the base to be placed at unique poses based on task demand and also to consider time efficiency
and range of applicability.

Collision in the base placement planning is vital, as the output base location may be in colli-
sion with manipulated workspace, worktable, and other surroundings. Also, the base placement
does not depend only on the reachability of the task poses, the cost or minimum joint motions
should also be considered.

To manage these collisions, we come up on a strategy for safety of user in ICI scenario. We
consider multiple control factors like planning, workspace /working posture and placement.

Planning: There are two aspects of planning of robot motion in this scenario: (i) reaching the desired next ICI location safely, (ii) being able to track the user hand at the frontier of the virtual environment without having collision with the user, to prepare a future motion of the ICI to its desired location as soon as possible. Having a better planning control helps in avoiding collisions.

Workspace / Working posture: Constraining the robot’s working posture for the entire workspace reduces the risk of collisions. In practice this avoid reconfiguration of the robot to change of working posture (such as elbow up or elbow down configuration) and such large amplitude motion can lead to collision.

Placement: The placement of the robot with respect to the user and the environment has to be defined in order that the robot is able to carry the prop to the limit of the virtual environment that the user wants to explore while avoiding collision with the human. It is better if all the part of the robot is keep outside of the interior of the virtual car.

Find a convenient location for the robot is an not easy task and it is the main goal of this
chapter.

## 2.2 Robot configurations

For the UR-5 Workspace there are eight configurations that form a complete workspace.
The computation of these configurations is based on the number of inverse kinematics solutions
obtained for a given point of end-effector.

**Definition: Configuration Space:** All the possible values of $\theta_i$ it can take. For a six-axis robot is the set of all $\theta_i$ attainable with out collision.

**Definition: Workspace:** The set of points that can be reached by its end-effector without having any collision. The physical volume it can cover.

### 2.2.1 Inverse kinematics

In this section we will start with an analytical model that give all the solution of the inverse kinematic model. The analytic inverse kinematics problem is to find the set of joint configurations that satisfies desired end-effector position and orientation. Given the input of the end-effector position and orientation, we get the different set of possible joint configurations to reach the same end-effector position and orientation (For UR-5, 8 possible set of joint configurations). A detailed explanation of the UR kinematics and inverse kinematics can be seen in Appendix A. In this section we will only deal with the key final results to understand the classification of the configurations.

From inverse kinematics, for finding $\theta_1$ results in:

$$ \theta_1 = \operatorname{atan2}(^0p_{05y}, ^0p_{05x}) \pm \cos^{-1}\left(\frac{d_4}{R} + \pi/2\right) \quad (2.1) $$

There exist two solutions for $\theta_1$, where the shoulder is left or right. Using the function atan2 is essential for insuring correct signs and behavior when $^0p_{05y} = 0$.

Given a particular $\theta_1$, we can solve for $\theta_5$. The final equation for $\theta_5$ is:

$$ \theta_5 = \pm \cos^{-1} \frac{p_x s_1 - p_y c_1 - d_4}{d_6} \quad (2.2) $$

Again, there are two solutions for $\theta_5$, which correspond to configurations where the wrist is in/down or out/up. After computing $\theta_5$, $\theta_6$ can be computed based on the equation below:

$$ \theta_6 = \operatorname{atan2}(\frac{yyc_1 - ys_1}{s_5}, \frac{xyc_1 - ys_1}{s_5}) \quad (2.3) $$

The other three joints can be derived easily, considering that they act as a 3-RRR planar arm. Once the previous 3 joints found, the location of the base and end-effector of this 3-RRR arm is available, then these 3 joints can be solved. There is two possible configurations, elbow up or elbow down. No solutions exist when the distance to the 4th joint exceeds the sum $|a_2 + a_3|$ or is less than the difference $|a_2 + a_3|$. If $a_2 = a_3$, a singularity exists when $\theta_3 = \pi$, making $\theta_2$ arbitrary.

$$ \theta_2 = \alpha_1 - \alpha_2 = \arctan(-^1p_{14z}, -^1p_{14x}) - \arcsin\left(\frac{-a_3 \sin \alpha_3}{|^1p_{14xz}|}\right) \quad (2.4) $$

$$ \theta_3 = \pm \arccos \frac{|^1p_{4xz}|^2 - a_2^2 - a_3^2}{2a_2a_3} \quad (2.5) $$

$$ \theta_4 = \arctan(^3x_{4y}, ^3x_{4x}) \quad (2.6) $$

Based on the IK computation, we can summarize a total of 8 solutions exist in general for the general inverse kinematic problem of the UR-5: $2\theta_1 \times 2\theta_5 \times \theta_6 \times 2\theta_3 \times \theta_2 \times \theta_4$. The different configurations of robot as shown in Table 2.2. Figure 2.8 shows the visualization of the different
<table><caption>Table 2.2 – Different configurations for UR-5</caption><thead><tr><th>Configs</th><th>q<sub>1</sub></th><th>q<sub>234</sub></th><th>q<sub>5</sub></th></tr></thead><tbody><tr><td>Config0</td><td>Left</td><td>Up</td><td>Out</td></tr><tr><td>Config1</td><td>Left</td><td>Down</td><td>Out</td></tr><tr><td>Config2</td><td>Left</td><td>Down</td><td>In</td></tr><tr><td>Config3</td><td>Left</td><td>Up</td><td>In</td></tr><tr><td>Config4</td><td>Right</td><td>Up</td><td>Out</td></tr><tr><td>Config5</td><td>Right</td><td>Down</td><td>Out</td></tr><tr><td>Config6</td><td>Right</td><td>Down</td><td>In</td></tr><tr><td>Config7</td><td>Right</td><td>Up</td><td>In</td></tr></tbody></table>

configurations based for a single end-effector position and orientation. It can be seen that to have maximum safety for the user, it is better to avoid some configurations like 0, 1, 4 and 5 as these configuration have the part of the robot (yellow component) closer to the human than in other configurations. Also configuration 1, 2, 5 and 6 are not desirable as they have an elbow down configuration which is prone to collision with the robot platform and human. From Table 2.2 and Figure 2.8 it can be summarized that Configuration 3 and 7 are ideal, with key difference being orientation of $\theta_1$ (left/right). The selection of configuration is very important because the maximum reachability of the workspace is crucial to avoid changing configurations between execution of task. A detailed study on the workspaces generation based on configurations and workspace reachable spheres will be dealt in further sections.

## 2.3 Generate workspace

The next step is to construct the workspace of the robot. Further we constrain the workspace to one configuration and analyze them. The construction of the workspace is done taking into consideration both the base table and prop of the robot. Adding this information in universal robot description file (URDF). URDF is an XML file format in ROS to describes all elements of the robot. First we construct a workspace with a single fixed configuration (eg: Config 7). From the URDF we have the information of the dimensions of the robot, creating a rough spherical workspace W with those dimensions (Algorithm 1). Now, the workspace W is discretized with squares of size of 5 cm (Figure 2.9a).

Creating squares of 5 cm and creating a sphere at its center (Figure 2.9b). Now for each sphere, we have a set of poses that are to be checked. By poses, we mean different orientations to reach the same sphere. These poses are shown using the arrows in Figure 2.9c. A range of poses is considered, with an interval of 30°. The range of Azimuth is from [0,360°] and elevation of [0,180°]. So for every pose of the sphere, we find the IK solutions. If a solution exists for a given pose then the sphere is reachable. But for the present case, we want a specific solution to exist, as we want to construct the workspace for a desired configuration defined among config 0 to config 7, illustrated in Figure 2.8. So we check if the solution we get is of the desired configuration, then we store the sphere $s_i$ and pose $p_i$, and next, we compute the reachability index (RI). This is a factor that determines how many poses are reachable in a given sphere with the desired configuration. Using this RI value, we color code the whole workspace.

$$RI = \frac{(\text{No. of poses with solution})}{(\text{No. of total poses checked})} \times 100 \qquad (2.7)$$
<figure><img src="image_52.png" /><figcaption>(a) Configuration 0</figcaption></figure>

<figure><img src="image_53.png" /><figcaption>(c) Configuration 2</figcaption></figure>

<figure><img src="image_54.png" /><figcaption>(e) Configuration 4</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>(g) Configuration 6</figcaption></figure>

<figure><img src="image_56.png" /><figcaption>(b) Configuration 1</figcaption></figure>

<figure><img src="image_57.png" /><figcaption>(d) Configuration 3</figcaption></figure>

<figure><img src="image_58.png" /><figcaption>(f) Configuration 5</figcaption></figure>

<figure><img src="image_59.png" /><figcaption>(h) Configuration 7</figcaption></figure>

<caption>Figure 2.8 – All the configurations of the robot for same end-effector position and orientations.</caption>
<figure><img src="image_60.png" /><figcaption>(a) $V_i$ of workspace W</figcaption></figure>

<figure><img src="image_61.png" /><figcaption>(b) sphere $s_i$ for $V_i$</figcaption></figure>

<figure><img src="image_62.png" /><figcaption>(c) poses P for sphere $s_i$ in $V_i$</figcaption></figure>

<caption>Figure 2.9 – Discretization of the workspace into voxels.</caption>

**Algorithm 1 Construction of workspace**

**Input:** URDF of Robot, discretization size - n

**Output:** create hypothesized workspace Z.

$V_i$ - Discretization of W, with size n

**for** each $V_i$ in W **do**

create spheres $s_i$ for $V_i$

check for self collision

save $s_i$ in S

**end for**

**for** each $s_i$ in W **do**

Generate poses P

**for** each $p_i$ in P **do**

Find ik solution

**if** solution **then**

store $(s_i, p_j)$

**end if**

compute $ri = \frac{\text{reachable poses}}{\text{total number of poses}}$

store $ri$ with $(s_i, p_j)$

**end for**

**end for**
<table><caption>Table 2.3 – Color code for reachable sphere in workspace</caption><thead><tr><td>Color</td><td>Percentage</td></tr></thead><tbody><tr><td>Blue</td><td>91-100 %</td></tr><tr><td>Light Blue</td><td>51-90 %</td></tr><tr><td>Green</td><td>31-50 %</td></tr><tr><td>Yellow</td><td>6-30 %</td></tr><tr><td>Red</td><td>1-5 %</td></tr></tbody></table>

<figure><img src="image_63.png" /><figcaption>Figure 2.10 – Construction of complete workspace of UR-5.</figcaption></figure>

A workspace construction comprising of all the configurations was done (Figure 2.10). Further workspaces were constructed for each configuration (from config 0 to config 7). These configurations and the corresponding reachable spheres in the workspace is summarized in Table 2.4.

Figure 2.11 shows the workspace generated for Config7. The full workspace, and the dissected one. We consider the prop and the base table as our robot collision objects. A comparison of workspace for different configurations (in Table 2.4) shows that not many spheres are lost when selecting one specific configuration (Config7). The total number of spheres in Figure 2.10 are 4584, and that in Figure 2.11 is 4519. The color code (Table 2.3, in this case of configuration 7 is based in the reachability index.

So after the workspace has been constructed for a specific configuration the next step is to select the task points and find the base placement based on these task point. This will be dealt in the next section.
<table><caption>Table 2.4 – Reachable spheres in workspace.</caption><thead><tr><th></th><th>Reachable Spheres</th></tr></thead><tbody><tr><td>Whole Workspace</td><td><b>4584</b></td></tr><tr><td>Config0</td><td>3311</td></tr><tr><td>Config1</td><td>3311</td></tr><tr><td>Config2</td><td>4484</td></tr><tr><td>Config3</td><td>4484</td></tr><tr><td>Config4</td><td>3310</td></tr><tr><td>Config5</td><td>3310</td></tr><tr><td>Config6</td><td><b>4519</b></td></tr><tr><td>Config7</td><td><b>4519</b></td></tr></tbody></table>

<figure><img src="image_64.png" /><figcaption>(a) Complete workspace for config 7</figcaption></figure>

<figure><img src="image_65.png" /><figcaption>(b) Half sliced Workspace for config 7</figcaption></figure>

<caption>Figure 2.11 – Generated workspace for Config7, considering the prop and base table.</caption>
## 2.4 Regions of interest

To interact between the virtual environment and robot tool, we can specify two fixed orientations for the TCP. These orientations under test are obtained by considering three major regions in the interior of the car, for interaction. These regions are defined as $S_1$, $S_2$ and $S_3$. The reason for selecting these regions for computing the base location is, that they are the extreme regions in the virtual environment: the door, the dashboard, and the seat next to the user/driver.

Also, we will be discussing the surfaces of the prop that will be used for these tests. The prop that we consider has 6 surfaces. Figure 2.12 shows the major frames of interest in this test scenario. The TCP frame is the frame at the end of the robot without any attachment. We will need to consider two transforms from the TCP to the surface of the prop in contact. These 2 surfaces are Surface 5 and Surface 6 shown in Figure 2.12. Since joint $q_6$ of robot (UR-5) has [-2π,2π] limits, all five sides (Surface 5) of the prop are reachable by simple rotation. So when one surface is reachable, all the rest side surfaces of the prop are also reachable. So the problem is simplified and only one surface (Surface 5) is used.

So, transforming from TCP to Surface 6 is a simple translation in Z direction,

$$T = \text{TransZ}[15 \text{ cm}] \qquad (2.8)$$

Transform from TCP to Surface 5 is a translation in Z, followed by rotation in Y and a final translation in Z.

$$T = \text{TransZ}[2.87 \text{ cm}] \cdot \text{RotY}[63.44^{\circ}] \cdot \text{TransZ}[7.75 \text{ cm}] \qquad (2.9)$$

<figure><img src="image_66.png" /><figcaption>Figure 2.12 – The prop used in the experiment.</figcaption></figure>

For $S_1$ region as shown in Figure 2.13a, which is the door close to the user, we want for the robot being able to reach this surface with the prop surface 6 orientations for the given set of points $P_1$ ... $P_4$. This correspond to the orientation 1. The points must belong to the workspace $W_1$ shown in Figure 2.15a.

For the $S_2$ Region as shown in Figure 2.14, we can have a maximum surface inter-action with robot prop, we imposed that the surface 6 or surface 1 to 5 of the prop can be presented in order to test several material. A rotation of the last axis of the robot allows to change the surface of the prop from surface 1 to 5. To achieve this, the points has to be reachable by the two orientations. The points must belong to the workspace $W_3$ shown in Figure 2.16.

Finally, when considering $S_3$ region, the seat next to the user/driver, as shown in Figure 2.13b all the 5 surfaces of the prop were tested. So the orientation selected is Surface 5. In all these
<figure><img src="image_67.png" /><figcaption>Figure 2.13 – Different regions in the virtual environment.</figcaption></figure>

<figure><img src="image_68.png" /><figcaption>Figure 2.14 – S₂ - Region</figcaption></figure>
<figure><img src="image_69.png" /><figcaption>(a) Workspace build for orientation 1 (denoted $W_1$). (b) Workspace build for orientation 2 (denoted $W_2$).

Figure 2.15 – Workspaces with $S_1$ and $S_3$ region orientations.</figcaption></figure>

3 regions $S_1$, $S_2$ and $S_3$ we define 4 points $P_1... P_4$ that we want the robot to reach in each region. For this task, the points must belong to workspace $W_2$ shown in Figure 2.15b.

### 2.4.1 Workspace for given orientation

Now, as the region of interest for robot is set up and the workspace of the robot is created. The next step is to create workspace of the robot based on the fixed orientations for the robot.

The workspace describes the reachability of a given robot model by discretizing its environment, creating poses in the environment, and calculating valid IK solutions for the poses. The poses which are reachable by the robot are associated with discretized spheres. The reachability of each sphere in the environment is parameterized by a Reachability index.

Workspace is generated for a given set of orientations and a given configuration of the robot. The output workspace map will be stored all the sphere the robot can reach. Figure 2.15a and Figure 2.15b show the created workspace for the selected orientations (Surface 5 and Surface 6) using the configuration 7 of the robot.

Then, we generate the intersection of both workspaces that can reach both orientations. Figure 2.16 show the region that is reachable with both the orientations of the prop (all six surfaces of the prop). This intersection will be used in our further tests for the base placement of the Robot.

## 2.5 Base placement of the robot

### 2.5.1 Requirement

The base placement of the robot, is defined in order to be able to place the prop at the point that the human want to explore. Since the robot arm studied has dimension which are close to the dimension of human arm, the ideal robot location would be in the location where the user is seated. However to avoid collisions, we cannot place the robot there. In fact, we would like
<figure><img src="image_70.png" /><figcaption>Figure 2.16 – Workspace for both orientation by intersection of $W_1$ and $W_2$ to obtain $W_3$.</figcaption></figure>

the robot to be located outside the passenger compartment of the virtual car. It is expected that the user body (specially the chest and neck) should be outside the workspace of the robot, such that there will be no possibility of contact between body parts of the user and the robot’s arm. The robot placement should be such that it can reach all the desired contact regions in the required orientation.

## 2.5.2 Methodology

The cobot used is UR-5 from Universal Robots [99]. It is a 6 DOF robot able to interact with humans in a shared space or to work safely in close proximity. We constrain the posture of the robot to stay in a single aspect, i.e. a particular configuration. In this case, it is elbow up. This helps us in avoiding crossing singularities, during motions.

Now to find the best base placement of the robot, we discretize a set of all the possible base locations that can reach a given point. The set of possible location are expressed as a set of sphere of 8 cm of diameter that represent the paving of the space.

If the robot is fixed on the floor, the paving can include only sphere at the level of the floor. In more general case, as the one studied here, a 3D paving can be considered. The vertical coordinate of the best pose of the robot will defined the good height for the table supporting the robot.

The task is defined as previously by the set of points that we want to reach with a given orientation. The workspace generated for each orientation of region is also used. A point can be reach if it belongs to the workspace associated to the desired orientation (for example $W_1$).

The methodology can include any number of task points. For each point associated with one orientation i.e. one workspace $W_i$, all the sphere inside of the workspace is covered and placed at the point studied, and the corresponding pose of the robot is registered. The location of the base is associated to the sphere paving the possible placement of the robot. The number of point reachable from this base location is increased.

After considering all the point of interest (12 point in our study), the possible placement of
the robot to achieve the task is defined as the sphere where the number of reachable is maximum.

From this set of base locations, we remove the base locations that coincide with the location of the user. Depending on the number of desired points that can be reached, the points are categorized. This gives the possible robot base location more suited for this scenario.

The algorithm developed is based on the library Reuleaux [85] available via ROS and used the tool [70, 39].

<caption>**Algorithm 2** Base placement of the robot</caption>

**Input:** Reachability Map, Task Points.

**Output:** Set of robot base locations.

**function PROCEDURE 1 (create basemap)**

**for** each task point $P_i$ **do**

INVtransform from $P_i$ and obtain $b$ the base location of the robot

cluster $b$ and assign the spheres $S_k$

increment the index associated to sphere $S_k$

save $S_k$

**end for**

**end function**

**function PROCEDURE 2(base placement)**

access spheres $S_k$

find spheres with max base index

exclude the spheres from inside the car

**end function**

### 2.5.3 Results

Tests were performed to find the base placement for a given set of points the robot has to reach with a defined orientation. Results were found in robot base placement locations, outside the interior of the VR car. Different combinations of the regions were taken to analyze how the base location of the robot changes.

In the tested scenario, the dimension of the table under consideration is 75 cm high and 53 cm long and wide. The base of the robot is located in the middle of the table to ensure its stability. A total of 12 task points have been defined: four task points in $S_3$ region (Chair), four task points in $S_1$ region (Door), and four task points in $S_2$ region (Dash board) with two sets of different orientation. The possible robot base location are illustrated by blue spheres. In all the test cases the base location are outside the passenger compartment of car. Multiple robot base locations were found that could reach all the desired positions.

#### $S_1$ and $S_2$ Region

A total of 8 task points have been defined: 4 task points in $S_1$ region (Door), 4 task points in $S_2$ region (Dash Board), with 2 sets of different orientations. All six surfaces of the prop were tested to be reachable. It can be seen that the base location is not inside the interior of the car. This analysis also helps in deciding the base table dimensions which will be discussed in further sections.
<figure><img src="image_71.png" /><figcaption>Figure 2.17 – *S*₁ and *S*₂ region task points and base location.</figcaption></figure>
<figure><img src="image_72.png" /><figcaption>Figure 2.18 – $S_2$ and $S_3$ region task points and base location.</figcaption></figure>

## $S_2$ and $S_3$ Region

A total of 12 task points have been defined: 4 task points in $S_3$ region (Chair), 4 task points in $S_2$ region (Dash Board), with 2 sets of different orientations. All six surfaces of the prop were tested to be reachable. It can be seen that the base location is not inside the interior of the car. each sphere here is at a discretization of 8cm.

However when compared to the previous result the position of the base to more towards the right (Figure 2.18), unlike it was left in the previous case (Figure 2.17).

## $S_1$, $S_2$ and $S_3$ Region

A total of 12 task points have been defined with 4 task points in $S_3$ region (Chair), four task points in $S_1$ region (Door), and four task points in $S_2$ region (Dash Board) with two sets of different orientations. All six surfaces of the prop were tested to be reachable. It can be seen that the base location is not inside the interior of the car, also the number of base locations, when compared with Figure 2.17 and Figure 2.18, is reduced.

Unlike the previous results, the base location is a bit closer to the user. The maximum distance of the robot from the car is not ideal for the given base dimension of the table.
<figure><img src="image_73.png" /><figcaption>(a) Base Placement for regions $S_1$, $S_2$ and $S_3$.</figcaption></figure>

<figure><img src="image_74.png" /><figcaption>(c) Front View.</figcaption></figure>

<figure><img src="image_75.png" /><figcaption>(b) Side View.</figcaption></figure>

<figure><img src="image_76.png" /><figcaption>(d) Top View.</figcaption></figure>

<caption>Figure 2.19 – $S_1$, $S_2$ and $S_3$ region task points and base locations.</caption>

## 2.6 Obstacle avoidance and dimensions of base table

In the previous study, the obstacle of the environment (the chair and the human) was not taken into account since the location of the robot was not known. Thus the next step is to check for the selected pose location (in Figure 2.19a) that the points can be reached when the real environment are taken into account.

In this environment, we introduce a model of the chair and the human in rest position. For the user's body, the dimensions considered, for user cover 75 percentile of men and women of age group 25-55 in France. The volume dimensions of the cuboid are 185 cm in height and 60 cm in width and 30 cm in thickness (Figure 2.20c).

First the location of the robot corresponding to the actual height of the table is considered. For each pose location, the following task is considered: go from home position to each points successively.

From several candidate poses of the robot, the task cannot be achieved. For others, the task can be achieved. Two main problems were encountered:

An illustration of the result is shown for the case illustrated in Figure 2.20a. The configuration of the robot to reach a point on the passenger chair is shown in Figure 2.20c, the robot is close to the legs of the human. The configuration of the robot to reach a point on the dashboard is shown in Figure 2.20d, the robot configuration is convenient, it is far from the human.

To improve, the robot base placement, we can change the height of the table used. Several tests were done to determine the effect of changing the height of the table on the position of the base and its effect on the inclusion of the robot in the passenger compartment of the car. For
performing the tests, the task points shown in Figure 2.21a are considered.

When the robot tries to reach the chair task points, we can see from Figure 2.20c that the robot goes to the passenger compartment of the car. Conversely, the same problem does not occur in Figure 2.21c, as the height to the table is increased by 5 cm. Similar result is shown in Figure 2.21d with modified base table dimensions. It can be concluded that the current size of the table is not ideal for the scenario, and that a better design of the table can be done to improve the results.

It can be concluded that it is better to use a table of 80 cm height than 75 cm height for the given scenario.

## 2.7 Conclusions

Complete modeling of a well-known robotic manipulator UR5 was presented. For this purpose, first the robot's kinematic was characterized and calculated. Workspace and different configurations of the Robot were analyzed and visualized. The selection of configuration is very important because the maximum reachability of the workspace is crucial to avoid changing configurations between the execution of tasks.

The work presented allows to choose a robot placement (Figure 2.22) so that the end-effector reaches the areas of interest in the environment while ensuring the absence of collision between the robot and the user. An algorithm has been proposed to define convenient placement of the robot base in order to reach several areas of interest. First results are obtained using a given height of the table that support the robot base. The results show an accessibility of the task but also show a robot insertion in the passenger compartment. Changing this height setting prevents this intrusion and therefore improves user safety.

Based on the study performed the following results were found. The ideal location of the robot is 80 cm above ground level (Figure 2.23).
<figure><img src="image_77.png" /><figcaption>(a) Robot base locations in blue and task points with desired orientations in pink</figcaption></figure>

<figure><img src="image_78.png" /><figcaption>(b) Robot interaction at chair without user model</figcaption></figure>

<figure><img src="image_79.png" /><figcaption>(c) Robot interaction at chair with user model</figcaption></figure>

<figure><img src="image_80.png" /><figcaption>(d) Robot interaction at dash with user model</figcaption></figure>

<caption>Figure 2.20 – Introduction of User model into environment - Table height of 75 cm.</caption>
<figure><img src="image_81.png" /><figcaption>(a) New table height in Blue and task points in pink</figcaption></figure>

<figure><img src="image_82.png" /><figcaption>(b) Robot interaction at chair without user model</figcaption></figure>

<figure><img src="image_83.png" /><figcaption>(c) Robot interaction at chair with user model</figcaption></figure>

<figure><img src="image_84.png" /><figcaption>(d) Robot interaction at dash with user model</figcaption></figure>

<figure><img src="image_85.png" /><figcaption>(a) The computed robot placement</figcaption></figure>

<figure><img src="image_86.png" /><figcaption>(b) Robot interaction with user</figcaption></figure>

<caption>Figure 2.22 – Placement analysis of the base table of the robot.</caption>
<figure><img src="image_87.png" /><figcaption>Figure 2.23 – New Table design, 80 cm height.</figcaption></figure>
# Chapter 3

# Design of a user mannequin model for
ROS

## 3.1 Introduction

In this chapter we will introduce the design of the user model used in this thesis. The main objective of this approach is to ensure the safety of the operator. Collecting knowledge of the scene and creating a model of the human. Alongside we also want to use the information to predict the goal state of the robot trajectory. The goal state can be predicted via the final place of the human hand knowing its current pose.

This can be analyzed by observing the movement of a human through a motion capture system. A motion capture system is a system able to track the motion of an object moving in space. The purpose of motion capture is to record only the movements of the actor, not his or her visual appearance. This animation data is mapped to a 3D model so that the model performs the same actions as the actor. In our case, the motion of interest will be that of the different parts of the body of the subject. We will use a motion capture system based on rigid bodies.

The scene (Figure 3.1) is composed of the chair of the car which is fixed, the support of the robot is fixed and known. The robot movement is known. The human is seating in the chair (torso and legs fixed), its arms are moving.

This system will provide us with the position of each of the parts of the body of the subject at different instants. The whole process of analyzing the motion of a human subject can be divided into three different steps. The first step consists of building a model of the subject. In order to do this, we need to get different measurements from the subject, which will allow us to characterize the Human model.

The second step consists to estimate the necessary joint values in order to achieve the observed movement. Finally, the third step consists of visualizing them. We want to capture or track the motion of the user's hand in the given environment. To achieve this, we use the VIVE trackers and the HTC VR set.

### 3.1.1 Using kinect to know the environment

Using Kinect is useful for detecting any unexpected objects in the environment. It helps to get a rough idea of the objects in the environment. However, the location of the camera is difficult to cover the entire scene and also Kinect's constructed information has no information
<figure><img src="image_88.png" /><figcaption>Figure 3.1 – Environment scene setup.</figcaption></figure>

about an object obstructed by another object.

As shown in Figure 3.2, the information on the wall behind the objects is not known. However, this information can be used to know the form information of any object in the environment. This knowledge can be used for emergency stops if the robot is interrupted by an obstacle. These shortcomings in using Kinect has prompted to design a new approach to know the user presence in the environment.

## 3.2 Selection of human model

A human can be represented as an articulated rigid body. The arm can be a classical 7 dof arm. The pose of specific bodies can be measured via sensors (Vive tracker). In the studied scenario, it is assumed that the user is seated in the chair (the motion of the trunk is limited, the leg is assumed fixed). His/her hand motion is of specific interest.

In this section, we will describe the process of building an approximated model of the subject.

The model that has been chosen to describe the body of the subject is based on the modified Hanavan model (Figure 3.3b). The proposed model is a quite simple enough for the purpose of this project. This model approximates the human body by a set of 6 different rigid body segments, as shown in Figure 3.4.

These segments are defined as simple geometric shapes. This way, this model allows us to describe the body of a subject. For this test case, we want to measure only one dimension and relate the rest of the dimensions based on it. For this, we use Winter's model shown in (Figure 3.3a). This relation helps in measuring only the height and relating all other dimensions based on it.

From the height (measured using the HMD of the Vive set), the dimensions that define each of the segments can be directly obtained as a relation to height (Figure 3.3a). Then, from the measurements of the total height of the subject, the length of each of the segments can be estimated through those relations. This feature helps in having a model that can be scaled based on the user's height (Figure 3.12).
<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /><figcaption>Figure 3.2 – The environment scanned using Kinect.</figcaption></figure>

<figure><img src="image_91.png" /><figcaption>(a) Anthropomorphic human dimensions, c
Winter, DA, 1990.</figcaption></figure>

<figure><img src="image_92.png" /><figcaption>(b) The modified Hanavan model.</figcaption></figure>

<caption>Figure 3.3 – Models referenced in this experiment.</caption>
## 3.3 Description of the motion capture system

In this section, we will describe the motion capture system used to capture the movements of the subject. This motion capture system is an indoor whole-body measurement system based on HTC Vive Technology and located in a dedicated room under some specific conditions, as we explain below. The system is used to help mimic the user motion based on a custom user model that will be designed and explained in further sections. The system returns five frames (corresponding to the four segments of the body's model plus an additional one corresponding to the floor), as a function of time.

The placement of the sensors is shown in Figure 3.6. The motion capture system consists of two base stations (timed infrared emitting stations), two Vive-trackers are fixed to the different parts of the body of the subject, and a calibration tool.

- Two base stations also known as the lighthouse tracking system are two black boxes that create a 360-degree virtual space. The base stations emit timed infrared pulses at 60 pulses per second that are then picked up by the headset and controllers with sub-millimeter precision. They are placed at the corners of the motion capture area (delimiting the space where the subject can move) and looking towards the middle of the defined space.

- The VIVE trackers are a set of frames designed to be attached to different parts of the body. These trackers (as can be seen in Figure 3.6 and Figure 3.5) that are easily traceable by the base stations. The trackers are rigid and transmit information on the position and orientation of the segment.

- The calibrating tool is another Vive-tracker used to calibrate the position and orientation of the cameras w.r.t the real world. In order to do so, it is necessary to place this tool in a specific location, to link both the real and virtual environments. This has to be done every time the system is initialized, or for some reason if one of the base stations has somehow been moved with respect to the other.

## 3.4 Construction of human model

For the given scenario, we use four Vive-Trackers and one HMD. These act as markers for the tracking system. The major measurements required to construct the user are his arm dimensions and the distance of head w.r.t to base and shoulder. The model constructed (Figure 3.4), assumes that the user is seated on a chair. The whole model is defined in a URDF. Since the human arm is a 7 dof system, with two spherical joints each at the shoulder and wrist and a single revolute joint near the elbow. We use two trackers each on each arm. These trackers are positioned one on the hand, and the other closer to the elbow. From the trackers the position and orientation are collected. The positioning of the trackers is based on the key position information required (eg: accurate position of the user hand). Figure 3.7 shows the frames of the model and the trackers. The tracker frames $V_s$ and $V_w$ can be seen at the translation in Z of 5 cm and 2 cm respectively. This translation is the approximated distance of the tracker position from the bones of the user arm. Other key dimension like W, $L_1$ and $L_2$ are calculated from the height of the user, on the basis of the Winter model.

Calculation of the joints values is done based on the tracker values. Each tracker gives the rotation and translation with respect to a fixed frame.
<figure><img src="image_93.png" /><figcaption>Figure 3.4 – The model visualized in Rviz.</figcaption></figure>

<figure><img src="image_94.png" /><figcaption>Figure 3.5 – Tracking system used.</figcaption></figure>
<figure><img src="image_95.png" /><figcaption>Figure 3.6 – Front and side view of the subject attached with the trackers.</figcaption></figure>
<figure><img src="image_96.png" /><figcaption>Figure 3.7 – Model for right arm.</figcaption></figure>

### 3.4.1 Calculation of the configuration of the arm

We are concerned mainly about the hand of the subject. So in this section, we will deduce the equations that obtain the joint values of the arm, for a given set of positions transmitted by the trackers in time.

We will analyze the mechanism of the arm system shown in Figure 3.7. This system consists of 7 joints to be controlled using two trackers. We collect information of position and orientation from both the trackers (6 by each tracker, 3 position and 3 orientations) and the objective is to obtain the 7 joint values of the arm model. The mechanism starts with the shoulder frame $F_s$ followed by three revolute joints ($q_1, q_2$ and $q_3$). The shoulder and elbow are separated by a distance $L_1$. Then there is the elbow joint $q_4$. Frame $V_s$ is related to the vive tracker frame near the user’s elbow. From the elbow we have again three revolute joints ($q_5, q_6$ and $q_7$). These frames constitute the wrist joints. The elbow and wrist are separated by a distance $L_2$. From the wrist frame we have a hand link separated by W. The frame $V_w$ is related to the vive tracker on the hand of the user. Now to compute the joints and mimic the user movement we need to actuate the joint based on the information from the vive trackers attached to the user.

Computing the transformation matrix from the $F_s$ frame to the joint $\theta_3$ of the arm model leads to a $T_{03}$ matrix. Tracker information ($V_s$) is known and is connected to the upper-arm (the tracker information in $V_s$ is w.r.t reference frame). The transform from reference frame (base tracker) to $F_s$ is known. The vive tracker connected to upper arm allows to define the three first rotation of the shoulder based on the orientation information given by the vive tracker.

$$R_{03} = \begin{bmatrix} -\cos(\theta_1) \sin(\theta_2) \cos(\theta_3) - \sin(\theta_1) \sin(\theta_3) & \cos(\theta_1) \sin(\theta_2) \sin(\theta_3) - \sin(\theta_1) \cos(\theta_3) & -\cos(\theta_1) \cos(\theta_2) \\ -\cos(\theta_2) \cos(\theta_3) & \cos(\theta_2) \sin(\theta_3) & \sin(\theta_2) \\ -\sin(\theta_1) \sin(\theta_2) \cos(\theta_3) + \cos(\theta_1) \sin(\theta_3) & \sin(\theta_1) \sin(\theta_2) \sin(\theta_3) + \cos(\theta_1) \cos(\theta_3) & -\sin(\theta_1) \cos(\theta_2) \end{bmatrix} \quad (3.1)$$

$$\theta_1 = 2 \tan^{-1}(R_{03}(3, 3), R_{03}(1, 3)) \quad (3.2)$$

$$\theta_2 = 2 \tan^{-1}(R_{03}(2, 3), \sqrt{(R_{03}(2, 1))^2 + (R_{03}(2, 2))^2}) \quad (3.3)$$

$$\theta_3 = 2 \tan^{-1}(-R_{03}(2, 2), R_{03}(2, 1)) \quad (3.4)$$
For joint $\theta_4$, we know the transform matrix information of tracker $V_s$ and joints $\theta_1, \theta_2$ and $\theta_3$. For a point on the forearm segment the translation in x-axis (in $F_s$ frame) will be same for $T_{04}$ and $V_s$. Using the tracker information from $V_s$ and taking that position in x value and equating it with the transform matrix from $T_{04}$. Gives the equation 3.5, where $x_1 = 2$ cm.

$$P_x = -\left(-\left(\cos \theta_1 \sin \theta_2 \cos \theta_3 - \sin \theta_1 \sin \theta_3\right) \sin \theta_4 - \cos \theta_1 \cos \theta_2 \cos \theta_4\right)L_2 - \left(-\left(\cos \theta_1 \cos \theta_2 \sin \theta_3 - \sin \theta_1 \cos \theta_3\right)\right)L_1 + x_1 \quad (3.5)$$

This equation can be written as:

$$P_x - (C) = (A) \sin \theta_4 + (B) \cos \theta_4, \quad (3.6)$$

with,

$$A = (\cos \theta_1 \cos \theta_2 \cos \theta_3 + \sin \theta_1 \sin \theta_3)L_2$$

$$B = -\cos \theta_1 \sin \theta_2 L_2$$

$$C = (\cos \theta_1 \cos \theta_2 \sin \theta_3 + \sin \theta_1)L_1 + x_1$$

Equation 3.6 is a type 2 equation [71] and the angle $\theta_4$ can be deduced :

$$\theta_4 = \operatorname{atan2}(\sin \theta_4, \cos \theta_4) \quad (3.7)$$

Equation 3.7 gives two solutions but only one can be true due to the limited angle of the elbow joint ($180^\circ > \theta_4 > 0$).

Now, for computing $\theta_5$, $\theta_6$ and $\theta_7$ using rotation part of $T_{47}$, which is

$$R_{47} = \begin{bmatrix} -\cos(\theta_5) \sin(\theta_6) \cos(\theta_7) - \sin(\theta_5) \sin(\theta_7) & \cos(\theta_5) \sin(\theta_6) \sin(\theta_7) - \sin(\theta_5) \cos(\theta_7) & -\cos(\theta_5) \cos(\theta_6) \\ -\cos(\theta_6) \cos(\theta_7) & \cos(\theta_6) \sin(\theta_7) & \sin(\theta_6) \\ -\sin(\theta_5) \sin(\theta_6) \cos(\theta_7) + \cos(\theta_5) \sin(\theta_7) & \sin(\theta_5) \sin(\theta_6) \sin(\theta_7) + \cos(\theta_5) \cos(\theta_7) & -\sin(\theta_5) \cos(\theta_6) \end{bmatrix} \quad (3.8)$$

Since transform matrix information of tracker $V_w$ and joints $\theta_1, \theta_2, \theta_3$ and $\theta_4$ are known, $T_{47}$ can be computed.

$$T_{47} = T_{40}T_{07} \quad (3.9)$$

$T_{07}$ is taken from the $V_w$ tracker and $T_{40}$ can also be known as we know $\theta_1, \theta_2, \theta_3$, and $\theta_4$.

$$\theta_5 = 2 \tan^{-1}(R_{47}(3, 3), R_{47}(1, 3)) \quad (3.10)$$

$$\theta_6 = 2 \tan^{-1}(R_{47}(2, 3), \sqrt{(R_{47}(2, 1))^2 + (R_{47}(2, 2))^2}) \quad (3.11)$$

$$\theta_7 = 2 \tan^{-1}(-R_{47}(2, 2), R_{47}(2, 1)) \quad (3.12)$$

## Extend the model for both arms

The model explained so far has been extended for both hands, to get complete motion capture information of both arms. This setup gives complete information about the arms that can be viewed as dynamic obstacles for the robot in the environment.

This can help in improving the safety of the user. Also, an additional joint is introduced at the waist to introduce two rotations at the hip and to model the rotation of the torso. These joints are calculated according to the information provided by the trackers. Figure 3.8 shows the complete structure of frames. As explained before in section 3.3 five sensors are used. The
<figure><img src="image_97.png" /><figcaption>Figure 3.8 – Structure for both arms.</figcaption></figure>

<figure><img src="image_98.png" /><figcaption>(a) Transform frames for left hand.</figcaption></figure>

<table><caption>(b) D-H parameter table.</caption><thead><tr><th>Link</th><th>σ<sub>i</sub></th><th>q<sub>i</sub></th><th>a<sub>i</sub></th><th>α<sub>i</sub></th><th>d<sub>i</sub></th></tr></thead><tbody><tr><td>s - 1</td><td>0</td><td>q<sub>1</sub></td><td>0</td><td>0</td><td>0</td></tr><tr><td>1 - 2</td><td>0</td><td>q<sub>2</sub>+π/2</td><td>0</td><td>π/2</td><td>0</td></tr><tr><td>2 - 3</td><td>0</td><td>q<sub>3</sub>+π/2</td><td>0</td><td>π/2</td><td>0</td></tr><tr><td>3 - V<sub>s</sub></td><td>2</td><td>0</td><td>L1-0.05</td><td>3π/2</td><td>0</td></tr><tr><td>3 - 4</td><td>0</td><td>q<sub>4</sub></td><td>L1</td><td>3π/2</td><td>0</td></tr><tr><td>4 - 5</td><td>0</td><td>q<sub>5</sub></td><td>L2</td><td>π/2</td><td>0</td></tr><tr><td>5 - 6</td><td>0</td><td>q<sub>6</sub>+π/2</td><td>0</td><td>π/2</td><td>0</td></tr><tr><td>6 - 7</td><td>0</td><td>q<sub>7</sub>+3π/2</td><td>0</td><td>π/2</td><td>0</td></tr><tr><td>7 - V<sub>w</sub></td><td>2</td><td>0</td><td>W</td><td>3π/2</td><td>0</td></tr></tbody></table>

<caption>Figure 3.9 – Model for left arm.</caption>

connection of these sensors to the user model and physical world is shown. A base sensors acts as the reference frame for the real and virtual world elements.

A change between the models for each arm (Figure 3.7a and Figure 3.9a) is in the direction of axis, especially for the tracker frames $V_w$ and $V_s$. Applying the similar approach explained in section 3.4.1 based on the model shown in Figure 3.9. Gives these below equations:

$$R_{03} = \begin{bmatrix} \cos(\theta_1) \sin(\theta_2) \cos(\theta_3) + \sin(\theta_1) \sin(\theta_3) & \cos(\theta_1) \sin(\theta_2) \sin(\theta_3) + \sin(\theta_1) \cos(\theta_3) & \cos(\theta_1) \cos(\theta_2) \\ \cos(\theta_2) \cos(\theta_3) & \cos(\theta_2) \sin(\theta_3) & \sin(\theta_2) \\ \sin(\theta_1) \sin(\theta_2) \cos(\theta_3) - \cos(\theta_1) \sin(\theta_3) & -\sin(\theta_1) \sin(\theta_2) \sin(\theta_3) - \cos(\theta_1) \cos(\theta_3) & \sin(\theta_1) \cos(\theta_2) \end{bmatrix} \quad (3.13)$$

$$\theta_1 = 2 \tan^{-1}(R_{03}(3, 3), R_{03}(1, 3)) \quad (3.14)$$

$$\theta_2 = 2 \tan^{-1}(R_{03}(2, 3), \sqrt{(R_{03}(2, 1))^2 + (R_{03}(2, 2))^2}) \quad (3.15)$$

$$\theta_3 = 2 \tan^{-1}(R_{03}(2, 2), R_{03}(2, 1)) \quad (3.16)$$
$$P_x = -((\cos \theta_1 \sin \theta_2 \cos \theta_3 - \sin \theta_1 \sin \theta_3) \sin \theta_4 - \cos \theta_1 \cos \theta_2 \cos \theta_4)L_2 - (\cos \theta_1 \cos \theta_2 \sin \theta_3 - \sin \theta_1 \cos \theta_3)L_1 + x_1 \quad (3.17)$$

where $x_1 = 2$ cm,

$$P_x - (C) = (A) \sin \theta_4 + (B) \cos \theta_4, \quad (3.18)$$

with,

$$A = -(\cos \theta_1 \cos \theta_2 \cos \theta_3 + \sin \theta_1 \sin \theta_3)L_2 \\ B = \cos \theta_1 \sin \theta_2 L_2 \quad (3.19)$$

$$C = -(\cos \theta_1 \cos \theta_2 \sin \theta_3 + \sin \theta_1)L_1 + x_1$$

This gives the equations for $\theta_4$ as,

$$\theta_4 = \operatorname{atan2}(\sin \theta_4, \cos \theta_4) \quad (3.20)$$

Now, for computing $\theta_5$, $\theta_6$ and $\theta_7$ using rotation part of $T_{47}$, which is

$$R_{47} = \begin{bmatrix} \cos(\theta_5) \sin(\theta_6) \cos(\theta_7) - \sin(\theta_5) \sin(\theta_7) & \cos(\theta_5) \sin(\theta_6) \sin(\theta_7) - \sin(\theta_5) \cos(\theta_7) & \cos(\theta_5) \cos(\theta_6) \\ \cos(\theta_6) \cos(\theta_7) & -\cos(\theta_6) \sin(\theta_7) & -\sin(\theta_6) \\ \sin(\theta_5) \sin(\theta_6) \cos(\theta_7) - \cos(\theta_5) \sin(\theta_7) & \sin(\theta_5) \sin(\theta_6) \sin(\theta_7) + \cos(\theta_5) \cos(\theta_7) & \sin(\theta_5) \cos(\theta_6) \end{bmatrix} \quad (3.21)$$

Since transform matrix information of tracker $V_w$ and joints $\theta_1$, $\theta_2$, $\theta_3$ and $\theta_4$ are known, $T_{47}$ can be computed.

$$T_{47} = T_{40}T_{07} \quad (3.22)$$

where $T_{07}$ is taken from the 2nd tracker and $T_{40}$ can also known as we know $\theta_1$, $\theta_2$, $\theta_3$, and $\theta_4$.

$$\theta_5 = 2 \tan^{-1}(R_{47}(3, 3), R_{47}(1, 3)) \quad (3.23)$$

$$\theta_6 = 2 \tan^{-1}(-R_{47}(2, 3), \sqrt{(R_{47}(2, 1))^2 + (R_{47}(2, 2))^2}) \quad (3.24)$$

$$\theta_7 = 2 \tan^{-1}(-R_{47}(2, 2), R_{47}(2, 1)) \quad (3.25)$$

### 3.4.2 Calculation of movement in torso

So far we have calculated the hand pose and arm configurations of the human model. We are also interested in the torso motion of the user seated on the chair. For torso motion, we consider a simple rotation between the hip and head/neck. We assume that the user can bend forward, backward, sideways, and also be able to turn. We assume that when he/she bends there is no curvature of his/her back.

To achieve these values we use the information from the trackers placed near the elbow. From the position of the tracker $V_s$ near the elbow we deduce the position of the shoulder frame $F_s$ of the model. This frame $F_s$ is used as the main reference information. We know the distance between the two shoulder frames from the winter model, based on the user height. Now we have the position of both the shoulder frames $F_s$ from both trackers on each hand.

The neck frame is based on the two shoulder frames. It is known that human can have cartesian displacement at shoulder and we assumed that the displacement are symmetric at the two shoulders. The hip frame is assumed to be constant since the user is seated. For the user model, we assume that the shoulder frames are always aligned in a line.
<figure><img src="image_99.png" /><figcaption>Figure 3.10 – Transform frames and D-H parameters for chest.</figcaption></figure>

In Figure 3.10 shows the connection between the frames. The shoulder frames Left and Right are computed based on the position of the vive trackers. Now the goal is to compute the chest_psi ($\psi$), chest_theta ($\theta$) and chest_psi ($\phi$) to have the information about rotations about the torso. It is to be noted that the model designed is disconnected at the shoulders. So the position of the user torso and legs is based on the base tracker that is used (explained in Figure 3.8).

Based on the shoulder frames the neck frame is calculated. From the base reference sensor, using the DH table in Figure 3.10 the position of the hip frame is defined. First, using the shoulder frames a line is drawn passing through the three frames (two shoulder frames and neck). This line S is used to compute the angles of rotation for the torso.

Projecting line S onto the chest frame gives the rotation information of the torso. Projection of line 'S' onto XY, YZ, and ZX planes of chest frame. The dot product of line 'S' and the projected line on the planes gives rotation about the respective axis. Now have all the required motions of the user, next tests were carried out to check the accuracy of the model.

# 3.5 Analysis of user model

## 3.5.1 Accuracy of arms

Within the model, two small points have been created within the humerus and palm links, which represent the location of the sensors in the user as represented in Figure 3.13. To test the accuracy in arms of the model, two additional frames ($U_s$ and $U_w$) were introduced in the model which are positioned such that they align with the real frames for the tracker ($V_s$ and $V_w$). The frames $U_s$ and $U_w$ are child frames to the frames $q_3$ and $q_7$ of the model. So the hypothesis is using the information from the vive trackers ($V_s$ and $V_w$), the joints of the arms are found. So based on the joint values the position of the frames $U_s$ and $U_w$ are found. So to check the accuracy of the model an error between the values of the frames $U_s$ and $V_s$ is computed.

This analysis was done and the resulting measure of the error computed for $U_s$ and $V_s$ for each arm (right and left) can be seen in Figure B.1 and Figure B.2. The errors computed are magnitude less than 1 cm.
<figure><img src="image_100.png" /><figcaption>Figure 3.11 – Transform frames for estimation of error between shoulder frames.</figcaption></figure>

Similar analysis was done and the resulting measure of the error computed for $U_w$ and $V_w$ for each arm (right and left) can be seen in Figure B.3 and Figure B.4. The errors computed are magnitude less than 1 cm.

### 3.5.2 Accuracy of torso

The key frames computed to simulate the torso are the shoulder frames as explained in section 3.4.2. It was known that using the tracker we compute the shoulder frames ($F_s$). Now to make the error analysis we add two additional frames on the torso part of the user model. These are the expected shoulder frames based on the Hanavan model (ideally we want the shoulders to be connected with the body, Figure 3.11). Calculation of the error between the real and the computed position of shoulder frames was done to test the accuracy of the model. Figure B.5 shows the error computed between the real and estimated position for the right shoulder. It can be seen that the error is close to 0.1 cm. The same can be observed for the left shoulder in Figure B.6.

A video demonstrating the working of the user model can be found here.

### 3.5.3 Scalability of the model

The mannequin also has the option of scalability (Figure 3.12), having a different size proportionate to the user's height to have a better relationship between the real world and the model.
<figure><img src="image_101.png" /><figcaption>Figure 3.12 – Scalability of the mannequin model for a height of 1.6m and 2m.</figcaption></figure>

## 3.6 Conclusions

To model the user, a mannequin was defined in a URDF robot model. The hip of the model is fixed, while the arms are structured as 7 DoF serial robot with all revolute joints, where the first 3 constitute the shoulder, the 4th joint represents the elbow and the last 3 rotating joints represent the wrist of the arm.

User test was done to test the accuracy and response of the model designed. In the scenarios the user was seated and was asked to move the hand and the torso. Two types of analyses were performed. The first one was to check the accuracy of the sensors position in real life and in the model. The second was to find the accuracy in the shoulder frames computed during the calculation of the torso rotations.
<figure><img src="image_102.png" /><figcaption>Figure 3.13 – User with the HTC vive trackers</figcaption></figure>

<figure><img src="image_103.png" /><figcaption>Figure 3.14 – Complete mannequin model of the user.</figcaption></figure>
# Chapter 4

## System architecture and robot
trajectory planning

### 4.1 Introduction

So far we have established the base location of the robot and the occupancy of user in the environment. This chapter is intended to provide an insight into the tools used in the development of the experiments, such as the software architecture, data flow between softwares used, a description of the system's environment, and communication between the systems. Then we will concentrate on the trajectory planning of the robot. Assuming that we have predicted the user's intention (which will be explained in next Chapter) through his/her gaze direction and the position of his/her dominant hand (the one touching the object) Now, the goal is to move the robot to the appropriate location. The robot motion to be performed must avoid collision with the user and also with itself.

When the user uses HMD vision interfaces and has to perform haptic evaluations, he/she no longer sees the real scene, but only a virtual world. His/her physical reference points quickly disappear except for objects he/she touches such as his/her seat and the dashboard. It is challenging to move the robot in the physical environment such that the robot doesn't cause any harm to the user.

The outline of this chapter is as follows. In section 4.2 an introduction to the proposed architecture and data flow is explained. A brief description of all the components in the system is explained. Section 4.3 deals with the preliminary setup of the systems explained in the architecture. Section 4.4 deals with path planning for the robot. A complete analysis is made between different planning algorithms and compared to find the best planning algorithm. Finally, general remarks and conclusions are commented in section 4.5.

### 4.2 Proposed architecture and data flow

As established in chapter 1, there are multiple systems 1) Unity in Windows which has the display of the virtual environment and the user prediction strategy (explained in chapter 5), 2) ROS in Ubuntu which is used for the robot trajectory planning (will be studied in this chapter) and user occupancy (explained in chapter 3) and 3) the robot system itself. Figure 4.1 show the flow of data between these systems. A detailed explanation of modules inside ROS (like MoveIt) will be dealt in detail in following sections of this chapter.
<figure><img src="image_104.png" /><figcaption>Figure 4.1 – System’s Architecture.</figcaption></figure>

The proposed architecture provides an overview of how the instances share information and communicate with each other, the HTC Vive sensors connected in Windows communicate their positions and orientation. The same information is passed between Windows and Ubuntu system. Additional information of the predicted desired point of intersection is also sent from Unity to ROS in Ubuntu. From the information of the desired point and location of the user model, robot planning is done and the pre-computed planned trajectories are sent to the robot controller for executing the trajectory.

ROS [5] is the middleware that communicates with the robot and Unity. Based on this information, pre-computed trajectories (using MoveIt) are selected so that the robot reaches the desired positions while knowing the current states of obstacles in the scene. Once the trajectory is selected, we communicate with the UR-5 robot. Thus, we can move the UR-5 robot with the ROS control, and send as output the current states of the robot's joints for visualization in Unity.

To move the robot to a specific position and orientation, the position and orientation of the cube are sent through a ROSBridge connection to a robot controller at a frequency of 50 Hz. This controller runs on a thread library that consists of two threads, which read data from Unity VR software and write data to the real-time data interface of the UR-5 at a frequency of 100 Hz.

The proposed architecture of the project describes the different interactions each element of the system has and provides an insight into how the instances share the information and communicate with each other as shown in Figure 4.1. The architecture is a description of the way the system works and which tasks are taken care by which instances.

This chapter explains the ROS section described in the architecture, where it receives as input the desired goal and the sensor's data, with the sensor's data we can compute the kinematics of the mannequin model, which allows the "planning场景" (in MoveIt) to spawn the objects that correspond to the mannequin in the scene, for them to be kept up to date by the