# Beware of the Robot: A Highly Interactive and Immersive Virtual Reality Training Application in Robotic Manufacturing Systems

Elias Matsas, Dimitrios Batras, George-Christopher Vosniakos

► To cite this version:

Elias Matsas, Dimitrios Batras, George-Christopher Vosniakos. Beware of the Robot: A Highly Interactive and Immersive Virtual Reality Training Application in Robotic Manufacturing Systems. 19th Advances in Production Management Systems (APMS), Sep 2012, Rhodes, Greece. pp.606-613, 10.1007/978-3-642-40352-1_76 . hal-01472298

HAL Id: hal-01472298

https://inria.hal.science/hal-01472298

Submitted on 20 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Beware of the robot: a highly interactive and immersive
Virtual Reality training application in robotic
manufacturing systems

Elias Matsas<fnref n="1" />, Dimitrios Batras<fnref n="2" />, George-Christopher Vosniakos<fnref n="1" />

**Abstract.** A highly interactive and immersive Virtual Reality Training System (VRTS) is developed, in terms of an educational serious game that simulates the cooperation between industrial robotic manipulators and humans, executing manufacturing tasks. “BeWare of the robot” application ultimately aims at studying the acceptability of human-robot collaboration, when both human and robot share the same workspace. The initial version of the application was evaluated by a group of users. Experimental results on usability and technical aspects are presented and several remarks about users’ experience and behavior in the virtual world are discussed.

**Keywords:** Virtual Reality, Safe Human-Robot Cooperation, Manufacturing Training, Interaction, Serious Game

## 1 Introduction

In not-too-distant future manufacturing systems a need for cooperation and work-
space sharing of industrial robots and humans to execute manufacturing tasks will
arise. For a long time now, safety of the human interacting with industrial robots is
addressed by segregation between humans and robots [1]. Cooperation of human and
robot may present an interesting solution that balances productivity, quality, initial
capital cost, running cost and flexibility. However, all important physical and “men-
tal” safety issues that arise must be successfully dealt with. Serious games and highly
interactive and immersive Virtual or Mixed Reality training applications are preferen-
tially deployed in such cases.

In terms of functionality, VR allows users to be extracted from physical reality in
order to virtually change time, space, and (or) interaction type [2]. Virtual Reality-
based training systems (VRTSs) are advanced computer-assisted, interactive training
systems using VR technology, e.g. allowing proper test and operation of new equip-

<footnote n="1">1 National Technical University of Athens, School of Mechanical Engineering, Manufacturing Technology Division, Heroon Polytechniou 9, 15780, Athens, Greece
{imatsas, vosniak}@central.ntua.gr</footnote>

<footnote n="2">2 Presence & Innovation Lab, Arts et Métiers ParisTech, Ingénierium, 4 rue de l' Ermitage,
53000, Laval, France
dbatras@yahoo.gr</footnote>
ment before it is actually installed [3]. They need to have the necessary physical fidelity to mimic the resolution of the physical world in order to be effective for tasks that are characterized by a significant perceptual and/or motor component [4]. Shukla [5] points out the significance of VR in manufacturing training. All VRTSs can be decomposed into three distinct functional parts: (i) output devices, (ii) input devices (mainly for interaction), and, (iii) a VR engine including virtual scene, interaction models, and a graphical representation of the user ( avatar) [6].

In the literature, [7] first propose a deictic mode of sensory-motor control with gestures and a HMD for teleassistance, that can be useful for human/robot interaction and control. Morioka [8] developed a new cell production assembly system, in which physical and information supports are provided to the human operators to double productivity. Krüger [9] gives a survey about human-robot cooperation and all available technologies that support the cooperation. Oborski [10] points out the most important factors of man-machine interaction in advanced manufacturing systems and proposes a method of human attributes modeling for human-machine (robot) interaction. Corrales [11] presents the implementation of real-time proximity queries between humans and robotic manipulators. From another point of view, [12] deal with mental strains of human operators in human-robot cell production systems and propose metrics for a physiologically comfortable collaboration.

In this paper a serious game application named “BeWare of the robot” is presented, simulating tasks of human-robot collaborative tape laying for building aerospace fabric reinforced composite parts. Profiled fabric layers (patches/cloths) are stacked successively inside a die, one on top of the other, until the desired thickness is reached.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Avatar's hands trying to pick-up the part from the robot (left) and user trying to put the part in the metallic die (right).</figcaption></figure>

Technical aspects of the application development are explained centering on safety issues, such as contacts and collisions that are tackled through “emergencies”, i.e. visual stimuli and sound alarms. Preliminary evaluation results by a group of users are presented. Mental safety, i.e. human’s awareness of the robot’s motion, is the primary objective.
## 2 The Virtual Reality Training System Description and Analysis

Shop-floor environment and its components were developed using Rhinoceros™ and 3ds Max™ for 3D part design and mesh creation. Unity 3d™ game engine was used for assembly, rendering, lighting, physics, simulation, building compilation and programming. “BeWare of the robot” VRTS is a PC standalone build that can run in a typical PC with Windows XP™ and an nVidia™ graphics card supporting 3D stereoscopic vision. An immersive device (HMD) supporting head tracking and a Kinect™ sensor are needed to fully exploit navigation, tracking and immersion capabilities of the system. Communication between Unity 3d and the Kinect™ sensor is implemented with the OpenNI framework. The VRTS incorporates (i) the virtual model of a composites hand layout work-cell, (ii) the model of a Staübli RX90L robotic manipulator, (iii) the skinned model of an avatar created online in evolver.com, with a 3ds Max™ biped attached to it, (iv) interaction scripts in C#, (v) real-time shadows and lighting, and (vi) image, video and audio textures from the real working space. Interaction is mainly based on collision and ray-casting scripts, child/parenting functions, and skeletal tracking of 18 avatar’s points.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** UML Use Case Diagram and Activity Diagram</figcaption></figure>

In order to describe, analyze and decompose the system, Object-Oriented Analysis (OOA) tools and UML 2.0 expressions are used. OOA methods divide the system to
its components and make every relation explicit [13]. The Use Case Diagram presented in Fig. 2 defines the required tasks and interactions between the user and the system. The Activity Diagram is a graphical representation of system behavior: flow of actions, activities and transitions. The Structure Diagram provides a detailed analysis of system entities, their attributes, the methods used and the associations between the entities as shown in Fig. 3. OOA tools in VEs can be used as a path for the creation and/or the improvement of the virtual world. In addition, through OOA methods, system knowledge can be captured to build intelligence.

<figure><img src="image_4.png" /><figcaption>Fig. 3. OOA Structure Diagram showing entities, attributes and associations.</figcaption></figure>

## 3 Experiment Set-Up and Results

To evaluate both the effectiveness and the usability of the application, as well as to investigate users' experience, a series of experiments were carried out. The participants of the experiment are a group of 30 senior mechanical engineering students (aged between 21 and 31 years), sufficiently familiar with theoretical Robotics and Manufacturing Systems design. None of the students had tested the application before the experiments.
The experimental task required subjects to individually test the “beware of the robot” application; that is to virtually collaborate in sequence with an industrial robot, while sharing the same physical environment. The experimental goal was to test/validate the following hypothesis:

- Highly-interactive and immersive VRTSs may be a successful tool for motor skill training, such as human-robot collaboration testing.

- Immersion and head tracking the user experiences with the use of the HMD boosts the feel of presence and realism. Manufacturing education is therefore enhanced when physical access to equipment is either not possible or potentially hazardous.

- The use of additional audiovisual stimuli (e.g. HD video textures and 3d spatialized sounds from the real environment, real-time shadows and differed lighting) enhances user's experience of presence in the virtual world.

The testing platform consisted of: the “beWare of the robot” application, a PC running Windows XP™ equipped with an nVidia Quadro™ FX1700 graphics card, an eMagin Z800 HMD with stereo ear buds, a Microsoft Kinect™ sensor, keyboard and mouse. Furthermore, in order to be able to reproduce what the users sees through the HMD and to record the experiments with a video camera, we used a projector cloning the displays of the HMD in a wall behind the user.

Experiment's duration was 10 minutes and participants were given detailed instructions orally. After wearing the HMD, subjects were asked to stand in “Y” posture towards the Kinect sensor in order to be detected and calibrated. They, then, had to “discover” system's tracking, moving and rotating capabilities, as well as the objects of the virtual environment. Afterwards, participants had about 6-8 minutes to complete the tasks described in the Use Case Diagram (Fig. 2).

At the end of the experiment, each participant was asked to fill out and submit the online form (questionnaire). This was developed from scratch in order to examine user experience as well as system effectiveness. The questionnaire consisted of 42 questions, grouped into 3 thematic categories (i) participant's personal information, level and experience (14 questions), (ii) immersion, presence and realism (10 questions), (iii) usability, effectiveness, tracking and interaction quality, (18 questions).

Analysis was conducted on three levels: (i) online questionnaires, (ii) task execution video recording and a posteriori observation, and (iii) discussions with each participant.

## 3.1 Results and Discussions

**Presence and Immersion.** Table 1 shows the results of selected questions concerning users' experience in the VRTS. Some interesting findings rise from this table regarding the concepts of presence and immersion. Slater et al. [14] define presence as a subjective phenomenon such as the sensation of being in a VE, while immersion as an objective description of aspects of the system such as field of view and display resolution. In the literature, several theories have been proposed on the nature of presence in immersive VR [15]. Our research focuses more on “presence by involvement”; that is, that both involvement and immersion are thought to be necessary for
experiencing presence. Moreover, [16] claim that “presence” and “situation awareness” are overlapping constructs.

In our experiment, a vast majority of participants were feeling as they were really moving in the scene, “involved and present” in the virtual activity. In addition, 93% of the subjects answered that they did not lose their concentration at all during the test. The results also show that a large number of subjects felt like they were really moving an object with their hands, despite the fact that the object did not have physical mass. A posteriori video observation emphasizes the above finding with an intrinsic proprioceptive users’ reaction: although subjects were told that they should use one hand, 20% of them used both hands in order to grasp and to carry the workpieces. We also noticed that some users (17%) spontaneously closed their hands and/or fingers in order to grasp the parts (as they would have reacted in the real world), although they were told that our system does not support fingers tracking.

<table><caption>**Table 1.** Users' experience selected results.</caption><thead><tr><th>Item</th><th>SD (1)</th><th>D (2)</th><th>N (3)</th><th>A (4)</th><th>SA (5)</th></tr></thead><tbody><tr><td><strong>Presence & Immersion</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>I feel like I was really moving an object with my hand, despite the fact that the object did not have physical mass.</td><td>1</td><td>7</td><td>8</td><td>12</td><td>2</td></tr><tr><td>Wearing the HMD I feel immersed in the virtual environment.</td><td>1</td><td>0</td><td>7</td><td>13</td><td>9</td></tr><tr><td>I feel like I was really moving in the scene.</td><td>0</td><td>1</td><td>3</td><td>5</td><td>21</td></tr><tr><td>Wearing the HMD I feel present and involved in the virtual activity.</td><td>2</td><td>1</td><td>5</td><td>15</td><td>7</td></tr><tr><td>I feel like my behavior in the virtual world didn't change, compared to my behavior in the real world.</td><td>3</td><td>10</td><td>9</td><td>8</td><td>0</td></tr><tr><td><strong>Usability & Effectiveness</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>I did not encounter any difficulties during the initial calibration process with the Kinect sensor.</td><td>0</td><td>3</td><td>3</td><td>6</td><td>18</td></tr><tr><td>Audiovisual stimuli (alarms) helped me being aware of a potentially hazardous workspace.</td><td>0</td><td>1</td><td>1</td><td>8</td><td>20</td></tr><tr><td>I feel like the avatar followed my movements precisely.</td><td>1</td><td>2</td><td>12</td><td>14</td><td>1</td></tr><tr><td>It was easy to pick-up the workpieces from the robot's end-effector.</td><td>0</td><td>0</td><td>9</td><td>13</td><td>8</td></tr><tr><td>It was easy to navigate/move in the virtual world (ease of movement, restraint).</td><td>0</td><td>3</td><td>12</td><td>12</td><td>3</td></tr><tr><td>I feel that my eyes were tired because of the use of the HMD.</td><td>15</td><td>10</td><td>3</td><td>2</td><td>0</td></tr><tr><td>I feel more like I was participating in an amusing game.</td><td>0</td><td>2</td><td>5</td><td>10</td><td>13</td></tr><tr><td>After the experiment I went through, I believe that human-machine training tasks can be more attractive with the use of "serious games".</td><td>1</td><td>0</td><td>2</td><td>8</td><td>19</td></tr></tbody></table>

**Usability and System effectiveness.** Although 77% of the subjects had never used the Kinect™ sensor before, only 10% of them encountered some difficulties during the initial detection and calibration process. During the human-robot collaboration procedure, almost all subjects (97%) replied that they easily perceived the red transparent sphere that represented the robot's workspace, and 70% of subjects managed not to enter into the workspace (which was the potentially hazardous area). The survey results show that most of the subjects favorably accepted the use of visual and auditory stimuli (alarms) for the robot's workspace awareness. Note that the red sphere and visual alarms in general may be contradictory with fidelity and realism of VEs, but realism is not always the ultimate goal. In VRTSs learning procedure for
example, exaggeration or deformation of real events is “authorized” for complex situations understanding [6].

Concerning the Kinect's™ tracking quality, most of the participants considered that the avatar was following their (tracked) movements precisely, 23% of them, though, pointed out a slight “vibration” in the avatar's movements. On the other hand, observations revealed that 53% of the users sometimes “lost control” of their hands, even instantaneously. That is because experiment tasks required user body rotation towards the sensor, and at the extremes hands tracking is confusing: Kinect™ sensor cannot easily distinguish the left from the right hand. In addition when users turn their body at 180 degrees towards the Kinect sensor, skeletal tracking is lost and although they may continue moving, the avatar remains “frozen” for a while in the last detected posture. Furthermore, 43% of the subjects felt like their movements were altered compared with the real ones; with emphasis in hands “roll” rotation (50%) and body rotation (47%). Nevertheless, 70% of the subjects answered that they easily managed to pick-up the workpieces from the robot’s end-effector, while 50% said that it was easy to navigate (walk, rotate, bend etc.) in the virtual world.

Concerning user motivation, 76% of the participants replied that during the experiment they were feeling more as if they were participating in an amusing game, and 90% of the subjects answered that training tasks requiring human-robot collaboration can be more attracting with the use of “serious games”.

Eventually, the results suggest a positive prospect for the use of VR for training. In addition, moving around the scene and using real gestures (skeletal tracking) to manipulate objects and to complete the tasks makes learning more active and impressive for learners.

# 4 Conclusion and Future Work

Due to the notions of presence and realism VRTSs and highly-interactive serious games are probably an efficient and safe learning tool to study human-robot collaboration with overlapping workspaces. Through multi-level “invisible” interactions with the Kinect™ sensor skeletal tracking, user experience and behavior in the virtual world can be recorded, studied and analyzed, leading to new knowledge acquisition.

Experiment questionnaire analysis permitted to identify several remarks about user experience in the VE, and helped us pose the question of how virtual world affects user temporal identity in the VE, which will be tackled in an upcoming study.

Moreover, a Mixed Reality version of the application where the user would interact physically with real objects while being tracked and immersed in the virtual world is expected in the near future. Authors are also working on solving the skeletal tracking problems with the use of a second Kinect™ sensor.

**Acknowledgements.** This research has been co-financed by the European Union (European Social Fund–ESF) and Greek national funds through the Operational Program "Education and Lifelong Learning" of the NSRF - Research Funding Program: Heracleitus II, Investing in knowledge society through the European Social Fund.
We wish to thank the participants in the experiments for sharing their experience with us, as well as Dr. Dimitrios Nathanael for his helpful comments concerning the experiment set-up and the questionnaire. We would also like to thank Prof. Simon Richir and Matthieu Lépine for accepting and hosting our application in the exhibition section of Laval Virtual 2012 conference.

<figure><img src="image_5.png" /></figure>

European Union
European Social Fund

<figure><img src="image_6.png" /></figure>

OPERATIONAL PROGRAMME
EDUCATION AND LIFELONG LEARNING

DEPARTMENT OF EDUCATION, CULTURE AND SPORTS OF GREECE

MINISTRY OF EDUCATION, LIFELONG LEARNING AND REHABILITATION APPLICANTS

MANAGING AUTHORITY

<figure><img src="image_7.png" /></figure>

NSRF
2007-2013

EUROPEAN SOCIAL FUND

Co-financed by Greece and the European Union

## References

1. Helander M.G.: Ergonomics and Safety considerations in the design of robotics workplaces: A review and some priorities for research, Int. J. of Ind. Ergonomics. 6, 127-149 (1990)

2. Fuchs Philippe et al.: Le traité de la réalité virtuelle, Volume 2, Les Presses de l'Ecole des Mines de Paris (Eds.) (2006)

3. Lin F., Ye L., Duffy V.G., Su C-J.: Developing virtual environments for industrial training, Information Sciences 140, pp. 153-170 (2002)

4. Nathanael D., Vosniakos G.-C., Mosialos S.: Cognitive task analysis for Virtual Reality Training: the case of CNC tool offsetting. Proceedings of ECCE 2010 Conference, pp. 241-244, Delft, The Netherlands (2010)

5. Shukla Ch., Vazquez M., Chen F.F.: Virtual Manufacturing: An overview. Computers and Industrial Engineering, 31, No. 1/2, pp. 79-82 (1996)

6. Burkhardt J.-M., Lourdeaux D., Fuchs P.: Conception d'un système de RV pour la formation des agents de conduites aux opérations en milieu ferroviaire. In: A. Grumbach & N. Richards (Eds.), Journées Réalité Virtuelle et Cognition, pp. 123-132, ENST, Paris (1999)

7. Pook P.K., Ballard D.: Deictic human/robot interaction, Robotics and Autonomous Systems, 18, 259-269 (1996)

8. Morioka M., Sakakibara S.: A new cell production assembly system with human-robot cooperation, CIRP Annals - Manufacturing Technology, 59, 9-12 (2010)

9. Krüger J., Lien T.K., Verl A.: Cooperation of human and machines in assembly lines, CIRP Annals - Manufacturing Technology 58, 628-646 (2009)

10. Oborski P.: Man-machine interactions in advanced manufacturing systems, Int. J. Adv. Manuf. Technol., 23, 227-232 (2004)

11. Corrales J.A., Candelas F.A., Torres F.: Safe human-robot interaction based on dynamic sphere-swept line bounding volumes, Robotics and CIM, 27, 177-185 (2011)

12. Arai T., Kato R., Fujita M.: Assessment of operator stress induced by robot collaboration in assembly, CIRP Annals - Manufacturing Technology, 59, 5-8 (2010)

13. Deacon J.: Object-Oriented Analysis and Design, Pearson Education, Essex (2005)

14. Slater, M., & Wilbur, S.: A framework for immersive virtual environments (FIVE): Specifications on the role of presence in virtual environments. Presence, 6, 603-616 (1997).

15. Schuemie M.J. et al.: Research on Presence in Virtual Reality: A Survey. CyberPsychology & Behavior, 4(2), 183-201, Mary Ann Liebert, Inc. (2001)

16. Prothero, J.D., Parker, D.E., & Furness, T.: Towards a robust, quantitative measure for presence. Proceedings of the Conference on Experimental Analysis and Measurement of Situation Awareness, pp. 359-366 (1995)