# Application of Virtual Reality in the Training of Operators and Servicing of Robotic Stations

Magdalena Muszyńska, Dariusz Szybicki, Piotr Gierlak, Krzysztof Kurc, Andrzej Burghardt, Marek Uliasz

► To cite this version:

Magdalena Muszyńska, Dariusz Szybicki, Piotr Gierlak, Krzysztof Kurc, Andrzej Burghardt, et al.. Application of Virtual Reality in the Training of Operators and Servicing of Robotic Stations. 20th Working Conference on Virtual Enterprises (PRO-VE), Sep 2019, Turin, Italy. pp.594-603, 10.1007/978-3-030-28464-0_52 . hal-02478804

HAL Id: hal-02478804

https://inria.hal.science/hal-02478804

Submitted on 14 Feb 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Application of Virtual Reality in the Training of Operators and Servicing of Robotic Stations

Magdalena Muszyńska<fnref n="1" />,*, Dariusz Szybicki<fnref n="1" />, Piotr Gierlak<fnref n="1" />, Krzysztof Kurek<fnref n="1" />, Andrzej Burghardt<fnref n="1" /> and Marek Uliasz<fnref n="2" />

**Abstract.** The article discusses the subject of the use of virtual reality in training the service and maintenance of robots and robotic stations. The developed trainings use virtual reality, thus they are interactive and allow advanced operations without the risk of damaging expensive equipment. The idea of the training program and the application of virtual reality was based on the collaborative network defined in the article. The individual chapters discuss the idea of virtual reality, tools used to develop the training system and examples of activities carried out. Real tools used to obtain virtual reality are shown. Methods of developing interactive elements necessary in simulations are presented.

**Keywords:** virtual reality, robotic stations, operator training, robot service.

## 1 Introduction

The article presents the concept of training operators and service technicians in the support, maintenance and servicing of robotic stations. The training system is implemented as part of a collaborative network consisting of a university and companies forming part of the Aviation Valley Association (Fig. 1). This network consists of autonomous components that vary with respect to the operational environment, social capital and culture [1], [2]. The aim of the university is to conduct scientific research, educate students and influence the technological development of the region. The goals of companies are widely known to be the satisfaction of the needs of other entities of social life through the production of products and services. In the Podkarpackie region of Poland, the Aviation Valley Association was established to strengthen the role of south-eastern Poland as one of the leading regions in the aviation industry in Europe.

<footnote n="1">¹ Rzeszow University of Technology, Rzeszow, Poland
{ magdaw, dszybicki, pgierlak, kkurc, andrzejb}@prz.edu.pl</footnote>

<footnote n="2">² Pratt&Whitney Rzeszow S.A., Rzeszow, Poland
Marek.Uliasz@pwrze.utc.com</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Diagram of the functioning of the discussed collaborative network.</figcaption></figure>

The detailed goals of Aviation Valley are:

* The cooperation with universities of technology, which would promote new ideas and scientific research within the aerospace industry.

* The further development of aerospace research, aptitudes and skills.

* The organization and development of a low cost supply chain.

* The creation of favorable conditions in order to enhance the development of aerospace industry enterprises in this region.

* The protection of enterprise and businesses in the aerospace industry.

Aviation Valley connects over 160 companies from the aviation industry. Cooperation with higher education institutions, especially with the Rzeszow University of Technology, is very important for this association. Joint research projects are carried out as part of this cooperation. An example of cooperation with the companies of Aviation Valley presented in this work focuses on designing advanced robotic stations and robotization of various types of processes. One such jointly developed process is the robotic quality control of jet engine components presented in the works [3], [4], [5]. Robotic machining processes in the aviation industry and diagnostics of the components of robotic stations are the subject of publications [6], [7], [8], [9], [10], [11], [12], [13], [14]. The existence of this collaborative network allows companies to implement advanced robot-related solutions, and academic staff can co-create and publish interesting works. A very important component of the university's cooperation with companies is the education of students within the so-called dual courses
conducted simultaneously at the university and in companies, as well as
commissioned work related to training. An example of such work is the training of
robotic operators and servicers. The idea of organizing training in the field of robot
service and servicing appeared during meetings of the University employees with
representatives of associated companies in the Valley. The university has staff
specializing in robot programming, designing robotic stations and station servicing. In
addition to the staff at the Faculty of Mechanical Engineering and Aeronautics, 10
different types of robots, grouped into dedicated stations, are available (Fig. 2).

<figure><img src="image_3.png" /><figcaption>Fig. 2. A photograph of one of the robotic stations.</figcaption></figure>

Companies in Aviation Valley are currently investing in robotization, more and more advanced stations are being created, and more processes are becoming robotized and automated. Due to ongoing robotization, training and raising of awareness among staff regarding robots are necessary. After analysing the needs of companies, it was decided to prepare a common training program for robot operators and service technicians. The program has been prepared so that it can also be used in an engineering study program. Market research was carried out and due to the advantage of robotic stations with ABB robots, training related to the company's solutions was prepared. The first training was carried out for Pratt & Whitney Rzeszow S.A. (P&W). After consultation with the company's representatives, analysis of the most frequent support and servicing procedures and review of the types of stations, it was decided to use virtual reality to prepare interactive training. The use of virtual reality made it possible to adapt the training program to specific robots that exist in P & W without interrupting production. It was possible to develop alternative scenarios for support, repairs and maintenance. In addition, in the safe simulation environment operators could perform advanced maintenance activities.
## 2 Virtual Reality

Virtual reality (VR) is an interactive computer simulation that detects and simulates the presence and actions of the user in a virtual environment and sends sensory feedback to one or more senses in such a way that the user has the feeling of immersion in the simulation [15]. The virtual environment is represented by the description of computer-generated objects in the simulation as well as the principles and relations of managing them. These objects have such physical characteristics as size, shape, colour, mass and behaviour. The features of virtual environment objects depend on the degree of interactivity of a given component and the complexity of the physical model of the virtual environment. For the virtual reality to be realistic, it must react to the user's actions, i.e. be interactive. The user's ability to influence the environments generated by the computer is one of the forms of interaction. Another option is to change the location and orientation from which the user sees the environment. If multiple users can exist in one virtual environment, the virtual reality system should allow interaction between them. Flight training simulators are one of the best examples of the practical application of virtual reality [16]. Pilots can practice flying in different conditions. Flight exercises in difficult weather conditions are particularly beneficial for the training process, and the occurrence of airplane system failures can be simulated. Pilots can thus experience how the machine behaves in certain situations and their errors will not cause a plane crash. Driving simulators have been developed for a similar purpose – they enable safe driving lessons in various conditions (rain, ice, congestion) or tests of new cars [17]. In a virtual environment, you can change any of the car's functions (both aesthetic and functional) and then observe how real drivers react to the changes. The simulation allows to test cars designed before building a prototype.

Thanks to widespread access to VR equipment, new applications using virtual reality are constantly emerging. Scientific institutions also contribute to the development of virtual reality applications. The Fraunhofer Institute for Factory Operation and Automation IFF develops high-level interactive VR environments that can be specifically used in a wide range of industrial training programs [18]. The work [18] presents a methodology for creating an example of a virtual educational platform for operators. Realistic models of components used by the company for which the platform was built were used to build the virtual environment. On the other hand, in [19], a virtual reality system was proposed for training employees of industrial plants on sites where they cooperate with robots. The motivation to build the station was to improve security. Thanks to this, an inexperienced employee will be able to learn the principles of safe cooperation with machines. The work [20] presents the use of VR technology and computer graphics by industry in order to reduce production costs, minimize learning curves and eliminate hazardous situations. In addition, examples of procedural training and VR simulators are shown, as well as their use for military training and support for soldiers. The authors of work [21] built a virtual reality application to teach the management of industrial processes, with particular emphasis on pneumatic systems. The virtual application includes the environment of a virtual laboratory and a virtual industrial plant. The proposed solution allows the participation of multiple users in one virtual environment and interaction between them.
## 3 Designing of a Training Program and Virtual Robotic Stations

For the construction of a training program together with industrial partners, it was decided to use currently available tools for designing and programming robots off-line. The computer tools offered on the market allow for the three-dimensional design of robotic stations and the creation of simulation models through which the way a new station will look and function can be seen in detail before its launch. Due to the previously accepted assumption that the training is to concern ABB robots, the company's tool, RobotStudio, was used to build virtual stations (Fig. 3).

<figure><img src="image_4.png" /><figcaption>Fig. 3. View of a station designed in the RobotStudio software.</figcaption></figure>

The RobotStudio software from ABB is a robot programming environment. It can be used to build robotic stations and program robots both off-line and on-line. Virtual controller technology is implemented in the environment, allowing for a precise reflection of the actual work of the controller, and hence, among others, accurate determination of the actual work cycles of the device. The latest versions of the software work with VR glasses, allow to program robots in the virtual world and interact with their components (Fig. 4).
<figure><img src="image_5.png" /><figcaption>Fig. 4. The concept of cooperation of RobotStudio software with VR tools.</figcaption></figure>

High-performance PCs with RobotStudio software and Oculus Rift glasses were used to build training stations. Oculus Rift is a set of virtual reality goggles. After they are placed on the user's head, they display the image of a computer-generated world or a film recorded in 360 degrees. The user can look around them in a natural way, moving their head or body. The image displayed in front of the eyes is then transformed to create the illusion of being in the virtual world or in the centre of the action taking place in the film. The Oculus Rift version used was equipped with a screen displaying to each eye independent images with a resolution of 1080 × 1200 pixels. Connection with the computer is made by means of HDMI and USB interfaces. The movements of the user's head are tracked by two sets of sensors that are responsible for tracking the rotation and position of the head in space. The idea is that the device should properly interpret both shaking the head from side to side and, for example, leaning it forward.

RobotStudio makes available, for example, three-dimensional models of robots offered or their controllers. Unfortunately, they are not detailed models, there are no components inside; they only show the dimensions and external appearance. For the purpose of the training program, precise, interactive models of components of robotic stations were built. 3D CAD software was used to build the models. Next, the models prepared were exported to the RobotStudio software using the SAT standard (Fig. 5).

<figure><img src="image_6.png" /><figcaption>Fig. 5. Diagram of the export of detailed models.</figcaption></figure>
Due to the adopted training program, the focus was on support and servicing of 6-axis robots and the latest IRC5 controller. ABB's technical and service documentation was used for the building of detailed models of station components (Fig. 6).

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Construction of the IRC5 controller.</figcaption></figure>

Based on photographs and available units, detailed models of the most important components of the IRC5 controller were built, such as, for example, an axis computer, a main computer and drive controllers. Models of IRC5 components are equipped with interactive components (e.g. LEDs) informing about status, operating mode and failures. The built-in components are connected programmatically with a virtual controller that allows the execution of robot programs. In addition, the models built were equipped with a mechanism that randomly generate failures, display error messages and prompt methods for solving problems. The picture of the upper computer, its model and information displayed using the VR technique is shown in Fig. 7.

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** The idea of building interactive components of a robot controller.</figcaption></figure>
Similarly to the IRC5 controller model, models of the most common robots were built. They are equipped with gears, brakes and components related to the calibration and replacement of oils in the gears (Fig. 8).

<figure><img src="image_9.png" /><figcaption>Fig. 8. A section of the virtual environment developed.</figcaption></figure>

As part of the work, scenarios of failures, repairs and servicing of robotic stations were prepared using VR. The person being trained, equipped with glasses and hand controllers, deals with support and servicing of the station. Their tasks include identification of the type of controller failure based on the symptoms, operation of the robot consisting in the calibration, checking of shock absorbers and the replacement of oil in the gears. The structure of functioning and the components developed allow the training program to be developed in the future.

## 4 Conclusion

The article presents a developed system of training of operators and service technicians in the field of operation, maintenance and servicing of robotic stations. This system uses virtual reality, thanks to which the training is interactive and allows advanced operations without the risk of damage to expensive equipment. As part of subsequent works, a training program with the use of virtual reality will be expanded. There will be added next activities performed on the robots and the expanded base of possible system failures. The number of stations whose service can be simulated will be increased. The students will be more involved in the project and a database of virtual-interactive objects subject to servicing and service will be built. In subsequent works, we plan to use the augmented reality as an element complementing the training system. The concept and the training program were created thanks to the existence of a collaborative network based on the Aviation Valley Association. At present, no company, university or government has the time or resources for continuous innovation. The global market places higher and higher demands, and there is a tendency towards specialization and continuous development. In the case of the collaborative network discussed, the cooperation of the university, bringing tools and specialist knowledge, with companies, which have experience and high
technology, can bring mutual benefits and create new value. The presented cooperation stems from a common belief that together network members can achieve goals that would not be possible to achieve separately or would require more time and resources.

## References

[1] Camarinha-Mato, s. L. M., Afsarmanesh, H.: Collaborative networks: a new scientific discipline. Journal of Intelligent Manufacturing, 16(4-5), pp. 439-452, (2005).

[2] Shuman, J., Twombly, J.: Collaborative networks are the organization: an innovation in organization design and management. Vikalpa, 35(1), pp.1-14, (2010).

[3] Burghardt, A., Kurc, K., Szybicki, D., Muszyńska, M., Szczęch, T.: Robot-operated inspection of aircraft engine turbine rotor guide vane segment geometry. Tehnički vjesnik, 24 (Supplement 2), pp. 345-348, (2017).

[4] Burghardt, A., Kurc, K., Szybicki, D., Muszyńska, M., Nawrocki, J.: Software for the robot-operated inspection station for engine guide vanes taking into consideration the geometric variability of parts. Tehnički vjesnik, 24(Supplement 2), pp. 349-353, (2017).

[5] Burghardt, A., Kurc, K., Szybicki, D., Muszyńska, M., Nawrocki, J.: Robot-operated quality control station based on the UTT method. Open Engineering, 7(1), pp. 37-42, (2017).

[6] Teti, R., Jemielniak, K., O'Donnell, G., Dornfeld, D.: Advanced monitoring of machining operations. CIRP annals, 59(2), 717-739 (2010).

[7] Gierlak P.: Hybrid position/force control in robotised machining. Solid State Phenomena, vol. 210, pp. 192-199, Switzerland (2014).

[8] Burghardt, A., Kurc, K., Szybicki, D., Muszyńska, M., Szczęch, T.: Monitoring the parameters of the robot-operated quality control process. Advances in science and technology-research journal, 11(1), pp.232-236, (2017).

[9] Burghardt, A., Szybicki, D., Kurc, K., Muszyńska, M., Mucha, J.: Experimental Study of Inconel 718 Surface Treatment by Edge Robotic Deburring with Force Control. Strength of Materials, 49(4), pp.594-604, (2017).

[10] Barnfather, J.D., Goodfellow, M.J., Abram, T.: A performance evaluation methodology for robotic machine tools used in large volume manufacturing. Robot. CIM-Int. Manuf. 37, 49-56 (2016).

[11] Kurc K., Burghardt A., Gierlak P., Szybicki D.: Non-contact Robotic Measurement of Jet Engine Components with 3D Optical Scanner and UTT Method. In: Hanus R., Mazur D., Kreischer C. (eds) Methods and Techniques of Signal Processing in Physical Measurements. MSM 2018. Lecture Notes in Electrical Engineering, vol 548, pp. 151-164. Springer, Cham, (2019).

[12] Hui-Ping, Liu, Dai-Min, C., Miao, Y.: Communication of multi-robot system on the TCP/IP In: mechatronic science, electric engineering and computer (MEC). In: International Conference on. IEEE, 1432-1435 (2011).

[13] Szybicki D., Burghardt A., Gierlak P., Kurc K.: Robot-Assisted Quality Inspection of Turbojet Engine Blades. In: Hanus R., Mazur D., Kreischer C. (eds) Methods and Techniques of Signal Processing in Physical Measurements. MSM 2018. Lecture Notes in Electrical Engineering, vol 548. pp 337-350. Springer, Cham, (2019).

[14] Szybicki, D., Burghardt, A., Kurc, K., Pietruș, P.: Calibration and verification of an original module measuring turbojet engine blades geometric parameters. Archive of Mechanical Engineering vol.: 66. pp: 97-109, (2019).

[15] Burdea, G. C., Coiffet, P.: Virtual reality technology. John Wiley & Sons.
[16] Shao, F., Robotham, A. J., Hon, K. K.: Development of a 1: 1 Scale True Perception Virtual Reality System for design review in automotive industry, (2019).

[17] Gallagher, A. G., Ritter, E. M., Champion, H., Higgins, G., Fried, M. P., Moses, G., Satava, R. M.: Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Annals of surgery, 241(2), 364,(2005).

[18] Kožlak, M., Kurzeja, A., Nawrat, A.: Virtual reality technology for military and industry training programs. In Vision Based Systemsfor UAV Applications (pp. 327-334). Springer, Heidelberg, (2013).

[19] Matsas, E., Vosniakos, G. C.: Design of a virtual reality training system for human-robot collaboration in manufacturing tasks. International Journal on Interactive Design and Manufacturing (IJIDeM), 11(2), pp. 139-153, (2017).

[20] Ortiz, J. S., Sánchez, J. S., Velasco, P. M., Quevedo, W. X., Carvajal, C. P., Morales, V. Andaluz, V. H.: Virtual training for industrial automation processes through pneumatic controls. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics, pp. 516-532. Springer, Cham, (2018).

[21] De Pace, F., Manuri, F., Sanna, A., Zappia, D.: An Augmented Interface to Display Industrial Robot Faults. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics, pp. 403-421. Springer, Cham, (2018).