# VRQUEST: Designing and Evaluating a Virtual Reality System for Factory Training

Khanh-Duy Le, Saad Azhar, David Lindh, Dawid Ziobro

► To cite this version:

Khanh-Duy Le, Saad Azhar, David Lindh, Dawid Ziobro. VRQUEST: Designing and Evaluating a Virtual Reality System for Factory Training. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.300-305, 10.1007/978-3-030-85607-6_26 . hal-04291214

HAL Id: hal-04291214

https://inria.hal.science/hal-04291214v1

Submitted on 17 Nov 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# VRQUEST: Designing and Evaluating a Virtual Reality System for Factory Training

Khanh-Duy Le<fnref n="1" />, Saad Azhar<fnref n="1" />, David Lindh<fnref n="2" />, and Dawid Ziobro<fnref n="1" />

**Abstract.** Training is vital in factories to ensure the quality of workers' technical expertise but can be costly due to various physicality constraints. The emergence of virtual reality (VR) opens the opportunities to address this issue by providing affordable solutions where workers can freely learn by doing without compromising safety or risking damage. This paper presents an industrial research project carried out in a company, focusing on designing and evaluating a VR application for training a procedural task in factories. Insights from the evaluation suggest multiple design considerations that need to be taken into account in designing future VR interfaces for factory training in the company.

**Keywords:** VR · factory training · immersive environment

## 1 Introduction

Technical training plays an important role in manufacturing industries. Both experienced or newly-hired workers need to be trained regularly to ensure up-to-date technical knowledge and skills for reliable and timely operation or troubleshooting in factories. However, technical training can be costly. In traditional training approaches, trainees need to typically work directly on a piece of real equipment in order to help them acquire hands-on experience. This might be impractical in many scenarios where the real equipment is expensive, thus economically difficult for trainees to freely explore. Sometimes trial-and-error exploration with actual devices should even be avoided as they may cause life-threatening accidents. Traditional approaches also typically require experts to be locally present to supervise, assist or teach trainees, which is becoming challenging due to the shortage of experts as well as difficulties of traveling.

Recent developments of VR technologies promise novel low-cost solutions for technical training in factories. Modern VR headsets can provide powerful visualization capabilities and intuitive interactions based on hand gestures conveyed through handheld controllers, in a relatively compact form factor with affordable prices. Intuitive gesture-based interactions currently supported on modern VR systems might also allow to simulate user interactions with virtual artefacts similarly as in the real world, thus helping train necessary motor skills for the tasks. This opens opportunities for industry to easily simulate complex and costly systems and environments at a high fidelity in virtual worlds, where trainees can freely explore, manipulate, make errors and learn without taking risks.

<footnote n="1">¹ ABB Research, Västerås, Sweden</footnote>

<footnote n="2">² Linköping University, Linköping, Sweden</footnote>
ABB is a global manufacturer and supplier of complex industrial systems including industrial robotics, power and heavy electrical equipment. Technical training is thus a crucial part in the organization's business, not only in securing the company's production but also in supporting the company's customers. With recent advancements of VR, ABB was interested in exploring the applicability of this technology to benefit technical training activities in and offered by the company. We thus carried out a research project focusing on designing, developing and evaluating a VR application called *VRQUEST* to support training of a technical task in factories. The main goal was to examine the advantages and disadvantages of currently common VR devices for industrial technical training.

## 2 Design and Development of VRQuest

To frame design and research problems, we interviewed several subject matter experts (SMEs) in the company to understand more about their current training practices. The SMEs were managers and engineers from different departments such as electrical, mechanical, system design, installation, site management and maintenance with various levels of experience working for the company. We also reviewed multiple research articles on training, learning and instructional design [1,2] to understand more about what should be scientifically considered when designing a system for training. Based on contextual understandings gathered from the SMEs and knowledge from the literature review, we ran a design workshop to ideate on different design features of the system. We chose a maintenance task where a user needs to procedurally perform multiple steps to change a filter of a cooling system as an exemplary use-case for the *VRQUEST* system. This task was chosen because it was a common one found in the literature as well as currently existing in the company. The design features were iteratively refined during the prototyping process following the five aspects below.

**Physical fidelity:** this aspect includes: *representational fidelity* - describing how the virtual environment should resemble the look and even the feel of the environment, and *interaction fidelity* - describing that users interact with virtual artefacts as if they were in the real world [3]. This aspect is to familiarize users with the visual appearance of the corresponding real environment and the necessary psychomotor skills to perform the actual task. *VRQUEST* implements this by presenting a virtual room that resembles an existing cooling substation. A 1:1 scale 3D model of a standard cooling system of ABB was also placed in the virtual room (Figure 1a). *VRQUEST* also tries to ensure that the gestures used to operate different components of the cooling system should be as close to reality as possible, given the commercial VR hardware (i.e. 6-DOF touch controllers) that we had (see Figure 1d and Figure 2). Haptic feedback like vibrations and sound effects were also strategically employed to improve the realism of user interactions with virtual artefacts.

**Sequential task description:** *VRQUEST* describes the main goal users need to achieve by the end of the training task, followed by step-by-step descriptions of the sub-tasks needed to be completed to reach the main goal. The descriptions are presented as texts on a virtual screen placed at a corner in the virtual room (Figure 1b).
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Overview of VRQUEST system (a) 1:1 scale 3D model of the cooling system (b) a virtual screen allowing the user to choose the training mode (**Tutorial** or **Practice**) and view the instructions in **Tutorial** mode (c) summary of user performance when completing the task in **Practice** mode (d) a participant bending down and reaching his hand over a virtual pipe to turn a valve located close to the floor.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Examples of user interactions in VRQUEST. (a) An arrow helps the user locate a switch (b) the user's virtual hand forming a pointing gesture with the index finger placed on the switch (c) then flicks down to turn it off (d) an arrow indicating the location and the rotation direction of a valve (e) grabbing the valve handle to rotate it (f) removing the nut of the filter using a wrench, the arrow indicates the rotation direction (g) the user turning the wrench in the indicated direction.</figcaption></figure>

**Attention guidance:** in technical training, it is important to guide trainees to focus their attention in order to reduce frustration and tiredness caused by open exploration. VRQUEST supports this by employing visual indicators such as colored arrows to hint users about which artefacts they should interact with at the moment and how to interact with them (e.g. turn a valve in a particular direction) (Figure 2).

**Practice:** after completing the task with guidance, users need to be able to repeatedly practice it, with or without guidance, to build the psychomotor memories needed for the task. VRQUEST supports this by offering two training modes: **Tutorial** mode, allowing users to practice with guidance provided by the system, and **Practice** mode, allowing users to practice without guidance so that they have to recall the steps by themselves.

**Motivating trainees:** VRQUEST employs gamification features such as badges of seniority level or experience scores to motivate users to keep practicing. Once completing a training session in the **Practice** mode, the user will see a summary of their performance such as task completion time or the number of errors, coupled with the corresponding experience score and seniority badge.

The prototype of VRQUEST was developed using Unity and Steam VR and running on an Oculus Rift VR headset connected to a Windows 10 Enterprise gaming PC.
# 3 Evaluation and Insights

Research insights from this project needed to be transferred to ABB businesses to be employed in the company's potential products and services in the future. Because of that, *VRQUEST* needed to be evaluated holistically to examine the advantages and disadvantages of common off-the-shelf VR technologies in their application for factory training. Design considerations also needed to be distilled based on these findings so that designers and developers in the company can take over and implement in future systems.

We conducted a user study with 10 participants working at ABB to evaluate if *VRQUEST* can help users effectively learn to perform a factory task as well as how the design affects users' experience. Each participant performed the filter changing task of *VRQUEST* in two trials, first in *Tutorial* mode and then in *Practice* mode with a 30-minute coffee break in between. Our intention was to examine if participants can recall correctly by themselves all the steps to successfully complete the task after learning it once with guidance and especially being distracted from the task during the break. We collected the task completion times of the participants in both trials, the numbers of errors (e.g. missed steps, changed orders of steps in the task) the participants made when performing the task in the second trial, participants' perceived usability (using System Usability Scale (SUS) questionnaire) and their qualitative feedback after finishing both trials.

In general, we saw that *VRQUEST* could help participants successfully learn to perform a factory task. In the second trial, participants spent significantly less time than in the first one (on average, 4.7 ± 1.1 minutes vs 12.2 ± 1.1 minutes, pair-wise t-test showed *p* < 0.001). On average, each participant made 0.8(±0.4) errors (over 21 steps in total to complete the task) when performing it in the second trial. Regarding usability, *VRQUEST* received 65.8(±3.1) on average for SUS scores, which can be considered as marginally acceptable [4].

Participants' qualitative feedback provided us better understandings on the quantitative data as well as the benefits of VR for factory training and how VR training systems can be better in the future. Participants reported that compared to training using user-manual printouts they traditionally had, the immersive environment of *VRQUEST* using 1:1 virtual copies of the real artefacts allowed them to easily grasp the overall picture of where to locate the cooling system's components as well as how to manipulate them. Hands-on experiences as well as the spatiality and immersion provided by VR helped users efficiently remember the operations of the task. Besides that, VR encouraged participants to interact and explore, helping them remember the training contents better. Knowing the artefacts were virtual made the participants feel more confident to interact, manipulate and explore without the fear of causing damages. The gamification features also had positive effects on users. Most of the participants reported that they liked the task-completion time and the number of errors index as it kept them informed about their performance. After finishing the task in the second trial, three participants tried performing the task again to beat their current
results. However, experience scores and seniority badges received little attention
from participants.

Even though VR can visually and spatially provide immersive experience, participants reported the lack of certain haptic feedback such as weight or heat, which were often important for them to sense where they are relative to an equipment (e.g. a heated pipe) or to feel the tool they are holding (e.g. a wrench or a screw driver). Currently it is difficult for developers to integrate those features in similar systems due to the lack of off-the-shelf supportive technologies. However, with enormous on-going academic research efforts in this topic and rapid industrial adoptions, we believe such an integration would be soon feasible. Besides that, even though it is important to replicate the real environment in the virtual world to ensure physical fidelity, VR training systems should also leverage the strengths of virtual environments to visualize certain information that is not visible in the real world to benefit training outcomes. For example, the system can provide an on-demand visualization of the flow of water in the pipes to help trainees understand deeply why they should turn off a valve before another. This can be further enhanced by providing pre-recorded videos of experts explaining or giving tips to remember at critical points in the training task. Furthermore, having effective tools for users to take and review notes directly in VR is also important for trainees to personalize their learning process.

4 Conclusion

In this paper, we present an industrial research project exploring the applicability
of common off-the-shelf VR technologies into factory training. We designed an
exemplary VR training system supporting a representative maintenance task
called VRQUEST based on design considerations distilled from interviews with
SMEs and literature reviews. We then conducted a user study to examine the
strengths and the limitations of current VR technologies in supporting factory
training. Design insights derived from this research project were delivered to
different internal businesses of the company for adoptions in the future.

References

1. Sawyer, T., White, M., Zaveri, P., Chang, T., Ades, A., French, H., Anderson, J., Auerbach, M., Johnston, L. and Kessler, D.: Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Academic Medicin **90**(8), 1025–1033 (2015)

2. Jamet, E., Gavota, M. and Quaireau, C.: Attention guiding in multimedia learning. Learning and instruction **18**(2), 135–145 (2008)

3. Pence, T.B., Dukes, L.C., Hodges, L.F., Meehan, N.K. and Johnson, A.: The effects of interaction and visual fidelity on learning outcomes for a virtual pediatric patient system. In: 2013 IEEE International Conference on Healthcare Informatics, pp. 209–218. IEEE (2013)

4. Bangor, A., Kortum, P.T. and Miller, J.T.: An empirical evaluation of the system usability scale **24**(6), 574–594 (2008)