<figure><img src="image_1.png" /></figure>

HAL
open science

# AR/VR-Based Live Manual for User-Centric Smart Factory Services

Jae Yeol Lee, Duck Young Kim, Minseok Kim, Kyeong-Beom Park, Sung Ho Choi

► To cite this version:

Jae Yeol Lee, Duck Young Kim, Minseok Kim, Kyeong-Beom Park, Sung Ho Choi. AR/VR-Based Live Manual for User-Centric Smart Factory Services. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2018, Seoul, South Korea. pp.417-421, 10.1007/978-3-319-99707-0_52. hal-02177894

HAL Id: hal-02177894

https://inria.hal.science/hal-02177894

Submitted on 9 Jul 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
AR/VR-based Live Manual for User-centric Smart
Factory Services

Minseok Kim¹, Kyeong-Beom Park¹, Sung Ho Choi¹, Jae Yeol Lee¹*,
and Duck Young Kim²

¹ Chonnam National University, Gwangju 61186, South Korea

² UNIST, Ulsan 44919, South Korea

jaeyeol@jnu.ac.kr

**Abstract.** Although several physical implementations of the smart factory have been introduced, a viable implementation or platform has not been proposed to make human workers perceive smart factory services more effectively and naturally with respect to task assistance. In this paper, we propose an AR/VR-based Live Manual, a new approach to provide workers with user-centric smart manufacturing services in IoT-enabled smart factory testbeds. The proposed live manual can make both on-site and remote workers do their tasks more effectively by utilizing and synchronizing AR with VR, which can help them to make better decisions. The synchronized dual view between AR and VR can help workers to understand the situation more easily and clearly depending on the worker's contexts and manufacturing situations.

**Keywords:** AR/VR-based Live Manual, Smart Manufacturing Services, Augmented Reality, Virtual Reality, Smart Factory.

# 1 Introduction

One of the key themes in Industry 4.0 is to build a new manufacturing ecosystem (e.g., smart factory) through the fusion of smart manufacturing systems and information communication technology (ICT). The new manufacturing ecosystem refers to a customer-oriented production system that can incorporate ICT technology into the existing manufacturing services to effectively and actively reflect the user's demands and requirements [1,2].

To manufacture products, various production processes such as machining, assembly and inspection should be performed [3,4]. In order to be more productive and safe while performing tasks, the worker must be able to accurately and effectively recognize the manufacturing situation and the operation status of related facilities. The operating conditions of the facilities change in real time due to internal factors such as facility failure and external factors such as production status or customer requirement change. For this reason, it is very difficult for the worker to recognize all working situations and to acquire all related knowledge. Therefore, it is essential to provide relevant and user-centric visual information to workers in a form appropriate to their working situations [5,6].
With the advent of smart augmented reality (AR) and virtual reality (VR) devices, AR and VR have been widely applied to a variety of fields such as entertainment, game and education. In particular, in manufacturing industries, AR and VR are considered to play an important role in providing more user friendly visual information to the worker for task assistance and monitoring [7-9]. However, it is almost impossible to provide user-oriented manufacturing services without considering the worker's location, the status of manufacturing facilities, and worker's role and situation [5,6,8].

In this paper, we propose an AR/VR-based Live Manual, a new approach to provide
workers with user-centric manufacturing services regarding worker's contexts and sit-
uations by combining AR, VR, and deep learning in IoT-enabled smart factory testbeds.
The proposed live manual can be executable in various mobile and wearable devices,
and has been successfully applied to two smart factory testbeds. Section 2 overviews
the proposed approach. Section 3 presents the proposed AR/VR-based Live Manual
with some implementation results. Section 4 concludes the paper with some remarks.

2 Proposed AR/VR-based Live Manual

This paper proposes an AR/VR-based Live Manual to more effectively provide user-centric manufacturing services such as information visualization, monitoring, and sharing for task assistance and maintenance in the IoT-enabled smart factory environment. Fig. 1 presents the research goal and framework of the proposed live manual. We aim to develop an AR/VR-enabled mediator to workers in smart factory testbeds built in two cities such as Ulsan and Gumi, South Korea as shown in Fig. 1.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Proposed AR/VR Live Manual for supporting various kinds of user-centric task assistance.</figcaption></figure>
The AR/VR-based Live Manual supports the visual augmentation of manufacturing information acquired from the related manufacturing execution system (MES) in real time into physical artifacts through AR/VR technology depending on the user context or situation. The deep learning technology has also been utilized to find the worker's location and to detect physical facilities in the factory to overcome the limitation of the conventional marker-based AR. The AV/VR Live Manual consists of five main modules: Human-Machine Interface, AR/VR Visualization, Detection and Recommendation, Information View Management, and Interaction with Information modules. Based on the five modules, the AV/VR Live Manual enables to track and recognize physical facilities and products using deep learning-based AR, to analyze the detected results and relevant manufacturing information retrieved from MES and working environment, and to embed the analyzed information into the worker's display of smart devices, rendered in AR and VR.

## 3 AR/VR-based Live Manual with Deep Learning-based Object Detection

One of the key features of the AR/VR Live Manual is to make workers do their tasks more effectively by combining and synchronizing AR and VR. Furthermore, the deep learning method is applied to support physical object detection and to suggest recommendation for task assistance by analyzing manufacturing information, user contexts and result of the object detection.

For the traditional marker-based AR, it is essential that the facility or its surrounding environment must have a planar surface to attach AR markers. To make worse, to support different perspective-based recognition, multiple markers must be attached, which is almost impossible in the real environment. For this reason, the object detection without markers is also supported by applying a deep learning method such as YOLO [10] as shown in Fig. 2. This method can overcome the inherent problem of the marker-based AR.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Applying the deep learning-based object detection to AR [10].</figcaption></figure>
Based on the detected facilities and the worker's context, the live manual can retrieve related manufacturing information, filters the relevant information required for the worker, and augments it into corresponding physical facilities or products through smart and wearable devices. For this purpose, another machining learning method with an auto encoder (AE) is used, which can recommend a necessary task or suggest an appropriate information depending on the worker's role [11].

Another key feature of the AR/VR Live Manual is to provide a synchronized dual view in addition to the main AR view. In particular, the dual view can be switched, and the VR can be the main view if the worker needs alternative information than augmented information, which can complement the AR view. Therefore, the worker can more effectively monitor and analyze the manufacturing information depending on the situation of the working environment. Fig. 3 presents the implementation of the AR/VR-based Live Manual in two different modules of the smart factory testbeds.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Implementation of AR/VR-based Live Manual using handheld devices that has been applied to two different smart factory modules; the live manual supports dual views that can complement AR and AR as they are visually synchronized each other.</figcaption></figure>

We have developed the AR/VR-based Live Manual for providing user-centric manufacturing services, which has been applied to the two real smart factory testbeds built in two cities. The two testbeds can produce different types of BLDC motors and handpieces, respectively. In addition to the AR-based task assistance with the dual view and the deep machine learning, the proposed approach can also support different functionalities such as VR-based smart factory simulation with a high immersion using VR HMDs such as HTC Vive™, world-in-miniature (WIM)-based visualization, mixed reality (MR)-based visualization and monitoring of physical artifacts using wearable glasses such as MS HoloLens™. In particular, when a remote collaboration is performed, it is possible to interoperate with the on-site worker with the AR module. Using different level of functionalities, on-site workers and remote workers can monitor real-time manufacturing information more easily and perform required tasks more effectively.
## 4 Conclusion

In this paper, we proposed the AR/VR-based Live Manual, a unified approach to provide human workers with smart factory services in IoT-enabled testbeds. The proposed live manual system can be executable in various smart and wearable devices by taking advantage of combining and synchronizing AR and VR. In addition, it can make workers do their tasks more effectively as the proposed approach can provide situation-adaptive visualization with respect to the worker's surrounding contexts and manufacturing situations. The dual view conversion between AR and VR helps on-site workers to switch the live manual to either VR- or AR-focused mode using smart and wearable devices. We have also developed several AR/VR-based live manual modules using smart glasses. We are currently designing a more concrete platform for supporting various AR/VR-based task assistance. We are also evaluating the usability and usefulness of the proposed approach.

## Acknowledgements

This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03934697) and the Development of IIoT-based manufacturing testbeds for the Korean manufacturing equipment industry funded by the Ministry of Science and ICT (2015-0-00374).

## References

1. Lee, J., Bagheri, B., Kao, H.-A.: A cyber-physical systems architecture for Industry 4.0-based manufacturing systems. Manufacturing Letters 3, 18-23 (2015).

2. Wang, S., Wan, J., Li, D., Liu, C.: Knowledge reasoning with semantic data for real-time data processing in smart factory. Sensors 18(2), 471 (2018).

3. Peniche, A., Diaz, C., Trefftz, H., Paramo, G.: Combining virtual and augmented reality to improve the mechanical assembly training process in manufacturing. Proc. the 6th WSEAS International Conference on Computer Engineering and Applications, pp. 292-297 (2012).

4. Wang, X., Ong, S.K., Nee, A.Y.C.: Multi-modal augmented-reality assembly guidance based on bare-hand interface. Advanced Engineering Informatics 30(3), 406-421 (2016).

5. Henderson, S., Feiner, S.: Exploring the benefits of augmented reality document for maintenance and repair. IEEE Trans. Visualization and Computer Graphics 17(10), 1355-1368 (2011).

6. Webel, S., et al.: An augmented reality training platform for assembly and maintenance skills. Robotics and Autonomous Systems 61, 398-403 (2013).

7. Zhu, Z., Branzoi, V., Wolverton, M., Murray, G., Vitovitch, N., Yarnall, L.: AR-mentor: augmented reality based mentoring system. Proc. ISMAR, 17-22 (2014).

8. Liu, C., Cao, S., Tse, W., Xu, X.: Augmented reality-assisted intelligent window for cyber-physical machine tools. Journal of Manufacturing Systems 44, 280-286 (2017).
9. Manca, D., Brambilla, S., Colombo, S.: Bridging between virtual reality and accident simulation for training of process-industry operators. Advances in Engineering Software 55, 1-9 (2013)

10. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You Only Look Once: unified, real-time object detection. Proc. CVPR’16, 779-788 (2016).

11. Zhang, S., Yao, L., Xu, X., Wang, S., Zhu, L.: Hybrid collaborative recommendation via semi-AutoEncoder. arXiv preprint arXiv:1706.04453 (2017).