# Human Digital Twin System for Operator Safety and Work Management

Goo-Young Kim, Donghun Kim, Sang Do Noh, Hong Ku Han, Nam Geun Kim, Yong-Shin Kang, Seung Hyun Choi, Dong Hyun Go, Jungmin Song, Dae Yub Lee, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Goo-Young Kim, Donghun Kim, Sang Do Noh, Hong Ku Han, Nam Geun Kim, et al. Human Digital Twin System for Operator Safety and Work Management. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.529-536, 10.1007/978-3-031-16411-8_61. hal-04768979

HAL Id: hal-04768979

https://inria.hal.science/hal-04768979v1

Submitted on 6 Nov 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Human Digital Twin System for
Operator Safety and Work Management

Goo-Young Kim¹[0000-0003-3337-8782], Donghun Kim¹[0000-0001-8734-4559], Sang Do Noh¹*[0000-0002-5610-5560], Hong Ku Han², Nam Geun Kim³, Yong-Shin Kang⁴, Seung Hyun Choi⁴, Dong Hyun Go⁵, Jungmin Song⁵, Dae Yub Lee⁵ and Hyung Sun Kim⁵

¹ Sungkyunkwan University, Suwon 16419, Republic of Korea
sdnoh@skku.edu

² Shinsung E&G, Seongnam 13543, Republic of Korea

³ Shinsung CS, Seongnam 13543, Republic of Korea

⁴ Advanced Institute of Convergence Technology, Suwon 16229, Republic of Korea

⁵ DEXTA Inc., Suwon 16677, Republic of Korea

**Abstract.** The value-driven Industry 5.0 has brought a shift in the approach towards worker well-being. However, the understanding of the effects on workers due to technological advancements of Industry 4.0, based on a human-centric approach, is limited. The reason for this limitation is that the tools are scarce, which is quantitatively evaluating and analyzing various factors in the workplace. To solve this problem, we propose a human digital twin system supporting decision-making regarding safety management and work management of workers. The human digital twin system consists of a digital twin module, an analysis module, and a visualization module. The proposed system connects a physical human and a virtual digital human model; analyzes the location, posture, and motion-time of workers; and delivers information about safety and work management. This information enables workers and managers to improve the work environment by making them resilient to workplace factors.

**Keywords:** Industry 5.0, Operator 4.0, Human Digital Twin System.

# 1 Introduction

Industry 5.0, announced by the European Commission, has modified the approach of industries towards worker well-being [1]. Technological advancements of Industry 4.0 based on a human-centric approach affect workers who are essential resources in manufacturing systems are considered [2]. For allowing workers to respond flexibly to various factors existing in a workplace, a method was proposed, which is making the manufacturing system more agile and flexible [3]. Additionally, it was argued that future research to create a worker-centered work environment is necessary through reliable and quantifiable technology [4]. In other words, the problem is the lack of skills to quantitatively evaluate and analyze factors in the work environment for workers. To solve this problem, we propose a human digital twin system that supports decision-
making regarding safety management and work management of workers. The human
digital twin mirrors the physical environment into a virtual world and enables the mon-
itoring of workers of the physical environment. Furthermore, the digital twin aids deci-
sion-making for workers and managers by synchronizing and analyzing field data. This
study utilizes motion capture devices and mobile devices to implement the digital hu-
man model (DHM) of the human digital twin and connects physical and virtual workers.
Through visualization, the workers and managers confirm quantitatively evaluated and
measured analysis results. The analysis results, such as safety level and process smooth-
ness, support improvement in the work environment for workers by making workers
and managers resilient to workplace factors.

## 2 Research Background

2.1 Operator 4.0

Under the technology-driven Industry 4.0, a human-centric approach has attracted global attention because of the value-driven Industry 5.0 announced by the European Commission [1]. Operator 4.0 was presented as a smart and skilled operator of the future that involves leveraging the collaboration between cooperative robots and accurate machinery and the human being’s ingenious unique potential [5]. In the Operator 4.0 typology, the healthy and smart operator has the ability to detect problems and suggest solutions in machines and systems that build predictive models for proactive actions [6]. In addition, Healthy Operator 4.0 architecture was suggested, which is composed of smart connection, integration and communication, modeling, and cognition layer [7]. Through this architecture [7], the gathered data sources are analyzed in real-time with advanced modeling technologies to derive insightful inferences. Similarly, a framework of human cyber-physical system was presented for symbiosis between Operator 4.0 and artificial intelligence to facilitate feedback and decision-making from the shop-floor [8]. Moreover, with the upcoming Industry 5.0 phase, the focus of research is shifting towards Operator 5.0. An argument is being made that a new collaborative framework for Operator 5.0 should be designed using advanced digital technologies such as mixed reality, deep learning, and digital twins [9].

## 2.2 Digital Twin

Digital twin is a data-driven digital replica model of a product, machine, and manufac-
turing process that reflects the property and state of the object and explains how a phys-
ical and virtual model behave. According to [10], the digital twin not only supports
real-time monitoring and tracking of diverse situations in the physical environment by
reflecting it in the cyber environment, but also the interaction between the physical and
cyber environments through repetitive communication and convergence. Based on
these characteristics of the digital twin, digital twin-based services assist shop-floor
workers and factory managers to share information about the production sites and cre-
ate a safe environment for workers by predicting collisions through real-time location
information [11]. Through the digital twin application, workers' motions were synchro-
nized between the physical and cyber environments and analyzed by using motion cap-
ture technology [12]. Meanwhile, a framework was proposed to support monitoring and
decision-making by implementing human digital twin [13]. In addition, a digital twin
concept and prototype were proposed to represent human operators in the cyber envi-
ronment for acquiring and analyzing data by using a simulation-based framework [14].
Similarly, a human-centric digital twin architecture was suggested to improve operator
capabilities by using derived context information as a solution [15].

3 Human Digital Twin System

<figure><img src="image_4.png" /><figcaption>Fig. 1. Human Digital Twin System for Operator Safety and Work Management</figcaption></figure>

Human digital twin system (HDTS) for operator safety and work management consists of digital twin, analysis, and visualization modules, as shown in Figure 1. The digital twin module composes DHM, an important element of the human digital twin, by connecting the worker in the real environment with the worker object in the virtual environment. The analysis module analyzes real-world worker-related data acquired through a virtual environment for worker safety and work management. The visualization module delivers the analysis results to the workers and managers who make decisions by checking and reflecting on the data from results.
## 3.1 Digital Twin Module

To connect the physical world and virtual world, it is necessary to create an object in the virtual world for an object in the physical world and synchronize data between the two objects. The proposed architecture introduces mobile devices and motion capture equipment to synchronize between the real world and virtual world. Real-time location coordinates of workers are acquired through mobile devices and access points installed in the real environment. Musculoskeletal data is acquired through the accelerometer and gyroscope sensor values of the mobile device and the data of the motion capture equipment worn by the operator. The data acquired through mobile sensors and motion capture equipment connects real-world workers and virtual-world objects. The position of the worker object in the virtual environment is matched according to the position of the worker in the real environment, and the movement of the worker object in the virtual environment is also synchronized according to the movement of the worker in the real environment. Finally, the DHM in the virtual environment is implemented through the acquired sensor data of the worker.

## 3.2 Analysis Module

The acquired data is not only used to implement the DHM, but also for data analysis, such as identification of the worker's workplace, posture and work fatigue analysis, and safety assessment. The location coordinate data of the worker is used to identify specific locations of the worker through the localization module; that is, through the localization module, it is possible to identify the work-center where the operator and tools are located. The accelerometer and gyroscope of the mobile device are used to analyze the movement and posture of the operator through the posture detection module. The worker's location and surrounding tools, motion, and posture data analyzed through the localization module and posture detection module embedded in the expert system are used to evaluate worker safety through rule-based reasoning. Furthermore, the motion capture equipment's skeleton data are used to analyze the fatigue of workers and the execution time of the operator's standard work using a fatigue analysis module, which is a rapid upper limb assessment (RULA), and the motion-time analysis module.

## 3.3 Visualization Module

The accident safety level of workers and work performance fatigue obtained through the analysis of the acquired data is used by safety managers and process managers to monitor shop-floor workers. Shop-floor workers detect the safety of accident and adjust their posture according to the safety level determined by their posture in relation to their location and the work they conduct. The safety manager performs safety management for all workers located on-site and takes precautions against possible accidents. The process manager understands the fatigue and work smoothness of the shop-floor workers and ensures the process productivity and safety from musculoskeletal injuries. Furthermore, standardized worksheets required for production planning can be prepared and used by standardizing worker processing time through working time analysis.
# 4 Case Study

## 4.1 Experiment

We implemented the proposed HDTS for the human-based assembly process of fan filter-producing units. A worker performs a task using the upper body to take assembly tools and materials from a workbench connected to a conveyor belt. To create DHM, workers on the shop-floor wear motion capture devices and carry mobile devices to acquire joint data such as accelerometer and gyroscope data from their skeletons and coordinate data from their shop-floor map. For the shop-floor map, we analyzed the production line and factory layout and developed a virtual factory using Unity. Figure 2 shows workers wearing motion capture devices in a physical factory and avatars representing shop-floor workers in a virtual factory. The avatar of the virtual environment is connected with the worker of the physical environment through the data acquired from the physical environment. The avatar's data, representing the worker's data, is transferred to the analysis module of the HDTS.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Implementation of Digital Human Model. (a) Actual Worker. (b) Virtual Avatar</figcaption></figure>

The coordinates of a shop-floor map and sensor values such as accelerometers and gyroscopes are used for safety management. The indoor localization model yields a workplace and equipment that the worker can locate by using shop-floor map coordinates. The posture detection model returns the actions of the worker, such as standing, walking, walking down, walking up, sitting, and lying down by using the accelerometer and gyroscope of the mobile device. The rule-based reasoning model of safety management generates the safety level of the worker by utilizing the outcome of the indoor localization and posture detection model. The skeleton data from the motion capture device is used for work management. The fatigue analysis model utilizes data from the accelerometer, gyroscope, and magnetometers, which are acquired from sensors attached to the joints of the human body. The fatigue level of DHM is calculated through RULA, and the change in fatigue level flow during the process is checked to calculate the work smoothness of the process. These sensor data are also used in the motion-time analysis model. The motion-time analysis model identifies standard tasks that compose the assembly process and calculate the working time of each standard task. By analyzing the pattern of the acquired data, each standard operation is classified, and the working time is calculated by evaluating the start and end times of each standard operation of the
DHM. The visualization module aggregates and shows the results of the analysis module, helping the manager monitor the current safety level of all workers by expressing the safety level of workers in real time, as shown in Figure 3.

<figure><img src="image_6.png" /><figcaption>**Fig. 3. Dashboard for Worker's Safety Level Visualization**</figcaption></figure>

## 4.2 Empirical Results

The result of applying the HDTS helps managers' decision-making by visualizing the analysis module. Figure 4 depicts (a) worker's posture in physical environment, (b) avatar's posture in virtual environment depending on worker's posture in physical environment, and (c) the workplace, surrounding equipment, posture and accuracy, and safety level. According to the current situation, the worker's safety level is "MIDDLE." The safety level is considered appropriate because there are ladders around the worker in the physical environment and the worker is in a sitting position. Likewise, safety managers are able to monitor the safety levels of all workers even if they are not on the shop-floor sites.

<figure><img src="image_7.png" /><figcaption>**Fig. 4. Analysis Results of Rule-based Inference in Safety Management**</figcaption></figure>

Work management shows work smoothness, standard tasks, and standard processing times. Figure 5 demonstrates the work smoothness of workers in the assembly process.
Each worker performs the same assembly process but with different work smoothness because they have dissimilar detailed tasks. Conversely, each worker has a different level of work smoothness even when assigned to the same assembly process due to the variety of standard tasks performed by the workers, and the safety from injury is also different because the workers experience different fatigue levels while performing the same process.

<figure><img src="image_8.png" /><figcaption>Fig. 5. Analysis Results of Work Smoothness in Work Management</figcaption></figure>

# 5 Conclusions

This study presented and implemented an HDTS to support decision-making about safety management and work management of operators. The proposed system, composed of digital twin, analysis, and visualization modules, implemented a DHM of an actual worker, calculated safety and fatigue levels, and expressed analysis results. Future research could focus on verifying results by enhancing the accuracy of machine learning models used in safety and work management. Furthermore, the digital twin module not only has a chance to apply DHM with a collaborative robot in a factory, but also to utilize augmented and virtual reality devices for workers in the physical environment to accept feedback.

# Acknowledgement

This research was financially supported by the MOTIE and KIAT through the International Cooperative R&D program [P0009839] and supported by project for Smart Manufacturing Innovation R&D funded Korea Ministry of SMEs and Startups in 2022 [RS-2022-00140261].
References

1. Breque, M., De Nul, L., & Petridis, A. Industry 5.0: towards a sustainable, human-centric and resilient European industry. Luxembourg, LU: European Commission, Directorate-General for Research and Innovation (2021).

2. Pinzone, M., Albè, F., Orlandelli, D., Barletta, I., Berlin, C., Johansson, B., & Taisch, M. A framework for operative and social sustainability functionalities in Human-Centric Cyber-Physical Production Systems. Computers & industrial engineering, 139, 105132. (2020).

3. Romero, D., & Stahre, J. Towards The Resilient Operator 5.0: The Future of Work in Smart Resilient Manufacturing Systems. Procedia CIRP, 104, 1089-1094. (2021).

4. Lu, Y., Zheng, H., Chand, S., Xia, W., Liu, Z., Xu, X.,... & Bao, J. Outlook on human-centric manufacturing towards Industry 5.0. Journal of Manufacturing Systems, 62, 612-627. (2022).

5. Romero D, Bernus P, Noran O, Stahre J, Fast-Berglund Å. In: The operator 4.0: human cyber-physical systems & adaptive automation towards human automation symbiosis work systems. Cham. p. 677-86. Springer. (2016).

6. Romero, D., Stahre, J., Wuest, T., Noran, O., Bernus, P., Fast-Berglund, Å., & Gorecky, D. Towards an operator 4.0 typology: a human-centric perspective on the fourth industrial revolution technologies. In proceedings of the international conference on computers and industrial engineering (CIE46) pp.29-31, Tianjin, China, (2016).

7. Sun, S., Zheng, X., Gong, B., Garcia Paredes, J., & Ordieres-Meré, J. Healthy operator 4.0: A human cyber-physical system architecture for smart workplaces. Sensors, 20(7), 2011. (2020).

8. Bousdekis, A., Apostolou, D., & Mentzas, G. A human cyber physical system framework for operator 4.0-artificial intelligence symbiosis. Manufacturing letters, 25, 10-15. (2020).

9. Mourtzis, D., Angelopoulos, J., & Panopoulos, N. Operator 5.0: A survey on enabling technologies and a frame-work for digital manufacturing based on extended reality. Journal of Machine Engineering, 22. (2022).

10. Park, K. T., Nam, Y. W., Lee, H. S., Im, S. J., Noh, S. D., Son, J. Y., & Kim, H. Design and implementation of a digital twin application for a connected micro smart factory. International Journal of Computer Integrated Manufacturing, 32(6), 596-614. (2019)

11. Kim, G. Y., Flores-García, E., Wiktorsson, M., & Noh, S. D. Exploring Economic, Environmental, and Social Sustainability Impact of Digital Twin-Based Services for Smart Production Logistics. In IFIP International Conference on Advances in Production Management Systems. pp. 20-27. Springer, Cham. (2021).

12. Nam, Y. W., Lee, S. H., Lee, D. G., Im, S. J., & Noh, S. D. Digital Twin-based Application for Design of Human-Machine Collaborative Assembly Production Lines. Journal of Korean Institute of Industrial Engineering, 46(1), 42-54. (2020).

13. Greco, A., Caterino, M., Fera, M., & Gerbino, S. Digital twin for monitoring ergonomics during manufacturing production. Applied Sciences, 10(21), 7758. (2020)

14. Sharotry, A., Jimenez, J. A., Wierschem, D., Mediavilla, F. A. M., Koldenhoven, R. M., Valles, D.,... & Aslan, S. A digital twin framework for real-time analysis and feedback of repetitive work in the manual material handling industry. In 2020 Winter Simulation Conference (WSC). pp. 2637-2648. IEEE. (2020).

15. Löcklin, A., Jung, T., Jazdi, N., Ruppert, T., & Weyrich, M. Architecture of a Human-Digital Twin as Common Interface for Operator 4.0 Applications. Procedia CIRP, 104, 458-463. (2021).