# Worker in the Loop: A Framework for Enabling Human-Robot Collaborative Assembly

Sotiris Makris, Panagiotis Angelakis, George Veloudis, Christos Gkournelos, Sotiris Makris

► To cite this version:

Sotiris Makris, Panagiotis Angelakis, George Veloudis, Christos Gkournelos, Sotiris Makris. Worker in the Loop: A Framework for Enabling Human-Robot Collaborative Assembly. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.275-283, 10.1007/978-3-030-85874-2_29. hal-04030415

HAL Id: hal-04030415

https://inria.hal.science/hal-04030415

Submitted on 16 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Worker in the loop: a framework for enabling Human-Robot collaborative assembly

Eleni Tzavara¹, Panagiotis Angelakis¹, George Veloudis¹, Christos Gkournelos¹[0000-0001-6834-3075] and Sotiris Makris¹[0000-0001-9687-5925]

¹ Laboratory for Manufacturing Systems and Automation (LMS), Department of Mechanical Engineering and Aeronautics, University of Patras, Patras 26504, Greece
makris@lms.mech.upatras.gr

**Abstract.** Industry has taken a big leap forward by placing a human in the center of interest by turning the working areas into a collaborative environment between operators and robots. In this environment, human behavior is a major uncertainty factor that can affect operator's safety and execution status. Furthermore, the creation of a digital twin including the whole workstation area, the operators and the procedures that take part in there, is a way to design and integrate collaborative systems using a virtual space. This paper aims to overview the current state of the technological trends in human detection, human task monitoring and digital twin integration. Also, the design of the upcoming solution of a case study from the automotive industry will be represented.

**Keywords:** human-robot collaboration, human task monitoring, digital twin, robot behavior.

## 1 Introduction

In the existing assembly systems, the ability to offer more variants per model and introduce new models faster is limited by the current technologies and equipment of mass production processes, which are incapable of supporting product diversity [1]. One of the most promising approaches over the last years is to increase the sensitivity of the production system to internal and external changes. Several paradigms such as flexible [2], reconfigurable [3], lean [4], holonic [5], self organizing [6] assembly systems have been realized in the last decades to meet these requirements.

In the last years, hybrid production systems [7], that combines flexibility and reconfigurability, enabling the collaboration between humans and robots is gaining increasing interest by the research community and the industrial world [8]. The desired result from creating human-robot collaborative environments is to utilize in their full extent the skills of human operators such as intelligence and cognitive capabilities supporting them through robot's strength and dexterity. The main concept is that in a shared workplace, several tasks are assigned to operators and robots following a production order. Those tasks' instructions must be followed strictly. There are numerous issues though, that need to be addressed to create a completely human-friendly working environment in the manufacture [9].
The major factor that is mainly unpredictable in such a collaborative environment, is human behavior, which can cause changes or issues in both production execution and safety. To ensure the prevention of issues in the production line, human detection, and task prevision are crucial. Last decades there is extended research on human modeling and monitoring [10]. In terms of monitoring the execution of human tasks, Andrianakos et al. [11] proposed a solution for monitoring the execution of human tasks. This solution is based on object detection and hand detection using in parallel machine learning techniques. Another way to achieve human modeling was proposed by M. Urgo et al. [12] combining a human pose estimation with a statistical model for operator's task identification. In [13] researchers suggested a solution based on the Dynamic Time Wrapping (DTW) algorithm, which is based on the measurement of similarity between two temporal sequences. Similar approach, we can see in [14]. In this work a product assembly task has been modeled as a sequence of human motions and the human motion prediction problem is solved by Hidden Markov Model (HMM).

In parallel with human task monitoring, robot behavior may need to be adapted to its current state. Considering many cases in which something can change the predefined robots' trajectory, it is clear that the adaptation is crucial for both human safety and task execution [15]. This need for adaptation leads to the creation of a virtual representation of the system that can monitor the behavior of the involved resources [16]. The use of the Digital Twin (DT) concept has gained a lot of attention given the advantages that it may offer towards more autonomous and intelligent systems [17]. This technology has promising results and seems to be compulsory component in smart manufacturing systems [18].

As previously stated, the current level of technology allows for a rather advanced model of Human Robot Collaboration (HRC). Different ways of human detection and task monitoring have been developed in recent years, which in conjunction with Digital Twins could lead to a flexible, reconfigurable, and safe production system. This paper presents a framework that includes the design of the abovementioned components for enabling the synergy between humans and robots in flexible manufacturing systems. The desirable solution will be deployed in an assembly use case from the automotive industry. The paper is organized as follows: Section 2 describes the main approach and the design of the framework; in Section 3, the way that the whole framework will be implemented is analyzed; Section 4 presents the automotive case study in which this work will be deployed and finally Section 5 reports the conclusion and the future work.

## 2 Approach

Following the aforementioned challenges in the creation of a collaborative and at the same time safe workplace, a framework will be presented in the current work that aims to include the workers in the control loop of the production with robots. Fig. 1 presents the conceptual architecture of the proposed framework. The whole procedure is represented as a loop that starts and ends with the human worker.

The two modules that extract information from the worker are the Human Body Detection (HBD) and Human Task Prediction (HTP). These are capturing the data from
different types of sensors, such as 3D vision sensors 2D laser scanners and wearable IMUs. The output data from those two modules alongside with the robot state, become inputs in the DT module which is responsible for informing Assembly Execution Controller and also Human side interfaces. More specifically, Human side interfaces are informed from the DT about human's and robot's current task and give feedback to the human. The loop ends when the operator is notified about errors in the assembly, execution status of the whole procedure, and warnings regarding human safety.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Worker in the loop of Human-Robot collaborative assembly</figcaption></figure>

## 2.1 Human Body Detection

The detection of human existence is a mandatory component in a H-R collaborative assembly system for ensuring the safety of the operator. Aiming beyond the standard means of safety detection that causes emergency stops on robots when the worker is near, advanced information is needed alongside with direct interface with robot trajectory execution. Robot follows a predefined trajectory in order to execute its task. By detecting the accurate position of the operator, it must be ready to adjust the trajectory to avoid the operator and ensure human's safety. Beyond safety of operator, replan of robot's trajectory can lead to a more efficient production, since the stoppage time of emergency is eliminated. HBD provides more detailed information regarding the human body posture and position on the workspace. 2D and 3D data are fused for providing the position of the whole human body.
## 2.2 Human Task Prediction

In the actual assembly line, operators and robots follow a predefined order of tasks. While the tasks that are executed by robots can be monitored, human processes cannot be fully tracked. Operators have several degrees of freedom which means that the way a task is executed, defers from person to person. In addition to this, an operator can make additional moves such as touch his head or adjust his uniform. This is a deviation from the predefined order of actions which should not be considered an error. This component aims to predict human intention and task status taking into consideration the different ways a task can be executed and also the minor deviations that may occur during execution. In order to be able to deal with the uncertainties and the noise in the measurements HTP module consists of probabilistic models.

Through this procedure, useful information is retrieved for enhancing the collaboration of the worker. Such information could be that operator completes a task so the robot could continue the procedure or that operator needs the assistance of a robot for completing a task. This “communication” between human and robot currently established through buttons that directly informs the system about the worker’s status. HTP provides indirect communication, and it is not imposing on the operators a strict workflow.

## 2.3 Digital Twin

A DT is considered as a bridge between the real and digital world. To effectively represent the whole workstation area this module must interact directly with the real world. More specifically, the DT will take as input information about the workstation area layout, the resources (robot operations, robot state), and the different parts (consumables, assembly parts, fixtures) that exist in the real world. The proposed DT is hardware agnostic and could integrate seamlessly multiple robots and sensors. Furthermore, the real-time awareness of the workstation is achieved with the use of different sensors. As shown in Fig. 1, sensor data needs to be captured and fitted in order to provide information for keeping up to date with the digital models. Apart from retrieving data, DT provides feedback to the operators through User Interfaces. The operator is informed about the execution status, robot's tasks, warnings, and errors that may occur in the assembly.

# 3 Implementation

As described in Section 2, the whole framework consists of three modules that need to be implemented, HBD, HTP, and DT. The implementation is based on ROS [19] principles. Each of the included modules is implemented as ROS nodes which exchange information by using the ROS interfaces (topics, services, actions). The initial setup of the collaborative workplace consists of a stationary 3D camera placed in front of the operator, to have good visual coverage of the human, and four IMU sensors attached on the operator's hands.
HBD requires the use of a 3D data in order to detect the operator's position. Skeleton tracking is a complicated aspect that is solved with the use of OpenPose [20] which is a real-time software that detects multi-person human body-parts key points such as hands, legs, core, and head which are provided in a quite accurate approach. Since OpenPose provides only 2D detection, the use of an algorithm that synchronizes these body parts with a Point Cloud (PCL) is used to extract the 3D location of these points. Fig. 2 presents the extracted result first using only the OpenPose and then the combination of OpenPose with Pointcloud.

<figure><img src="image_4.png" /><figcaption>Fig. 2. a) Pointcloud data combined with skeleton information b) detected skeleton points</figcaption></figure>

The HBD provides information about 25 human body key points. Focusing on tasks that are performed with hands the HTP module extracts the positions of human wrists from points 5 (right hand) and 8 (left hand). Additionally, the operator is equipped with two IMUs on his wrists. These sensors smooth the detected body parts coming from the HBD and most usefully augment these parts' 3D position with orientation information in the form of quaternions.

The HTP module apart from the data that is captured from the human, uses semantic information from the workplace. The workstation is divided into several schematic areas which are labelled according to heuristic knowledge on use case. Examples of such areas could be “kitting” areas, “assembly table” “machines’ fixtures” etc. The combination of the abovementioned extracted information is fed into a probabilistic Hidden Markov Model (HMM) that achieves human task monitoring. The use of this model requires state modelling. An example of this modelling for a “Pick-n-Place” task presented in Table 1.

In Fig. 3 it is provided to the left the simulation environment of observation “O1” and its DT on the right. Simulation environment is built in GAZEBO physics simulation [21]. By combining the output data from HBD, HTD, and the robot state, DT is responsible for informing operator about the current state of the assembly. More specific ROS services and actions are sent to the operator as actions or goals in order to inform him about the assigned task or warnings. DT also provides interfaces for robot
integration and connection with external software that will monitor the production system. These will be detailed described on future publications.

<table><caption>**Table 1.** Observation table of HMM for a Pick-n-Place task</caption><thead><tr><td>Observation</td><td>Description</td></tr></thead><tbody><tr><td>s</td><td>Initial state</td></tr><tr><td>O1</td><td>Grab cylinder</td></tr><tr><td>O2</td><td>Pick cylinder</td></tr><tr><td>O3</td><td>Place cylinder</td></tr><tr><td>t</td><td>End of "pick and place"</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** DT representation alongside with the simulation environment</figcaption></figure>

# 4 Automotive case study

The proposed framework will be implemented in an automotive mechanical plant at the production line of the electric motors. This use case is focusing on the assembly of the inverter with the e-motor. The e-motors are transferred in the specific workstation on a conveyor and inverters are stored on a kitting area inside the workstation. The inverter needs to be placed on the e-motor and screwed with eight screws. A robot and an operator is allocated on this workstation for carrying the assembly tasks. The main goal of this implementation is seamless human-robot collaborate, the dynamic allocation of tasks to each resource for maximizing the production efficiency always ensuring the operator's safety.

For the initial testing of our developments a simple pick and place scenario has been deployed. This scenario includes a block with twelve holes that travels on the conveyor and a kitting area with cylinders that should be placed inside the holes. Both human worker and robot is suitable to pick and place these cylinders. Robot is aware of the execution progress of the operator's task and the human existence. By monitoring the status execution of the human tasks, the robot will be informed about the failure or the completion and it will proceed to its task.

In case the operator misses a task, robot will be informed, and it will execute the task instead of interrupting the whole process. From the other side operator is aware of robot's status and could intervene in case of a failure. The information about next task
will be extracted from the DT. To test the whole design described above, an initial scenario has been deployed in GAZEBO simulation as shown the Fig. 4.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Automotive case study workstation simulation</figcaption></figure>

# 5 Conclusion

The latest trends in manufacturing, pose the need to create more human friendly assembly by embracing the Human-Robot collaboration. Reorganization of tasks can lead to a more efficient assembly basically because of the elimination of stoppage time in case of errors. The development and the continuous research on this topic can lead to a more efficient and autonomous assembly. It is also important to mention that those developments can affect the line performance. Driven by this need, an initial description of framework that will improve the H-R collaboration was presented in the previous sections. By the development of this framework, the issues that may occur can be detected and resolved in real time, to prevent adjustment and reschedule of the production plan or the process.

As future plans, we intend to create a testbed in the laboratory for the actual implementation of the design that was described above. Numerous of tests will be executed in order to validate the whole use case on the testbed and also on the actual factory premises.

# Acknowledgements

This work has been partially funded by the EC research project "ASSISTANT – Learning and robust decision Support systems for agile manufacturing environments" (Grant Agreement: 101000165) (www.assistant-project.eu).
References

1. Chryssolouris, G.: Manufacturing Systems: Theory and Practice. Springer New York, New York, NY (2006). https://doi.org/10.1007/0-387-28431-1.

2. Chryssolouris, G., Georgoulias, K., Michalos, G.: Production systems flexibility: Theory and practice. In: IFAC Proceedings Volumes (IFAC-PapersOnline). pp. 15–21. IFAC Secretariat (2012). https://doi.org/10.3182/20120523-3-RO-2023.00442.

3. Koren, Y., Shpitalni, M.: Design of reconfigurable manufacturing systems. J. Manuf. Syst. 29, 130–141 (2010). https://doi.org/10.1016/J.JMSY.2011.01.001.

4. Houshmand, M., Jamshidnezhad, B.: An extended model of design process of lean production systems by means of process variables. Robot. Comput. Integr. Manuf. 22, 1–16 (2006). https://doi.org/10.1016/j.rcim.2005.01.004.

5. Giret, A., Botti, V.: Engineering Holonic Manufacturing Systems. Comput. Ind. 60, 428–440 (2009). https://doi.org/10.1016/j.compind.2009.02.007.

6. Scholz-Reiter, B., Freitag, M.: Autonomous Processes in Assembly Systems. CIRP Ann. - Manuf. Technol. 56, 712–729 (2007). https://doi.org/10.1016/j.cirp.2007.10.002.

7. Kousi, N., Michalos, G., Aivaliotis, S., Makris, S.: An outlook on future assembly systems introducing robotic mobile dual arm workers. Procedia CIRP. 72, 33–38 (2018). https://doi.org/10.1016/j.procir.2018.03.130.

8. Makris, S.: Cooperating robots for flexible manufacturing. Springer (2021). https://doi.org/10.1007/978-3-030-51591-1.

9. Wang, L., Gao, R., Váncza, J., Krüger, J., Wang, X. V., Makris, S., Chryssolouris, G.: Symbiotic human-robot collaborative assembly. CIRP Ann. 68, 701–726 (2019). https://doi.org/10.1016/j.cirp.2019.05.002.

10. Tsarouchi, P., Michalos, G., Makris, S., Athanasatos, T., Dimoulas, K., Chryssolouris, G.: On a human-robot workplace design and task allocation system. Int. J. Comput. Integr. Manuf. 30, 1272–1279 (2017). https://doi.org/10.1080/0951192X.2017.1307524.

11. Andrianakos, G., Dimitropoulos, N., Michalos, G., Makris, S.: An approach for monitoring the execution of human based assembly operations using machine learning. In: Procedia CIRP. pp. 198–203. Elsevier B.V. (2020). https://doi.org/10.1016/j.procir.2020.01.040.

12. Urgo, M., Tarabini, M., Tolio, T.: A human modelling and monitoring approach to support the execution of manufacturing operations. CIRP Ann. 68, 5–8 (2019). https://doi.org/10.1016/j.cirp.2019.04.052.

13. Maderna, R., Lanfredini, P., Zanchettin, A.M., Rocco, P.: Real-time monitoring of human task advancement. IEEE Int. Conf. Intell. Robot. Syst. 433–440 (2019). https://doi.org/10.1109/IROS40897.2019.8967933.

14. Liu, H., Wang, L.: Human motion prediction for human-robot collaboration. J. Manuf. Syst. 44, 287–294 (2017). https://doi.org/10.1016/j.jmsy.2017.04.009.

15. Sankar, S., Tsai, C.Y.: Ros-based human detection and tracking from a wireless controlled mobile robot using kinect. Appl. Syst. Innov. 2, 1–12 (2019). https://doi.org/10.3390/asi2010005.

16. Váncza, J., Monostori, L.: Cyber-physical Manufacturing in the Light of Professor Kanji Ueda's Legacy. In: Procedia CIRP. pp. 631–638. Elsevier B.V. (2017). https://doi.org/10.1016/j.procir.2017.04.059.

17. Kousi, N., Gkournelos, C., Aivaliotis, S., Giannoulis, C., Michalos, G., Makris, S.: Digital twin for adaptation of robots' behavior in flexible robotic assembly lines. Procedia Manuf. 28, 121–126 (2019). https://doi.org/10.1016/J.PROMFG.2018.12.020.

18. Alexopoulos, K., Nikolakis, N., Chryssolouris, G.: Digital twin-driven supervised machine learning for the development of artificial intelligence applications in manufacturing. Int. J.
Comput. Integr. Manuf. 33, 429-439 (2020).
https://doi.org/10.1080/0951192X.2020.1747642.

19. Quigley, M., Gerkey, B., Conley, K., Faust, J., Foote, T., Leibs, J., Berger, E., Wheeler, R., Ng, A.: ROS: an open-source Robot Operating System. In: International Conference on Robotics and Automation (2009).

20. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Trans. Pattern Anal. Mach. Intell. 43, 172-186 (2021). https://doi.org/10.1109/TPAMI.2019.2929257.

21. Gazebo Robot Simulation, http://gazebosim.org/, last accessed 2021/07/14.