# Analyzing Human Robot Collaboration with the Help of 3D Cameras

Robert Glöckner, Lars Fischer, Arne Dethlefs, Hermann Lödding

► To cite this version:

Robert Glöckner, Lars Fischer, Arne Dethlefs, Hermann Lödding. Analyzing Human Robot Collaboration with the Help of 3D Cameras. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2019, Austin, TX, United States. pp.642-649, 10.1007/978-3-030-30000-5_78. hal-02419196

HAL Id: hal-02419196

https://inria.hal.science/hal-02419196

Submitted on 19 Dec 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Analyzing Human Robot Collaboration with the Help of
3D Cameras

Robert Glöckner¹, Lars Fischer², Arne Dethlefs²

and Hermann Lödding¹

¹ Hamburg University of Technology, 21073 Hamburg, Germany

² Garz und Fricke, Tempowerkring 2, 21079 Hamburg, Germany

robert.gloeckner@tuhh.de

**Abstract.** Recent developments in robotics allow the design of work systems with enhanced human robot collaboration (HRC) for assembly tasks. Productivity improvements are a common aim for companies that look into the implementation of HRC. To harvest the full productivity potential of these work systems, an analysis of the HRC work processes is essential. However, a dedicated method for the analysis of productivity in HRC is missing. This paper presents an approach using 3D-cameras to observe the employee in HRC. The approach links this information to robot states. The resulting analysis aims at improving the productivity of the work system e.g. by identifying and reducing balancing losses in HRC. The method tracks the movements of the employees in the HRC area and matches the corresponding robot activities.

**Keywords:** Human Robot Collaboration, Productivity Analysis, 3D cameras

# 1 Introduction

Smart manufacturing has a huge impact to on the design and operation of manufactur-
ing systems and accordingly the work of employees in manufacturing. Romero et al.
[1] identify 8 types of technology induced employee augmentation. One aspect of this
typology is collaborative automation or “the collaborative operator”. Recent develop-
ments in robotics allow the design of human robot collaboration work systems with a
very deep integration of robot activities in manual operations. As described by Romero
et al. [1], HRC can transfer tedious, non-ergonomic and vulnerable tasks to the robot
and achieve productivity and quality improvements as well as an increased operator
satisfaction. Of the manifold reasons for the implementation of HRC, improvements in
productivity are most important for many companies [2]. To ensure a high productivity
in HRC work systems, an adequate analysis of work processes is essential.

Many approaches exist to analyze the productivity of manual work (work sampling, time studies) as well as of machines (overall equipment efficiency) [3, 4]. A dedicated approach to analyze HRC work systems that supports continuous improvement actions is missing.
This paper presents an approach to analyze HRC work systems with the help of 3D-cameras and a robot interface. The camera tracks the position and the movement of the employees and a software matches corresponding robot data to create a digital twin of the HRC work system. The recorded data is analyzed with the help of state type diagrams as proposed by Schweitzer [5].

## 2 Collaborative robots and human robot collaboration

When referring to collaborative robots or cobots, many authors describe robots that are capable of working within the same environment as human employees without external safety barriers such as fences [6]. In order to do this, these robots usually provide enhanced safety features like rounded edges, reduced weight, limited power and force and protective skins that can detect collisions or the proximity of the employee. In addition to these safety requirements, Romero et al. [1], Mautua et al. [6] and Wang et al. [7] refer to enhanced interaction mechanisms as necessary feature for HRC.

Many authors describe HRC regarding the temporal and spatial relation of collaborating humans and robots. Human robot collaboration is assumed to have at least partially no spatial and no temporal separation of human and robot. This includes simultaneous and supportive work of human and robot on the same workpiece at the same time [2, 7, 8]. In addition to these forms of collaboration, employees and robots may also work independent when completing tasks with spatial separation or with temporal separation.

Deploying HRC in a way that increases productivity is a complex design task [2]. While the initial design and task allocation is addressed by some authors (for instance in [9]), a methodology that supports analysis of work in the operation of HRC for continuous improvement is missing.

## 3 Productivity and work analysis

### 3.1 Productivity and conventional work analysis

In the context of HRC, both the productivity of the employee (labor productivity) and of the robot (machine productivity) are important. Due to the deep integration of human and robot activities, breakdowns or waiting times of either the robot or the employee can lead to losses in the overall productivity of the work system.

While measurement of machine productivity can be automated in many cases, the vast majority of existing methods for assessing labor productivity uses observers to analyze the work processes (see [3]). In order to draw conclusions, states of robot and operator have to be captured simultaneously. This usually exceeds the cognitive capabilities of an observer, also because the robot state is not obvious (especially when deploying external equipment) and robot control information is not available for the observer.
## 3.2 Analysis of work processes with 3D cameras

Motion capturing systems track human positions and movements and make this data available for further processing. These systems can be used to analyze work processes without the need of an observer. Both electro-mechanical systems that use special suits to determine human motions and optical systems as 3D cameras can be used to track human movements. 3D cameras often emit an infrared light, which is reflected by the human body or special markers and captured by a stereoscopic sensor set. Optical systems require special software to detect human bodies. While electro-mechanical systems usually provide a superior data quality, the additional equipment is often expensive, has to be carefully attached to the employee and may impede the work process. Optical systems often suffer from a lower data accuracy, especially in environments that partly conceal the observed employee [10]. We opted for 3D-cameras, because their accuracy is good enough for the purpose of productivity analysis and because they are easy to use.

Various approaches have been developed to analyze work processes in regards to productivity with the help of optical systems. Escoria et al. [11] developed a method to detect characteristic work processes on construction sites on the basis of specific body positions. Ying et al. [12] propose an approach that uses convolutional neural networks in order to detect different activities of workers in a scaffolding process with the help of regular cameras. Benter [10] uses the Microsoft Kinect sensor and the corresponding software set to determine stopping points of the whole body or single body joints. Benter uses these stopping points to distinguish different movements during manual processes, which enables him to automatize a detailed productivity analysis of manual assembly processes.

While the existing approaches provide methods for the analysis of manual work processes, they do not integrate machine or robot data in the analysis.

# 4 Analysis of human robot collaboration work systems

## 4.1 Methodology

To analyze the work processes of HRC work systems, our approach aims at mapping states of employees and robots in the work system on a synchronized timeline with the help of state type diagrams. The methodology aims to reveal situations that will reduce the work systems productivity and result from the collaboration of employee and robot such as:

* balancing losses that result from an unbalanced task duration of robot and employee,

* breakdowns of the robot or related peripheral systems,

* disturbances in manual processes,

* speed losses and minor stops that result from the close proximity of employee and robot.
Balancing losses occur if either the robot or the employee needs to wait for the counterpart. Consequently, balancing losses either decrease the robot or employee productivity.

Breakdowns of the robot or a related system will force the employee to repair the disturbed system or to wait for an expert to resolve the issue. Breakdowns therefore cause productivity losses for employees and robots.

Disturbances in manual work processes will force the employee to resolve the disturbances while the robot has to wait for the employee in order to proceed. Thus these situations will cause productivity losses for employee and robots.

Due to safety reasons, collaborative robots will stop if they get in contact with the employee or will work with decreased speed in their close proximity. This is inevitable if robot and employee are about to collaborate. However, these situations may also occur if robot and employee work independently in close proximity. Those situations should be avoided by a better workplace design.

## 4.2 Exemplary work system

In order to illustrate the idea of the analysis, an exemplary HRC work system for research purposes was developed at Garz and Fricke GmbH in Hamburg. The work system consists of a manual work station and a robot work station for collaborative processes and of two handling points where employees take materials for processing or place already processed materials. In this case a Universal Robots UR 10 robot is used. Figure 1 shows the set-up.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Set-up of the HRC work system</figcaption></figure>

In this exemplary work system, touch display components are glued to plastic frames. This involves the preparation and manual inspection of the frame, the cleaning of the components with ISO-propanol, activation of the frames surface with a plasma activator, application of the glue and the joining and adjustment of the components. The preparation and manual inspection is performed by the employee as this involves the handling of small parts. Due to the higher repetitive accuracy, cleaning, surface activation and the application of glue is performed by the robot. Joining and adjustment of the components is done by employee and robot in a collaborative step. The exemplary set up therefore contains independent, synchronized and collaborative work steps.
## 4.3 Relevant states

Schweitzer [5] suggests an analysis approach for concatenated production lines consisting of states and state diagrams. These approach uses the states regular activity (tact), disturbance, waiting, blocking and set up. These states are also suited for the assessment of HRC, but need some additions. The activity needs to be tracked with greater detail, since robots can perform a wider range of activities. Therefore not only the activity, but also the kind of activity e.g. cleaning, surface activation or glue application has to be detected. In addition to machine warnings or disturbance types, also safety information must be recorded.

The robot states can be applied to the employee with limitations in the states of set up and disturbance. For the robot as well as for the employee it is important to distinguish between different regular activities, since both can perform a variety of tasks. It is also necessary to identify collaborative activities. Employees can also encounter disturbances. Usually these disturbances are not directly related to the employee, but are related to missing material, information and tools, that is required for the task processing. This is the same with set up activities, since employees may be involved in the set up of the robot or of the work station.

<table><caption>**Table 1.** Robot, employee and collaborative states of the exemplary work system</caption><thead><tr><th>robot states</th><th>collaborative states</th><th>employee states</th></tr></thead><tbody><tr><td>activity</td><td>activity</td><td>activity</td></tr><tr><td>- clean display</td><td>- transfer frame</td><td>- prepare frame</td></tr><tr><td>- clean frame</td><td>- join components</td><td>- join components</td></tr><tr><td>- activate frame</td><td>- transfer frame</td><td>- transfer frame</td></tr><tr><td>- apply glue</td><td></td><td>- join components</td></tr><tr><td>- prepare display</td><td></td><td></td></tr><tr><td>waiting</td><td></td><td>waiting</td></tr><tr><td>disturbance</td><td></td><td>disturbance</td></tr><tr><td>safety state</td><td></td><td></td></tr><tr><td>- reduced speed</td><td></td><td></td></tr><tr><td>- safety stop</td><td></td><td></td></tr></tbody></table>

## 4.4 Data recording and analysis

We use an optical tracking system based on the Intel RealSense D400 sensor family and the Nuitrack SDK [13] to capture the employee position and motion. A robot TCP socket interface is used to capture robot data. The following section describes the underlying data structure, the state detection approach and the resulting analysis HRC work systems.

### Data structure

The robot interface provides basic raw data such as the position of the tool in the robot coordinate system, robot control variables for external devices, robot state messages
and safety messages. This data is transformed into robot states in a later step. Based on the infrared depth map of the Intel real sense 3D-camera, the Nuitrack SDK provides information about the position and orientation of 18 human joints of a tracked body. In order to detect different employee states, we intend to define work stations that are linked with activities. A workstation data set consists of the position of the work station and the associated activities. In addition to the work station other areas may be defined in order to detect other states as waiting or disturbances.

## State detection

To analyze the work process of the HRC work system, it is necessary to detect the states discussed in the prior section. The state detection is carried out by a self-developed software that stores the detected robot, employee and collaborative states in a spreadsheet. To detect a robot state, the user defines a single parameter or a combination of parameters provided by the robot interface for a certain state. The activity state “apply glue” for instance can be detected if the robot tool position is set in the area of the glue dispenser and the robot control variable that activates the glue dispenser is set to “high”.

The employee activity is mainly derived from the employee position, since the employee states are connected to work stations and additional areas. For example the state “prepare frame” in table 1 is associated with the assembly table work station. It is detected if the employee is present at the assembly table and is moving arms and hands.

So far the detection of robot and employee states has not been tested in the exemplary work system. However, the employee state detection was tested in the learning factory of the institute. The robot state detection was tested with the help of an offline simulation software for the UR 10 robot.

## Analysis

In order to identify the losses described in section 4.1 and to find ways to prevent these losses, an analysis of the captured state data is necessary. The state type diagram (figure 2) displays the states of the employee and of the robot over time adapting a diagram introduced by Schweitzer [2] for production lines to HRC work systems.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Exemplary state type diagram for HRC processes</figcaption></figure>

Collaborative states are placed in the center, employee states in the upper, and robot states in the lower half of the diagram. Activity states of employee and robot are placed close to the center in order to better illustrate the workflow. Waiting, disturbances, and safety states are positioned more to the outside of the diagram. In the example of figure
2, the diagram reveals balancing losses after cleaning the display (robot) and after preparing the display (employee) and shows the underlying circumstances of these balancing losses. Employee and robot states can also be displayed in pie charts to show the distribution of all states, so that the user can easily identify the percentage of value adding states and of collaborative activities (fig. 3).

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Exemplary state distribution of employee and robot states</figcaption></figure>

# 5 Summary and outlook

This article presents an approach to analyze the productivity of HRC work systems. The method uses a 3D-camera to track the employee and several types of robot data. The data is used to detect the states of robots and employees and to map them in state type diagrams. This enables the user to identify productivity losses due to an insufficient balancing of employee and robot tasks, disturbances of the robot or the employee processes and mutual disturbances of employee and robot. The proposed method supports continuous improvement of productivity in a HRC environment, e.g. the reallocation of tasks between employee and robot in case of balancing losses.

However, it is not designed for the use of planning a completely new HRC work system. This requires a more detailed analysis and the integration of other aspects as safety, implementation costs, flexibility of the automation, etc.. While a detailed test and evaluation of the presented methodology for the entire work system is still pending, the functionality of single components was tested with positive results. A detailed evaluation in HRC work systems is therefore an important next step.

# Acknowledgements

The authors would like to thank the Hamburgische Investitions- and Förderbank (IFB) for funding the project "Automated gluing processes with human robot collaboration for high-mix, low-volume production of customized human machine interfaces (HMI)" and Garz und Fricke GmbH.
References

1. Romero D, Stahre J, Wuest T, Noran O., Bernus P, Fast-Berglund A, Gorecky D (2016) Towards an Operator 4.0 Typology: A Human-Centric Perspective on the Fourth Industrial Revolution Technologies CIE 46: International Conference on Computers & Industrial Engineering, S 1–11

2. Ranz F, Komenda T, Reisinger G, Hold P, Hummel V, Sihn W (2018) A Morphology of Human Robot Collaboration Systems for Industrial Assembly. Procedia CIRP 72:99–104. doi:10.1016/j.procir.2018.03.011

3. Aft LS (1992) Productivity measurement and improvement, 2. Aufl. Prentice Hall, Englewood Cliffs, NJ

4. Nakajima S (1988) Introduction to TPM. Total productive maintenance. Productivity Press, Cambridge, Mass.

5. Schweitzer T (2011) Nutzungsgradsteigerung verketteter Produktionslinien. (English title: Improving the Utilization Ratio of Chained Production Lines), Dissertation, TU Hamburg-Harburg

6. Mautua I, Ibarguren A, Kildal J, Susperregi L, Sierra B (2017) Human-robot collaboration in industrial applications. International Journal of Advanced Robotic Systems 14(4):172988141771601. doi:10.1177/1729881417716010

7. Wang XV, Kemény Z, Váncza J, Wang L (2017) Human-robot collaborative assembly in cyber-physical production: Classification framework and implementation. CIRP Annals - Manufacturing Technology 66(1):5–8. doi:10.1016/j.cirp.2017.04.101

8. Marvel JA, Falco J, Marstio I (2015) Characterizing Task-Based Human-Robot Collaboration Safety in Manufacturing. IEEE Trans. Syst. Man Cybern, Syst. 45(2):260–275. doi:10.1109/TSMC.2014.2337275

9. Malik AA, Bilberg A (2018) Digital twins of human robot collaboration in a production setting. Procedia Manufacturing 17:278–285. doi:10.1016/j.promfg.2018.10.047

10. Benter M (2015) Analysis of manual work with 3D cameras Advances in Production Management Systems. Value Networks: Innovation, Technologies, and Management. IFIP WG 5.7 International Conference, APMS 2015, Tokyo, Japan. Springer Berlin Heidelberg, Berlin Heidelberg

11. Escorcia V, Dávila MA, Golparvar-Fard M, Niebles JC (2017) Automated Vision-Based Recognition of Construction Worker Actions for Building Interior Construction Operations Using RGBD Cameras. In: Cai H, Kandil A, Hastak M, Dunston PS (Hrsg) Construction Research Congress 2012. American Society of Civil Engineers, Reston, VA, S 879–888

12. Ying W, Changzhi W, Jun W, Wenchi S, Xiangyu W (2018) A Framework for Vision-Based Automatic Scaffolding Productivity Measurement. In: Amor R, Dimyadi J (Hrsg) 18th International Conference on Construction Applications of Virtual Reality (CONVR 2018), S 423–432

13. 3divi Inc. (2018) Nuitrack. Body Tracking Software. 3divi Inc.