# A Framework for Process Model Based Human-Robot Collaboration System Using Augmented Reality

Hwaseop Lee, Yeeyeng Liau, Siku Kim, Kwangyeol Ryu

► To cite this version:

Hwaseop Lee, Yeeyeng Liau, Siku Kim, Kwangyeol Ryu. A Framework for Process Model Based Human-Robot Collaboration System Using Augmented Reality. IFIP International Conference on Advances in Production Management Systems (APMS), Aug 2018, Seoul, South Korea. pp.482-489, 10.1007/978-3-319-99707-0_60. hal-02177889

HAL Id: hal-02177889

https://inria.hal.science/hal-02177889v1

Submitted on 9 Jul 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A Framework for Process Model Based Human-robot Collaboration System using Augmented Reality

Hwaseop Lee, Yeeyeng Liau, Siku Kim, and Kwangyeol Ryu#

¹Pusan National University, Busan 46241, South Korea

¹{hslee, yeeyeng85, siku.kim, kyryu#}@pusan.ac.kr

**Abstract.** The concept of smart factory is being applied into traditional manu-facturing system. Since factory automation is applied gradationally, Human-ro-bot collaboration (HRC) system is becoming an important issue. In order to con-struct an effective HRC system, clear communication with the human workers and the robots has to be considered. This research proposed a conceptual frame-work of process model based HRC system for efficient human-robot collabora-tion in a semi automation process to produce electric motors. We applied a pro-cess modeling methodology for capturing collaborative features, activity and re-source flow in the manufacturing process. The model defined by the proposed methodology is the data storage to contain process information and interface be-tween the human workers and robots to provide accurate information in the ap-propriate context. Furthermore, machine vision technology is implemented to recognize specifications of work in process (WIP) parts. The recognized parts are mapped with the correct work order and work instruction manual defined by the proposed model. In order to reduce human worker's errors, the extracted work information is transmitted to the worker through the augmented reality device. The proposed HRC system is expected to be able to support the construction of a semi automation system that can reduce errors of human workers and ensure production flexibility.

**Keywords:** Augmented reality, Human-robot collaboration, Machine vision, Process modeling.

## 1 Introduction

This research proposed a conceptual framework of human-robot collaboration (HRC) system for semi-automated process to produce electric motors. This process consists of automated machines, a handling robot, and a human worker. The automated machines conduct processing operations such as cut off, drilling, facing and etc. The robot assists the human worker in the material handling operation. The robot picks up the parts from each automated machine and locates on the pallets, then transfers the partial processed parts to the assembly section which performed by a human worker. In the current prac-tice, there are no interaction between the robot and human worker on the part infor-mation. Human worker does not receive information of the type of parts that are picked up by the robot in advanced, i.e. before the part arrives at assembly section, especially
when changeover of part model. Hence, delay occurred when the human worker needs to measure and identify the part.

In this paper, machine vision technology is applied to identify the part in collaboration process. When image processing through machine vision is completed, accurate work order and part information corresponding to the extracted image should be defined. We defined the work order and part information using Part-flow based Manufacturing Process Modeling (PMPM) which can express process flow and part flow at the same time [1]. In order to transmit the tracked data to the worker immediately, we used the augmented reality (AR) device to transmit work order information and product specification information. This study proposes a conceptual framework of HCR system using a process model and AR interface.

# 2 Literature Reviews

## 2.1 Human-robot collaboration in Assembly Operation

Human-robot collaboration is one of new trends in the field of industrial robots as a part of strategy Industry 4.0. Full automation has limitation in achieving the required production demand due to the high complexity in process and flexibility of equipment to cope with the rapid response. Hence, HRC which has advantage to maximize the flexibility in production became one of the important trends in current industrial application. Human-robot collaboration is a system in which a robot and a human are collaborating on a same task which combines benefits from both human and robot in a joint task [2]. HRC system plays an important role to increase flexibility to cope with wide variety of products and the frequent changeover in equipment to cope with the fluctuating demand [3]. In HRC workstation, the collaborative robot must have the capability to sense the existence of human, i.e. with appropriate safety mechanism that defines the ability to collaborate with human without safety cage.

**Machine vision for Assembly.** Machine vision system consists of image capture device, lighting and computer with image processing software [4]. Machine vision systems enable flexibility in automation by providing the machine capability of “see” and think. From the existing researches, the main application of machine vision system is quality monitoring for wide range of industries such as electromechanical parts [5], textile manufacturing [6], pharmaceutical [7], food packaging [8], steel making industry [9] and etc. Teck et al. [10] integrated machine vision system into automation systems to provide product data which assists the decision making of the production system. Gao et al. [11] developed machine vision to track and calibrate the coordinates of fast moving battery lid in automated sealing rings assembly system. The application of machine vision with the robot in the assembly system increased the efficiency and robot adaptability to the environmental changes and enable real-time adjustment of robot motion.

**AR for Assembly.** AR is a technology derived from a field of VR, which refers to a computer graphics technique that superimposes virtual contents created in 3D in the real world of the user [12]. Since the augmented reality technology models and matches
only the required virtual objects based on the real-time image of the actual manufactur-
ing environment, the construction cost and time for 3D models necessary for imple-
menting a new manufacturing system can be drastically reduced.

Kollatsch et al. [13] discussed about an AR based application for mobile devices realizing a user-friendly and problem-oriented visualization of information. However, this research has limitations in simply suggesting the AR-based concept introduced for visualizing the process value of mobile devices. Further, Wang et al. [14] presented about a novel human Cognition-based interactive Augmented Reality Assembly Guidance System is proposed to investigate how AR can provide various modalities of guidance to assembly operators for different phases of user cognition process during assembly tasks. In addition, Danielsson et al. [15] implemented an environment in which an untrained worker using AR in a human-robot collaboration environment. This research, however, has a problem that various errors occur during assembly due to misunderstandings of the instructions of the test person.

Although many studies have been conducted to utilize the strength of the AR tech-
nology in the manufacturing system implementation, there is a lack of an AR system
which can be properly utilized in the manufacturing system.

## 2.2 Part-flow based Manufacturing Process Modeling: PMPM

PMPM is a modeling tool for visualizing and visualizing man (worker), machine, ma-
terial (parts), and method (activity) of collaborative manufacturing process. It can
clearly define the order of the manufacturing operations that make up the process and
its execution objects. This method consists of 6 notations for activity expression and 10
notations for part expression. It can provide functions to record parts’ history and to
manage parts status by using part notations. and this method can define collaborative
activities in each process. The flow of parts and the flow of work can be determined
identically or independently. It is possible to define changes in characteristics such as
changes in the number of management units of parts and merging of parts. It also allows
a clear definition of objects using manufacturing facilities. It is manufacturing process-
oriented modeling methodology. In this paper, we used PMPM to store process infor-
mation and track part information.

3 Process model based HRC system

3.1 System framework

Fig. 1. shows data flow diagram of the proposed process model based HRC system. Electronic motor manufacturing process consists of four automated workstations which perform coil forming and cap machining and assembly of electric motor by human worker. Two types of material handling methods are used in the process: 1) using conveyor to transfer parts between automated workstations; 2) a robot collects the partial complete components from these workstations on a pallet and transfers to assembly workstation. PMPM modeling methodology is used in this study to build the digital model for this process. The PMPM model stores information about work order, parts
flow, and task performing objects. When the part arrives at the assembly process, the operator recognizes the part through the AR device. In this paper, the machine vision is integrated in AR interface to identify the parts in the assembly model. After the parts are identified, the work order such as assembly sequence, parts required and production quantity corresponds to the identified part is extracted from the PMPM model and transmitted to the AR interface. The PMPM modeling simulates the start time of each workstation in the order of minimizing the production cycle time. The coordination of processing time among the workstations and the material handling time by robot i.e. part pick-up time at each automated workstation and delivery time to human assembly workstation by robot is important to ensure the smooth flow and avoid stacking of part and idle time at each station especially the human assembly workstation. Hence, the real-time production status at the human assembly workstation has to be sent to the robot via AR interface to “call” for parts and “stop” input of parts. Besides, this method also can reduce overproduction at each automated workstation and part shortage with the updated finished parts quantity.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Data flow of system framework.</figcaption></figure>

## 3.2 Implementation

**PMPM modeling.** The first step is to model the electronic motor assembly process using PMPM. Fig. 2. shows the result for modeling the target process. Rectangles are the tasks performed by automated machines. Rounded rectangles and octagons are the collaborative works of the robot and the worker. As shown in Fig. 2., the type and number information of the input and output parts are defined in the model. Therefore, this model plays a role as an important data engine that has main information of HRC system and is used to track part information and verify image processing results. From PMPM modeling, the human worker who performs the assembly is updated with the real-time type of model and quantity during the production in each automated workstation via the AR device. Hence, PMPM modeling acts as communication interface between the automated workstation and human worker who performs the assembly.
<figure><img src="image_3.png" /><figcaption>Fig. 2. PMPM model for electric motors.</figcaption></figure>

**Image processing.** The electric motor models are differentiated according the size of case and the size of cap. The dimension of cap and case are shown in Table 1. Since the outer shapes of the parts are similar, there are possibility for human worker assemble the wrong combination. The measurement using Vernier caliper every time before the assembly process reduce the productivity of the operation. Hence, part measuring using machine vision system is proposed to increase the accuracy of part identification and increase the productivity of the assembly process. In order to achieve these, the vision system must be capable to acquire the image for part detection as close as human vision [16]. The steps in image processing involved to identify cap and measure the dimension of the cap arrived at the assembly section are shown in Fig. 3. First, the image is captured by the Augmented Reality interface and send to MATLAB image processing software. Image processing is performed using image processing toolbox in MATLAB. The image processing starts with converting the image acquired to gray scale image. Then, the image is converted to binary image for edge detection using Canny method. Since the cap is in round shape, the ‘imfindcircles’ function is used to identify the shape with defined radius range. Finally, the identified cap is marked and the radius is measured in pixels (see Fig. 4). The algorithm needs further improvement to remove the shadow of the image captured and to provide measurement in centimeters.

<table><caption>Table 1. Types of raw material and measurement for processing.</caption><thead><tr><td>Magnet</td><td>Coil</td><td>Inner Diameter of Bearing</td><td>Outer Diameter of Case</td><td>Length of Case</td></tr></thead><tbody><tr><td>6ø</td><td>6.5ø</td><td>1.5mm</td><td>14ø</td><td>35mm</td></tr><tr><td>8ø</td><td>8.5ø</td><td>2mm</td><td>16ø</td><td>37mm</td></tr><tr><td>-</td><td>-</td><td>-</td><td>-</td><td>39mm</td></tr></tbody></table>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Image processing for identify and measure cap.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Visualization of image processing steps.</figcaption></figure>

**AR interface development.** In this study, app development environment is built. Based on Android, an app is developed that uses an image target and a virtual button which are linked to a tracker by camera rendering by using Vuforia and Unity. The developed application visualizes actual factory data and pre-analyzed simulation data to the manager and visualizes them to the human worker in real time through the AR device. Managers can make manufacturability decisions and verify production plans through pre-simulation. In addition, real-time log data preprocessing and data exchange environment can be constructed by data linkage between log data from manufacturing facilities and MES.

To build an AR interface, we used the Vuforia module by using the Unity engine. Unity is a virtual/augmented reality production tool which has the advantage of creating mobile AR programs regardless of iOS or Android. This is expected to be effective in the manufacturing field where various AR equipment are combined. The user can monitor real-time information of the identified part. The information is predefined and can be freely set according to the characteristics of the operator and each part.

Fig. 5. shows the pilot program screen of the AR interface. It provides the assembly recipes and process information stored in the database to the operator using the AR through the part information verified by image processing. The developed AR application allows process values to be stored and analyzed to show process-related information to the operator or administrator. Thus, the user does not need extensive knowledge of the machine and the control device, and the production process is not disturbed. The collected data can be displayed to the user in real time or can be evalu-
ated later. Process data is stored in the database and can be retrieved from other appli-
cations. The traditional monitoring system provides operators with a simple signal on
the designated board, but the AR based interface provides the operator with real-time
status of the plant and makes the work more efficient. The operator can virtually overlap
the plant parameters and process state at the actual location of the sensor / equipment /
actuator. It can also visualize information about areas that are generally inaccessible or
dangerous.

<figure><img src="image_6.png" /><figcaption>Fig. 5. AR interface pilot program.</figcaption></figure>

4 Conclusion

In this paper, we proposed process model based HRC system using AR interface. We
defined the work order, parts flow, and collaboration information of the electronic mo-
tor assembly process using the part-flow based manufacturing process modeling
(PMPM). The PMPM model is used to act as an interface between human worker, au-
tomated processing machines and material handling robot through the identification of
parts and corresponding work sequences. Hence, integration of AR devices and image
processing is needed to capture parts information efficiently. Image processing tech-
nique used to extract features of component from the image captured using AR device
and feedback the result of part identification to human workers via AR device. The
proposed system is expected to reduce human error in part model identification and
work sequence. Furthermore, production flexibility that able to cope with frequent
model changeover can be achieved. The main contributions of this framework are the
data management using process model and the use of image processing to increase the
part recognition of AR. However, since this study is an early stage, it is necessary to
verify the proposed framework and define practical problems through application.
References

1. Lee, H., Ryu, K., Son, Y. J., & Cho, Y.: Capturing green information and mapping with MES functions for increasing manufacturing sustainability. International Journal of Precision Engineering and Manufacturing 15(8), 1709-1716 (2014).

2. Faber, M., Bützler, J., & Schlick, C. M.: Human-robot cooperation in future production systems: Analysis of requirements for designing an ergonomic work system. Procedia Manufacturing 3, 510-517 (2015).

3. Thomas, C., Matthias, B., & Kuhlenkötter, B.: Human-robot collaboration-new applications in industrial robotics. In: International Conference on Competitive Manufacturing (COMA), pp. 293-299, Stellenbosch (2016).

4. Malamas, E. N., Petrakis, E. G., Zervakis, M., Petit, L., & Legat, J. D.: A survey on industrial vision systems, applications and tools. Image and Vision Computing 21(2), 171-188 (2003).

5. Di Leo, G., Liguori, C., Pietrosanto, A., & Sommella, P.: A vision system for the online quality monitoring of industrial manufacturing. Optics and Lasers in Engineering 89, 162-168 (2017).

6. Cho, C. S., Chung, B. M., & Park, M. J.: Development of real-time vision-based fabric inspection system. IEEE Transactions on Industrial Electronics 52(4), 1073-1079 (2005).

7. Možina, M., Tomažević, D., Pernuš, F., & Likar, B.: Automated visual inspection of imprint quality of pharmaceutical tablets. Machine Vision and Applications 24(1), 63-73 (2013).

8. Duan, F., Wang, Y., & Liu, H.: A real-time machine vision system for bottle finish inspection. In: ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, vol. 2, pp. 842-846. IEEE, Kunming, China (2004).

9. Yun, J. P., Choi, D. C., Jeon, Y. J., Park, C., & Kim, S. W.: Defect inspection system for steel wire rods produced by hot rolling process. The International Journal of Advanced Manufacturing Technology 70(9-12), 1625-1634 (2014).

10. Teck, L. W., Sulaiman, M., Shah, H. N. M., & Omar, R.: Implementation of shape-based matching vision system in flexible manufacturing system. Journal of Engineering Science & Technology Review 3(1), 128-135 (2010).

11. Gao, M., Li, X., He, Z., & Yang, Y.: An automatic assembling system for sealing rings based on machine vision. Journal of Sensors 2017 (2017).

12. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B.: Recent advances in augmented reality. IEEE Computer Graphics and Applications 21(6), 34-47 (2001).

13. Kollatsch, C., Schumann, M., Klimant, P., Wittstock, V., & Putz, M.: Mobile augmented reality based monitoring of assembly lines. Procedia CIRP 23, 246-251(2014).

14. Wang, X., Ong, S. K., & Nee, A. Y. C.: Multi-modal augmented-reality assembly guidance based on bare-hand interface. Advanced Engineering Informatics 30(3), 406-421 (2016).

15. Danielsson, O., Syberfeldt, A., Brewster, R., & Wang, L.: Assessing instructions in augmented reality for human-robot collaborative assembly by using demonstrators. Procedia CIRP 63, 89-94 (2017).

16. Peña-Cabrera, M., Lopez-Juarez, I., Rios-Cabrera, R., & Corona-Castuera, J.: Machine vision approach for robotic assembly. Assembly Automation 25(3), 204-216 (2005).