# Guidelines for Implementing Augmented Reality Procedures in Assisting Assembly Operations

Viviana Chimienti, Salvatore Iliano, Michele Dassisti, Gino Dini, Franco Failli

► To cite this version:

Viviana Chimienti, Salvatore Iliano, Michele Dassisti, Gino Dini, Franco Failli. Guidelines for Implementing Augmented Reality Procedures in Assisting Assembly Operations. 5th IFIP WG 5.5 International Precision Assembly Seminar (IPAS), Feb 2010, Chamonix, France. pp.174-179, 10.1007/978-3-642-11598-1_20. hal-01055687

HAL Id: hal-01055687

https://inria.hal.science/hal-01055687

Submitted on 13 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Guidelines for Implementing Augmented Reality Procedures in Assisting Assembly Operations

Viviana Chimienti¹, Salvatore Iliano¹, Michele Dassisti², Gino Dini¹, Franco Failli¹

¹ Dipartimento di Ingegneria Meccanica, Nucleare e della Produzione, Università di Pisa,
Via Bonanno Pisano 25/b - 56126 Pisa (Italy)

² Dipartimento di Ingegneria Meccanica e Gestionale, Politecnico di Bari
Viale Japigia 182 - 70126 Bari (Italy)

{viviana.chimienti, salvatore.iliano, dini, f.failli}@ing.unipi.it; m.dassisti@poliba.it

**Abstract.** The use of Augmented Reality (AR) in training or assisting operators during an assembly task can be considered an innovative and efficient method in terms of time saving, error reduction, and accuracy improvement. Nevertheless, the implementation of an AR-based application is quite difficult, requiring to take into account several factors. This paper provides a general procedure to follow for a correct implementation, starting from an assessment of the assembly task, until the practical implementation. To assess the procedure, it has been applied to the training of unskilled operators during the assembly of a planetary gearbox, with the help of a hand-held device.

**Keywords:** Assembly, Augmented Reality, Training

## 1 Introduction

Training and assisting operators in assembly procedures represents an important point in different industrial situations: precision assembly, use of temporary personnel in assembly lines, etc. Up to now, two different training methods have been adopted: on-the-job and face-to-face training. The former is accomplished directly on assembly stations and requests the continuous attendance of an experienced operator placed side by side to the unskilled one. The latter consists in face-to-face lessons given in different modalities (traditional or using computer-assisted procedures). These methods are undoubtedly quite effective, but serial, in that they consume productive time of skilled operators. A new training method is represented by the application of AR techniques, overcoming the above mentioned cons in assisting unskilled operators all over the assembly procedure, providing step-by-step instructions and thus assuring an immediate capability to accomplish the task by himself.

## 2 Augmented Reality in assembly

Augmented Reality (AR) is a concept developed in the last decades, consisting in improving information content in a real environment. The basic idea, used also in
non-industrial scenarios, is to supplement a real scene by synthetic images superimposed on it. The goal of AR is therefore to generate images somewhere on the optical path between the eyes of the operator and real objects in the working area.

Among the techniques used to achieve the augmentation, Video Mixing enables the user to watch the real scene indirectly through a video camera; a computer acquires the information and includes the digital content. Real and virtual objects coexist as two separate video streams, and the result is shown on a display. In [1] other techniques (i.e. Optical Combination and Image Projection) are described. The hardware used is[1][2]: a camera, to frame the real environment; a computer, that creates virtual contents and mixes real and virtual video streams together; a display, which shows the results of the augmentation; a tracking system, to detect operator's mutual position with the camera. A taxonomy of the different displays used can be found in [3]; considering their position compared with operator, the following can be distinguished: Head Mounted Display (HMD), Hand Held Display (HHD) and Spatial Display (SD).

Several applications have been investigated in the field of assembly so far (e.g. [4][5]). A first example was applied in aircraft industry, to electrical wiring assembly: the path of the wire is shown to the operator on a HMD in order to follow the visual track to perform the wiring operations. In automotive industry, AR methods were used in door lock assembly, with the target of creating a training instrument.

One of the main problems in this kind of applications is represented by the complexity of the AR implementation procedure: an effective application of this innovative technique requests the analysis of several aspects related to the assembly procedure. The aim of this work is to propose standard guidelines for a correct implementation of AR systems for guiding operators while assembling products, obtaining the advantages of time saving, error reduction and accuracy improvement.

## 3 Proposal of implementation procedure

The implementation procedure proposed in this paper is illustrated in Fig.1. The goal is to create a "standard procedure" to be followed whenever an AR method has to be applied for supporting an operator in performing an assembly task. The procedure is described in the following paragraphs using, as test case, a planetary gearbox (Fig.2) consisting of over than 200 different parts. Despite the specific application, the procedure is general enough to be adopted for other assembly operations.

**Preliminary Analysis of the Assembly Procedure.** The process starts with the analysis of the product, its parts, the assembly sequence, presence of subassemblies and the identification of assembly relationships among components. The purpose of the analysis is to have a clear vision of the components and of the process under study, checking all the elements to be manipulated and the information the operator might need in addition. During this step, the list of components related to the gearbox has been created; 4 assembly groups (output, 2^nd^ stage, 1^st^ stage, input) and 2 assembly subgroups (1^st^ stage and 2^nd^ stage planetary gear) were identified.

**Subdivision in Tasks, Sub-Tasks and Elementary Operations.** The assembly
process is hierarchically divided in tasks, sub-tasks and elementary operations. Going down in level of detail, the operation becomes more and more elementary and indivisible in other sub-operations. Each action will be then described accurately and all relevant information to perform correctly the operation will be identified, such as tools, devices, equipment, safety requirements, and organized in a table, as shown in Table 1, referring to the assembly of the output group of the gearbox.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Standard procedure for implementing an AR application for assisting operators during assembly tasks.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Case study: planetary gearbox</figcaption></figure>

**Creation of Logic Flow-Charts.** Each task and sub-task should be represented by means of logic flow-charts, including the assembly sequence, check points, variants and alternative procedures. These charts will be used to carry out the software implementation. Compared to the definition of the assembly cycle described above, this diagram allows to go through alternative paths, if there are checkpoints. Fig.3 reports an example of the flow chart regarding task #2, namely the assembly of the roller bearing on the shaft.

**Definition of Assembly Instructions.** For each elementary assembly operation,
suitable instructions have to be identified. This step includes the selection of textual messages, icons, 2D pictures or 3D models which have to be positioned in the real environment through AR. An example is reported in Table 2.

<table><caption>**Table 1.** Tasks, Sub-Tasks and Elementary Operations in assembly of the output group.</caption><thead><tr><th>Task</th><th>#</th><th>Sub-task</th><th># Elementary operation</th><th>Tools</th><th>Safety requirements</th><th>Remarks</th></tr></thead><tbody><tr><td rowspan="10">Assembling ball bearing on the casing</td><td rowspan="3">1</td><td rowspan="3">Press equipping</td><td>1 Take output casing</td><td></td><td rowspan="3">Wear gloves</td><td></td></tr><tr><td>2 Place it on the base</td><td>Base "K"</td><td></td></tr><tr><td>3 Lubricate bearing housing</td><td>Oil</td><td></td></tr><tr><td rowspan="6">2</td><td rowspan="3">Pressure on the bearing</td><td>1 Pump oil to bring piston near the punch</td><td></td><td>No contact!</td><td></td></tr><tr><td>2 Ensure the centering</td><td></td><td>Visually</td><td></td></tr><tr><td>3 Close protection cover</td><td></td><td>Close cover!</td><td></td></tr><tr><td rowspan="3">3</td><td rowspan="2">Coupling check</td><td>1 Check if the feeler gauge goes through bearing and housing</td><td>0.05mm</td><td></td><td></td></tr><tr><td>2 Check if the feeler gauge goes through bearing and housing</td><td>0.05mm</td><td></td><td></td></tr><tr><td>3</td><td>Check if the feeler gauge goes through bearing and housing</td><td>0.05mm</td><td></td><td></td></tr><tr><td rowspan="2">Assembling output shaft-roller bearing</td><td rowspan="2">1</td><td rowspan="2">Press equipping</td><td>1 Take output shaft</td><td></td><td rowspan="2">Wear gloves</td><td></td></tr><tr><td>2 Take output shaft</td><td></td><td></td></tr></tbody></table>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Flow chart of the assembly of the roller bearing on the shaft.</figcaption></figure>

<table><caption>**Table 2.** Correspondence between assembly operation and virtual elements to be visualized in AR environment.</caption><thead><tr><th>Elementary operation</th><th>Text message</th><th>Symbol</th><th>Image</th><th>3D model</th><th>Other images</th></tr></thead><tbody><tr><td>SHAFT LUBRIFICATION</td><td>Oil shaft where arrow points</td><td></td><td></td><td></td><td></td></tr><tr><td>INNER RING POSITIONING</td><td>Set the inner ring on the shaft</td><td></td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

**Hardware selection.** The AR hardware is selected according to the main features of the working environment and to the assembly process to be performed. Selection
charts as the one illustrated in Table 3 are used to choose the most appropriate device.

<table><caption>**Table 3.** Example of a selection chart used for choosing the most appropriate hardware device.</caption><thead><tr><th colspan="2">Hand Held Display</th><th colspan="2">Head Mounted Display</th><th colspan="2">Spatial Display</th></tr><tr><th>Pros</th><th>Cons</th><th>Pros</th><th>Cons</th><th>Pros</th><th>Cons</th></tr></thead><tbody><tr><td>Integrate in one device camera, display and processor</td><td>Low performance of processors used</td><td>Good integration between real and virtual</td><td>Low confort</td><td>No ergonomic problems</td><td>Occlusion of the projection by objects or the user</td></tr><tr><td>Easy to find or purchase</td><td>One hand is not free</td><td>Portable</td><td>Fixed image depth</td><td>No visual fatigue</td><td>Only for fixed applications</td></tr><tr><td>Non invasive</td><td></td><td></td><td></td><td>Wide displays</td><td></td></tr></tbody></table>

For the gearbox assembly, a handheld device has been chosen, consisting of an 8” touch screen monitor (Fig. 4). The tracking system selected was the optical marker-based tracking, with the software for the management of real and virtual streams.

<figure><img src="image_11.png" /><figcaption>**Fig. 4.** Hardware configuration.</figcaption></figure>

**User Interface Definition.** A very important step of the procedure is the creation of a graphical user interface (Fig.5). The system functionalities should be easily perceived by all kinds of users. So, it is a good choice to put more emphasis on clarity and abundance of information available, distinguishing between essential information for any type of operator, the ones that immediately appear on the screen, and information that are shown only if operations are carried out by less experienced staff.

**Software Implementation.** The planned procedure is implemented by programming the AR software and by preparing the working environment with AR tools and other devices (tracking system, hardware docking stations, etc). The lighting of the work environment needs to be considered accurately, since it influences software’s markers recognition: too much or insufficient lighting can cause problems. Depending on the chosen software, different ways of implementing can be adopted, such as writing a C# code, or using software’s already implemented *actions*, also written in C#, which have to be recalled graphically to create the workflow.

**Validation.** The AR system should be validated using a sample of users having different levels of experience and competences. A questionnaire can be proposed for collecting responses, comments and difficulties encountered in performing the
assembly tasks, to be used for enhancing the implementation made during the previous step. Questions must be formulated appropriately in order to analyze procedural fairness, clarity of instructions, ergonomics, effectiveness and efficiency of AR in assembly training.

<figure><img src="image_12.png" /><figcaption>Fig. 5. Graphical User Interface definition.</figcaption></figure>

## 4 Conclusions

The proposed guidelines offer an effective starting point for the implementation of an AR training system for assembly operations, and their general nature gives it the flexibility to make it applicable also in other fields. Compared to the training techniques used previously, the AR application seems to offer a set of advantages, such as a significant reduction of time and a lower investment in human resources.

Concerning the validation test performed on the gearbox, the specific solutions implemented was also successful thanks to the application of the procedure. Technical features selected were positively appreciated by operators. The whole sample of people tested was able to successfully conclude the procedure in timing between 20 and 35 minutes.

## 5 References

1. Azuma, R. T., A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments. Vol.6, 4 (August 1997), 355 - 385.

2. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B., Recent Advances in Augmented Reality. IEEE Computer Graphics and Applications (Nov/Dec 2001), 34-47.

3. Milgram, P., Kishino, F., A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems. Vol.E77-D No.12 (Dec1994), pp.1321-1329.

4. Pang, Y., Nee, A.Y.C., Ong, S.K., Yuan, M.L., Assembly Feature Design in an Augmented Reality Environment. Assembly Automation Journal, Vol. 26, No. 1 (2006) 34-43.

5. Y. Pang, A.Y.C. Nee, K. Youcef-Toumi, S.K. Ong and M.L. Yuan, Assembly Design and Evaluation in an Augmented Reality Environment. Singapore-MIT Alliance Symposium, (19-20 January 2005) Singapore.