# A Low-Cost Automated Fastener Painting Method Based on Machine Vision

Ran Zhao, Adrien Drouot, Joseph Griffin, Richard Crossley, Svetan Ratchev

► To cite this version:

Ran Zhao, Adrien Drouot, Joseph Griffin, Richard Crossley, Svetan Ratchev. A Low-Cost Automated Fastener Painting Method Based on Machine Vision. 8th International Precision Assembly Seminar (IPAS), Jan 2018, Chamonix, France. pp.92-100, 10.1007/978-3-030-05931-6_9. hal-02115839

HAL Id: hal-02115839

https://inria.hal.science/hal-02115839

Submitted on 30 Apr 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Low-cost Automated Fastener Painting Method Based on Machine Vision

Ran Zhao<fnref n="1" />, Adrien Drouot<fnref n="2" />, Joseph Griffin<fnref n="3" />, Richard Crossley<fnref n="3" /> and Svetan Ratchev<fnref n="3" />

**Abstract.** Fastener painting of aerospace structures, in particular legacy products, relies heavily on the skill or rather craftsmanship of the human operator. This process is time-consuming while automated operations with industrial robots can be a more efficient solution. Spray painting robots have been widely used in industry, however, they are not suitable for painting fasteners individually because it will cause a significant waste of materials. Thus, it is essential to develop proper tools and automated methods to replace manual work in order to reduce the cost and improve product quality. This research topic has been receiving more and more attention from both academia and industry.

In this article, we present a low-cost and flexible solution for automated fastener painting using a painting dabber and machine vision. A specific nozzle for the dabber is designed to apply paint on fasteners. The system locates the fasteners on aerospace structures with a Cognex camera, then painting is done by the robot with off-line programming. Experimental results show the effectiveness and practicality of automated painting system developed in this paper.

**Keywords:** Low-cost, Automated Fastener Painting, Machine Vision.

## 1 Introduction

Painting is widely used in industry, especially in automotive and aerospace manufacturing. It is an indispensable procedure for engineering machinery products in the process of surface manufacturing, which has various effects such as protection, beautification, water avoidance, antifouling, hidden and so on [1].

As intelligent equipment, painting robots can improve the efficiency and quality of painting operations, meanwhile reducing the labor costs [2]. However, painting robots are usually used in traditional spray painting that applies a coating (paint, ink, varnish, etc.) through the air onto a surface. They are often programmed to follow a specific

<footnote n="1">¹ College of Information and Electrical Engineering, China Agricultural University, Beijing, China, ran.zhao@cau.edu.cn</footnote>

<footnote n="2">² Institut FEMTO-ST, CNRS UMR 6174, UFC – ENSMM - UTBM, 25 000 Besançon, France, adrien.drouot@femto-st.fr</footnote>

<footnote n="3">³ Institute for Advanced Manufacturing, University of Nottingham, NG7 2TU, Nottingham UK, joseph.griffin, richard crossley, svetan.ratchev@nottingham.ac.uk</footnote>
path and to spray material precisely and consistently, as shown in Figure 1a. The main drawback of a spraying robot is that it is only suitable for surface painting. For the case of fastener painting where paint only needs to be only applied on several discrete regions on a surface, it is not competent anymore.

Meanwhile, traditional painting robots are usually programmed using a teach pendant which, although it is an intuitive programming approach and doesn't require high programming skills, does suffer from the reliance on the human programmer to make changes to the robot path. This manual teaching process often takes a long time, and during this stage, the robot cannot be used in the production. Moreover, workers are exposed to toxic and hazardous environments. Therefore, this type of automation is not cost effective for aircraft manufacturing where the geometry of the parts and the required operations vary considerably.

In this paper, a low-cost automated fastener painting system is presented. The traditional fastener painting process is usually carried by operators using brushes to paint every fastener on the product, as shown in Figure 1b. This is a highly repetitive task and the manual process is time-consuming and labourious. Moreover, the product quality is based on the operator's skills, experience, and knowledge of the production facilities. The operators must work in a hazardous environment. All such disadvantages will restrict the efficiency of the manual painting process. The proposed system emulates the human operation but in a fully automated manner. A painting dabber is used to apply paint instead of brushes which are unsuitable for an industrial robot to use. A machine vision system is implemented on the industrial robot to determine the position of the fasteners on the structure.

<figure><img src="image_2.png" /><figcaption>Fig. 1. (a): A FANUC industrial spray painting robot is applying the coating onto a curved surface. Source: http://www.fanuc.eu. (b): The manual process of fastener painting where the human operator paints each fastener on a wing structure using a brush.</figcaption></figure>

## 2 Related Work

One of the challenges in fastener painting is to locate each fastener quickly and accurately. In this perspective, good sensing ability is essential to provide robots with an increased level of autonomy. Vision systems are highly suited for providing the robot's environment information as well as for the recognition of objects and their features,
providing information about their position and pose, allowing it to more reliably act in
an unconstrained cell. Furthermore, to ensure any automated process is carried out ef-
fectively after the manual operation, vision can be used for in-line visual quality in-
spection to confirm that the assembly's sub-components are mounted within design
specification.

Machine vision has been used widely in manufacturing and production, in a number of applications [3]. Indeed, the state of the art presents various solutions that use different techniques (both 2D and 3D) such as monocular vision [4], CCD cameras [5] or laser range-imaging [6]. Many types of vision systems and range sensors are available today, but the design and integration of them needs to take consideration of many issues such as: accuracy, speed, reliability, and cost, as well as environmental operating constraints such as: lighting, temperature, and humidity.

Besides good sensing abilities, it is necessary to develop systems that can reduce the
time and difficulty associated with tool path programming. In this respect, offline pro-
gramming software (OLP) [7] has been developed in recent years, both as commercially
available products (such as KUKA.Sim for KUKA [8], RobotStudio for ABB [9] and
MotoSim for Motoman [10]) as well as in academic research field (for example
ROBOMO [11] and Robott [12]). Such software has the ability to model and to present
graphically the robot and other equipment, generate programs and then simulate the
robot behavior for a given task. However, most software still relies on the teach pendant
for robot motion generation, such as tag creation and trajectory planning. Therefore,
this paper proposed an automated OLP approach with the help of external sensors. It is
low-cost, efficient and sufficient for the use in the manufacturing operations of small
and medium-sized enterprises (SMEs).

The remaining content of this paper is organized as follow: Section 3 describes the machine vision-based fastener painting system, including the system overview and detailed techniques. Experimental results carried on an ABB industrial robot are presented in Section 4. Finally, this work is concluded in Section 5.

3 The Automated Fastener Painting System

**3.1 System overview**

The automation of the fastener painting has been thought as a partially automated pro-
cess where the human operator presents the parts to be operated by the robot, after
having performed other manual operations on it previously. Each part is provided by
an RFID tag that is scanned before presenting the part to the robot, and that allows
uploading from a database the corresponding part information. After the manual oper-
ation, the part will be transferred to the painting robot by a conveyor. As the part will
be placed randomly on the conveyor by the operator, painting robot must rely on exter-
nal sensors to detect the exact position and orientation of the part in front of it. Mean-
while, specific painting end-effector will need to be designed as well for different fas-
tener models in a real-world application.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** (a): The designed fastener painting cell with a conveyor and a manual workbench. (b): The experimental setup of the painting cell.</figcaption></figure>

Figure 2 shows the designed fastener painting workbench and the physical setup used for the experiments of the developed system. The robot used in this paper for implementation is the ABB IRB-14000 (also known as YuMi). It is a dual-arm manipulator platform which is designed for small parts assembly. It has only a 0.5 kg payload but this is adequate for small fastener painting. Compared with the large-sized industrial robot, YuMi is cheaper and the manufacturer specified accuracy of 0.02 mm is more than sufficient for painting applications. Also, YuMi is equipped with a built-in safety system, such as padded arms and light-weight design. Therefore, YuMi is quite suitable for working alongside human operators without the requirement for fully enclosed guarding. Meanwhile, YuMi has a built-in machine vision camera module in its right hand. The vision module contains a Cognex AE3 In-Sight 2D camera and provides powerful and reliable vision and identification tools. Table 1 details the camera specification.

<table><caption>**Table 1.** Specification of the integrated machine vision camera system.</caption><thead><tr><td>Description</td><td>data</td></tr></thead><tbody><tr><td>Resolution</td><td>1.3 Megapixel</td></tr><tr><td>Lens</td><td>6.2 mm f/5</td></tr><tr><td>Illumination</td><td>Integrated LED with programmable intensity</td></tr><tr><td>Software engine</td><td>Powered by Cognex In-Sight</td></tr></tbody></table>

## 3.2 Painting process

To emulate the human's skill in the fastener painting process, a painting dabber is used to apply paint on the fasteners, as shown in Figure 3. The painting dabber is composed of a bottle which contains the paint and a nozzle which is used to apply paint. During the usage, a piece of foam is placed inside the nozzle to control the flow of the paint.
The foam will soak up the paint and store it. When the dabber contacts the fastener, the paint will be squeezed out and then applied evenly on the fastener.

<figure><img src="image_4.png" /><figcaption>Fig. 3. (a): The working principle of the painting dabber. (b): The internal structure of the nozzle attached to the painting dabber. (c): A real painting dabber used in the cell.</figcaption></figure>

Nozzles of different sizes can be substituted with respect to different fasteners. Painting dabber is easy to obtain, therefore it is suitable for the proposed low-cost scenario.

According to the working principle of the painting dabber, the robotic fastener painting process is designed as follows:

*Step 1:* Locate each fastener with the machine vision system.

*Step 2:* The robot grabs the dabber and moves to the position that is 20 mm above the located fastener.

*Step 3:* The robot goes down and keeps the fastener is entirely covered by in the nozzle.

*Step 4:* Rotate the dabber clockwise by 180 degrees and then rotate 180 degrees counter-clockwise.

*Step 5:* The robot goes up and moves on to painting the next fastener.
## 3.3 Vision-guided programming

<figure><img src="image_5.png" /><figcaption>Fig. 4. (a): The system configuration of the vision-guided programming for controlling the robot moves to each fastener that is on a piece of part. (b): The image processing result. The green coordinates indicate the center position (x and y) of each fastener in the camera frame (the yellow coordinate).</figcaption></figure>

The system configuration of vision-guided programming is shown in Figure 4a. A hybrid position/vision control platform is developed to control the robot motion. The position control is used to maintain the tool orientation. Vision sensing is used to locate the fasteners on the part. The part image is captured by a single camera. Then the image is processed using the PatMax [13] algorithm for the 2D (x and y) coordinates. Figure 4b shows an example of the image processing results. The depth coordinate (z) cannot be obtained from the camera. However, it is able to be detected by the contact between the painting dabber and the fastener, according to the variation of the robot's motor torque.

## 3.4 System calibration

One of the most important steps of using vision system is the camera-robot calibration. The calibration consists of two basic steps. First the camera calibration which converts the image pixels to mm, and second the camera to robot calibration which relates the camera coordinates to a robot frame. Camera calibration is the process of estimating the parameters of a pinhole camera model approximating the camera that produced a given photograph. For 2D cameras, a checkerboard with fiducial is usually used for calibration. The calibration process will find the origin in the camera frame $F_c$ which is used to determine the position of the fastener in $F_c$. We denote this position as $P_F^c$. Then this position needs to be transformed to the robot frame $F_R$. Finally, we will get the fastener's position in the robot coordinate $P_F^R$.

## 3.5 Singularity avoidance

During the movement from the initial robot position to the painting position (the position where the fastener is), the robot may pass through the singularity pose. It is because
the fastener is a symmetrical object, the orientation from the vision system may vary from each result although the fastener's position never changes. As shown in Figure 5, two camera results give the same position of the object but completely different orientation. As the motion path is planned by the robot's own controller, it can provide the shortest solution but it cannot avoid the singularity. For example, in Figure 5b, the robot's wrist joint (joint 6) has to rotate a large angle during the movement. Singularity may occur in this case. Thus, to avoid the singularity, we can simply fix the orientation value that comes from the vision result as zero. Consequently, the robot's wrist joint will not rotate during *Step 2*. If the initial position of joint 6 is programmed properly (for example, $j_6 = 5°$), the singularity will be avoided in the whole painting process. The joint value is set approximately to zero so that the joint boundary of the 6th joint will not be exceeded during the rotation in *Step 4* in Paragraph 3.2. Another advantage of fixed orientation is that the execution time is largely reduced. For multi fasteners, the robot doesn't need to rotate to the exact results coming from the camera as the orientation result of each fastener can be completely different (It can be clearly seen from Figure 4b).

<figure><img src="image_6.png" /><figcaption>Fig. 5. Different orientation results from the same object.</figcaption></figure>

# 4 Experimental Results

To evaluate the proposed painting system, a test part with five fasteners was used in the experiment. Software running on the robot is written by the ABB RAPID programming language. Experiments were conducted with the set-up as shown in Figure 2b. Experimental results are illustrated in Figure 6. From the results, we can see that the fasteners can be successfully painted with the proposed system.
<figure><img src="image_7.png" /><figcaption>**Fig. 6.** (a): The robot is painting the first located fastener. (b): The robot moves to the next fastener. (c): The robot is painting the second fastener. (d): The final result after all of the fasteners have been painted.</figcaption></figure>

It was observed that the vision accuracy could be affected by the ambient light. 5 camera acquisitions of the part were made at the shortest possible time span one after the other at four different times during the day (10 am, 1 pm, 2 pm, 5 pm) in order to test the camera with different levels of light and different light directions. The average errors between the camera results and the real positions are measured for all cases. Meanwhile, we also calculated the rate that the system can successfully recognize all fasteners. Table 2 shows the accuracy and the success rate of each condition. The minimum error is 1 mm in the morning and in the late afternoon while the maximum error is 4.5 mm at 5 pm. The light conditions in the workshops were partially controllable because of a skylight in the ceiling. To achieve greater accuracy, it has been decided to leave the available neon lights always turned on.

<table><caption>**Table 2.** Results of the calculations of camera accuracy under each light conditions<fnref n="1" />.</caption><thead><tr><td>Time</td><td>Error</td><td>Success rate</td></tr></thead><tbody><tr><td>Morning light (10 am)</td><td>1 mm</td><td>99%</td></tr><tr><td>Lunch light (1 pm)</td><td>1.8 mm</td><td>98%</td></tr><tr><td>Early afternoon light (2 pm)</td><td>2.2 mm</td><td>98%</td></tr><tr><td>Dusk light (5 pm)</td><td>4.5 mm</td><td>90%</td></tr></tbody></table>

<footnote n="1">¹ The experiments were done under the day light condition of mid-November in Nottingham, UK. Results may vary at different time and different places.</footnote>
# 5 Conclusion and Future Work

In this paper, an automated fastener painting system was presented. It is based on a simple machine vision system and a painting dabber to emulate the skilled human operation. An ABB YuMi robot with built-in vision sensor is deployed, which reduces the total system cost. The proposed system is currently able to locate and paint fasteners on flat surfaces. This platform could enable SMEs to cost efficiently automate fastener painting or similar processes such as small part sorting and sealing. The preliminary experimental results presented in this paper validate the effectiveness of the proposed approach.

In future work, we will consider the integration of more sensors, such as the force/torque sensor. Such sensors are able to provide the essential contact feedback information, which can be used to compensate for the error from the camera. For 3D curved surfaces a low-cost 3D sensor (such as the Kinect RGB-D sensor) could be added. To fully evaluate the proposed system, more experiments with standard aerospace structures are required with additional quality control measures to enable process certification.

# References

1. Xuewei C, et al.: Rapid generation of spraying instructions for painting robot basing on automatic programming technology. In: proceeding of IEEE International Conference Mechatronics and Automation, pp. 669-674, IEEE, New York, (2014).

2. Hepling, C.: Automated industrial robot path planning for spray painting process: a review. In: Proceedings of 4th IEEE conference on Automation Science and Engineering, pp. 522-527. IEEE, New York, (2008).

3. Elias N. M, et al: A survey on industrial vision systems, applications and tools. Image and vision computing 21(2), 171-188, (2016).

4. Liangyu L.: A machine vision system for inspecting bearing-diameter. In: Proceedings of the fifth World Congress on Intelligent Control and Automation. pp. 3904-3906, IEEE, New York, (2004).

5. Andrew KC W, Rong L and Liang X.: Robotic vision: 3D object recognition and pose determination. In: Proceedings of International Conference on Intelligent Robots and Systems. Pp.1202-1209, IEEE, New York, (1998).

6. Antonio Ramon J, Ceres R and Pons JL.: A vision system based on a laser range-finder applied to robotic fruit harvesting. Machine Vision and Applications 11(6), 321-329, 2000.

7. Zengxi P, et al.: Recent progress on programming methods for industrial robots. Robotics and Computer-Integrated Manufacturing 28(2), 87-94, (2012).

8. KUKA Homepage, http://www.kuka.com, accessed on 10-12-2017.

9. ABB Homepage, http://new.abb.com/products/robotics/fr/robotstudio, accessed on 09-12-2017.

10. Motoman Homepage, https://www.motoman.com/, accessed on 01-01-2018.

11. Chang-Sei K, et al.: PC-based off-line programming using VRML for welding robots in shipbuilding. In: Proceedings of IEEE Conference on Robotics, Automation and Mechatronics, pp.949-954, IEEE, New York, (2004).
12. Bottazzi V.S and Fonseca J.C.: Off-line robot programming framework. In: proceedings of the joint international conference of Autonomic and Autonomous Systems and International Conference on Networking and Services, pp. 71-76, IEEE, New York, (2005).
13. http://www.cognex.com/pattern-matching-technology.aspx?pageid=11368&lang-type=1033, accessed on 26-12-2017.