# Machine Vision Systems for Industrial Quality Control Inspections

Ricardo Luhm Silva, Marcelo Rudek, Anderson Luis Szejka, Osiris Canciglieri
Junior

► To cite this version:

Ricardo Luhm Silva, Marcelo Rudek, Anderson Luis Szejka, Osiris Canciglieri Junior. Machine Vision Systems for Industrial Quality Control Inspections. 15th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2018, Turin, Italy. pp.631-641, 10.1007/978-3-030-01614-2_58 . hal-02075633

HAL Id: hal-02075633

https://inria.hal.science/hal-02075633

Submitted on 21 Mar 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Machine Vision Systems for Industrial Quality Control
Inspections

Ricardo Luhm Silva¹, Marcelo Rudek¹, Anderson Luis Szejka¹ and
Osiris Canciglieri Junior¹

¹Industrial and Systems Engineering Graduate Program, Pontifical Catholic University of Parana, Curitiba, Brazil
{marcelo.rudek, anderson.szejka, osiris.canciglieri}@pucpr.br

**Abstract.** In this paper we introduce Machine Vision System (MVS) for industrial quality control inspections presenting new perspectives with the recent developments of Artificial Intelligence (AI). A brief literature review is provided which indicates a substantial growth of machine vision new studies and an improved workflow is proposed to include these findings. Besides already existing machine vision solutions there is space to increase detection in quality control inspection and reduce current implementation constraints and technical limitations. The paper shows MVS new development and evinces that a deeper understanding of AI,MVS limitations is needed to provide a clearer path for future studies.

**Keywords:** Machine Vision, Industrial Inspection, Machine learning, Artificial Intelligence.

# 1 Introduction

Visual inspection is an important process in an industry to recognize defective parts, to assure quality conformity of a product and fulfill customer demands [1][2]. In assembly and manufacturing activities, product and process inspection are usually performed by human inspectors, but due fatigue, small parts, small details, hazardous inspection conditions and process complexity this task may not achieve the desired quality or be almost impossible to detect some types product non-conformities. In this cases a machine vision solution is recommended [1] [3][4].

Machine Vision System (MVS) consist in applying computer vision to industrial solutions [5]. MVS can be used to perform visual inspection and fulfill industrial and factory performance, consequently improving product quality outcomes. To meet industrial expectations, MVS has been used to reduce product quality problems through improved inspections. Inspection system must be adapted to a scenario which have a wide variety of product features and high production speed assembly lines with complex environment variables from the MVS perspective [5][6].

The purpose of this article is to provide an overview of MVS concepts and theirs current status for industrial inspection, identifying the main Artificial Intelligence (AI) concepts and application to quality in the automotive industry.
# 2 Theoretical foundation

## 2.1 Machine vision with artificial intelligence techniques Machine Vision System concepts and applied technologies

MVS systems have become imperative in many modern manufacturing facilities as forms of automatic quality inspection. These systems are integrated with manufacturing process where all products must pass through. Most systems consist of a camera (or cameras), PC and usually a controlled lighting environment within an enclosure [7].

## 2.2 Machine vision with artificial intelligence techniques

Machine learning and deep learning are data-driven artificial intelligence techniques, which may be applied to MVS. Both techniques uses neural network architecture concepts, which transforms raw data into representative information for decision making. Machine learning can be applied to feature extraction and classification where each step is constructed separately, and it may be highly dependent of an expert knowledge. In the other hand deep learning is applied in both feature extraction and classification steps as an unified neural network solution with requires minimum human interference. Figure 2 shows a comparison between machine learning and deep learning [8].

<figure><img src="image_2.png" /><figcaption>Fig. 2. Comparison between machine learning and deep learning. Adapted from [8].</figcaption></figure>

One major drawback of MVS system without artificial intelligence techniques is that it cannot learn with all the processed images. Every image detail or new fault detection that may occur after the MVS initial setup may not be detected, which can lead to an incorrect information output while MVS with learning features has the potential to learn with new images incomes.

Golbani and Asadpour [4] proposed a block diagram for a typical vision system operation when artificial intelligence techniques were still being developed for MVS, as shown in Figure 3. A new diagram was built containing the original block together with new AI techniques identified in recent literature. In this new framework is important to emphasize the need of an image knowledge database, which contains object features and quality criteria definitions, which may be used to assist and improve the current learning methods [1][8].
<figure><img src="image_3.png" /><figcaption>**Fig. 3.** A block diagram for a typical vision system operation and applicable AI techniques. Adapted from [1][4][8].</figcaption></figure>

## 2.3 1-D and 2-D MVS industrial applications

One (1-D) and two dimensional (2-D) MVS have a wide range of applications such as measurement, surface and depth inspection, thermal inspection and robot vision. Each kind of application has its own characteristic equipment with different image gathering source, such as photoelectric sensor, lasers, cameras and so on. Some MVS source types and applications are shown in Figure 4.

<figure><img src="image_4.png" /><figcaption>**Fig. 4.** Conventional MVS types and detection modes. Adapted from [1][9].</figcaption></figure>

Camera based MVS for industrial applications are commonly used to verify presence or absence of components, verify if the components are in their correct position and orientation, verify if components has the desired colors, analyze and recognize image
content such as code bars and inspect size and measure of parts and assembly
components [4].

Photoelectric and laser-based MVS inspections can also be used to presence or
absence of components, check component positioning and verify desired colors, mainly
to measure parts.

2.4 3-D MVS industrial applications

Optical non-contact 3-D measurement technique has been used to measure an image of an object and extract its geometrical information. It can be divided in passive and active 3-D sensing systems, where passive works with natural lighting from the scene without controlling the light that goes to the inspected object. While active sensing systems uses an external light, such as laser or a known projected light, by measuring speed of light, laser coherence or applying triangulation techniques [10].

Structured light is an active 3-D sensing system, which illuminates the object with predefined patterns and analyses how these patterns are deformed by the object when observed from a different angle of the projection. Some systems adopt non-visible structured light to avoid interfering with other computer vision [11].

Stereo vision profilometry techniques simulates human vision through two camera setups angled with each other, which aims to identifying and match common features of an object images from multiple allowing it to be reconstructed through triangulation techniques [11]. Stereo vision normally is a passive 3-D sensing system but there are new camera setups which also uses a projected structured light in the object that turns them into active stereo vision [7].

Another active vision techniques are through time of flight light measurement. This technique uses light pulses with a known camera range so the time for the emitted light to travel from the camera and hits the object and it reflected to the camera is measured, based a fixed and known light speed the distance can be calculated [7].

Light coding imaging is also an active 3-D sensing system but instead of using light pulses it keeps light source constantly turned on. It also uses an infrared spectrum emitter and receiver, which analyses lens distortion, the emitted light patter and the distance between object, emitter and receiver and the deformation of the light over the inspected object [7].

2.5 MVS industrial evaluation

Pérez et al. [7] compared several 3-D machine vision techniques applied to industrial environments emphasizing which factors need to be considered in order to select the most adequate vision, considering, system accuracy, working distance, image output, system advantages and limitations. An overview of this evaluation is shown in Figure 5 along with the current status of machine vision system in the automotive industry.
<table><caption>Fig. 5. MVS system assessment. Adapted from [7].</caption><thead><tr><th>MVS TYPE</th><th>TYPE</th><th>PRECISION</th><th>WORKING DISTANCE</th><th>ADVANTAGES</th><th>CURRENT LIMITATIONS</th><th>AUTOMOTIVE INDUSTRY PERSPECTIVE</th></tr></thead><tbody><tr><td>Traditional Camera System</td><td>2D</td><td>up to 0,1mm (*1)</td><td>18 mm up to 2m (*1)</td><td>Solutions are commercially available with customizable lenses and lighting accessories</td><td>Influenced by environment light conditions<br>Object and Camera must be static<br>Fewer image processing features which limits some defect type and object detection</td><td>Already in use (*3).<br>AI applications under technical viability evaluation</td></tr><tr><td>Photoelectric or Laser System</td><td>2D</td><td>(*2)</td><td>60mm up to 5m</td><td></td><td>Influenced by environment light conditions<br>Object and Camera must be static</td><td>Already in use (*4)</td></tr><tr><td>Laser Triangulation</td><td>3D</td><td>(*2)</td><td>-</td><td></td><td>Influenced by environment light conditions</td><td>Under technical viability evaluation</td></tr><tr><td>Time of Flight Triang.</td><td>3D</td><td>10mm</td><td>0,25m up to 3m</td><td>Independent of ambient light<br>Static Object and Camera are not necessary</td><td>Low accuracy</td><td>Under technical viability evaluation</td></tr><tr><td>Light Coding Imaging</td><td>3D</td><td>10mm</td><td>1m up to 3m</td><td>Static Object and Camera are not necessary</td><td>Low accuracy</td><td>Technology not evaluated yet</td></tr><tr><td>Structured Light</td><td>3D</td><td>34 µm up to 0,12mm</td><td>157mm up to 480mm</td><td>High Accuracy</td><td>Short working distance / Sensor can be quite large<br>It may be influenced by ambient light depending on structured light type<br>Static Object and Camera are needed</td><td>Technology not evaluated yet</td></tr><tr><td>Stereo Vision and Photogrammetry</td><td>3D</td><td>up to 50 µm</td><td>0,25m up to 3m</td><td>High Accuracy</td><td>Influenced by environment light conditions<br>Physical marks necessary<br>Point cloud density can be low<br>Object and Camera must be static<br>Intensive image processing required and it may be time consuming</td><td>Under technical viability evaluation</td></tr><tr><td>Projected Texture Stereo Vision</td><td>3D</td><td>up to 0,1mm</td><td>0,25m up to 3m</td><td>No physical marks necessary</td><td>Influenced by environment light conditions<br>Object and Camera must be static<br>Intensive image processing required and it may be time consuming</td><td>Technology not evaluated yet</td></tr></tbody><tfoot><tr><td colspan="7">(*1) - Depend on Lens Selection, Camera Resolution, Working distance, object size and object tolerances<br>(*2) - Increasing working distance decreases inspection accuracy.<br>(*3) - Used for part detection, poka yoke - New process or process modification greatly affects its performance. New products may not be detected with the current<br>(*4) - Used for part detection and precision measurement</td></tr></tfoot></table>

Environment light influence is one of the major problem to MVS, only a few systems are not subject to external light conditions. Another constraint that may affect your MVS selection is the necessity of both the camera and the object to remain static which, in some cases, are an exception in the industry. Accuracy and working distances are variables that must be taken in account, because depending on the precision needed the MVS solutions options narrows down.

## 2.6 MVS applications to Industry 4.0

The fourth industrial revolution or Industry 4.0 aims to develop intelligent factories with upgraded manufacturing technologies through new features such as cyber-physical systems (CPSs), the Internet of Things (IoT), Big Data and cloud computing. New manufacturing systems propose simultaneous monitoring of physical processes with being controlled by digital technologies, being able to make smart decision through real-time communication and interaction between humans, machines, or any smart device [12].

Figure 6 contains a simplified 4.0 Industry diagram, adapted from more complex diagrams available in the literature. Machine vision is located in an IoT layer. Its function is to provide image data through a connected network to a big data cloud server. This data will be subject to mining and cleaning procedures, removing unnecessary information. This information can be used as an input for machine learning techniques, allowing an integrated system to detect and describe what happened to the product, determining why that happened, predict what may happen and prescribe which actions must be taken.
<figure><img src="image_5.png" /><figcaption>Fig. 6. MVS integrated with a simplified 4.0 Industry diagram. Adapted from [8][13].</figcaption></figure>

MVS future solutions, such as knowledge-driven decision-making, real time control, online advanced analytics and artificial intelligence in CPS, are considered as challenging implementation process. It may take from 3 to 10 years for industries to achieve a concrete degree of maturity and obtain a fully operational system with these functionalities. [14]

## 3 Methodological approach

In order to obtain a detailed comprehension of the theme and to direct future studies, a systematic review is proposed. Based on the keywords identified in the theoretical foundation, they were grouped in three fields of study: Industrial Quality Control (IQC), AI and Machine Vision (MV). Papers available through Scopus database were chosen, covering the period between 2007 to 2017.

Figure 1 shows each keyword used for each group. Keywords combinations from same group were not used in this paper, in order to increase the likelihood of significant search results.
<figure><img src="image_6.png" /><figcaption>**Fig. 1.** Proposed field groups with each composing keyword.</figcaption></figure>

The primary search results totalized 60.176 articles but they contained duplicate values. Before removing duplicate values, results which belonged of at least more than one search group were labelled as 'AI+MV+IQC'. After that duplicates were removed, 42.858 articles remained.

Figure 2 shows these non duplicated publications distributed over the analysed years. The graphic results indicates an increased number of publications for each group. Artificial intelligence and machine vision keywords combination presented the most relevant increase, mainly after 2012. It can be observed that vision systems applied to industrial quality through AI technologies tend to be revelant in the following years.

<figure><img src="image_7.png" /><figcaption>**Fig. 2.** Publications in each field of application over the years.</figcaption></figure>

We filtered our search results to article labelled by SCOPUS as Journals, reducing to 16760 articles. Through ISSN information of each journal and the database extracted from SCIMAGO, a link between database could be established with the remaining search results. With SCIMAGO information another filter was applied, selecting journals that has been evaluated by SCIMAGO, totalizing 8549 articles.
A content analysis with detailed keyword identification was made using the remaining articles. Titles and abstracts with keywords related to medicine, human features identification and non industrial applications were used to create an exclusion filter, because they were not related to this paper, resulting in 3350 articles.

New relevant keyword were also identified and included in initial keywords used to create the first database. With the updated groups, a filter with articles considering at least one keyword of each group was made and 289 results were found.

Table 1 contains the keywords used in the last filter and Table 2 summarizes all filtering steps database results.

<table><caption>**Table 1.** Search results for machine vision applications and trends.</caption><thead><tr><th colspan="2">Vision</th><th colspan="3">Industrial/Industrial Quality</th><th>AI</th></tr></thead><tbody><tr><td>Image</td><td>Feature Matching</td><td>Assembly line</td><td>Flaw</td><td>Product Analysis</td><td>Artificial Intelligence</td></tr><tr><td>Object Recognition</td><td>SURF</td><td>Assembly</td><td>Flaw Detection</td><td>anomaly detection</td><td>Deep Learning</td></tr><tr><td>Shape Recognition</td><td>ORB</td><td>industrial</td><td>Fault Detection</td><td>anomaly</td><td>Machine Learning</td></tr><tr><td>Computer Vision</td><td>3D</td><td>Manufac</td><td>Fault</td><td>measurement</td><td>Neural Network</td></tr><tr><td>Machine Vision</td><td>3-D</td><td>Product</td><td>Product Quality</td><td>quality prediction</td><td></td></tr><tr><td>Part Recognition</td><td>scanning</td><td>Production</td><td>Quality Control</td><td>Visual Quality</td><td></td></tr><tr><td>2D</td><td>Image Processing</td><td>industrial applications</td><td>quality assessment</td><td>measure</td><td></td></tr><tr><td>2-D</td><td>Imaging technology</td><td>industry</td><td>Defect detection</td><td>automated quality control</td><td></td></tr><tr><td>Tracking object</td><td>Laser Triangulation</td><td>factories</td><td>Quality testing</td><td>industrial inspection</td><td></td></tr><tr><td>Track object</td><td>Photogrammetry</td><td>facility</td><td>Inspection</td><td>Total Quality</td><td></td></tr><tr><td>SIFT</td><td>Stereo Vision</td><td>manufactory</td><td>weld</td><td>Automotive Industry</td><td></td></tr><tr><td>Feature Detection</td><td>Structured Light</td><td>vehicle</td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>**Table 2.** Search results for machine vision applications and AI and IQC trends.</figcaption></figure>

## 3.1 Initial Results

During abstract reading step, solutions which integrates AI and IQC applied to MVS were found. One proposed solution used genetic programming with machine learning to develop and modify preprocessing programs. It has the ability to adapt to new production parameters and light conditions changes with an automated preprocessing framework. This solution has a potential to solve common MVS problems [15].
One common aspect of most of the articles is that they only provide a solutions according to a specific type of industry, such as welding systems, PCB fault detection, and sometimes in does not have the potential to be applied to other scenarios. One important factor detected in the articles is the absence of a common key performance indicators (KPI) in order to evaluate how accurate is each MVS solution. Another point identified is the absence of a method for selecting the best AI solution framework for each kind of solution. Figure 3 shows that from 289 results, only a few articles provides a framework or method as an important part of the abstract.

<figure><img src="image_9.png" /><figcaption>Fig. 3. Publications in each field of application over the years.</figcaption></figure>

## 4 Conclusion

MVS solutions applied to quality inspection for the industry can be improved through existing and new technologies such as AI, increase inspection detection processing speed and the capacity detect new types of defects.

There are many project restrictions and technical constraints in MVS implementation, which must be considered during selection, implementation and validation steps for industrial solutions, otherwise the system may not perform accordingly.

Despite existing MVS solutions with AI applied to IQC, they tend to be exclusive for each scenario and they lack of a common KPI to evaluate its performance or the ability the replicated in other industrial scenarios. A method to select the most appropriate type of AI technique and how to integrate with existing or new MVS systems it is still lacking.

## 5 Future work

A detailed review initial of the current research is necessary, in order to identify which are main adopted AI frameworks and techniques to MVS and IQC, which are the KPI's to evaluate MVS performance, how much MVS with AI are improving industrial KPIs and which are the main researches.

**Acknowledgments.** This work was supported by Araucaria Foundation for Science and Technology / FA-PR under Grant 40/2017 and Renault Brazil.
References

1. Vergara-Villegas, O. O., Cruz-Sánchez, V. G., de Jesús Ochoa-Domínguez, H., de Jesús Nandayapa-Alfaro, M. and Flores-Abad, Á. Automatic Product Quality Inspection Using Computer Vision Systems. In Lean Manufacturing in the Developing World, Springer, Cham., pp. 135-156 (2014). https://doi.org/10.1007/978-3-319-04951-9_7

2. Satorres, S., Gómez, J., Gámez, J. and Sánchez, A. A machine vision system for defect characterization on transparent parts with non-plane surfaces. *Machine Vision and Applications*, 23, (1), 1–13 (2012). https://doi.org/10.1007/s00138-010-0281-0

3. Szkilnyk, G. Vision-based fault detection in assembly automation (Thesis, Queen's University (Canada)) (2012).

4. Golnabi, H., and Asadpour, A. Design and application of industrial machine vision systems. *Robotics and Computer-Integrated Manufacturing*, 23, (6), 630-637 (2007).
https://doi.org/10.1016/j.rcim.2007.02.005

5. Labudzki, R., Legutko, S. and Raos, P. The essence and applications of machine vision. *Tehnicki Vjesnik*, 21, (4), 903-909 (2014).

6. Lerones, P., Fernández, J., García-Bermejo, J., and Zalama, E. Total quality control for automotive raw foundry brake disks. *International Journal of Advanced Manufacturing and Technology*, 27(3–4), 359–371 (2005). https://doi.org/10.1007/s00170-004-2165-9

7. Pérez, L., Rodríguez, I., Rodríguez, N., Usamentiaga, R., and García, D. F. Robot guidance using machine vision techniques in industrial environments: A comparative review. *Sensors*, 16, (3), 335 (2016). https://doi.org/10.3390/s16030335

8. Wang, J., Ma, Y., Zhang, L., Gao, R. X., and Wu, D. Deep learning for smart manufacturing: Methods and applications. *Journal of Manufacturing Systems*, in-press (2018).
https://doi.org/10.1016/j.jmsy.2018.01.003

9. Chauhan, V. and Surgenor, B. A comparative study of machine vision based methods for fault detection in an automated assembly machine. *Procedia Manufacturing*, 1, 416-428 (2015). https://doi.org/10.1016/j.promfg.2015.09.051

10. Su, X., and Zhang, Q. Dynamic 3-D shape measurement method: a review. *Optics and Lasers in Engineering*, 48, (2), 191-204 (2010).
DOI: https://doi.org/10.1016/j.optlaseng.2009.03.012

11. Van der Jeught, S., & Dirckx, J. J. Real-time structured light profilometry: a review. *Optics and Lasers in Engineering*, 87, 18-31 (2016).
DOI: https://doi.org/10.1016/j.optlaseng.2016.01.011

12. Zhong, R. Y., Xu, X., Klotz, E. and Newman, S. T. Intelligent Manufacturing in the Context of Industry 4.0: A Review. *Engineering*, 3, (5), 616-630 (2017).
DOI: https://doi.org/10.1016/J.ENG.2017.05.015

13. Pal, A., Dasgupta, R., Saha, A., and Nandi, B. Human-Like Sensing for Robotic Remote Inspection and Analytics. *Wireless Personal Communications*, 88, (1), 23-38 (2016).
https://doi.org/10.1007/s11277-016-3239-3

14. Leitão, P., Colombo, A. W. and Karnouskos, S. Industrial automation based on cyber-physical systems technologies: Prototype implementations and challenges. *Computers in Industry*, 81, 11-25 (2016).
DOI: https://doi.org/10.1016/j.compind.2015.08.004

15. Nagato, T, Koezuka, Automatic generation of image-processing programs for production lines. *Fujitsu Sci. Tech, Journal*, 52, (2016)