# Automating Quality Control Based on Machine Vision
## Towards Automotive 4.0

Dimitris Mourtzis, John Angelopoulos, Angelos Nektarios Arvanitis, Nikos Panopoulos

► To cite this version:

Dimitris Mourtzis, John Angelopoulos, Angelos Nektarios Arvanitis, Nikos Panopoulos. Automating Quality Control Based on Machine Vision Towards Automotive 4.0. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.126-134, 10.1007/978-3-031-16407-1_16. hal-05386038

HAL Id: hal-05386038

https://inria.hal.science/hal-05386038v1

Submitted on 27 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Automating Quality Control Based on Machine Vision Towards Automotive 4.0

Dimitris Mourtzis✉, John Angelopoulos✉, Angelos Nektarios Arvanitis, and Nikos Panopoulos✉

Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and Aeronautics, University of Patras, 26504 Rio Patras, Greece

mourtzis@lms.mech.upatras.gr

**Abstract.** By virtue of the increased automation, digitization, and digitalization of manufacturing and production systems a constantly growing amount of data, also known as Big Data Sets, are produced. Thus, techniques, in particular Machine Learning (ML) and Artificial Intelligence (AI), to utilize and organize Big Data Sets for the benefit of contemporary industries are in need. High-quality products are in great demand, especially under the framework of Mass Customization. Machine Vision (MV) algorithms in combination with precision manipulation offered by robotics can be valuable towards this direction. Therefore, the contribution of this research work is focused on the design and development of a framework for automating the quality inspection of parts towards the creation of an Intelligent Manufacturing Cell (IMC). The applicability of the proposed framework is validated in a car differential case study derived from the automotive industry, achieving mesh quality of 98.7%, with minimal voids based on 64 image samples.

**Keywords:** Manufacturing systems · Artificial Intelligence (AI) · Machine Learning (ML) · Machine Vision (MV) · Automotive 4.0

## 1 Introduction and State of the Art

Lately, the digitization and the digitalization of manufacturing has obtained a fair amount of scientific consideration [1, 2]. The key drivers/challenges for modern manufacturing and production systems can be summarized to are the volatility in demand, the requirements for increased quality, customized products, as well as the emerge of flexible supply chains [3]. Hence, companies are focusing on the development of new strategies based on the integration of automation and flexibility, such as AI, ML, MV and Computer-Aided Technologies (CAx). The main challenges for AI integration are, i) reasoning, ii) representation, iii) knowledge, learning, iv) natural language processing, v) planning, perception and vi) precision object manipulation. The gap between AI and natural intelligence has led to the creation of two models, in particular the structuralist and the functionalist. Concretely, structuralist models loosely mimic human mind, in areas of basic intelligence operations, such as logic and reasoning. On the other hand,
functionalist models correlate data to its computer counterpart [4]. Overall, AI constitutes a set of technologies and techniques enabling computers and machines to develop intelligence [5, 6]. Consequently, machine tools must become more intelligent, capable of constructing a solid correspondence framework among them and supporting Human-Machine Interface (HMI) [7, 8]. Furthermore, communication networks and structures like Wireless Sensor Networks and Industrial Networks (ModBus, ProfiBus), are important for establishing communication channels between various systems. Cyber-Physical Systems (CPS) consist of smart machines and other production facilities aiming at a more organized production [9]. MV is the mix of strategies and innovation providing an imaging-based programmed review and examination for applications, e.g. investigation, process control, quality control, robot guidance etc. [10] The rising of automation has transformed complex activities manufacturing into direct step by step methodologies that can be repeated by machines [11]. Ma et al. [12], performed an optical inspection for blister defect detection for lithium-ion batteries based on the development of a “DenseNet” architecture. Staar et al. in [13] proposed a modern triplet architecture for Convolutional Neural Networks (CNN), applied two data augmentation methods and trained the algorithm on three diverse datasets, for improving the algorithm’s accuracy. Similarly, Pachner et al. [14] and Lokrantz et al. [15], focused on the entirety of a production process instead of the final visual inspection. Lokrantz model was a Bayesian network capable of recognizing quality deviations in two dissimilar manufacturing processes. On the other hand, Pachner, create a model of a decision tree to gain insights about the effects of several process parameters on the floss of the final product for the optimized manufacturing process of powder coating [14]. Sumesh et al. [16] in their research work have proposed a method for the quality classification of welding processes by utilizing a random forest and (J48) decision tree. Based on the literature investigation, quality control in manufacturing is still an open challenge. Furthermore, robotics have not yet been fully integrated to support such activities in modern manufacturing environments. Thus, the contribution of this research work is focused on the design and development of a framework for automating the quality inspection in modern manufacturing and production systems in the automotive industry, utilizing MV and ML techniques, that will be embedded in an IMC [7, 17].

## 2 Manufacturing Cell Model

The IMC is a combination of three machines, as illustrated in Fig. 1. The MV module consists of a depth camera and a robot. A 3D printer is also integrated for the production of additively manufactured components. Third, a CNC milling machine is utilized for light machining operations, as resulted by the CAD comparison module.
<figure><img src="image_3.png" /><figcaption>Fig. 1. a) Cell layout, b) STL of the differential, c) CAD of the differential</figcaption></figure>

Taking into consideration the increased volume and variation of produced parts in the automotive industry the proposed framework is based on the utilization of a differential, due to its complicated geometries. The differential consists of 6 gears, 1 input shaft, 2 output shafts and a housing for the gears and bearings.

## 3 Problem Formulation and System Architecture

The proposed framework relies on the development of methods for the creation of a point cloud with the use of a robot, and the conversion to CAD file, in order to compare it versus the CAD file of the manufactured part. The general system architecture is presented in Fig. 2.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Architecture of the proposed framework</figcaption></figure>

As illustrated in Fig. 2, the overall procedure of the framework is divided into two sections: pre-processing and processing. In the pre-processing stage, the user imports the CAD file to the Cloud Database, then the file is converted to an STL and automatically imported to the 3D-Printer. After the component is printed, the robot automatically transfers it to the scanning position. The robot then scans the component, with the use of the depth camera, and 64-point cloud images are captured and uploaded to the Cloud. Afterwards, the noise elimination and the alignment of the images into groups
are performed. The point cloud groups are automatically converted to 3D meshes. This conversion starts by applying the Poisson disk sampling in which each point of the cloud is subjected to an independent Bernoulli trial in order to determine if an element should be part of the sample, using Eq. 1.

$$ \pi_{ij} = \pi_i \times \pi_j \qquad (1) $$

where $\pi_i$ is the point population, $\pi_j$ is the point population that is sampled and $\pi_{ij}$ is the single sample of points of the Poisson disk sampling. The next steps are the computation of Normals for the aforementioned sets of points (n) and the procedure of rendering. Finally, the Ball Pivoting Algorithm (BPA) is applied [18], in order to reconstruct the surface, and ultimately create the mesh. Starting by determine a seed triangle ($\tau = \sigma_i, \sigma_j, \sigma_k$, where $\sigma_i, \sigma_j, \sigma_k$ the vertices of the triangle), which contains three datasets relative to the radius R, the BPA pivots a (virtual) ball of a given radius R (in this paper case R = 0.03 mm) around the already formed edges ($e(i, j), e(j, k)$ and $e(k, i)$) until it touches another point, forming another triangle. The interaction continues until all reachable edges have been attempted. In this manner the BPA is producing a large number of equal triangles, that finally form the mesh.

<figure><img src="image_5.png" /><figcaption>Fig. 3. a) Ball Pivoting Formation, b) Join operation of the Ball Pivoting Algorithm, c) Pseudocode for Ball Pivoting Algorithm</figcaption></figure>

The BPA uses the existing points without creating new ones. A pseudocode is illustrated in Fig. 3c, in order to depict the reasoning of the BPA. Then the file is extracted as an STL. Hence, 16 meshes in an STL formation are generated. Following, is the bonding of the 16 meshes to form one single, three-dimensional mesh, that will be geometrical optimized, by the smoothing of the edges and the filling of holes that may occur, from the bonding procedure. The BPA uses the existing points without creating new ones. In Fig. 4, it is visible that, both the point cloud and mesh files are immediately acquiring a three-dimensional shape, due to the bonding procedure.

The last phase of the pre-processing is the conversion of the STL file to a 3D CAD file, which will then pass in the processing phase. During the processing phase, the original CAD and the one the user acquired from the pre-processing phase are compared, using the deviation algorithm. If any deviations are detected, the UR10 will automatically transfer
<figure><img src="image_6.png" /><figcaption>Fig. 4. a) Point cloud of differential, b) mesh of differential</figcaption></figure>

the differential to the CNC-working center, to apply the deviations. If there are not any deviations, then the part is transferred to the buffer. The proposed method is based on the implementation of a k-nearest neighbor (k-NN) classification algorithm. The algorithm automatically detects the center of the CAD file in the Cartesian coordinate system. Afterwards, based on the user-selected accuracy (n), the edges of the original CAD are divided with n dots and the distance between each point and the origin is calculated. Then the algorithm detects if there are any deviations (Fig. 5).

$$C_n = [k_n(x_n, y_n, z_n) - d_n(x_n, y_n, z_n)] \quad (2)$$

where Cn is the value of the n-th (x, y, z) point of the deviation matrix, and k_n and d_n are the cartesian distances of the original and the scanned item respectively. The deviation results are saved to matrix C (Eq. 2).

<figure><img src="image_7.png" /><figcaption>Fig. 5. Deviation algorithm principle: a) scanned item converted to CAD, b) original CAD</figcaption></figure>

This matrix has n rows and 3 columns (for axes X, Y, Z).

$$C = \begin{bmatrix} -1.2 & -2.43 & -1.44 & \cdots & 0 & \cdots & 0 \\ -1.75 & -1.86 & -2.22 & \cdots & 0 & \cdots & 0 \\ 0 & 0 & 0 & \cdots & 0 & \cdots & -1.11 \end{bmatrix} \quad (3)$$

Afterwards, the algorithm shorts the negative coordinates, and convert them into g-code to be uploaded to the milling machine controller via the Cloud Database. This
algorithm operates similarly with the BPA. The deviation algorithm rolls a virtual ball, starting from the center of the CAD towards every edge of the three-dimensional CAD file. After each algorithm recursion, the deviation algorithm measures the distance from center-to-edge. Every time a distance is measured for both the original and printed CAD, the Cartesian coordinates of this edge, are imported to their respective distance matrix. Upon completion of measurements the two distance matrixes of both the original and printed CAD are created ([D<sub>O</sub>] – [D<sub>P</sub>]), the final deviations matrix is formed by their subtraction.

$$[C] = [D_{O,i,j,k}] - [D_{P,i,j,k}] \qquad (4)$$

where, [C], is the deviation matrix, [D<sub>O,i,j,k</sub>] is the distance matrix of the original CAD, and [D<sub>P,i,j,k</sub>] is the distance matrix of the printed CAD.

## 4 Method Implementation

Regarding the IMC, the machines used are a 3D Systems Cube Trio 3D-Printer, an UR 10 robot integrated with an Intel RealSense D435 depth camera for the 3D scanning, and an XYZ RMX 2500 milling machine. From a software point of view, the GUI have been developed in Unity 3D game engine, both as a mobile application and a desktop stand-alone application. The code scripts for the GUIs have been developed in MS™ Visual Studio IDE (Integrated Development Environment) in C#, and in Python for the CAD comparison algorithm. MeshLab has also been utilized as a middleware for parsing

<figure><img src="image_8.png" /><figcaption>Fig. 6. 1) CAD selection GUI; 2) CAD comparison GUI, with deviation highlighted</figcaption></figure>
the 3D point cloud and the conversion to CAD format. The robot's path was designed in the RoboDK software and exported as a universal report (urp) file for uploading to the controller. The GUIs are displayed in Fig. 6.

# 5 Results and Discussion

The quality of the mesh grid is essential for the successful operation of the framework, which is also depended to the point cloud quality. Concretely, a more detailed point cloud data can result to a more solid final mesh. The experimentation began by capturing only four images of the differential (one image per side). The resulting mesh is of poor quality (see Fig. 7a) with multiple voids. Thus, several experiments were conducted, increasing the sampling rate each time. The best results have been obtained for 64 sample images. Beyond this sampling rate, the quality improved, but cavities were formed. As illustrated in Fig. 7, for 64 samples the quality of the mesh is 98.7%, with minimal voids.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Mesh Quality regarding the Number of Samples</figcaption></figure>

# 6 Concluding Remarks and Outlook

In this manuscript a method has been proposed for improving the automation level of an IMC via the utilization of a MV algorithm and a robotic arm for the quality inspection of manufactured products. The key contribution of this research work lies in the development of an algorithm for the comparison of CAD files prior and postproduction of the products/components. Furthermore, the results presented in the previous paragraphs indicate that the precision and repeatability of robotic arms has to be further exploited towards the direction of 3D scanning and quality control. The presented method is part of a bigger solution, towards the complete automation of the shop-floor level, limiting human intervention to the bare minimum required, towards the realization of what is denoted as “Lights-out Factory”. However, there is still room for further elaboration. More specifically, since robotic manipulators can multipurposed, future research work,
will be focused on the development of an intelligent robotic end effector, which will enable the robotic arm to handle different geometries, and use a separate end effector for the depth camera. Concretely, the cell presented, will be integrated with an automatic tool changer for the end effector, enabling the robot to select through a wider variety of tools. By extension the cell can be further optimized in order to utilize the robot in light machining operations instead of using the CNC milling machine, and thus achieve greater energy economy. Future work will also be head towards the further elaboration of the CAD conversion and comparison algorithm, in order to reduce geometry voids and eliminate surface imperfections (e.g. casting marks) that do not affect the operation of the component but appear as deviation from the original CAD (which has a totally smooth surface).

## References

1. Liu, C., Zheng, P., Xu, X.: Digitalisation and servitisation of machine tools in the era of Industry 4.0: a review. Int. J. Prod. Res. 1–33 (2021). https://doi.org/10.1080/00207543.2021.1969462

2. Mourtzis, D.: Simulation in the design and operation of manufacturing systems: state of the art and new trends. Int. J. Prod. Res. 58(7), 1927–1949 (2020). https://doi.org/10.1080/00207543.2019.1636321

3. Wang, X., Wang, Y., Tao, F., Liu, A.: New paradigm of data-driven smart customisation through digital twin. J. Manuf. Syst. 58, 270–280. https://doi.org/10.1016/j.jmsy.2020.07.023

4. Lieto, A., Lebiere, C., Oltramari, A.: The knowledge level in cognitive architectures: current limitations and possible developments. Cogn. Syst. Res. 48, 39–55 (2018). https://doi.org/10.1016/j.cogsys.2017.05.001

5. Aydin, N., Ray, Y., Zhong, X.L., Bogdan, I.-E.: Review of machine learning technologies and artificial intelligence in modern manufacturing systems. In: Mourtzis, D. (ed.) Design and Operation of Production Networks for Mass Personalization in the Era of Cloud Technology. Elsevier, Amsterdam, pp. 317–348 (2022). https://doi.org/10.1016/B978-0-12-823657-4.0002-6

6. Yang, T., Yi, X., Lu, S., Johansson, K.H., Chai, T.: Intelligent manufacturing for the process industry driven by industrial artificial intelligence. Engineering 7(9), 1224–1230 (2021). https://doi.org/10.1016/j.eng.2021.04.023

7. Dogan, A., Birant, D.: Machine learning and data mining in manufacturing. Expert Syst. Appl. 166, 114060 (2021). https://doi.org/10.1016/j.eswa.2020.114060

8. Mourtzis, D.: Machine Tool 4.0 in the Era of Digital Manufacturing (2020). https://doi.org/10.46354/i3m.2020.emss.060

9. Weichert, D., Link, P., Stoll, A., Rüping, S., Ihlenfeldt, S., Wrobel, S.: A review of machine learning for the optimization of production processes. Int. J. Adv. Manuf. Technol. 104(5–8), 1889–1902 (2019). https://doi.org/10.1007/s00170-019-03988-5

10. Yin, S., Ji, W., Wang, L.: A machine learning based energy efficient trajectory planning approach for industrial robots. Procedia CIRP 81, 429–434 (2019). https://doi.org/10.1016/j.procir.2019.03.074

11. Mourtzis, D., Angelopoulos, J., Dimitrakopoulos, G.: Design and development of a flexible manufacturing cell in the concept of learning factory paradigm for the education of generation 4.0 engineers. Procedia Manuf. 45, 361–366 (2020). https://doi.org/10.1016/j.promfg.2020.04.035
12. Ma, L., Xie, W., Zhang, Y.: Blister defect detection based on convolutional neural network for polymer lithium-ion battery. Appl. Sci. 9(6), 1085 (2019). https://doi.org/10.3390/app9061085

13. Staar, B., Lütjen, M., Freitag, M.: Anomaly detection with convolutional neural networks for industrial surface inspection. Procedia CIRP 79, 484–489 (2019). https://doi.org/10.1016/j.procir.2019.02.123

14. Pachner, S., Miethlinger, J.: Smart data analysis for optimized manufacturing of powder coatings on co-rotating twin screw extruders. AIP Conf. Proc. 2055(1), 070010 (2019). https://doi.org/10.1063/1.5084854

15. Lokrantz, A., Gustavsson, E., Jirstrand, M.: Root cause analysis of failures and quality deviations in manufacturing using machine learning. Procedia CIRP 72, 1057–1066 (2018). https://doi.org/10.1016/j.procir.2018.03.229

16. Sumesh, A., Rameshkumar, K., Mohandas, K., Babu, R.-S.: Use of machine learning algorithms for weld quality monitoring using acoustic signature. Procedia Comput. Sci. 50, 316–322 (2015). https://doi.org/10.1016/j.procs.2015.04.042

17. Tsurumine, Y., Cui, Y., Uchibe, E., Matsubara, T.: Deep reinforcement learning with smooth policy update: application to robotic cloth manipulation. Robot. Autonom. Syst. 112, 72–83 (2019). https://doi.org/10.1016/j.robot.2018.11.004

18. Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., Taubin, G.: The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Visualization Comput. Graph. 5(4), 349–359 (1999). https://doi.org/10.1109/2945.817351