<figure><img src="image_1.png" /></figure>

HAL
open science

# Product Quality Inspection Combining with Structure Light System, Data Mining and RFID Technology

Kesheng Wang, Quan Yu

► To cite this version:

Kesheng Wang, Quan Yu. Product Quality Inspection Combining with Structure Light System, Data Mining and RFID Technology. 6th Programming Languages for Manufacturing (PROLAMAT), Oct 2013, Dresden, Germany. pp.205-220, 10.1007/978-3-642-41329-2_22. hal-01485816

HAL Id: hal-01485816

https://inria.hal.science/hal-01485816

Submitted on 9 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Product Quality Inspection Combining with Structure
Light System, Data Mining and RFID Technology

Kesheng Wang and Quan Yu

Knowledge Discovery Laboratory, Norwegian University of Science and Technology, Norway
{kesheng.wang, quan.yu}@ntnu.no

**Abstract.** 3D vision inspection has been rapidly developed and increasingly applied in product quality inspection. Structure Light System (SLS) is one of these technologies with good performance and lower cost. To achieve the automated quality inspection, a decision support system is required to provide with the SLS together to automatically perform the quality check. However, for extracting quality information, the quality inspection system has to be combined together with a data logging system, so that the quality information of the product is available for production query and quality checking. In this paper, Radio Frequency Identification (RFID) technique is used to accomplish the production tracing and tracking together with the quality inspection system. By assigning a RFID tag to each inspected part, it is possible to identify the type of the product and write the quality inspection result decided by the data mining classifier for the real-time quality query. The proposed approach will be an alternative for SMEs considering the fast product type update with respect to the fast changing market.

**Keywords:** Quality inspection, Structure Light System, Data mining, RFID

# 1 INTRODUCTION

Product quality inspection is a nontrivial procedure during the manufacturing process both for the semi-finished and end products. 3D vision inspection has been rapidly developed and increasingly applied in product quality inspection, with the advantages of high precision and applicability comparing with commonly used 2D vision approaches. 3D vision is superior in inspection of multi-features parts due to providing the height information. 3D vision techniques comprise approaches on the basis of different working principles [1]. Among various approaches, the Structure Light System (SLS) is a kind of cost effective techniques for the industrial production [2, 3, 4]. By projecting specific patterns on the inspected product, the camera captures corresponding images. The 3D measurement information of the product is retrieved in the form of the point cloud on the basis of the images. With the generated 3D point cloud, automated quality inspection can be performed with less human interference, where data mining approach is commonly used [5, 6].

However, although the quality information is able to be decided on the basis of the SLS and data mining approaches, it will not become more valuable until it is used to
improve the production process and also achieve the real-time data access and the
quality traceability.

In this paper, Radio Frequency Identification (RFID) technique is used to integrate the quality information together with the product. RFID uses a wireless non-contact radio system to identify objects and transfer data from tags attached to movable items to readers, which is fast, reliable, and does not require physical sight or contact between reader/scanner and the tagged objects. By assigning a RFID tag to each inspected product, it is possible to identify the product type and query the quality inspection history. An assembly quality inspection problem is selected as a case study to test the feasibility of proposed complex system. The proposed approach will be an alternative for SMEs considering the fast product type update with respect to the fast changing market.

The paper is organized as the following: Section 1 introduces the general applica-
tions of 3D vision in manufacturing and the importance of combining SLS, Data min-
ing and RFID technology. Section 2 introduces the architecture of the complex sys-
tem and the working process in each system level respectively. Section 3 shows a case
to study the feasibility of the system combining the three techniques. Section 4 comes
to the conclusion of the applicability of the complex system.

2 QUALITY INSPECTION SYSTEM ARCHITECTURE ON
THE BASIS OF STRUCTURE LIGHT SYSTEM, DATA
MINING AND RFID

The RFID 3D quality inspection system combines the function of product quality
inspection and RFID tracing and tracking. By attaching the RFID tag to the product,
the system takes the pictures of the inspected product as the inputs, generates the 3D
point cloud and finally writes the quality related information of the product in the
RFID tag. Thus, it is available to monitor the product quality along the production
line and achieve the real-time quality control.
## 2.1 System architecture

<figure><img src="image_3.png" /><figcaption>Fig. 1. System architecture</figcaption></figure>

The quality inspection system comprises 4 levels as shown in Figure 1, which are 3D vision level, data processing level, computational intelligence level and RFID level respectively. Within each level of the system, the data is converted as the sequence of the point cloud, the feature vector, the quality information and the writable RFID data. Each level is introduced as following:

1. 3D vision level consists of the Structure Light System, which uses camera together with the projector to generate the point cloud of the inspected product.

2. Data processing level comprises quality related feature determination and extraction. The product quality is quantified on the basis of the point cloud according to the design requirements. The feature vector is generated after the processing as the input of the next level.

3. Computational intelligence level uses data mining approaches to achieve the automated quality classification on the basis of the feature vector.

4. RFID level comprises the RFID hardware and software, which achieves the product tracking and controlling by writing and reading RFID tag attached on the product.

## 2.2 Introduction of the Structure light system

Structured Light System (SLS) is one of typical 3D vision techniques. SLS accomplishes to point cloud acquisition by projecting specific patterns onto the measured object and capturing the corresponding images. The point cloud of the object surface can be generated with the images analysis approach.

The hardware of a SLS consists of a computer, an image capture device and a projector. Figure 2 shows the working process of a typical SLS, which can be divided into 4 steps.
<figure><img src="image_4.png" /><figcaption>Fig. 2. General working process of a Structured Light System</figcaption></figure>

1. Step 1 is the pattern projection. A coded light pattern is projected onto the scene by the projector. The pattern can be either a single one or series with respect to the type of the code.

2. Step 2 is the image recording. The inspected object is captured by the camera, and then captured images are stored in sequence if pattern series are used. The scene is captured previously as references without the presence of the object. Comparing the images with the inspected object to the one without it, it is observed that the pattern is distorted due to the existence of the object, which indicates the height information.

3. Step 3 is the phase map generating. Images captured in step 2 are analysed by the computer with fringe analysis techniques on the basis of the pattern encoding rule. The wrapped phase maps are obtained firstly, and to be unwrapped to obtain the map with continuous phase distribution.

4. Step 4 is the transformation from phase to height. The height value of each image pixel is derived from its phase by phase calibration or phase-height mapping comparing with the reference obtained from step 2. After the calibration, the pixels in the image are transformed to points in metric unit, and the height value in each pixel is calculated, so that the 3D point cloud of the inspected object is formed.

## 2.3 Decision support using data mining approaches

To achieve the automated product quality inspection, it is of importance to use computational intelligence approaches to classify the product quality with less artificial interference. Data mining method is often used to analyze the massive data and find out the knowledge to ascertain the product quality [7]. Data mining based quality inspection requires input variables related to product quality. Although the point cloud is able to be acquired using SLS in the form of 3D coordinates, it generally includes mega points and is unlikely possible to use the data driven methods to solve the classification problem. In this case, the point cloud generated by the SLS has to be
processed according to the specifics of the product. Mega points are converted to be a
vector including the most useful quality related parameters. It is effecient to focus on
the partial point cloud mostly representing the product quality, which can be seen as
the Region of Interest (ROI) of the point cloud. Furtherly, a vector X = {x₁,x₂,...,xₙ}
is extracted from the mega points, which comprises feature values xi calculated ac-
cording to the point cloud. Thus, the mega points are converted to a single vector
including geometrical features which most represent the product quality information.
With this step of simplification, it becomes feasible to select the most suitable data
mining approaches on the basis of the extracted feature vectors to do the quality clas-
sification. For example, three typical data mining approaches are common used in the
classification problem, which are Artificial Neural Networks (ANN), Decision tree
and Support Vector Machines (SVM) respectively.

1. Decision Tree

A decision tree is one of data mining approaches applied in many real world applica-
tions as a solution to classification problems. A decision tree is a flowchart-like tree
structure, where each internal node denotes a test on an attribute, each branch repre-
sents an outcome of the test, and each leaf node holds a class label. The construction
of decision tree classifiers does not require any domain knowledge or parameter set-
ting, and therefore is appropriate for exploratory knowledge discovery.

C4.5 is a classic algorithm for decision tree induction. The succedent algorithm
C5.0 is available in IBM SPSS Modeler®. By using this software, it is easy to ac-
complish the decision tree induction and test.

2. Artificial Neural Networks

As another effective data mining approach, an Artificial Neural Network consists of
layers and neurons on each of them. Parameters are adjustable in an ANN such as the
number of the hidden layers and neurons, the transfer functions between layers and
the train method etc. A powerful ANN toolbox is available in Matlab® and can be
highly customized to get the best result by the user.

3. Support Vector Machines (SVM)

A Support Vector Machine (SVM) is a supervised learning method for data analysis and pattern recognition. The standard SVM is designed for binary classification. Given a set of training examples, each marked as belonging to one of two categories, several SVM training algorithms are available to build a model that assigns examples into corresponding category. New examples are then predicted to belong to a category based on the constructed model. For multi-classification problem, a commonly used approach is to construct K separate SVMs, in which the kth model yk(x) is trained using the data from class Ck as the positive examples and the data from the remaining K-1 classes as the negative examples, which is known as the one-versus-the-rest approach.
## 2.4 RFID level

Radio Frequency Identification (RFID) is one of numerous technologies grouped under the term of Automatic Identification (Auto ID), such as bar code, magnetic inks, optical character recognition, voice recognition, touch memory, smart cards, biometrics etc. Auto ID technologies are a new way of controlling information and material flow, especially suitable for large production networks [8]. RFID is the use of a wireless non-contact radio system to transfer data from a tag attached to an object, for the purposes of identification and tracking. In general terms, it is a means of identifying a person or object using a radio frequency transmission. The technology can be used to identify, track, sort or detect a wide variety of objects [9]. RFID system can be classified by the working frequency, i.e. Low Frequency (LF), High Frequency (HF), Ultra High Frequency (UHF) and Microwave. Different frequency works for various media, e.g. UHF is not applicable to metal but HF is metal friendly. Thus, the working frequency has to be used on the basis of tracked objects.

Hardware of RFID system includes RFID tag, RFID reader and RFID antenna. RFID tag is an electronic device that can store and transmit data to a reader in a contactless manner using radio waves, which can be read-only or read-write. Tag memory can be factory or field programmed, partitionable, and optionally permanently locked, which enables the users save the customized information in the tag and read it everywhere, or kill the tag when it will not be used anymore. Bytes left unlocked can be rewritten over more than 100,000 times, which achieves a long useful life. Moreover, the tags can be classified by power methods i.e. passive tags without power, semi-passive tags with battery and active tags with battery, processor and i/o ports. The power supply increases the cost of the tag but enhance the readable performance. Furthermore, a middleware is required as a platform for managing acquired RFID data and routing it between tag readers and other enterprise systems. Recently, RFID become more and more interesting technology in many fields such as agriculture, manufacturing and supply chain management.

# 3 CASE STUDY

In this paper, a wheel assembly problem is proposed as the case study of the implementation for combining the SLS, data mining approaches and RFID technology. In the first step, the assembly quality classification is introduced. Secondly, the point cloud of the object is acquired using SLS and converted to be the feature vector, which is defined according to the assembly requirements and provided to the data mining classifier as the input. At last, the quality is decided by the classifier and converted to RFID data, which is saved in the RFID tag attached on the object.

## 3.1 Problem description

To certify the feasibility of proposed 3D vision based quality inspection, LEGO® wheel assembly inspection is taken as the example in this paper. In the supposed wheel assembly inspection, the object is to check the assembly quality. A wheel is
constituted of 2 components, the rim and the tire. Possible errors occur during the
assembly process as shown in Figure 3, which are supposed to be divided into 5 clas-
ses, according to the relative position of the rim and tire:

1. Wheel assembly without errors

2. Tire is compressed

3. There exists an offset for one side

4. Rim is detached from the tire

5. Rim is tilted

<figure><img src="image_5.png" /><figcaption>Fig. 3. Wheel assembly classification</figcaption></figure>

Each class has a corresponding inner layout, which is shown respectively in Figure 4.
The section views show the differences among the classes.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Inner layout of the wheel</figcaption></figure>
2D vision is not applicable to distinguish some cases because of the similarity of pictures, as shown in Figure 5.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Similarity from the top of the view</figcaption></figure>

It is noticed that there is not so much difference between the two wheels from the top view in the image. However, the height is different if seen from the side view. Structured Light System (SLS) is an effective solution which can get 3D point cloud of the inspected part, so that the real height value of the parts are obtained while errors can be recognized with the metric information directly.

## 3.2 Feature extraction for the classification

The hardware configuration is shown in Figure 6. The image capture device is used with a SONY XCG-U100E industrial camera, with UXGA resolution (1600×1200 pixels) and Gigabit Ethernet (GigE) interface, and together with a Fujinon 9mm lens. A BenQ MP525 digital projector is employed to project the patterns. The hardware control and the image processing are performed with commercial software Scorton®. After calibration, the accuracy of the measurement can achieve 0.01mm in this case study.
<figure><img src="image_8.png" /><figcaption>Fig. 6. SLS hardware</figcaption></figure>

In proposed assembly inspection, geometry features are extracted as the input attrib-
utes on the basis of the 3D point cloud and regression analysis, while the correspond-
ing class of the inspected wheel is predefined. 200 examples have been recorded for
each class. 180 of them are used as the train set while the left examples are used as the
test set.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Point cloud acquired with SLS</figcaption></figure>

Because the tire has much lower reflectivity comparing with the rim, more noises will be generated when acquiring the point cloud of both. To get a cleaner point cloud and simplify the feature extraction, the tire part is disregarded by lowering the contrast value. Thus, only the rim part is kept for the further analysis as shown in Figure 7. 6 line profiles are extracted from the point cloud of the rim part for feature extraction in the next step, with 3 horizontals and 3 verticals. Only the end sections of each line profile are kept for the regression analysis
<figure><img src="image_10.png" /><figcaption>Fig. 8. Section profile processing</figcaption></figure>

As shown in Figure 8, two sections of profile are used to define the attributes of the rim. Only the beginning and ending section of the profile are kept while other sections are ignored. The horizontal data of the points are considered as the predictors, which correspond to the x-axis and y-axis in terms of the horizontal and vertical profiles respectively, while the vertical data are considered as the responses, which means the height value of the points. It is possible to use statistics to estimate the regression line of the values corresponding to the profile section [10]. Taking into account the slope of the regression line, it is possible to estimate the height and the pose of the rim part.

The least squares linear regression is used to acquire the regression line. With the numerical data of a profile and the regression line corresponding to a wheel, 5 values are obtained, which are respectively

x1: slope of the regression line

x2: bias of the regression line

x3: mean height value of points in the beginning section

x4: mean height value of points in the ending section

x5: width between the starting point and the ending point

where x1 and x2 are acquired by linear regression analysis while x3, x4 and x5 are measured from the points of the profile sections directly shown in Figure 9. With all the extracted values, a data vector is constructed for each of the six profiles of the rim:

<figure><img src="image_11.png" /><figcaption>Fig. 9. Feature extraction</figcaption></figure>

Regarding the 5 classes in the wheel assembly problem, it is of importance to extract features most related to the assembly status from the point cloud. These 5 features extracted above are able to describe the pose of a wheel, which indicates the height and inclination. For each profile, 5 feature values are extracted. Thus, a vector $X_S$ is used to denote a wheel where $X_S = \{X_1, X_2, ..., X_5\}$ as the inputs of data mining approaches.
3.3 Quality information embedded using RFID tag

After the assembly quality has been approved using SLS and data mining decision
support system. The quality information will be written into the RFID tag placed in
the wheel. Thus, the quality information is combined with the product together for the
check in the future. In this wheel assembly quality inspection problem, the tag access-
ing is to be completed with the Reader Test Tool of the RFID reader as shown in Fig-
ure 10.

<figure><img src="image_12.png" /><figcaption>Fig. 10. RFID reader test tool</figcaption></figure>

In this case study, the OMNI-ID RFID tag, the SIRIT RFID reader and the IMPINJ
near-field antenna are used to construct the RFID system, as shown in Figure 11.
<figure><img src="image_13.png" /><figcaption>Fig. 11. RFID system hardware</figcaption></figure>

During the test, a RFID tag is placed in the tire in the first step as shown in Figure 12.

<figure><img src="image_14.png" /><figcaption>Fig. 12. Attach a RFID tag to a tire</figcaption></figure>

Then the tagged tire and a rim are assembled. Because the EPC code of each RFID tag is set to be unique, each wheel is given a unique EPC. Finally the assembly quality of the wheel is to be inspected using SLS and data mining based decision support system. After the classification, the quality information will be written into the tag and kept with the product as shown in Figure 13.
<figure><img src="image_15.png" /><figcaption>Fig. 13. Quality inspection and information writing</figcaption></figure>

The memory of the OMNI-ID RFID tag is divided into three parts, which are allocated for the EPC code of 96 bits, the user data of 512 bits and the TID data of 64 bits. The 512 bits user data are reserved for the customized information, which can be converted to 64 ASCII characters. In this case, the quality inspection related information is written in the user memory of the RFID tag. The information comprises the classified assembly quality, the inspection time and the inspection date, in the form of "QUALITY=X TIME=HH:MM DATE=DD.MM.YY". Because the tag memory is written in the form of hexadecimal, the texts have to be converted to HEX before writing. Supposing the quality is classified to be 1 and the inspection is taken at 14:05 on 30.04.13, the information is converted from the ASCII characters to hexadecimal as shown in Figure 14.

<table><caption>Fig. 14. Conversion from the ASCII characters to the Hexadecimal</caption><tr><td>Text</td><td>QUALITY=1 TIME=14:05 DATE=30.04.13</td></tr><tr><td>HEX</td><td>0x5155414C4954593D312054494D453D31343A303520444154453D33302E30342E3133</td></tr></table>

The information writing is completed using reader test tool as shown in Figure 15.
<figure><img src="image_16.png" /><figcaption>Fig. 15. RFID tag information control</figcaption></figure>

After the quality information is written in the user memory of the tag, it is able to be read out with any other RFID reader. Using the same decoding approach, the hexadecimal can be restored to characters again for later check.

## 3.4 Result and analysis

During the quality inspection test. Six measures are obtained for the classifier on the basis of 4 outcomes: True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN), which constructs a confusion matrix

<table><tr><td>TP (correctly accepted)</td><td>FN (incorrectly refused)</td></tr><tr><td>FP (incorrectly accepted)</td><td>TN (correctly refused)</td></tr></table>

For each validation, the confusion matrixes of each class are constructed respectively, and the final confusion matrix for each validation would contain the average values for all classes combined.

The 6 measures are defined as following

1. Correctly Classified Instances (CCI): Percentage of samples correctly classified.

2. Incorrectly Classified Instances (ICI): 100% -CCI.
3. True Positive Rate (TPR): TP / (TP + FN). Also called Recall measure.

4. False Positive Rate (FPR): FP / (FP + TN)

5. Precision (P): TP / (TP + FP)

6. F-Measure (FM): Defined as ( 2 * Precision *Recall) / (Precision + Recall)

<table><caption>Table 1 shows the performance SVM classifiers.</caption><thead><tr><td>Measures</td><td>Support Vector Machine</td></tr></thead><tbody><tr><td>CCI</td><td>95.80 %</td></tr><tr><td>ICI</td><td>4.20 %</td></tr><tr><td>TPR</td><td>0.853</td></tr><tr><td>FPR</td><td>0.011</td></tr><tr><td>Precision</td><td>0.958</td></tr><tr><td>F-measure</td><td>0.902</td></tr></tbody></table>

<caption>Table 1. Results of SVM</caption>

After the classification, the inspection related information is written in the RFID tag. The inspection time and date is available using reader test tool as shown in Figure 16.

<figure><img src="image_17.png" /><figcaption>Fig. 16. Acquiring the RFID tag information</figcaption></figure>

Combining with the quality classification result, the text to be written is

QUALITY=1 TIME=07:27 DATE=21.05.13

Which is converted to be HEX as following using compiler

0x5155414C4954593D312054494D453D30373A323720444154453D32312E3035

2E3133

The converted HEX is written in the user memory of the tag in command line of the reader test tool as shown in Figure 17.
<figure><img src="image_18.png" /><figcaption>Fig. 17. Information writing using RFID system</figcaption></figure>

## 4 CONCLUSIONS

The combination of Structured Light System, the data mining approach and RFID technology is tested in this paper. SLS is applicable in this proposed wheel assembly quality classification problem. According to the assembly requirements, the feature definition on the basis of the point cloud is suitable for similar product type. The feature vector extraction provides the SVM classifier available inputs and achieves 95.8% Correctly Classified Instances. Meanwhile, the RFID system successfully converts the quality inspection result to acceptable data format for the tag and writes the information in it. This step is able to improve the traceability of the product quality. Suppose multiple SLS inspection stations are assigned along the assembly line, the quality inspection results are saved in the RFID tag respectively. The earlier inspection result is available for the system before the product enters into the following processing station. The tag embedded information does not require the remote database access. The interruption due to the product quality is able to be avoided. In the following work, the middleware for integrating the three system will be developed.

## 5 REFERENCES

1. Barbero, B. R. & Ureta, E. S. (2011), "Comparative study of different digitization techniques and their accuracy", Computer-Aided Design, Vol. 43 No. 2, pp. 188-206.

2. Pernkopf, F. (2005), "3D surface acquisition and reconstruction for inspection of raw steel products", Computers in Industry, Vol. 56 No. 8-9, pp. 876-885.
3. Xu, J., Xi, N., Zhang, C., Shi, Q. & Gregory, J. (2011), "Real-time 3D shape inspection system of automotive parts based on structured light pattern", Optics & Laser Technology, Vol. 43 No. 1, pp. 1-8.

4. Skotheim, Ø. & Couweleers, F. (2004), "Structured light projection for accurate 3D shape determination". ICEM12-12th International Conference on Experimental Mechanics, Bari, Italy. 536-541.

5. Ravikumar, S., Ramachandran, K. I. & Sugumaran, V. (2011), "Machine learning approach for automated visual inspection of machine components", Expert Systems with Applications, Vol. 38 No. 4, pp. 3260-3266.

6. Lin, C.-S., Lin, C.-H., Chen, D.-C., Tien, C.-L. & Yeh, M.-S. (2011), "Measurement method of three-dimensional profiles of small lens with gratings projection and a flexible compensation system", Expert Systems with Applications, Vol. 38 No. 5, pp. 6232-6238.

7. Wang, K., (2007) Applying data mining to manufacturing: the nature and implications, Journal of Intelligent Manufacturing, Vol. 18, No. 4, pp.487-495.

8. Elisabeth, I., Zsolt, K., Péter, E. and László, M., 2006. The RFID Technology and Its Current Applications. Proceeding of The Modern Information Technology in the Innovation Process of the Industrial Enterprise-MITIP, pp.29-36.

9. Lewis, S., 2004. A Basic Introduction to RFID technology and Its Use in the Supply Chain. Laran RFID, White Paper.

10. Tellaeche, A., Arana, R., Ibarguren, A. & Martínez-Otzeta, J. M. (2010), "Automatic quality inspection of percussion cap mass production by means of 3D machine vision and machine learning techniques", Hybrid Artificial Intelligence Systems. San Sebastian: Springer.