# Research on Printing Defects Inspection of Solder Paste Images

Authors: Min Qi; Ting Yin; Gong Cheng; Yuelei Xu; Hongying Meng; Yi Wang; Shanshan Cui
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Solder paste printing is the first part of the surface mount process flow; its postprinting defect inspection is particularly important. In this paper, we focus on studying the printing defects inspection algorithm for solder paste on PCB (Printed Circuit Board) images. The work proposes a number of methods to enhance the defects inspection performance of solder paste printing: a regional multidirectional data fusion image interpolation method, which can achieve fast and high precision image interpolation; a method for detecting solder paste areas with better accuracy, efficiency, and robustness; an improved connected domain labeling method to reduce time complexity; and defects detection and types classification method, which extracts features and centroid of every solder paste region and completes the inspection by comparing with a standard image. The experiments show that the defects inspection algorithm can detect the most common types of defects with low time consumption, high inspection accuracy, and classification accuracy.


---

## Body

## 1. Introduction

Figure1(a) is an image of a PCB printed with solder paste and taken by a camera. Figure 1(b) is the corresponding standard image obtained by parsing the standard Gerber file of the PCB. As the image taken by the camera and the standard image do not match exactly in size and position, the registration operations for them need to be performed. One of the important steps in registration is image interpolation. After the registration is completed, the camera-taken image needs to be binarized to obtain a binary image with the solder paste area in white and the rest in black. In order to compare each solder paste region, a pass-through tagging operation is performed to assign a unique tag to each solder paste region in the binary image. Once the connected field is marked, the corresponding comparison inspection can be performed.Figure 1
PCB images. (a) The image taken using a camera. (b) The standard image.
(a)(b)Existing image registration methods are divided into two main categories: feature-based registration and grayscale-based registration [1]. Firstly, we extract the features of the image, find out the transformation model and parameters, and finally transform the image as a whole [2–4]. The latter is based on the similarity of pixel grayscale to complete the matching, without extracting other feature information, but when the image is rotated and scaled, it has a huge impact on the algorithm [5]. Image interpolation is an important step in the image registration process. Existing image interpolation methods are broadly classified as adaptive and nonadaptive. Nonadaptive algorithms apply a fixed pattern to each pixel, mainly nearest neighbor, bilinear, and bicubic image interpolation; adaptive techniques require consideration of image edge information, texture, and pixel intensity [6, 7].Image detection methods have their specific targets, and the corresponding detection methods are designed according to the application and image characteristics. For grayscale images, existing image detection methods are mainly based on two aspects: grayscale value discontinuity or grayscale value similarity. The former mainly relies on image edge detection and edge connectivity for implementation, such as improved Canny algorithm [8, 9], improved Prewitt operator [10], and Soble algorithm [11]. The latter, on the other hand, categorizes the image pixels according to a certain predefined rule, and typical algorithms include region generation and erosion algorithms and binarization algorithms.Existing defect classification and inspection algorithms can be broadly classified into two categories: reference comparison method and reference-free verification method. The former is to compare the board to be tested with the standard board for pixel comparison and determine the defects by the difference between the two images. This type of method can inspect more types of defects but requires a high standard image and the registration accuracy of the two images. The latter does not require a standard image but generally requires the extraction of a large amount of feature information and has a high algorithm complexity. Depending on the type of inspected defects, researchers have proposed different inspection schemes [12, 13].In this paper, the four aspects of image interpolation, solder paste area detection, connected domain labeling, and printing defect inspection are improved. The solution of this problem can enrich the connotation of machine vision in solder paste image and printing defect inspection and solve the urgent problem of SMT (Surface Mount Technology) production.


## 2. Methodology

### 2.1. Multidirection Data Fusion Image Interpolation Based on Regional Features

In order to meet the strict requirements of high-level industrial inspection in terms of accuracy and operation time, an image interpolation method with adaptive fusion calculation characteristics is proposed by referring to the idea of data fusion techniques, which is applied to edge regions in the images.Let the corresponding point of the point to be interpolated in the source image be pointP with coordinates (u,v), i and j are the largest integers not larger than u and v, respectively, and the grayscale values of the four pixel points in the 2 × 2 neighborhood of this corresponding point are f(i, j), f(i + 1, j), f(i, j + 1), and f(i + 1, j + 1), respectively.The mean grayscale valueE is(1)E=fi,j+fi+1,j+fi,j+1+fi+1,j+14.The grayscale variance Var is(2)Var=E−fi,j2+E−fi+1,j2+E−fi,j+12+E−fi+1,j+124According to the image characteristics, set the thresholdT. If Var < T, it is judged as a region of low grayscale variation, and the bilinear interpolation algorithm is used to interpolate the value. Otherwise, it is judged as an edge region with large grayscale variations, and interpolation is performed according to the following idea.First, the estimated values of pointP in the four interpolation directions of 45°, 135°, horizontal, and vertical are calculated and denoted as y1, y2, y3, and y4, respectively. The equations are as follows, where Sx,y is the distance of point P from the source image Euclidean distance in pixels from the pixel with coordinates (x, y) in the image, where x = i, i − 1, i + 1, i + 2; y = j, j − 1, j + 1, j + 2.(3)y1=12fi+2,j−1Si,j+12+fi,j+1Si+2,j−12Si,j+12+Si+2,j−12+fi+1,jSi−1,j+22+fi−1,j+2Si+1,j2Si−1,j+22+Si+1,j2,y2=12fi−1,j−1Si+1,j+12+fi+1,j+1Si−1,j−12Si+1,j+12+Si−1,j−12+fi,jSi+2,j+22+fi+2,j+2Si,j2Si+2,j+22+Si,j2,y3=12fi−1,jSi+1,j2+fi+1,jSi−1,j2Si+1,j2+Si−1,j2+fi,jSi+2,j2+fi+2,jSi,j2Si+2,j2+Si,j2,v−j<=0.5,12fi−1,j+1Si+1,j+12+fi+1,j+1Si−1,j+12Si+1,j+12+Si−1,j+12+fi,j+1Si+2,j+12+fi+2,j+1Si,j+12Si+2,j+12+Si,j+12,v−j>0.5,y4=12fi,j−1Si,j+12+fi,j+1Si,j−12Si,j+12+Si,j−12+fi,jSi,j+22+fi,j+2Si,j2Si,j+22+Si,j2,u−i<=0.5,12fi+1,j−1Si+1,j+12+fi+1,j+1Si+1,j−12Si+1,j+12+Si+1,j−12+fi+1,jSi+1,j+22+fi+1,j+2Si+1,j2Si+1,j+22+Si+1,j2,u−i>0.5.Next, the distances from the pointP to the four interpolation directions, denoted as r1, r2, r3, and r4, are calculated as follows:(4)r1=u−i+v−j−12,r2=u−i−v+j2,r3=v−j,j−v<=0.5,j−v+1,j−v>0.5,r4=u−i,u−i<=0.5,i+1−u,u−i>0.5.The normalized formula forr1, r2, r3, and r4 is as follows:(5)dk=rkr1+r2+r3+r4,k=1,2,3,4.Then, the mean values of the grayscale gradient in the four interpolation directions are calculated, which are noted asg1,g2,g3, and g4, respectively, with the following equations:(6)g1=fi+2,j−1−fi+1,j+fi+1,j−fi,j+1+fi,j+1−fi−1,j+23,g2=fi−1,j−1−fi,j+fi,j−fi+1,j+1+fi+1,j+1−fi+2,j+23,g3=fi−1,j−fi,j+fi,j−fi+1,j+fi+1,j−fi+2,j3,v−j<=0.5,fi−1,j+1−fi,j+1+fi,j+1−fi+1,j+1+fi+1,j+1−fi+2,j+13,v−j>0.5,g4=fi,j−1−fi,j+fi,j−fi,j+1+fi,j+1−fi,j+23,u−i<=0.5,fi+1,j−1−fi+1,j+fi+1,j−fi+1,j+1+fi+1,j+1−fi+1,j+23,u−i>0.5.Normalized tog1,g2,g3,g4, the formula is as follows:(7)tk=gkg1+g2+g3+g4,k=1,2,3,4.The method to determine the fusion coefficient is as follows:(8)ψk=e−λdk×e−1−λtk,k=1,2,3,4,where λ is a constant, 0 < λ < 1, used to adjust the interpolation distance and the weight value of the grayscale gradient in the fusion coefficient.The formula for normalizing Ψk is as follows:(9)ωk=ψkψ1+ψ2+ψ3+ψ4,k=1,2,3,4.After data fusion, the grayscale value of the pointP to be interpolated is as follows:(10)y=ω1y1+ω2y2+ω3y3+ω4y4.At this point, the interpolation of pointP is completed.


### 2.2. Solder Paste Area Detection

The PCB images used for inspection are taken with a monochrome industrial camera under red light irradiation and blue light irradiation, referred to as red monochrome images and blue monochrome images, respectively. The images taken under both color light irradiation are of the same size and correspond to the same area on the PCB. They are used to enhance the accuracy, efficiency, and robustness of the detection.Let the original imagef be a grayscale image with N rows and M columns, and the grayscale value of the pixel at the coordinates (x, y) is denoted as f(x, y). The average grayscale value Tave of the whole image is calculated by the following formula:(11)Tave=1MN∑y=1N∑x=1Mfx,y.Calculate the average grayscale valueTave1 between Tave and 255.(12)Tave1=1num∑x,y∈Sfx,y,where num is the total number of pixels in the image with grayscale values between Tave and 255 and S is the set of pixels with grayscale values between Tave and 255.A Gaussian model is created for the red monochrome image and blue monochrome image with pixel grayscale values in the intervalTave1 to 255, respectively, its grayscale mean value μ and standard deviation σ are found, and the binarization model is set as follows:(13)gx,y=255,ifμ−3σ<fx,y<μ+3σ,0,otherwise.Binarize the red monochrome images and the blue monochrome images according to the model shown in (14), respectively, and then according to formula (15) for the binary AND operation to obtain the binarized result image.(14)gox,y=255,ifgrx,y=255andgbx,y=255,0,otherwise,where gr (x, y) and gb (x, y) are the grayscale values of the pixel at (x, y) in the binarized red and blue monochrome images, respectively. go (x, y) is the grayscale value of the pixel at (x, y) in the resulting image. When grayscale values of pixels are 0, they are background pixels and will not be processed. If grayscale values of pixels are 255, the pixels are target pixels. The solder paste areas are obtained.


### 2.3. Connected Domain Labeling

The labeling of solder paste regions is a prerequisite for printing defect inspection of solder pastes. An improved method is proposed here to reduce the high time complexity in existing connected domain labeling algorithms.For a target pixel, the grayscale values of pixels in a specific neighborhood are detected according to the location of the target pixel, and the label is set according to specific rules. All the different location types of pixels in the binary image are shown in Figure2.Figure 2
Schematic diagram of different location types of pixels in binary images.Set a two-dimensional array label[y][x], which is used to store the label of the pixel at the location of (x, y) coordinates in the image, and set a one-dimensional array belong[label[y][x]], which is used to store the label of the connected domain to which the pixel at the location of (x, y) coordinates belongs, and set the coordinates of the current pixel as (x, y), in the following three cases:(1)
If the current pixel and its neighboring pixels both have different grayscale values, then label[y][x] is the maximum value of the existing initial concatenated domain label plus 1 and set belong[label[y][x]] = label[y][x](2)
If the current pixel has the same grayscale value as only one of its neighboring pixels, let the coordinates of the pixel with the same grayscale be (i, j) and then label[y][x] = label[j][i](3)
If the current pixel hasm (m ≥ 2) identical grayscale values with its neighboring pixels, take the pixel with the smallest initial connected domain label value among these m identical grayscale pixels, and note its coordinates as (i, j), then label[y][x] = label[j][i]At this point, after a complete scan of the image, the initially connected domain label and the belonging connected domain label are markedThen, set a one-dimensional array temp[T], where T is greater than or equal to the maximum of belong[label[y][x]], set a variable var, and initialize it to 1The initial concatenated domain labeled two-dimensional array label[y][x] is scanned in the following two cases:(1)
If label[y][x] = 0, the pixel at coordinates (x, y) is a background pixel and is not processed(2)
If label[y][x] ≠ 0, the pixel at coordinates (x, y) is the target pixel in the following two cases:(1)
If temp[belong[label[y][x]]] = 0, the connected domain label appears for the first time, set temp[belong[label[y][x]]] = var, label[y][x] = temp[belong[label[y][x]]], var = var + 1(2)
If temp[belong[label[y][x]]]≠0, then the label of the connected domain has already appeared, set label[y][x] = temp[belong[label[y][x]]]The label[y][x] stores the final label of the connected domain in the area where the pixel is located at coordinates (x, y).


### 2.4. Defects Detection and Types Classification

The types of defects studied here are Excessive Solder, Insufficient Solder, Solder Offset, Solder Bridge, Missing Printing, and Spattering Solder, as shown in Figure3. The paper studies the characteristics of different defects and compares the connected domains between the labeled result image and standard image to propose the inspection rules to find the defects and identify their types.Figure 3
Six types of defects: (a) Excessive Solder, (b) Insufficient Solder, (c) Solder Offset, (d) Solder Bridge, (e) Missing Printing, and (f) Spattering Solder.
(a)(b)(c)(d)(e)(f)Calculate the area and centroid of the solder paste areaA with the following equation:(15)S=∑x,y∈A1,x0=1N∑x,y∈Ax,y0=1N∑x,y∈Ay,where (x, y) are the coordinates of the pixels in the solder area, S is the area of the solder area, N is the total number of pixels in the solder area, and (x0, y0) is the centroid of the solder area. The connected domain with an area less than a certain threshold T0 is treated as noise and is not used for defect inspection.Step 1.
By means of the centroid coordinates, find the label of the corresponding connected domain in the standard image of all the solder-paste connected domains in the resulting image.
If the label is 0, it corresponds to the background area and is not processed. If the label is not 0, the following two cases can be distinguished:(1)
If there are duplicate labels, it means that Solder Bridge has occurred.(2)
For a nonrepeat label, it indicates that Solder Bridge has not occurred and continues with other defect type inspections.T1 and T2 are preset thresholds, T1 < 1, T2 > 1. Take one out of all the nonrepeated labels, suppose its connected component area is Sf, and the corresponding connected component area in the standard image is Sb.(1)
IfSf/Sb<T1, the connected domain is determined to be Insufficient Solder.(2)
IfSf/Sb>T2, the connected domain is determined to be Excessive Solder.(3)
IfT1≤Sf/Sb≤T2, then it means there are no Excessive Solder and Insufficient Solder and continues to Solder Offset inspection.Assuming that the centroid coordinates of the connected domain in the standard image are (xb, yb), the centroid coordinates of the corresponding connected domain in the resultant image are (xf, yf), T3 is a predetermined threshold, and the centroid position deviation d is expressed as follows:(16)d=xb−xf2+yb−yf2.①
Ifd ≤ T3, the connected domains are judged to be normal.②
Ifd > T3, the connected domains are judged as Solder Offset.Step 2.
Through centroid coordinates, find out the connected domain labels in the standard image corresponding to all the unprocessed solder paste connected domains in the resulting image, and record all the labels:(1)
If the label is 0, the connected area of solder paste is judged as Spattering Solder.(2)
If there is a nonzero repeated label in the label record, it means that multiple solder paste connected domains in the resulting image correspond to one solder paste connected domain in the standard image, and data merging is required. After data merging, the operation is similar to Step1.(3)
For nonzero and nonrepetitive labels, there is no need to merge data. The operation is exactly the same as the processing of nonrepetitive labels in Step1.Step 3.
After the processing of the above steps, if there are unprocessed connected domains, it indicates that a Missing Printing has occurred.


## 2.1. Multidirection Data Fusion Image Interpolation Based on Regional Features

In order to meet the strict requirements of high-level industrial inspection in terms of accuracy and operation time, an image interpolation method with adaptive fusion calculation characteristics is proposed by referring to the idea of data fusion techniques, which is applied to edge regions in the images.Let the corresponding point of the point to be interpolated in the source image be pointP with coordinates (u,v), i and j are the largest integers not larger than u and v, respectively, and the grayscale values of the four pixel points in the 2 × 2 neighborhood of this corresponding point are f(i, j), f(i + 1, j), f(i, j + 1), and f(i + 1, j + 1), respectively.The mean grayscale valueE is(1)E=fi,j+fi+1,j+fi,j+1+fi+1,j+14.The grayscale variance Var is(2)Var=E−fi,j2+E−fi+1,j2+E−fi,j+12+E−fi+1,j+124According to the image characteristics, set the thresholdT. If Var < T, it is judged as a region of low grayscale variation, and the bilinear interpolation algorithm is used to interpolate the value. Otherwise, it is judged as an edge region with large grayscale variations, and interpolation is performed according to the following idea.First, the estimated values of pointP in the four interpolation directions of 45°, 135°, horizontal, and vertical are calculated and denoted as y1, y2, y3, and y4, respectively. The equations are as follows, where Sx,y is the distance of point P from the source image Euclidean distance in pixels from the pixel with coordinates (x, y) in the image, where x = i, i − 1, i + 1, i + 2; y = j, j − 1, j + 1, j + 2.(3)y1=12fi+2,j−1Si,j+12+fi,j+1Si+2,j−12Si,j+12+Si+2,j−12+fi+1,jSi−1,j+22+fi−1,j+2Si+1,j2Si−1,j+22+Si+1,j2,y2=12fi−1,j−1Si+1,j+12+fi+1,j+1Si−1,j−12Si+1,j+12+Si−1,j−12+fi,jSi+2,j+22+fi+2,j+2Si,j2Si+2,j+22+Si,j2,y3=12fi−1,jSi+1,j2+fi+1,jSi−1,j2Si+1,j2+Si−1,j2+fi,jSi+2,j2+fi+2,jSi,j2Si+2,j2+Si,j2,v−j<=0.5,12fi−1,j+1Si+1,j+12+fi+1,j+1Si−1,j+12Si+1,j+12+Si−1,j+12+fi,j+1Si+2,j+12+fi+2,j+1Si,j+12Si+2,j+12+Si,j+12,v−j>0.5,y4=12fi,j−1Si,j+12+fi,j+1Si,j−12Si,j+12+Si,j−12+fi,jSi,j+22+fi,j+2Si,j2Si,j+22+Si,j2,u−i<=0.5,12fi+1,j−1Si+1,j+12+fi+1,j+1Si+1,j−12Si+1,j+12+Si+1,j−12+fi+1,jSi+1,j+22+fi+1,j+2Si+1,j2Si+1,j+22+Si+1,j2,u−i>0.5.Next, the distances from the pointP to the four interpolation directions, denoted as r1, r2, r3, and r4, are calculated as follows:(4)r1=u−i+v−j−12,r2=u−i−v+j2,r3=v−j,j−v<=0.5,j−v+1,j−v>0.5,r4=u−i,u−i<=0.5,i+1−u,u−i>0.5.The normalized formula forr1, r2, r3, and r4 is as follows:(5)dk=rkr1+r2+r3+r4,k=1,2,3,4.Then, the mean values of the grayscale gradient in the four interpolation directions are calculated, which are noted asg1,g2,g3, and g4, respectively, with the following equations:(6)g1=fi+2,j−1−fi+1,j+fi+1,j−fi,j+1+fi,j+1−fi−1,j+23,g2=fi−1,j−1−fi,j+fi,j−fi+1,j+1+fi+1,j+1−fi+2,j+23,g3=fi−1,j−fi,j+fi,j−fi+1,j+fi+1,j−fi+2,j3,v−j<=0.5,fi−1,j+1−fi,j+1+fi,j+1−fi+1,j+1+fi+1,j+1−fi+2,j+13,v−j>0.5,g4=fi,j−1−fi,j+fi,j−fi,j+1+fi,j+1−fi,j+23,u−i<=0.5,fi+1,j−1−fi+1,j+fi+1,j−fi+1,j+1+fi+1,j+1−fi+1,j+23,u−i>0.5.Normalized tog1,g2,g3,g4, the formula is as follows:(7)tk=gkg1+g2+g3+g4,k=1,2,3,4.The method to determine the fusion coefficient is as follows:(8)ψk=e−λdk×e−1−λtk,k=1,2,3,4,where λ is a constant, 0 < λ < 1, used to adjust the interpolation distance and the weight value of the grayscale gradient in the fusion coefficient.The formula for normalizing Ψk is as follows:(9)ωk=ψkψ1+ψ2+ψ3+ψ4,k=1,2,3,4.After data fusion, the grayscale value of the pointP to be interpolated is as follows:(10)y=ω1y1+ω2y2+ω3y3+ω4y4.At this point, the interpolation of pointP is completed.


## 2.2. Solder Paste Area Detection

The PCB images used for inspection are taken with a monochrome industrial camera under red light irradiation and blue light irradiation, referred to as red monochrome images and blue monochrome images, respectively. The images taken under both color light irradiation are of the same size and correspond to the same area on the PCB. They are used to enhance the accuracy, efficiency, and robustness of the detection.Let the original imagef be a grayscale image with N rows and M columns, and the grayscale value of the pixel at the coordinates (x, y) is denoted as f(x, y). The average grayscale value Tave of the whole image is calculated by the following formula:(11)Tave=1MN∑y=1N∑x=1Mfx,y.Calculate the average grayscale valueTave1 between Tave and 255.(12)Tave1=1num∑x,y∈Sfx,y,where num is the total number of pixels in the image with grayscale values between Tave and 255 and S is the set of pixels with grayscale values between Tave and 255.A Gaussian model is created for the red monochrome image and blue monochrome image with pixel grayscale values in the intervalTave1 to 255, respectively, its grayscale mean value μ and standard deviation σ are found, and the binarization model is set as follows:(13)gx,y=255,ifμ−3σ<fx,y<μ+3σ,0,otherwise.Binarize the red monochrome images and the blue monochrome images according to the model shown in (14), respectively, and then according to formula (15) for the binary AND operation to obtain the binarized result image.(14)gox,y=255,ifgrx,y=255andgbx,y=255,0,otherwise,where gr (x, y) and gb (x, y) are the grayscale values of the pixel at (x, y) in the binarized red and blue monochrome images, respectively. go (x, y) is the grayscale value of the pixel at (x, y) in the resulting image. When grayscale values of pixels are 0, they are background pixels and will not be processed. If grayscale values of pixels are 255, the pixels are target pixels. The solder paste areas are obtained.


## 2.3. Connected Domain Labeling

The labeling of solder paste regions is a prerequisite for printing defect inspection of solder pastes. An improved method is proposed here to reduce the high time complexity in existing connected domain labeling algorithms.For a target pixel, the grayscale values of pixels in a specific neighborhood are detected according to the location of the target pixel, and the label is set according to specific rules. All the different location types of pixels in the binary image are shown in Figure2.Figure 2
Schematic diagram of different location types of pixels in binary images.Set a two-dimensional array label[y][x], which is used to store the label of the pixel at the location of (x, y) coordinates in the image, and set a one-dimensional array belong[label[y][x]], which is used to store the label of the connected domain to which the pixel at the location of (x, y) coordinates belongs, and set the coordinates of the current pixel as (x, y), in the following three cases:(1)
If the current pixel and its neighboring pixels both have different grayscale values, then label[y][x] is the maximum value of the existing initial concatenated domain label plus 1 and set belong[label[y][x]] = label[y][x](2)
If the current pixel has the same grayscale value as only one of its neighboring pixels, let the coordinates of the pixel with the same grayscale be (i, j) and then label[y][x] = label[j][i](3)
If the current pixel hasm (m ≥ 2) identical grayscale values with its neighboring pixels, take the pixel with the smallest initial connected domain label value among these m identical grayscale pixels, and note its coordinates as (i, j), then label[y][x] = label[j][i]At this point, after a complete scan of the image, the initially connected domain label and the belonging connected domain label are markedThen, set a one-dimensional array temp[T], where T is greater than or equal to the maximum of belong[label[y][x]], set a variable var, and initialize it to 1The initial concatenated domain labeled two-dimensional array label[y][x] is scanned in the following two cases:(1)
If label[y][x] = 0, the pixel at coordinates (x, y) is a background pixel and is not processed(2)
If label[y][x] ≠ 0, the pixel at coordinates (x, y) is the target pixel in the following two cases:(1)
If temp[belong[label[y][x]]] = 0, the connected domain label appears for the first time, set temp[belong[label[y][x]]] = var, label[y][x] = temp[belong[label[y][x]]], var = var + 1(2)
If temp[belong[label[y][x]]]≠0, then the label of the connected domain has already appeared, set label[y][x] = temp[belong[label[y][x]]]The label[y][x] stores the final label of the connected domain in the area where the pixel is located at coordinates (x, y).


## 2.4. Defects Detection and Types Classification

The types of defects studied here are Excessive Solder, Insufficient Solder, Solder Offset, Solder Bridge, Missing Printing, and Spattering Solder, as shown in Figure3. The paper studies the characteristics of different defects and compares the connected domains between the labeled result image and standard image to propose the inspection rules to find the defects and identify their types.Figure 3
Six types of defects: (a) Excessive Solder, (b) Insufficient Solder, (c) Solder Offset, (d) Solder Bridge, (e) Missing Printing, and (f) Spattering Solder.
(a)(b)(c)(d)(e)(f)Calculate the area and centroid of the solder paste areaA with the following equation:(15)S=∑x,y∈A1,x0=1N∑x,y∈Ax,y0=1N∑x,y∈Ay,where (x, y) are the coordinates of the pixels in the solder area, S is the area of the solder area, N is the total number of pixels in the solder area, and (x0, y0) is the centroid of the solder area. The connected domain with an area less than a certain threshold T0 is treated as noise and is not used for defect inspection.Step 1.
By means of the centroid coordinates, find the label of the corresponding connected domain in the standard image of all the solder-paste connected domains in the resulting image.
If the label is 0, it corresponds to the background area and is not processed. If the label is not 0, the following two cases can be distinguished:(1)
If there are duplicate labels, it means that Solder Bridge has occurred.(2)
For a nonrepeat label, it indicates that Solder Bridge has not occurred and continues with other defect type inspections.T1 and T2 are preset thresholds, T1 < 1, T2 > 1. Take one out of all the nonrepeated labels, suppose its connected component area is Sf, and the corresponding connected component area in the standard image is Sb.(1)
IfSf/Sb<T1, the connected domain is determined to be Insufficient Solder.(2)
IfSf/Sb>T2, the connected domain is determined to be Excessive Solder.(3)
IfT1≤Sf/Sb≤T2, then it means there are no Excessive Solder and Insufficient Solder and continues to Solder Offset inspection.Assuming that the centroid coordinates of the connected domain in the standard image are (xb, yb), the centroid coordinates of the corresponding connected domain in the resultant image are (xf, yf), T3 is a predetermined threshold, and the centroid position deviation d is expressed as follows:(16)d=xb−xf2+yb−yf2.①
Ifd ≤ T3, the connected domains are judged to be normal.②
Ifd > T3, the connected domains are judged as Solder Offset.Step 2.
Through centroid coordinates, find out the connected domain labels in the standard image corresponding to all the unprocessed solder paste connected domains in the resulting image, and record all the labels:(1)
If the label is 0, the connected area of solder paste is judged as Spattering Solder.(2)
If there is a nonzero repeated label in the label record, it means that multiple solder paste connected domains in the resulting image correspond to one solder paste connected domain in the standard image, and data merging is required. After data merging, the operation is similar to Step1.(3)
For nonzero and nonrepetitive labels, there is no need to merge data. The operation is exactly the same as the processing of nonrepetitive labels in Step1.Step 3.
After the processing of the above steps, if there are unprocessed connected domains, it indicates that a Missing Printing has occurred.


## 3. Results and Discussion

### 3.1. Image Interpolation Experiment

The original image is shown in Figure4(a), which is sampled three times to reduce its length and width to 50%, 70%, and 30% of the original one, respectively. Then, the bilinear, bicubic, and the proposed methods in this paper are interpolated, respectively, and due to space limitation, only 30% of the resulting images are given as shown in Figures 4(b), 4(c), and 4(d).Figure 4
Interpolated image of the result of reducing the length and width to 30%. (a) Original image. (b) Bilinear interpolation. (c) Bicubic interpolation. (d) Method proposed in this paper.
(a)(b)(c)(d)Calculate the peak signal-to-noise ratio with the following equation:(17)PSNR=10log1025521/MN∑x=1M∑y=1Nfm,n−f^m,n2,where M and N are the horizontal and vertical sampling points of the image, respectively, f (m, n) is the original image, and f^ (m, n) is the processed image. The larger the PSNR value, the higher the accuracy of the interpolation algorithm. Due to space limitations, only the experimental results are given when the image aspect is reduced to 30%, as shown in Table 1. From the table, we can see that the algorithm in this paper has the highest PSNR, indicating that the algorithm has high interpolation accuracy.Table 1
PSNR of interpolation results for each algorithm (unit: decibels).
AlgorithmImage 1Image 2Image 3Image 4Image 5Bilinear24.04626.52424.66222.26433.798Bicubic26.62127.97326.73322.31336.131This paper27.32828.32426.99823.10337.323


### 3.2. Solder Paste Area Detection Experiment

Three sets of red and blue monochrome images were taken under normal, strong, and weak illumination, respectively, and the binarization method proposed in this paper was used to detect the solder paste area. Here, the detection and result images under weak illumination are given as an example which are shown in Figure5, where Figures 5(a) and 5(b) are the red and blue monochrome images to be detected, respectively, and Figure 5(c) is the detection result image.Figure 5
Original and test resulting images under weak illumination conditions. (a) Red monochrome image. (b) Blue monochrome image. (c) The resulting image.
(a)(b)(c)Ten types of solder paste areas were used for detection, as shown in Table2; due to space limitations, only the results of solder paste detection under weak illumination are given.Table 2
The results of solder paste detection under weak illumination.
TypeNumber of connected domainsPixel-number per connected domainAverage error of pixel-numberER (%)Round 11179040. 2Round 266112840. 3Round 35561440. 6Round 49625431. 2Rectangle 1161880120. 6Rectangle 220832121. 4Rectangle 3871067121. 1Rectangle 4267981. 2Square 14025683. 1Square 212014442. 7The error rate for each type of solder paste area detection is calculated according to the following formula:(18)ER=∑i=1Nni−nio∑i=1Nnio,where N denotes the number of solder areas of the type, ni denotes the number of pixels detected in the ith solder area of the type, and nio denotes the true number of pixels in the ith solder area of the type. The smaller the ER, the higher the detection accuracy. From the table, it can be seen that the adaptive threshold binarization method proposed in this paper has a low ER and can achieve high accuracy detection of solder paste areas.


### 3.3. Connected Domain Labeling Experiment

Six images were labeled with connected domains using the algorithm of this paper, the region growth method [14], and the tour-based coding method [15], respectively. They were executed 10 times consecutively, and the average time was recorded as shown in Table 3. As can be seen from the table, compared with the other two algorithms, the algorithm of this paper has the advantage of a shorter running time.Table 3
Running times for each image under different algorithms (unit: second).
AlgorithmImage 1Image 2Image 3Image 4Image 5Image 6Reference [14]0.0590.3380.1610.1470.1680.166Reference [15]0.2620.5090.2270.2210.3150.321This paper0.0310.1060.0420.0410.0630.063One of the six images is shown in Figure6(a). The image labeled using the improved algorithm in this paper is shown in Figure 6(b). Each connected domain in the resulting image is assigned a different grayscale value to show the distinction in labeling.Figure 6
The experiment of the improved algorithm of this paper. (a) The experiment image. (b) The labeled image.
(a)(b)


### 3.4. Defects Detection and Types Classification Experiment

Figure7(a) shows the image to be inspected, and Figure 7(b) shows the standard image obtained by parsing the Gerber file. The inspection results are shown in Figure 7(c), and the meanings of each letter are as follows: D for Excessive Solder, S for Insufficient Solder, P for Solder Offset, Q for Solder Bridge, L for Missing Printing, and J for Spattering Solder.Figure 7
(a) Solder paste image to be inspected. (b) Standard image. (c) Inspection result.
(a)(b)(c)Using the results of manual visual inspection as the standard, the inspection accuracy, classification accuracy, and false alarm rate of the defect inspection method studied in this paper were calculated, and the results are shown in Table4. As can be seen from the table, the proposed method can inspect six types of defects with a high inspection and classification accuracy and low false alarm rate.Table 4
The inspection results for different types of defects.
Type of defectExcessive Solder (%)Insufficient Solder (%)Solder Offset (%)Solder BridgeMissing Printing (%)Spattering SolderAverage (%)Inspection accuracy10010010099.17%100100%99.86Classification accuracy10096.7710096.67%97.78100%98.54False alarm rate0.030.040.0500.0200.14


## 3.1. Image Interpolation Experiment

The original image is shown in Figure4(a), which is sampled three times to reduce its length and width to 50%, 70%, and 30% of the original one, respectively. Then, the bilinear, bicubic, and the proposed methods in this paper are interpolated, respectively, and due to space limitation, only 30% of the resulting images are given as shown in Figures 4(b), 4(c), and 4(d).Figure 4
Interpolated image of the result of reducing the length and width to 30%. (a) Original image. (b) Bilinear interpolation. (c) Bicubic interpolation. (d) Method proposed in this paper.
(a)(b)(c)(d)Calculate the peak signal-to-noise ratio with the following equation:(17)PSNR=10log1025521/MN∑x=1M∑y=1Nfm,n−f^m,n2,where M and N are the horizontal and vertical sampling points of the image, respectively, f (m, n) is the original image, and f^ (m, n) is the processed image. The larger the PSNR value, the higher the accuracy of the interpolation algorithm. Due to space limitations, only the experimental results are given when the image aspect is reduced to 30%, as shown in Table 1. From the table, we can see that the algorithm in this paper has the highest PSNR, indicating that the algorithm has high interpolation accuracy.Table 1
PSNR of interpolation results for each algorithm (unit: decibels).
AlgorithmImage 1Image 2Image 3Image 4Image 5Bilinear24.04626.52424.66222.26433.798Bicubic26.62127.97326.73322.31336.131This paper27.32828.32426.99823.10337.323


## 3.2. Solder Paste Area Detection Experiment

Three sets of red and blue monochrome images were taken under normal, strong, and weak illumination, respectively, and the binarization method proposed in this paper was used to detect the solder paste area. Here, the detection and result images under weak illumination are given as an example which are shown in Figure5, where Figures 5(a) and 5(b) are the red and blue monochrome images to be detected, respectively, and Figure 5(c) is the detection result image.Figure 5
Original and test resulting images under weak illumination conditions. (a) Red monochrome image. (b) Blue monochrome image. (c) The resulting image.
(a)(b)(c)Ten types of solder paste areas were used for detection, as shown in Table2; due to space limitations, only the results of solder paste detection under weak illumination are given.Table 2
The results of solder paste detection under weak illumination.
TypeNumber of connected domainsPixel-number per connected domainAverage error of pixel-numberER (%)Round 11179040. 2Round 266112840. 3Round 35561440. 6Round 49625431. 2Rectangle 1161880120. 6Rectangle 220832121. 4Rectangle 3871067121. 1Rectangle 4267981. 2Square 14025683. 1Square 212014442. 7The error rate for each type of solder paste area detection is calculated according to the following formula:(18)ER=∑i=1Nni−nio∑i=1Nnio,where N denotes the number of solder areas of the type, ni denotes the number of pixels detected in the ith solder area of the type, and nio denotes the true number of pixels in the ith solder area of the type. The smaller the ER, the higher the detection accuracy. From the table, it can be seen that the adaptive threshold binarization method proposed in this paper has a low ER and can achieve high accuracy detection of solder paste areas.


## 3.3. Connected Domain Labeling Experiment

Six images were labeled with connected domains using the algorithm of this paper, the region growth method [14], and the tour-based coding method [15], respectively. They were executed 10 times consecutively, and the average time was recorded as shown in Table 3. As can be seen from the table, compared with the other two algorithms, the algorithm of this paper has the advantage of a shorter running time.Table 3
Running times for each image under different algorithms (unit: second).
AlgorithmImage 1Image 2Image 3Image 4Image 5Image 6Reference [14]0.0590.3380.1610.1470.1680.166Reference [15]0.2620.5090.2270.2210.3150.321This paper0.0310.1060.0420.0410.0630.063One of the six images is shown in Figure6(a). The image labeled using the improved algorithm in this paper is shown in Figure 6(b). Each connected domain in the resulting image is assigned a different grayscale value to show the distinction in labeling.Figure 6
The experiment of the improved algorithm of this paper. (a) The experiment image. (b) The labeled image.
(a)(b)


## 3.4. Defects Detection and Types Classification Experiment

Figure7(a) shows the image to be inspected, and Figure 7(b) shows the standard image obtained by parsing the Gerber file. The inspection results are shown in Figure 7(c), and the meanings of each letter are as follows: D for Excessive Solder, S for Insufficient Solder, P for Solder Offset, Q for Solder Bridge, L for Missing Printing, and J for Spattering Solder.Figure 7
(a) Solder paste image to be inspected. (b) Standard image. (c) Inspection result.
(a)(b)(c)Using the results of manual visual inspection as the standard, the inspection accuracy, classification accuracy, and false alarm rate of the defect inspection method studied in this paper were calculated, and the results are shown in Table4. As can be seen from the table, the proposed method can inspect six types of defects with a high inspection and classification accuracy and low false alarm rate.Table 4
The inspection results for different types of defects.
Type of defectExcessive Solder (%)Insufficient Solder (%)Solder Offset (%)Solder BridgeMissing Printing (%)Spattering SolderAverage (%)Inspection accuracy10010010099.17%100100%99.86Classification accuracy10096.7710096.67%97.78100%98.54False alarm rate0.030.040.0500.0200.14


## 4. Conclusion

In this paper, the techniques related to the inspection of solder paste images and printing defects are studied and some substantial results are achieved. The proposed regional multidirectional data fusion image interpolation method not only has a fast interpolation speed but also has a high interpolation accuracy, can protect the edge details of the image well, and is applicable to any level of interpolation transformation, and the interpolation method is applied to PCB image registration, which improves the speed and accuracy of registration. The proposed solder paste region detection scheme can accurately identify solder paste regions and can adapt well to changes in light source brightness with low time complexity. The proposed solder paste printing defect inspection method is able to detect six types of defects with low time consumption and has good performances in inspection and classification. However, the research work in this paper is aimed at external defects inspection and cannot deal with internal cavity defects detection. Moreover, the method is studied for the inspection work in two-dimensional space, which cannot detect those defects when needing height or depth information, for example, scars inspection, pits inspection, and so on. Three-dimensional defect inspection is worth studying, where highly accurate height or depth detection is the key technology.



---

*Source: 8651956-2022-03-25.xml*