# SMT Assembly Inspection Using Dual-Stream Convolutional Networks and Two Solder Regions
## 5 Solder Region Extraction
### Solder Region Extraction  
First, the chip component image \(I\), an RGB image, is converted into a grayscale image \({}^{I}V\in\mathbb{R}^{\text{M}\times N}\).  
\[{}^{I}V=\left[\begin{array}{ccc}v_{11}&\cdots&v_{1N}\\ \vdots&\ddots&\vdots\\ v_{M1}&\cdots&v_{MN}\end{array}\right] \tag{6}\]  
where,  
\[v_{mn}=\frac{r_{mn}+g_{mn}+b_{mn}}{3} \tag{7}\]  
The horizontal projection \(p_{H}(n)(n=1,\cdots,N)\) for \({}^{I}V\) is then obtained using Equation (8).  
\[p_{H}(n)=\sum_{m=1}^{M}v_{mn} \tag{8}\]  
Figure 8a,b show the chip component image and horizontal projection of the chip component image, respectively. As the solder region of the chip component is brighter than the package region, the horizontal projection is divided into two regions: dark (middle) and bright (top and bottom).  
From the horizontal projection, the split point (\(n_{TH}\)) for dividing the chip component region into two regions is calculated. The split point is the coordinate of the \(n\)-axis (\(y\)-axis) that divides the chip component into two regions. As shown in Figure 6, normal chip components have similar vertical projection values both above and below. However, since defects such as under-solder, over-solder, and Tombstone have different color distributions in the upper and lower solder regions, it is necessary to divide the chip component into two regions for accurate solder region extraction. In the horizontal  
Figure 8: Solder region extraction of SMT chip image where ‘px’ refers to pixels: (**a**) chip component image, (**b**) gray scale image showing horizontal split of chip component, (**c**) vertical and horizontal projections, and (**d**) result of solder region extraction.  
projection shown in Figure 8b, the chip component is divided into two regions. We then used the Otsu method [27] on the horizontal projection to obtain the threshold value and used it as the split point  
Based on the split point, the horizontal projection \(p_{H}(n)\) is divided into \(p_{H}^{1}(n)\) and \(p_{H}^{2}(n)\).  
\[\begin{array}{c}p_{H}^{1}(n)=\left\{p_{H}(n)\right\}1\ \leq n\ \leq\ n_{TH}-1 \right\}\\ p_{H}^{2}(n)=\left\{p_{H}(n)\right\}n_{TH}\ \leq n\ \leq\ N\end{array} \tag{9}\]  
The vertical projection \(p_{V}^{1}(m)\) and \(p_{V}^{2}(m)\ (m=1,\cdots,M)\) for \({}^{I}V\) are also obtained.  
\[\begin{array}{c}p_{V}^{1}(m)=\sum\limits_{n=1}^{n_{TH}-1}v_{mm}\\ p_{V}^{2}(m)=\sum\limits_{n=n_{TH}}^{N}v_{mm}\end{array} \tag{10}\]  
Figure 8c shows the horizontal and vertical projections of the chip component image divided into dark and bright sections. In the figure, the region where the brightness value changes abruptly in each projection denotes the boundary of the solder region. Let the minimum and maximum values of the horizontal projection be \(n_{min}^{i},\ n_{max}^{i}\ (i=1,2)\), respectively; and the minimum and maximum values of the vertical projection be \(m_{min}^{i},\ m_{max}^{i}\ (i=1,2)\), respectively. The solder region \(S^{i}\in\mathbb{R}^{P\times 3\text{Q}}(i=1,2)\) is then defined by Equation (11).  
\[S^{i}=\left\{(m,n)\right\}m_{min}^{i}\leq m\ \leq\ m_{max}^{i},\ \ \ n_{min}^{i}\leq n\leq\ n_{max}^{i} \tag{11}\]  
Figure 8d shows the solder region image extracted by the minimum and maximum values of the horizontal and vertical projections.