# SMT Assembly Inspection Using Dual-Stream Convolutional Networks and Two Solder Regions
## 6 Convolutional Neural Network (CNN) Structure
### Dual-Stream CNN  
Dual-stream CNN generates a classification model using two solder images (\(S_{1}\) and \(S_{2}\)), which consist of the color channel image (\({}^{S_{1}}R\), \({}^{S_{1}}G\), and \({}^{S_{1}}B\); and \({}^{S_{2}}R\), \({}^{S_{2}}G\), and \({}^{S_{2}}B\)). Figure 11 shows two different structure of dual-stream CNN. Each input datum has a size of \(32\times 32\) pixels. The output of the final stage of the dual-stream CNN is the probability value for each defect, similar to that of the single-stream CNN.  
Figure 11: _Cont._  
The feature value is the output value extracted by calculating the weight and bias value in a layer included in a deep-learning network. The feature value \(\theta_{mn}\) for the input data \(x\in\mathbb{R}^{M\times N}\) in the CNN is defined by Equation (12).  
\[\theta_{mn}=q\left(\sum_{k_{1}=0}^{K_{1}-1}\sum_{k_{2}=0}^{K_{2}-1}w_{k}[k_{1}, k_{2}]x[m+k_{1},n+k_{2}]\right) \tag{12}\]  
where \(w_{k}\) is the weight of the kernel, \(K_{1},\ K_{2}\) are the kernel sizes, \(\sigma\) is the activation function, and \(m\) and \(n\) are the location of the input data used to calculate the feature map. The feature matrix consists of output values of the deep-learning network in a 2D image. Let the kernel stride be \(S\), and padding be \(P\). The feature matrix \(f\in\mathbb{R}^{U\times V}(U,V\in\mathbb{N})\) is then defined as follows:  
\[f=\left[\begin{array}{cccc}\theta_{11}&\cdots&\theta_{1V}\\ \vdots&\ddots&\vdots\\ \theta_{U1}&\cdots&\theta_{UV}\end{array}\right] \tag{13}\]  
\[U=floor\left(\frac{M-K_{1}+2P}{S}+1\right) \tag{14}\]  
\[V=floor\left(\frac{N-K_{2}+2P}{S}+1\right) \tag{15}\]  
In general, the deep-learning network layer consists of a multichannel kernel. Therefore, the feature matrix is output in multiple channels called the feature map. Let the number of layers be \(L\), the number of kernels be \(C\), and the output feature matrix for each kernel be \(f_{c}\in\mathbb{R}^{U\times V}(c=1,\ 2,\cdots,C)\) make up the feature map of the \(l^{th}\) output layer \(F_{l}\in\mathbb{R}^{U\times V\times C}(l=1,2,\cdots,L)\).  
\[F_{l}=\left[f_{1}f_{2}\cdots\ f_{C}\right] \tag{16}\]  
Consequently, merging is the process of combining different feature maps into one map. To calculate for the merge result \({}^{S}F\in\mathbb{R}^{U\times V\times S}\), let the number of kernels of the \(l^{th}\) output layer be \(C_{l}\), and the feature map output from output layers with \(C_{l}\) kernel channels be \({}^{C_{l}}F_{l}\in\mathbb{R}^{U\times V\times C_{l}}\).  
\[{}^{S}F=\left[{}^{C_{1}}F_{1}\ {}^{C_{2}}F_{2}\ \cdots\ {}^{C_{l}}F_{L}\right] \tag{17}\]  
Figure 11: Structures for dual-stream CNNs. (**a**) Early-merge dual-stream CNN. The merge is located at the convolution layer of the network. (**b**) Late-merge dual-stream CNN. The merge is located at the front of the fully connected layer.  
where,  
\[S=\sum_{l=1}^{L}C_{l} \tag{17}\]  
By merging feature maps from different layers, multiple low-dimensional feature maps can be combined into a single high-dimensional feature map. This high-dimensional feature map can them be used to extract the global features directly related to defect classification, thereby improving the classification accuracy of the CNN model. Merging also simplifies the CNN layer by unifying feature maps, as dual-stream CNNs have a multi-stream structure with multiple layer branches. However, after merging, feature maps can be integrated into a single-stream structure with a single layer branch. Merging also reduces the weight of the CNN model by converting the CNN model from a two-stream structure to a single-stream structure.