# Mechanical Assembly Monitoring Method Based on Semi-Supervised Semantic Segmentation
## 3 Overall Framework
### Network Training  
Firstly, to train the discriminator network, it is necessary to minimize the spatial cross-entropy loss \(L_{D}\). The definition of \(L_{D}\) is as follows:  
\[L_{D}=\sum_{h,w}(1-y_{n})\log(1-D(G(X_{\mathbf{n}}))^{(h,w)})+y_{n}\log(D(Y_{ \mathbf{n}})^{(h,w)}), \tag{1}\]  
When the input of the discriminator network is the output of the segmentation network, \(y_{n}=0\); when the input of the discriminator network is the one-hot encoded ground truth, \(y_{n}=1\). Additionally, \(D(\bullet)^{(h,w)}\) refers to the confidence map at location (\(h\),\(w\)).  
To train the semantic segmentation network, it is necessary to minimize the multitask loss function. The definition of multitask loss is as follows:  
\[L_{\mathbf{seg}}=L_{\mathbf{ce}}+\lambda_{\mathbf{adv}}L_{\mathbf{adv}}+ \lambda_{\mathbf{semi}}L_{\mathbf{semi}}, \tag{2}\]  
where \(\lambda_{\mathbf{adv}}\) and \(\lambda_{\mathbf{semi}}\) are two weight coefficients and also two hyperparameters for minimizing the loss of multitasking. In this study, the semi-supervised training is divided into two steps: labeled image training and unlabeled image training. Thus, the use of its loss function also has two parts.  
When the semantic segmentation network is trained on images with labels, its cross-entropy loss \(L_{\mathbf{ce}}\) is defined as follows:  
\[L_{\mathbf{ce}}=\sum_{h,w}\sum_{e\in C}Y_{n}^{(h,w,c)}\log(G(X_{\mathbf{n}})^{ (h,w,c)}), \tag{3}\]  
where one-hot encoding is applied to convert the discrete real label information mapping into a \(c\)-channel probability mapping. Additionally, a fully convolutional discriminator network is used for adversarial learning. Its adversarial loss \(L_{\mathbf{adv}}\) is defined as follows:  
\[L_{\mathbf{adv}}=\sum_{h,w}\log(D(G(X_{\mathbf{n}}))^{(h,w)}), \tag{4}\]  
With the joint supervision of these two losses, the network finishes training on the labeled images.  
For unlabeled image training, no labeled images are used for training. Hence, the cross-entropy loss \(L_{\mathbf{ce}}\) is no longer used. However, the discriminator network is still needed, so the adversarial loss \(L_{\mathbf{adv}}\) is used. Here, the self-learning mechanism is adopted to send the segmentation results of unlabeled data into the discriminator network to generate a confidence map. Meanwhile, a threshold is used to binarize the confidence map, thereby better displaying the area close to the real distribution. Its mask cross-entropy loss \(L_{\mathbf{semi}}\) is defined as follows:  
\[L_{\mathbf{semi}}=-\sum_{h,w}\sum_{e\in C}I(D(G(X_{\mathbf{n}}))^{(h,w)}\geq T _{semi}*\overline{Y_{n}^{(h,w,c)}}\log(G(X_{\mathbf{n}}))^{(h,w,c)}), \tag{5}\]  
where \(I(\bullet)\) is the indicator function, and \(T_{semi}\) is the threshold for self-learning. In addition, if \(\mathbf{c}^{*}=\operatorname*{argmax}_{e}G(X_{\mathbf{n}})^{(h,w,c)}\), the self-learned one-hot encoded labeled image \(\overline{Y_{\mathbf{n}}}\) is set element-wise by \(\overline{Y_{n}^{(h,w,c^{*})}}=1\). The self-learn target \(\overline{Y_{\mathbf{n}}}\) and the indicator function \(I(\bullet)\) are regarded as constants. Experiments show that the network training has good robustness when \(T_{semi}\) = 0.2.