# Mechanical Assembly Monitoring Method Based on Semi-Supervised Semantic Segmentation
## 3 Overall Framework
### Structure of the Model  
This paper proposes an adversarial learning network for semi-supervised semantic segmentation of mechanical assembly images (AdvSemiSeg-MA) (Figure 1). The semantic segmentation network G consists of the FRASPP deep feature extraction module and the CoordConv spatial perception module. The discriminator network D consists of multiple layers of convolutions using spectral normalization (SN) [24]. Four types of losses are used in network training, namely, the adversarial loss \(L_{\mathbf{adv}}\) of the discriminator network D, the spatial cross-entropy loss \(L_{D}\), the mask cross-entropy loss \(L_{\mathbf{semi}}\), and the standard cross-entropy loss \(L_{\mathbf{ce}}\) of annotated images. This network uses the semantic segmentation network as the generator of the generative adversarial network. The output class probability map is sent to the discriminator network D. The discriminator network outputs the true and false discriminant values of each pixel to improve the precision of the semantic segmentation network G.  
As shown in Figure 1, a mechanical assembly image \(X_{1}\) with a size of H \(\times\) W \(\times\) 3 is input to G. The image is first input to the CoordConv module so that the convolution can perceive the spatial position information and enable G to be position-sensitive. Then, the obtained features are input to the RseNet101 module to extract the features of the mechanical assembly image. Subsequently, the obtained features are input to the FRASPP module to extract the deep features of the image. Finally, the deep features extracted by the FRASPP module, the features output by the 7 \(\times\) 7 convolution of RseNet101, and the features output by the first layer of RseNet101 are input to the ASFF module to fuse the above-mentioned multiscale features. A class probability map of dimension H \(\times\) W \(\times\) C is the output, denoted as G (\(X_{1}\)), where C is the class count of the semantic segmentation. In this study, the discriminator network (D) adopts four sets of convolutions including spectral normalization (SN). In addition, a set of convolutions consisting of a 3 \(\times\) 3 convolution with a step size of 1 and a 4 \(\times\) 4 convolution with a step size of 2 are added. In D, multilayer spectral normalization (SN) can make D satisfy the Lipschitz continuity. This prevents the function from drastic changes and improves the stability of the training  
Figure 1: The overall structure of the network. process. The input of D is the class probability map (G (X\({}_{1}\))) output by G or the one-hot encoded ground truth (Y\({}_{n}\)). The output of D is the confidence map of size H \(\times\) W \(\times\) 1. For each pixel of the confidence map, it is set to zero if it comes from G and is set to one if it comes from Y\({}_{n}\). During the semi-supervised training of the AdvSemiSeg-MA network, 1/8 of the assembly image training dataset is randomly used as labeled data, and the remaining data is taken as unlabeled data. Firstly, the G and D are trained using annotated images. The training of [PERSON] is jointly supervised by the cross-entropy loss \(L_{\mathbf{ce}}\) of the annotated images and the adversarial loss \(L_{\mathbf{adv}}\) of D. Meanwhile, the training of [PERSON] is only supervised by the spatial cross-entropy loss \(LD\). Then, the unlabeled images are used to train the network again. The unlabeled images are passed through G to output the class probability map, which is input to D to obtain the confidence map. Finally, the confidence map and mask cross-entropy loss \(L_{\mathbf{semi}}\) are used as supervised information to train [PERSON] in a self-learning way. The innovations of the proposed AdvSemiSeg-MA network include: (1) A multiscale output fusion method of ASFF is proposed. This fusion method can make full use of the deep semantic and shallow detail features extracted by the network to improve the accuracy of the model for segmenting small parts in mechanical assembly.