# MU R-CNN: A Two-Dimensional Code Instance Segmentation Network Based on Deep Learning
## 3 Our Contributions
### Training of MU R-CNN
#### 3.2.2 End-to-End Training of MU R-CNN  
The proposed MU R-CNN is mainly composed of Mask R-CNN and UNet branch. Among them, Mask R-CNN branch consists of two stages, and they are termed as region proposal network (RPN) and region-based convolutional neural network (R-CNN), respectively. During training of MU R-CNN, the input image first passes through RPN to obtain coarse screening candidate regions. Then, RoIAlign is used to extract features from each candidate region, and classification, boundary box and mask regression are carried out. We set the output candidate region of R-CNN as \(R_{i}\), and represent by formula (12):  
\[\{R_{i}|i=0,1,2,\cdots,N\} \tag{12}\]  
where N is the total number of candidate regions output by Mask R-CNN. \(R_{i}\) is then sent to the pre-trained \(M_{UNet}\) for texture segmentation, as shown in Figure 4. After the texture segmentation of all candidate images, binarization is carried out. Then, the binary image was combined with its ground truth to calculate \(L_{IoU}\) according to formula (1). Finally, the loss of the entire MU R-CNN network is calculated according to Equation (2). The training process is an end-to-end procedure. The training data set and network parameter setting information will be presented in the experimental section. We summarized the training process of the MU R-CNN network as shown in Table 2.  
We noticed that [PERSON] [69] also proposed a similar method to improve the quality of instance mask by adding IoU measure branch (named MaskIoU) to Mask R-CNN. The difference between our proposed MU R-CNN and Mask Scoring R-CNN (MS R-CNN for short) in [69] is that the weight of MaskIoU branch in MS R-CNN is trained at the same time as Mask R-CNN branch, while our UNet branch in MU R-CNN adopts the pre-trained optimal results, so as to ensure that \(L_{IoU}\) in MU R-CNN is accurate at the beginning of the MU R-CNN end-to-end training. In addition, MaskIoU in [69] uses the concatenation of features from RoIAlign layer and the bounded mask as the input of MaskIoU head. However, the input of the UNet branch in this paper uses the segmented candidate images from the input image. In this way, the UNet branch does not depend on the features of the Mask \begin{table}
\begin{tabular}{c l} \hline \hline
**Steps** & **Operating Process** \\ \hline
1 & Select 60 training images \\
2 & Mark ground truth manually \\
3 & The training images and their corresponding ground truth were augmented in pairs to \\  & obtain the training set after data augmentation \\
4 & The training set after data augmentation is sent to UNet for training \\
5 & Start training until the number of epochs reaches 1000, and the learning rate during \\  & training is set to 0.00001; eventually we got \(M_{UNet}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Training process of UNet branch. \begin{table}
\begin{tabular}{c l} \hline \hline
**Steps** & **Operating Process** \\ \hline
1 & First, the training image is input into the Mask R-CNN branch to obtain the candidate \\  & region \(R_{i}\) \\
2 & \(R_{i}\) is then sent to the pre-trained \(M_{UNet}\) to output a texture segmented QR tag image \(l_{i,anet}\) \\
3 & After the texture segmentation of all candidate region images, binarization is carried out \\
4 & Dice loss \(L_{IoU}\) is calculated according to the formula (1) \\
5 & Calculate the loss of the entire MU R-CNN according to Equation (2) \\
6 & Start end-to-end training \\
7 & After training, the MU R-CNN model is obtained. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Training process of total MU R-CNN. R-CNN, so the training process of the UNet branch can be carried out independently, in advance. In particular, as dice-loss is an approach for directly optimizing IoU measure in deep neural networks, we choose dice-loss to determine IoU loss, which makes it easier to achieve better training results.