# Dynamic Detection and Recognition of Objects Based on Sequential RGB Images
## 3 Framework of the System
### LSMNet  
For an image \(X\) and its annotation set \(\mathcal{M}=\{M_{1}^{0},M_{2}^{0},\cdots\}\), in which \(M_{k}^{0}\) is the binary mask for the \(k-th\) instance. The elements of \(M_{k}\) in the segmentation region are 1, and the remaining elements are 0. Let  
\[M_{k}^{n+1}=erode(M_{k}^{n},\Lambda) \tag{1}\]  
where \(erode(\cdot,\Lambda)\) is the eroding operation in morphology with the kernel \(\Lambda\). The region of \(M_{k}^{n+1}\) lies in the contour of \(M_{k}^{n}\). Then, the LSM of instance \(k\) can be obtained according to  
\[M_{k}^{LSM}=127\cdot M_{k}^{0}+h\cdot\sum_{k=1}^{K}M_{i}^{n} \tag{2}\]  
where \(K\) is the max step of eroding operation, and \(h\) is the biggest interval of contour lines. We can normalize \(M_{k}^{LSM}\) to \([0,255]\) in the following manner  
\[\overline{M}_{k}^{LSM}=\left(127+\frac{128\cdot\left(M_{k}^{LSM}-127\right)}{ \max\{M_{k}^{LSM}\}-127\right)}/255 \tag{3}\]  
where \(\max\{\cdot\}\) returns the maximum element of a 2D matrix. Denote the final LSM of \(X\) as \(\mathcal{M}^{LSM}\), then its element at \((i,j)\) is calculated as below  
\[\mathcal{M}_{i,j}^{LSM}=\max_{k}\Bigl{\{}(\overline{M}_{k}^{LSM})_{i,j}\Bigr{\}} \tag{4}\]  
A group of samples is depicted in Figure 3. In \(\mathcal{M}^{LSM}\), the elements out of contours are 0, and elements on the contours are equal to 127. We can set \(K=1000\), and stop Iteration (2) when \(M_{k}^{n+1}\) has been eroded totally; that is \(sum(M_{k}^{n+1})=0\). In an LSM, peaks represent objects and valleys represent the background. Once the LSM is predicted accurately, an appropriate threshold can be used to segment the peaks conveniently. Figure 2: Overall architecture of the proposed method. It is a typical pixel-level image translation task to predict \(\mathcal{M}^{LSM}\) with \(X\), and any pix2 pix architecture can be adopted. In this study, we chose an UNet as the basic architecture. The UNet contains 16 convolutional layers, where the number of convolutional kernels increases from 16 to 512 in the first 8 layers and decreases to 1 in the last 8 layers. The scale of downsampling and upsampling is 2, the input size is 256 \(\times\) 256 \(\times\) 3, and the output size of each branch is 256 \(\times\) 256 \(\times\) 1. To improve the quality of the LSM, we trained an LSM branch and a semantic segmentation branch simultaneously. Those two decoders have the same architecture. The framework of LSMNet is depicted in Figure 4. In the training process, the LSM loss and mask loss, together with a consistent constraint loss, were calculated and used to update the backbone. In the inferring process, small noisy regions of \(\mathcal{M}^{LSM}\) were filtered first, and the foreground regions were segmented with a threshold > 0.5. Each region corresponds to an instance, and the instance regions are dilated in a ratio proportional to its area. Denote the ground truth of the semantic segmentation mask as \(\mathcal{M}^{mask}\). Its elements are calculated according to  
\[\mathcal{M}^{mask}_{i,j}=\max_{k}\left\{\left(M^{0}_{k}\right)_{i,j}\right\} \tag{5}\]  
Figure 4: Framework of LSMNet. Figure 3: LSM with \(K=1000\) and \(h=2\). The output LSM and mask of LSMNet are written as \(\widetilde{\mathcal{M}}^{LSM}\) and \(\widetilde{\mathcal{M}}^{mask}\). Then, the final lost function consists of three sublosses as below  
\[ \begin{split}& l_{sum}=l_{LSM}+l_{mask}+l_{c}\\ & l_{LSM}=\omega_{1}\cdot\left|\mathcal{M}^{LSM}-\widetilde{ \mathcal{M}}^{LSM}\right|\\ & l_{mask}=\omega_{2}\cdot\left|\mathcal{M}^{mask}-\widetilde{ \mathcal{M}}^{mask}\right|\\ & l_{c}=\omega_{3}\cdot\left|\left(2\widetilde{\mathcal{M}}^{ mask}-1\right)\otimes\left(\widetilde{\mathcal{M}}^{mask}-0.5\right)-\left( \mathcal{M}^{mask}-0.5\right)\right|\end{split}  \tag{6}\]  
in which \(|\cdot|\) is the L1 norm, \(\otimes\) is the element-wise multiplying operation, and \(\omega_{1}\), \(\omega_{2}\), and \(\omega_{3}\) are the weights for sublosses.