# Multi-Object Tracking with Grayscale Spatial-Temporal Features
## 3 Method  
### Overview  
In order to better explain the proposed method, there are a few symbols that need to be defined:\(\mathbf{P_{t}^{i}}\) : represents a one-dimensional set at frame \(t\), and \(i\) is the size of this set, for example, \(\mathbf{P_{t}^{i}}=\left\{p_{t}^{0},p_{t}^{1},p_{t}^{2},\ldots,p_{t}^{i-1},\right\}\). \(\mathbf{RcID_{t,i}^{k}}\) : represents a two-dimensional set at frame \(t\): \(k\) is the size of a one-dimensional set, and \(i\) is the number of one-dimensional sets in this two-dimensional set, for example, \(\mathbf{RcID_{t,i}^{k}}=\left\{\left\{rcID_{t,0}^{0},\ldots,rcID_{t,0}^{k-1}, \right\},\ldots,\left\{rcID_{t,i-1}^{0},\ldots,rcID_{t,i-1}^{k-1},\right\}\right\}\). In the following, the region spatial-temporal feature is abbreviated as region-STF and target spatial-temporal feature is abbreviated as target-STF. Table 1 shows the interpretation of notation in the manuscript. As shown in Figure 1, our framework is based on tracking-by-detection, with embedding and tracking combined. It performs online tracking in four steps:  
In module1, continuous frames are input into the yolov5s module to obtain the target information. As the yolov5s module has image segmentation function, we can not only obtain the bounding box (\(\mathbf{Pb_{t}^{i}}\)) of each target (\(\mathbf{P_{t}^{i}}\)) but also obtain the segmentation mask (\(\mathbf{Pm_{t}^{i}}\)) of each target. In module2, we obtain the foreground mask map (\(\mathbf{FmI_{t}}\)), target-STF map (\(\mathbf{PsI_{t}}\)), and region-STF map (\(\mathbf{RsI_{t}}\)), which rely on \(\mathbf{Pb_{t}^{i}}\) and \(\mathbf{Pm_{t}^{i}}\). In module3, we match spatial-temporal features and use the \(\mathbf{Ps_{t}}\) and \(\mathbf{Rs_{t}}\) to save the matching results of the target-STF and region-STF, respectively. In module4, we classify targets by \(\mathbf{Ps_{t}}\). According to the classification results, the targets are matched hierarchically. Continuously detected targets are matched first, then re-detected targets are matched. We use a histogram to extract the appearance features of the target and use the yolov5 seg model to cut the target. This can remove background \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Interpretation** \\ \hline \(t\) & Frame \\ \(P_{t}^{i}\) & Detection targets \\ \(P_{t}^{i}\) & Bounding box for each target \\ \(Pm_{t}^{i}\) & Segmentation mask for each target \\ \(FmI_{t}\) & Foreground mask map \\ \(PsI_{t}\) & Target-STF map \\ \(RsI_{t}\) & Region-STF map \\ \(P_{t,i+1}^{k}\) & Matching results of the target-STF \\ \(Rrs_{t}^{i}\) & Matching results of the region-STF \\ \(Rs_{t}^{i}\) & The region to which the target belongs \\ \(PcId_{t}^{i}\) & The successfully matched IDs \\ \(RcID_{t,i}^{k}\) & The successfully matched IDs contained in each region \\ \(RID_{t,i}^{k}\) & Unmatched IDs of the previous frame in each region \\ \(\mathbf{UID_{t}^{i}}\) & Unmatched IDs of the previous frame do not belong to any region \\  & in the current frame \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The notation in the manuscript. Figure 1: The framework of our method. noise interference when extracting appearance features and remove interference between occluded targets. After matching, we update the trajectory of the target and record the ID for each target \(\mathbf{P_{t}^{i}}\) by using \(\mathbf{P_{t}}\mathbf{d_{t}^{i}}\). The successfully matched IDs contained in each region are recorded by the set \(\mathbf{R_{c}}\mathbf{ID_{t,i}^{k}}\). The unmatched targets in the previous frame are stored in \(\mathbf{RID_{t,i}^{k}}\) or \(\mathbf{UID_{t}^{i}}\). For the unmatched IDs in each region in the previous frame, if their region spatial-temporal features are not successfully matched in the current frame, these targets are recorded in \(\mathbf{UID_{t}^{i}}\), and if they are matched successfully, they are recorded in \(\mathbf{RID_{t,i}^{k}}\). The IDs contained in \(\mathbf{RID_{t}^{i}}\) only keep 200 frames. The IDs contained in \(\mathbf{UID_{t}^{i}}\) only keep 50 frames.