## 2 Materials and Methods
### Feature-Mapping Motion Estimation Module
#### 2.2.2. Displacement Prediction  
For optical flow estimation networks, the correlation volume indicates the pixel-wise similarity between \(\mathrm{f^{(t)}_{emb}}\) and \(\mathrm{f^{(t-r)}_{emb}}\). Treating the object-tracking task as a matching of center point feature pixels, the displacement matrix \(\mathrm{M}\in\mathrm{R^{Hc\times Wc}}\) could be predicted using the correlation volume. First, we calculated the four-dimensional all-pairs correlation volume \(\mathrm{C}\) by performing a matrix multiplication between the extracted features \(\mathrm{f^{(t)}_{emb}}\) and the preceding feature \(\mathrm{f^{(t-r)}_{emb}}\) using the following equation:  
\[\mathrm{C}=\left\langle\mathrm{f^{(t)}_{emb}},\mathrm{f^{(t-1)}_{emb}}\right\rangle \in\mathrm{R^{Hc\times Wc\times Hc\times Wc}},\mathrm{C(i,j,k,l)}=\mathrm{f^{( t)}_{emb}(i,j)\mathrm{f^{(t-1)}_{emb}}(k,l)}^{T}, \tag{5}\]  
where \(\left\langle\_\_\_\_\right\rangle\) denotes the matrix multiplication operation and \(\mathrm{{}^{T}}\) represents the transpose operation.  
Based on the matching similarity from the correlation volume, global matches of the target centers could be generated, which in turn allowed for the prediction of displacements, \(\mathrm{M}\). For a specific object \(\mathrm{O_{ij}}\) at time \(\mathrm{t}\) with its correlation volume \(\mathrm{C(i,j,k,l)}\), we utilized pooling convolutions with kernel sizes of \(\mathrm{Hc}\times 1\) and \(1\times\mathrm{Wc}\) to individually generate the normalized feature mapping \(\mathrm{C^{W}(i,j)}\in[0,1]^{1\times Wc}\) and \(\mathrm{C^{H}(i,j)}\in[0,1]^{Hc\times 1}\) in the vertical and horizontal dimensions, respectively. Then, we reflected these mappings to obtain the actual displacements in the original image. The process is as follows: we first defined the pixel-wise displacement vectors, \(\mathrm{U(i,j)}\in\mathrm{R^{1\times Wc}}\) and \(\mathrm{V(i,j)}\in\mathrm{R^{Hc\times 1}}\). The displacement prediction, \(\mathrm{M^{t}}\in\mathrm{R^{Hc\times Wc\times 2}}\), could be obtained by taking the element-wise product of  
Figure 6: Overview of the FMME module. The FMME module may predict the object displacement based on the inter-frame mapping constructed using a correlation volume and generate ReID features for trajectory management.  
the normalized feature mappings and corresponding displacement vectors of \(\mathrm{O_{ij}}\) in the following equation:  
\[\left\{\begin{array}{ll}\mathrm{U(i,j,k,*)}=(\mathrm{k-i})\times\mathrm{s},&1 \leq\mathrm{k}\leq\mathrm{Wc}\\ \mathrm{V(i,j,*,l)}=(\mathrm{l-j})\times\mathrm{s},&1\leq 1\leq\mathrm{Hc} \end{array},\right. \tag{6}\]  
\[\mathrm{M(i,j)}=\left[\mathrm{C^{H}(i,j)}^{T}\mathrm{V(i,j)},\mathrm{C^{W}(i,j )}\mathrm{U(i,j)}^{T}\right]^{T}, \tag{7}\]  
where \(\mathrm{s}=4\) and is the factor of recovery. For a specific object \(\mathrm{O_{ij}}\) at time \(\mathrm{t}\), the displacement prediction \(\mathrm{M(i,j)}\) reflects the real pixel displacement between the central feature \(\mathrm{f^{(t)}_{emb}(i,j)}\) of \(\mathrm{O_{ij}}\) and \(\mathrm{f^{(t-\tau)}_{emb}(k,l)}\) of the \(\mathrm{O_{k,l}}\) at time \(\mathrm{t-\tau}\), where \(\mathrm{O_{k,l}}\) represents, with the maximum similarity, to \(\mathrm{O_{ij}}\). This displacement is calculated in the original image size.  
To obtain the ReID loss under sparse labels, when the target \(\mathrm{O_{ij}}\) at time \(\mathrm{t}\) correctly matches the target \(\mathrm{O_{k,l}}\) of the same identity at the time \(\mathrm{t-\tau}\) in a certain dimension, we set \(\mathrm{Y_{ijd}}=1\), and \(0\) otherwise. We employed a focal loss for FMME training and only considered positive samples with their normalized similarity to define the displacement prediction loss, which increases the margin between the true identity targets and other different identity targets. This improvement supervises the training of the ReID sub-network and subsequently supervises the prediction process of the FMME module.