# Occlusion-Robust Multi-Target Tracking and Segmentation Framework with Mask Enhancement
## 3 Methodology
### Occlusion-Aware Mask Propagation Network
#### 3.2.2 Robust Mask Propagation Network  
To face the challenge of data association in congestion scenarios, we propose a novel approach for robust data association in such scenarios. The Robust Mask Propagation Network utilizes a constrained time window and captures the movement context effectively. The implementation steps of the method are as follows:  
(1) When the target is not occluded, the correlation between trajectories is realized based on the multi-hypothesis tracking algorithm.  
(2) When the occlusion sensor detects that a target is occluded, the first \(T\) frame of the current frame is input into the memory encoder, and the mask information of the network learning target is propagated based on the mask.  
(3) When the occlusion sensor detects that a target is occluded, the current image is input into the query encoder as a query after the memory encoder is constructed.  
(4) The mask propagation network builds a mask for the input query construction target, and the mask with the same size as the original query is obtained through decoder processing. Steps (3) and (4) are repeated for each frame where the occluded target is located.  
(5) When the target ends the occlusion stage, the correlation between trajectories is continued based on the multi-hypothesis tracking algorithm.  
Based on the steps above, the trajectory of the occluded target is generated by the mask propagation network. The relevant definition and network structure of the mask propagation network will be introduced in detail, and the workflow of mask propagation is shown in Figure 2.  
We defined memory encoder as \(En_{M}\), query encoder as \(En_{Q}\), and mask decoder as \(De\). We also defined \(S_{t}^{A}\) as the set of masks of target \(A\) in the previous \(T\) frames before frame \(t\), where trajectories are not occluded:  
\[S_{t}^{A}=\{m_{t-T-1}^{A},\ m_{t-T}^{A},\ \ldots,\ m_{t-1}^{A}\} \tag{21}\]  
We took a random target in frame \(t\) as an example to explain the mask propagation process. The input of the network was divided into two parts. In order to learn and memorize the coding pattern, the memory encoder \(En_{M}\) took the historical frames \(I_{j}^{t-T-1:t-1}\) and \(S_{t}^{A}\) as inputs, outputting \(F_{t}^{En_{M}}\), \(K_{t}^{En_{M}}\), and \(V_{t}^{En_{M}}\). \(F_{t}^{En_{M}}\), \(K_{t}^{En_{M}}\), \(V_{t}^{En_{M}}\) are the key-value pair set, key set, and value set corresponding to all the detections after being encoded. For a single frame input, the encoder \(En_{M}\) generated \(f_{t}^{En_{M}}\), \(k_{t}^{En_{M}}\), and \(v_{t}^{En_{M}}\). \(f_{t}^{En_{M}}\), \(k_{t}^{En_{M}}\), and \(v_{t}^{En_{M}}\) are the key-value pairs, keys, and values corresponding to the detections in frame \(t\) after being encoded.  
In order to generate new masks, the query encoder \(En_{Q}\) processed the current frame \(I_{j}^{t}\) and generated \(f_{t}^{En_{Q}}\), \(k_{t}^{En_{Q}}\), and \(v_{t}^{En_{Q}}\). In order to search based on the query key, the dot product was used between \(k_{t}^{En_{Q}}\) and \(k_{t}^{En_{M}}\). The method used normalization to obtain the similarity matrix \(k_{t}^{\text{QM}}\):  
\[k_{t}^{\text{QM}}=\text{softmax}(k_{t}^{En_{Q}}\cdot k_{t}^{En_{M}}) \tag{22}\]  
Figure 2: The workflow of the Robust Mask Propagation Network.  
The similarity matrix \(k_{t}^{QM}\) was multiplied by the memory values \(e_{t}^{En_{M}}\) to obtain the memory retrieval result matrix \(f_{t}^{QM}\). By concatenating \(f_{t}^{QM}\) with \(v_{t}^{En_{Q}}\), we obtained the mask feature matrix \(f_{t}^{concat}\). We then inputted \(f_{t}^{concat}\) in \(De\) for mask refinement.  
Eventually, the module restored \(f_{t}^{concat}\) to the original image size.