# Occlusion-Robust Multi-Target Tracking and Segmentation Framework with Mask Enhancement
## 3 Methodology
### Occlusion-Aware Mask Propagation Network
#### 3.2.1. Occlusion State Perception with Mask IoU  
The network proposes an occlusion state perception method. It defines four different occlusion states in crowded scenes. For frame \(t\), we defined target \(A\)'s bounding box as  
\[\text{Box}^{A}_{t}=(x^{A1}_{t},y^{A1}_{t},x^{A2}_{t},y^{A2}_{t},c^{A}_{t}), \quad A\in\mathcal{O}_{t} \tag{7}\]  
where \(\mathcal{O}_{t}\) is the target set, \((x^{1},y^{1})\) is target \(A\)'s top-left coordinate, \((x^{2},y^{2})\) is target \(A\)'s bottom-right coordinate, and \(c\) is the confidence. We defined target \(A\)'s mask as  
\[M^{A}_{t}=(m^{A}_{t},c^{A}_{t}),\quad m^{A}_{t}\in\{0,1\}^{H\times W} \tag{8}\]  
where \(H\) and \(W\) denote frame height and width, respectively. The method defines IoU between target \(A\) and target \(B\) in the same frame \(t\) as \(\alpha^{AB}_{t}\):  
\[\alpha^{AB}_{t}=\text{IoU}(\text{Box}^{A}_{t},\text{Box}^{B}_{t})=\frac{\text {Box}^{A}_{t}\cap\text{Box}^{B}_{t}}{\text{Box}^{A}_{t}\cup\text{Box}^{B}_{t}} \tag{9}\]  
The method then defines the mask area of target \(A\) as follows, where \(\beta^{A}_{t}\) stands for the mask area of target \(A\) in frame \(t\):  
\[\beta^{A}_{t}=\text{area}(M^{A}_{t})=\text{sum}(m^{A}_{t}) \tag{10}\]  
With the definitions above, we introduce the occlusion state of target \(A\) in frame \(t\) as \(o^{A}_{t}\in\{0,1,2,3\}\). \(o^{A}_{t}=0\) corresponds to \"no occlusion\", \(o^{A}_{t}=1\) corresponds to \"occluded by target\", \(o^{A}_{t}=2\) corresponds to \"occluding other targets\", and \(o^{A}_{t}=3\) corresponds to \"occluded by non-target objects\". The occlusion state from frame 1 to frame \(t\) of target \(A\) can be represented as \(o^{A}_{1:t}\), and the mask area from frame 1 to frame \(t\) of target \(A\) can be represented as \(\beta^{A}_{1:t}\):  
\[o^{A}_{1:t}=\{o^{A}_{1},\dots,o^{A}_{t}\} \tag{11}\]  
\[\beta^{A}_{1:t}=\{\beta^{A}_{1},\dots,\beta^{A}_{t}\} \tag{12}\]  
Eventually, we build the crowded perception model \(M_{oa}\) based on the least squares method:  
\[o^{A}_{t}=M_{oa}\Big{(}\max_{i\in\mathcal{O}_{t}}(\alpha^{Ai}_{t}),\beta^{A}_ {t-T:t}\Big{)} \tag{13}\]  
\[o^{A}_{t}=M_{oa}\Big{(}\alpha^{A,i}_{t},W_{k}\Big{)}, \tag{14}\]  
where \(\alpha^{A,i}_{t}\) represents the IoU between target \(A\) and other targets, and \(W_{k}\) is a weight matrix learned from the mask area. During the tracking process, \(\alpha^{A,i}_{t}\) is input to \(M_{oa}\) to determine the occlusion state of the target along with the mask area of previous \(T\) frames. The output is the occlusion perception state. The operation of \(M_{oa}\) consists of two stages: trend fitting and state determination. In the trend fitting stage, for each target \(A\) in frame \(t\), the input matrix is defined as  
\[X_{t-T:t}=[t-T,\;t-T+1,\;\dots,\;t]^{T}, \tag{15}\]  
And the response matrix is defined as  
\[V^{A}_{t-T:t}=\text{matrix}\Big{(}\beta^{A}_{t-T:t}\Big{)}^{T}=\Big{[}\beta^{ A}_{t-T},\;\beta^{A}_{t-T+1},\;\dots,\;\beta^{A}_{t}\Big{]}^{T}, \tag{16}\]The model learns the weight matrices \(W_{k}\in\mathbb{R}^{1\times 1}\) and \(W_{b}\in\mathbb{R}^{1\times 1}\) to perceive the crowding state of the target. Weight \(W_{k}\) is computed by minimizing the following objective function:  
\[M^{A}_{t}=(m^{A}_{t},c^{A}_{t}),\quad m^{A}_{t}\in\{0,1\}^{H\times W} \tag{8}\]0  
where \(\|\cdot\|_{F}^{2}\) is the Frobenius norm. Therefore, \(W_{k}\) can be solved with the linear regression equation as follows:  
\[M^{A}_{t}=(m^{A}_{t},c^{A}_{t}),\quad m^{A}_{t}\in\{0,1\}^{H\times W} \tag{8}\]1  
With the least squares method, \(W_{k}\) can be computed as  
\[M^{A}_{t}=(m^{A}_{t},c^{A}_{t}),\quad m^{A}_{t}\in\{0,1\}^{H\times W} \tag{8}\]2  
In the state determination stage, we designed a occlusion states classification strategy based on the IoU in current frame. The strategy utilized the following strategy to calculate \(o_{t}^{A}\):  
\[o_{t}^{A}=