# Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching
## 4 Methodology
### Deep Feature Aggregation  
Due to the presence of complex scenarios such as partial occlusions and motion blur in videos, it is difficult to detect occluded objects using only the clues from the current frame. Based on the consistency of object motion, historical information contains rich spatiotemporal contextual information that can assist in detecting occluded objects. By leveraging the abundant temporally complementary information, the description of occluded objects is enhanced, thereby reducing identity switches. Therefore, we first constructed the tracklet embedding module, which maintains a tracklet buffer area to save the image feature information and motion information contained in all historical tracklets. The buffer area is divided into different sub-areas according to the number of historical tracklets. The features in each area are represented as \(\mathbf{\mathsf{M}}^{t-1}_{j}=\left\{\tilde{\mathbf{\mathsf{F}}}^{t-\eta}_{j},\tilde{ \mathbf{\mathsf{F}}}^{t-\eta+1}_{j},\dots,\tilde{\mathbf{\mathsf{F}}}^{t-1}_{j}\right\}\), where \(\tilde{\mathbf{\mathsf{F}}}^{t-1}_{j}\) and \(\eta\) are the single-step embedding and cache length of the \(j\)-th object, respectively. The tracklet feature is updated to \(\mathbf{\mathsf{T}}^{t}_{j}=\gamma(\mathbf{\mathsf{M}}^{t-1}_{j},\tilde{\mathbf{\mathsf{F }}}^{t}_{j})\), where \(\gamma\) denotes the feature aggregation operation. Subsequently, an attention mechanism based historical tracklet feature enhancement module is adopted. When it is necessary to associate with the current frame object for data correlation, a memory feature fusion network is used to integrate all historical inputs of each tracklet, enhancing the features of the consistent areas between the historical tracklets and the current frame, to generate high-quality tracklets for subsequent modules. Figure 3: Framework overview of proposed method (JDTHM). JDTHM is composed of a feature extraction and object detection module, a feature aggregation module, and a data association module. Different colored circles represent the candidate bboxes and history tracklets. Setting \(\zeta(\mathbf{x}_{i},\mathbf{x}_{j})=\exp^{\phi(\mathbf{x}_{i})^{\mathrm{T}}\varphi( \mathbf{x}_{j})}\), the attention between the visual features of the current frame and the historical tracklet features is defined as follows:  
\[\mathcal{A}(\mathbf{F}_{i,\ast}^{l},\mathbf{T}^{l-1})=\sum_{k=1}^{m}\frac{ \alpha_{\ast,k}\sum_{\forall j}\xi(\mathbf{F}_{i,\ast}^{l},\mathbf{T}_{j,k}^{l -1})\theta(\mathbf{T}_{j,k}^{l-1})}{\sum_{\forall j}\xi(\mathbf{F}_{i,\ast}^{ l},\mathbf{T}_{j,k}^{l-1})}, \tag{2}\]  
where \(\mathbf{F}_{i,\ast}^{l}\) and \(\mathbf{T}_{j,k}^{l-1}\) represent random features from \(\mathbf{F}_{\ast}^{l}\) and \(\mathbf{T}_{k}^{l-1}\), respectively. \(\mathbf{T}_{k}^{l-1}\in\mathbf{T}^{l-1}\) represents the feature map of the \(k\)-th tracklet. \(\phi(\cdot)\), \(\varphi(\cdot)\) and \(\theta(\cdot)\) denote Convolution layers. \(\alpha_{\ast,k}\) is an indicator function, taking the value 1 if \(\mathrm{IoU}(\mathbf{P}_{\ast}^{t},\mathbf{B}_{k}^{t-1})>\alpha_{\mathrm{IoU}}\) and 0 otherwise. IoU stands for \"Intersection over Union\", which measures the degree of overlap between two regions, often used to compare the similarity between a predicted bbox or segmentation area and the ground truth bbox or segmentation area. \(\mathrm{IoU}(\mathbf{P}_{\ast}^{t},\mathbf{B}_{k}^{t-1})\) represents the geometric similarity between the candidate proposal \(\mathbf{P}_{\ast}^{t}\) and the final bbox \(\mathbf{B}_{k}^{t-1}\) in the tracklet \(\mathcal{T}_{k}^{t-1}\), where \(\alpha_{\mathrm{IoU}}\) is the geometric threshold. Furthermore, the features enhanced by the historical tracklets are as follows:  
\[\mathbf{\widetilde{F}}_{\ast}^{t}=\mathbf{F}_{\ast}^{t}\oplus\mathcal{A}( \mathbf{F}_{\ast}^{t},\mathbf{T}^{t-1}), \tag{3}\]  
where \(\oplus\) represents element-wise addition and subtraction operations for matrices.