# Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching
## 4 Methodology
### Data Association  
We transform the data association into a hypergraph matching problem between detection results \(\mathbb{B}^{t}=\{\mathbf{B}_{1}^{t},\mathbf{B}_{2}^{t},\ldots,\mathbf{B}_{n}^ {t}\}\) and historical tracklets \(\mathbb{T}^{t-1}=t\{\mathcal{T}_{1}^{t-1},\mathcal{T}_{2}^{t-1},\ldots, \mathcal{T}_{m}^{t-1}\}\), where \(n\) and \(m\) represent the number of detection results and historical tracklets, respectively. We first construct a detection hypergraph and a tracklet hypergraph following NHGM [51] to describe the relationships between different objects in the detection results and historical tracklets. The hypergraph matching process is shown in Figure 4.  
The detection hypergraph is defined as \(\mathcal{G}_{D}=(\mathcal{V}_{D},\mathbf{\varepsilon}_{D})\), where \(\mathcal{V}_{D}=\{(\mathbf{B}_{i}^{t},\mathbf{\hat{F}}_{i}^{t})\},i\in[1,n]\) represents the vertex set, with \(\mathbf{\hat{F}}_{i}^{t}\) representing the features of the bbox \(\mathbf{B}_{i}^{t}\). \(\mathbf{\varepsilon}_{D}\) represents the set of hyperedges. Similarly, the tracklet hypergraph is defined as \(\mathcal{G}_{T}=(\mathcal{V}_{T},\mathbf{\varepsilon}_{T})\), where  
Figure 4: Hypergraph matching process diagram. The computation process of vertex-to-vertex matching relationships in two hypergraphs is translated into a vertex classification task on the associated hypergraph, where edge weights can be induced by the association tensor. \(\mathcal{V}_{T}=\left\{(\mathbf{B}_{j}^{t-1},\mathbf{\Upsilon}_{j}^{t-1})\right\},j\in[1,m]\) represents the vertex set, with \(\mathbf{\Upsilon}_{j}^{t-1}\) representing the feature of final bbox \(\mathbf{B}_{j}^{t-1}\) of the \(j\)-th historical tracklet \(\mathcal{T}_{j}^{t-1}\). \(\mathbf{\epsilon}_{T}\) represents the set of hyperedges. The connectivity matrices of the detection hypergraph are denoted as \(\mathbf{\overline{P}}_{D}\) and \(\mathbf{\overline{Q}}_{D}\), and those of the tracklet hypergraph are denoted as \(\mathbf{\overline{P}}_{T}\) and \(\mathbf{\overline{Q}}_{T}\). The adjacency matrices obtained are \(\mathbf{\overline{A}}_{D}=\mathbf{\overline{P}}_{D}\mathbf{\overline{Q}}_{D}^{T}\) and \(\mathbf{\overline{A}}_{T}=\mathbf{\overline{P}}_{T}\mathbf{\overline{Q}}_{T}^{T}\), respectively. The representation of the hyperedges is constructed by concatenating the vertex features at both ends of the edges. \[\mathbf{\overline{E}}_{D}=[\mathbf{\overline{P}}_{D}^{T}\mathbf{\hat{r}}^{T}\mathbf{\quad \overline{Q}}_{D}^{T}\mathbf{\hat{r}}^{t}], \tag{5}\]  
\[\mathbf{\overline{E}}_{T}=[\mathbf{\overline{P}}_{T}^{T}\mathbf{T}^{t-1}\quad\mathbf{\overline {Q}}_{T}^{T}\mathbf{T}^{t-1}], \tag{6}\]  
We enhance the vertex features through cross-graph information transfer between the detection hypergraph and the tracklet hypergraph. \[\mathbf{\hat{r}}_{i}^{t,l+1}=\psi_{D}(\mathbf{\hat{r}}_{i}^{t,l}+\sum_{j=1}^{m}\left( \cos(\mathbf{\hat{r}}_{i}^{t,l},\mathbf{\Upsilon}_{j}^{t-1,l})+IoU(\mathbf{B}_{i}^{t,l}, \mathbf{B}_{j}^{t-1,l})\right)), \tag{7}\]  
\[\mathbf{\Upsilon}_{j}^{t-1,l+1}=\psi_{T}(\mathbf{\Upsilon}_{j}^{t-1,l}+\sum_{i=1}^{n} \left(\cos(\mathbf{\hat{r}}_{i}^{t,l},\mathbf{\Upsilon}_{j}^{t-1,l})+IoU(\mathbf{B}_{i}^{t,l},\mathbf{B}_{j}^{t-1,l})\right)), \tag{8}\]  
The enhanced vertex features are denoted as \(\mathbf{\hat{r}}_{i}^{t,L}\) and \(\mathbf{\Upsilon}_{j}^{t-1,L}\), meanwhile, the hyperedge features are denoted as \(\mathbf{\overline{E}}_{D,i12}^{L}\) and \(\mathbf{\overline{E}}_{T,i12}^{L}\). For hypergraph matching, the above QAP model is generalized to the higher-order case, adopting a tensor-marginalization-based model for \(p\)-order (\(p\geq 3\)) hypergraph matching, resulting in a higher-order assignment problem [51]:  
\[ \begin{array}{l}\mathcal{J}(\mathbf{x})=\mathbf{H}\otimes_{1}\mathbf{x} \otimes_{2}\mathbf{x}\ldots\otimes_{p}\mathbf{x}\\ s.t.\mathbf{X}\mathbf{1}_{n_{2}}=\mathbf{1}_{n_{1}},\mathbf{X}^{\text{T}} \mathbf{1}_{n_{1}}\leq\mathbf{1}_{n_{2}}\end{array}