# Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching
## 4 Methodology
### Data Association
, \tag{9}\]  
where \(\mathbf{x}=\text{vec}(\mathbf{X})\) is the column-vectorized form and \(\mathbf{H}\) is the \(p\)-order affinity tensor whose elements record the affinity between two hyperedges, operated by tensor product \(\otimes_{q}\), where \(\otimes_{q}\) can be regarded as tensor marginalization at dimension \(q\). To measure the similarity between vertices in the two hypergraphs, an affinity tensor, \(\mathbf{H}\), is constructed. The first-order vertex similarity, the second-order edge similarity, and the third-order angle similarity matrices of the affinity tensor \(\mathbf{H}\) can be represented as follows:  
\[\mathbf{H}_{ij}^{(1)}=\cos(\mathbf{\hat{r}}_{i}^{t,L},\mathbf{\Upsilon}_{j}^{t-1,L}), \tag{10}\]  
\[\mathbf{H}_{i12,j1j2}^{(2)}=\cos(\mathbf{\overline{E}}_{D,i12}^{L},\mathbf{\overline{ E}}_{T,i1j2}^{L}), \tag{11}\]  
\[\mathbf{H}_{\omega_{1},\omega_{2},\omega_{3}}^{(3)}=\exp\{-(\sum_{q=1}^{3}| \cos\beta_{D}^{\omega_{q}}-\cos\beta_{T}^{\omega_{q}}|)/\sigma_{3}\}, \tag{12}\]  
where \(\beta_{D}^{\omega_{q}}\) and \(\beta_{T}^{\omega_{q}}\) represent the angles between \(\mathbf{H}_{D}\) and \(\mathbf{H}_{T}\), respectively. Finally, we follow the computational process of hypergraph matching vertex updates as described in NHGM [51], utilizing hypergraph matching with high-order contextual information to match the detection hypergraph with the tracklet hypergraph. Hypergraph convolutional networks are used to embed the vertices of the association graph, capturing the topological structure information between nodes and generating feature representations for each node. A Sinkhorn network outputs a doubly stochastic matrix, transforming the vertex embeddings generated by HGCN into representations that meet the matching constraints. After each HGCN layer, the output of the Sinkhorn network (i.e., the predicted matching matrix) is concatenated with the vertex embeddings of the current layer. By introducing matching constraints in the embedding process, the matching-aware embeddings can not only utilize the topological structure information of the nodes but also more directly learn the node representations that satisfy the matching conditions, thereby improving the performance of the graph matching task. If the corresponding similarity in \(\mathbf{H}\) is higher than the affinity threshold \(\delta\), then two nodes between the detection hypergraph and the tracklet hypergraph are matched. Finally, optimal matching can be obtained. For detections that do not match any tracklet, if the detection confidence score is greater than a certain threshold, we will use it to initialize a new tracklet. Unmatched tracklet representations suggest that they may have left the scene or were not detected. During the training process, our proposed algorithm can be optimized in an end-to-end manner using the following loss function:  
\[\mathcal{L}=\mathcal{L}_{Det}+\mathcal{L}_{Emb}+\mathcal{L}_{HGM}, \tag{13}\]  
where \(\mathcal{L}_{Det}\), \(\mathcal{L}_{Emb}\), and \(\mathcal{L}_{HGM}\) represent the detection loss, feature loss, and hypergraph matching loss, respectively.