# Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching
## 2 Related Works
### Joint Detection and Embedding
In order to learn the appearance and motion features of the tracklets across multiple frames, model the temporal characteristics, and predict the tracklet direction, some researchers use a multi-frame network or a head aggregation network in the form of joint detection and tracking to learn spatiotemporal feature maps for tracking. For example, CTrack [44] follows the CenterNet anchor-free detection framework, splicing a pair of sequential frames and the previous frame's heatmap at the input end and predicting the object center position, size, offset, and inter-frame object displacement at the output end, which can achieve two-dimensional and three-dimensional MOT at the same time. Transcenter [45] also embeds Transformer into MOT, processing dense pixel-level, multi-scale detection and tracking queries in two query learning networks based on a variable-shaped Transformer encoder and decoder to obtain detection and predicted displacement. JDE's data association and object detection are carried out simultaneously, using a single neural network to process the input image and directly infer the tracklet of the object. The object location can be refined through the correlation between the tracklet features and the current feature map. Some researchers have integrated single-object tracking algorithms into multi-object algorithms [46], using the powerful feature extraction and positioning capabilities of single-object algorithms to achieve higher precision end-to-end tracking. SOTMOT [47] is built on a variant of DLA-34, solving the ridge regression coefficients of the previous frame for each object and using the learned coefficients to predict the labels of the next frame. Therefore, discriminative features can be learned in the local area. The disadvantage is that as the number of objects in the scene increases, assigning a single-object tracker to each object will cause efficiency problems. Recently, with the progress of Transformer [48], some researchers have also used the query vector mechanism in the Transformer model for MOT, to learn the tracklet features with the query vector for frame-by-frame positioning and object identification, implicitly completing data association, thereby simplifying the complex and multi-step architecture. [PERSON] et al. [49] were the first to introduce Transformer into MOT. They iterated the DETR (end-to-end object detection with transformers) detector [50] in the time direction. The first frame uses randomly initialized empty queries to detect objects in the image and output the corresponding queries for the objects. In subsequent frames, the object queries from the previous frame and the randomly initialized empty queries are input together into the decoder, where the object queries are responsible for detecting the position of the tracklet in the current frame and passing the identity, implicitly completing data association, while the empty queries are responsible for detecting newly appearing objects. The joint detection and tracking algorithms integrate different modules with the detection network for joint optimization, expecting the modules of the algorithm to work more collaboratively and simplifying the complex multi-step form, thereby improving the speed of MOT, which has received widespread attention in recent years.