# Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching
## 1 Introduction  
Multiple Object Tracking (MOT) is an important task in the field of computer vision, aiming to stably track multiple objects of interest in a video sequence and generate their motion tracklets. It has been widely applied in autonomous driving, action recognition, mobile robotics, and intelligent surveillance. Depending on whether or not the generation of motion tracklets depends on the information of subsequent frames, MOT tasks can be divided into two different modes: offline tracking and online tracking. Offline tracking uses all the detection results in a video segment to jointly infer the motion tracklets of objects in each frame, which can be seen as a global optimization problem, while online tracking generates new tracklets sequentially according to the current detection results and historical tracklets. Due to its ability to process input video sequences in real-time, online MOT is widely used in real-world scenarios. Early online MOT algorithms focused on complex association optimization methods, which had limited accuracy. With the rapid development of deep learning, online MOT algorithms based on deep learning have gradually become mainstream. At present, deep learning-based MOT methods are primarily divided into Detection-based-Embedding (DBE) [1, 2, 3, 4, 5, 6, 7, 8] and Joint Detection and Embedding (JDE) [9, 10, 11, 12, 13, 14]. DBE separates tracking into two independent tasks: object detection and data association. First, the detector outputs the detection results, which are then sent to the dataassociation module to match with the tracklets of the previous frame, thus forming new tracklet boxes. DBE, with its straightforward approach and excellent tracking accuracy, once became the mainstream paradigm for MOT. The object detection module and data association module are trained separately to achieve the best performance, but they usually have the disadvantage of not being able to propagate errors back through the entire MOT system. In other words, each module is optimized individually, rather than for the entire MOT system. Moreover, when there are a large number of objects in the detection frame, due to the lack of feature sharing between the two modules in the DBE paradigm, a Re-ID model is needed for each bounding box (Bbox) in the detection frame, leading to the disadvantage of not achieving real-time inference speed. JDE integrates the appearance embedding model used for data association into the detection module, so it can output detection results and corresponding appearance embeddings simultaneously. JDE uses a shared-weight, deep learning network to complete the tasks of object detection and data association, achieving good real-time performance. Figure 1a depicts the input data for an MOT task, which is a sequence of object tracking images where each frame contains multiple targets represented by rectangular boxes of different colors. Figure 1b is a schematic representation of the target tracklets abstracted from Figure 1a, where the circular dots of different colors correspond to the target rectangular boxes in Figure 1a, triangles represent similar targets, the gray square indicates the occurrence of an intersection between two targets at this time, causing visual occlusion, the red curve represents the correct tracklet of the target, and the blue curve represents the incorrect tracklet of the target. It can be seen from the figure that when there are similar targets nearby or severe occlusions occur, it is easy to accumulate errors or maintain visual consistency. If attention is paid to the higher-order interactions between multiple targets, as indicated by the purple part in the figure, it can effectively mitigate the impact of these situations.