# A Review of Deep Learning-Based Visual Multi-Object Tracking Algorithms for Autonomous Driving
## 3 Overview of Deep Learning-Based Visual Multi-Object Tracking
### MOT Datasets  
Deep learning has the benefit over more conventional machine learning techniques in that it can automatically identify data attributes that are pertinent to a specific task. For deep learning-based computer vision algorithms, data sets are crucial. The datasets and traits that are frequently utilized in the field of automatic driving tracking are outlined in the following. Due to their frequent updates and closer resemblance to the actual scene, the MOT datasets [25; 26] raise the most concerns in the field of visual multi-object tracking. On the MOT dataset, other cutting-edge tracking methods are also tested. The MOT16 dataset [25] is used exclusively for tracking pedestrians. There are a total of 14 videos, 7 practice sets, and 7 test sets. These videos were created using a variety of techniques, including fixed and moving cameras, as well as various shooting perspectives. Additionally, the shooting circumstances vary, depending on whether it is day or night and the weather. The MOT16 detector, called DPM, performs better in the area of detecting pedestrians. The video content of the MOT17 dataset [25] is the same as that of MOT16, but it also provides two detector detection results, namely SDP and Faster R-CNN, which have more accurate ground-truth annotations. The MOT20 dataset [26] has 8 video sequences, 4 training sets and 4 testing sets, and the pedestrian density is further increased, with an average of 246 pedestrians per frame. The KITTI dataset [27; 28] is currently the largest dataset for evaluating computer vision algorithms in autonomous driving scenarios. These data are used to evaluate 3D object detection and tracking, visual odometry, evaluation of stereo images, and optical flow images. The NuScenes dataset [29] provides a large dataset of full sensor data for autonomous vehicles, including six cameras, one lidar, five radars, as well as GPS and IMU. Compared with the KITTI dataset, it includes more than seven times more object annotations. For each scene, its key frames are selected for annotation, and the annotation rate is 2 Hz. However, it is worth noting that since 23 types of objects are marked, the class- The imbalance problem will be more serious. The Waymo dataset [30] is collected with five LiDAR and five high-resolution pinhole cameras. The entire dataset contains 1150 scenes, which are divided into 1000 training sets \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Tracking Algorithm** & **Principle** & **Advantage** & **Disadvantage** \\ \hline \multirow{4}{*}{TBD} & All objects of interest are & & \\  & detected in each frame of the & & \\  & video, and then they are & & \\  & associated with the detected & & \\  & objects in the previous frame & & \\  & to achieve the effect & & \\  & of tracking & & \\ \hline \multirow{4}{*}{JDT} & End-to-end trainable detection & \multirow{4}{*}{Multi-module joint learning, weight sharing} & Local receptive field, when the \\  & box paradigm to jointly learn & & & object is occluded, the \\  & detection and & & \\  & appearance features & & \\ \hline \multirow{4}{*}{Transformer-based} & \multirow{4}{*}{Transformer encoder-decoder} & Parallel computing; rich & \multirow{4}{*}{The parameters are too large} \\  & Transformer encoder-decoder & & \\  & architecture to obtain global & & \\  & and rich contextual & & \\  & interdependencies for tracking & & \\  & & & \\ \cline{1-1}  & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of characteristics of three types of visual multi-object tracking algorithms. and 150 test sets, with a total of about 12 million LiDAR annotation boxes and approx. 12 million image annotation boxes. The Mapillary Traffic sign dataset [31] is the largest and most diverse traffic sign dataset in the world, which can be used for research on the automatic detection and classification of traffic signs in autonomous driving.