# A Review of Deep Learning-Based Visual Multi-Object Tracking Algorithms for Autonomous Driving
## 6 MOT Algorithm Based on Transformer
### MOT Algorithm Based on Transformer Architecture  
DETR [15] is the first framework to introduce the transformer successfully into object detection. DETR uses CNN to extract image feature mappings, uses the extracted features as input to the transform encoder, and obtains detection results through the decoder of the transformer. TransTrack [87] introduced the attention mechanism into multi-object tracking for the first time. Firstly, the current frame image was input into the CNN backbone to extract the image feature maps and using the Query-Key mechanism for joint detection and tracking. This algorithm combines the features of two adjacent frames, object queries are used to learn to detect new objects, track queries are responsible for keeping the tracking track, one of the two decoders is responsible for detecting and generating a detection frame, and the other is responsible for generating a tracking frame for object propagation. Finally, the tracking frame and the detection frame are associated on the same frame using the IOU matching strategy to complete the multi-object tracking task. The algorithm first works to solve the MOT task in such a paradigm. It provides a novel perspective for multiple object tracking. Meinhardt et al. [88] proposed the TrackFormer algorithm, which treats multi-object tracking as a set of prediction problems with joint detection and tracking by attention. Similarly, based on the Query-Key mechanism, the architecture consists of a CNN for image feature extraction, a transformer encoder, and a transformer decoder for image coding. The image feature maps extracted using the CNN are encoded by self-attention features in the encoder and the queries are decoded by self-attention and cross-attention in the decoder, resulting in an output embedding with bounding box and class information. The incorporation of the attention mechanism ensures that the model simultaneously considers location, occlusion, and object recognition features and performs well in long tracking scenarios relative to TransTrack when tested under the dataset. Xu et al. [89] concluded that the bounding box-based tracking methods of both algorithms, TransTrack and TrackFormer, are not suitable for dealing with dense scenarios, and therefore proposed the TransCenter algorithm, which uses a dense multi-scale query to obtain a centroid-based representation of the object heat map and a two-dimensional Gaussian distribution to represent the position and size of the object, and in addition, considering the memory loss, proposed a variable decoder, and finally, combine the central heat map and a bidirectional decoding structure with geometric features and visual features of the decoder through the Hungarian algorithm for object association. Figure 6: The transformer’s encoder–decoder architecture. (Source: Data from DETR [15]). Chu et al. [90] proposed the TransMOT algorithm, which used the YOLOv5 algorithm as the detector model and the SiamFC network as the visual feature extraction sub-network, and then innovatively proposed a spatial and temporal graph transformer to model multiple objects in space and time, to achieve the purpose of long-term tracking. Specifically, the authors construct a series of sparse weighted graphs based on the spatial relationship of the object. Based on these weighted graphs, a spatial graph attention encoder, a temporal graph attention encoder, and a spatial attention decoder are established for modelling, a cascade association mechanism is also established to deal with low-confidence detections and long-term occlusions, to further optimize the speed and accuracy of TransMOT. Xie et al. [91] proposed a Siamese-like Dual-Branch network based solely on transformers (DualTFR) for visual object tracking in 2021. Dual branches are templates and search images, respectively, divided into non-overlapping facets and extracted a feature vector for each facet based on the results of matching each facet with the others within the attention window.