Multiple object tracking based on multi‐task learning with strip attention

IET Image ProcessingVolume 15, Issue 14 p. 3661-3673 ORIGINAL RESEARCH PAPEROpen Access Multiple object tracking based on multi-task learning with strip attention Yaoye Song, Yaoye Song orcid.org/0000-0003-2410-7729 School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorPeng Zhang, Corresponding Author Peng Zhang zh0036ng@nwpu.edu.cn School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. China corresponding author contributes the same as the first author Correspondence Peng Zhang, School of Computer Science, Northwestern Polytechnical University Department, Xi'an, China P.R. Email: zh0036ng@nwpu.edu.cnSearch for more papers by this authorWei Huang, Wei Huang School of Information Engineering, Nanchang University, Nanchang, P.R. ChinaSearch for more papers by this authorYufei Zha, Yufei Zha orcid.org/0000-0001-5013-2501 School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorTao You, Tao You School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorYanning Zhang, Yanning Zhang School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this author Yaoye Song, Yaoye Song orcid.org/0000-0003-2410-7729 School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorPeng Zhang, Corresponding Author Peng Zhang zh0036ng@nwpu.edu.cn School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. China corresponding author contributes the same as the first author Correspondence Peng Zhang, School of Computer Science, Northwestern Polytechnical University Department, Xi'an, China P.R. Email: zh0036ng@nwpu.edu.cnSearch for more papers by this authorWei Huang, Wei Huang School of Information Engineering, Nanchang University, Nanchang, P.R. ChinaSearch for more papers by this authorYufei Zha, Yufei Zha orcid.org/0000-0001-5013-2501 School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorTao You, Tao You School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this authorYanning Zhang, Yanning Zhang School of Computer Science, Northwestern Polytechnical University Department, Xi'an, P.R. ChinaSearch for more papers by this author First published: 09 November 2021 https://doi.org/10.1049/ipr2.12327 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Multiple object tracking (MOT) framework based on bifurcate strategy was usually challenged by data association of different model path, which work for object localisation and appearance embedding independently. By incorporating the re-identification (re-ID) as appearance embedding model, more recent studies on task combination of a single network have made a great progress in tracking performance. Unfortunately, the contributive improvement from re-ID model is hard to balance the accuracy and efficiency for the whole framework. For more effective enhancement of the overall tracking performance, a real-time detection needs to be taken into consideration with other auxiliary means for MOT modelling. Therefore, in this study, a one-shot multiple object tracking is proposed based on multi-task learning to obtain satisfactory performance in both speed and robustness. With updated re-training strategy for the backbone model of detection, a D2LA network is proposed to achieve more characteristic fine-grained feature extraction in branching task of pedestrian recognition. Additionally, a strip attention module is also introduced to further strengthen the feature discriminative capability of the tracking framework in occlusion. Experiments on the 2DMOT15, MOT16, MOT17, and MOT20 benchmark data sets have shown a superior performance in comparison to other state-of-the-art tracking approaches. 1 INTRODUCTION The main task of multiple object tracking (MOT) is to locate more than one interested targets in video to maintain their ID as well as to record their trajectory simultaneously. These targets can be pedestrians on the road, vehicles or multiple groups of animals, and even different parts of a specific single object. Most of the proposed multiple object tracking approaches can be divided into two steps by training the models of object detection [1-4] and re-identification (re-ID) [5], respectively. Firstly, the detection model determines the locations of interested objects with bounding boxed in each video frames, then the association model extracts re-ID features for each bounding box and employs the certain metrics between features to connect with the existing trackers. The limitation of such a strategy is that, if there are a lot of objects being appeared in the video, the error of the features association between two models are hard to be avoided because of the intrinsic challenge of sharing the obtained features in models. Meanwhile, to ensure a good performance of joint models, more complicated networks structure need to be utilised, which means much more inference time and the impossible real-time framework. To overcome above limitations, one-shot trackers which combine object detection module and appearance embedding module have been proposed. For instance, TrackR-CNN tracker [6] added a re-ID branch to the Faster R-CNN detector [2], it is able to share the re-ID feature and detection feature in ROI Pooling module. As shown in Figure 1, Faster R-CNN is a two-stage object detector, this framework starts with the generation of feature map by ResNet [7] or VGGnet [8]. Then region proposal network (RPN) of the first stage extracts the features with a fixed number of anchors and make classification and regression for those anchors. The target of the region proposal network is to obtain rough candidate bounding box of objects. The second stage is the ROI Align module which uses both the information of backbone and RPN, it can refine the position bounding boxes and classify the categories of objects. This strategy is able to achieve inference time reduction, but the tracking accuracy gets falling instead of rising in comparison with the two-step separated method, which demonstrates that the simply merging two branches does not certainly lead to positive effects. FIGURE 1Open in figure viewerPowerPoint Performance comparison between (a) Trackers of TrackR-CNN and (b) JDE and (c) FairMOT. The red boxes represent positive anchors and the green boxes are the ground truth. (a) TrackR-CNN tracker employs a two-stage anchor-based detection framework with a re-ID branch at the ROI Pooling module. In re-ID feature extraction, the obtained positive anchor may contain two different instances as shown on the right side of (a). (b) One-stage anchor-based detection framework is employed in JDE, which also has a re-ID branch following the backbone network, but multiple anchors with different image patches may still predict the same instance. (c) FairMOT builds up a one-stage anchor-free detection framework based on the points of objects. In the scenarios of two pedestrians coming too close, the discriminative capability of feature extraction would not be sufficient enough Unlike TrackR-CNN tracker [6], JDE tracker [9] is another one-shot tracker using YOLOv3 [3] as the detection branch (shown in Figure 1). Based on one-stage anchor-based detection model, JDE tracker has achieved a higher efficiency and accuracy than the typical two-stage method like Faster R-CNN [2]. One factor of the improvement is via updating the backbone network with DarkNet in YOLO v3 [3], which enables the process of classification and bounding box regression directly following the feature extraction. Besides using advanced detector, more contributive improvement of JDE tracker is introduced by paralleling the re-ID branch with the detection branch. This means that two branches can share the same set of low-level features to effectively save more inference time. For both of the discussed anchor-based methods above as shown in Figure 1, limitations are also obvious. The extracted anchor by TrackR-CNN might have multiple identities because most sampling locations in ROI-Pooling are possible to be the objects or background. Different from TrackR-CNN tracker, the JDE tracker rarely has the same problem, but one-stage anchor-based detector is usually prone to have multiple anchors corresponding to one identity. To handle these challenges, FairMOT [10] employed the CenterNet [4] as the detection branch by thinking of objects as points. An important representative work of one-stage anchor-free based tracking strategies, FairMOT tracker has greatly ameliorated the problems appeared in TrackR-CNN and JDE tracking. Figure 1 also shows that the backbone network in FairMOT is running with Deep Layer Aggregation (DLA) for feature extraction, and the function of DLA is to combine hierarchical deep aggregation (HDA) module and iterative deep aggregation (IDA). Thus, the low-level and high-level information for object representation can be used by HDA more characteristically, and further enables the final output to be obtained by the IDA with its iteration approach. The main differences between these real-time MOT frameworks are the variations of detectors and backbone networks. However, if a more complex detection model is incorporated, the enhancement of final output accuracy would be hard to guarantee the same expense of inference time. More promising way is to introduce a concise detector into the MOT framework to balance the performance improvement between accuracy and efficiency. Therefore, in this study, FairMOT is chosen as the baseline model for the proposed MOT framework, in which the backbone network has been changed from DLA34 to DLA60 owing to its deeper layer structure. By using bottleneck module instead of basic block module, this design is capable of pulling less parameters and free a considerable inference time. Further inspired by Res2Net, we replace the convolution layer of bottleneck and retrained the CenterNet detector on COCO [11] dataset. For the challenge of occlusion, which frequently happens in the crowded scenario containing pedestrians, the original backbone network easily fail to distinguish the specific target due to the similar shape of human. Therefore, to enhance the model discriminability among the pedestrian targets, a strip attention module is also proposed by training on the CrowdHuman [12] data set with self-supervised learning scheme [13]. In addition, the pre-trained self-surpervised learning tracking model also needs to be fine-tuned on a mixed data sets including Caltech [14], CityPersons [15], CUHK-SYSU [16], PRW [17], ETH [18], MOT17 [19] dataset. The main contributions of the proposed work can be summarised as follows: (1) a backbone update strategy based on DLA60 is proposed for detection model retraining; (2) a structure improvement D2LA60 has been proposed by incorporating a Res2net [20] based bottleneck convolutional layer to obtain better results; (3) a strip attention module has been introduced into the D2LA60 to improve the discriminative capability of the pedestrians targets; (4) a mixed-data sets training scheme is proposed to achieve state-of-the-art performance. The whole structure of our paper can be summarised as follows: (1) The related works of multiple tracking frameworks and different backbone networks on each tracking frameworks; (2) The proposed method which combines the backbone network, strip attention module, detection and Re-ID branch and the data association; (3) The experiments results which is based on our method and the baseline method. (5) The conclusion of our method. 2 RELATED WORKS As a preliminary knowledge, three typical kinds of multiple object tracking strategies need to be reviewed. (1) The online tracking based on the assumption that the incorporated detection is always available to pay attention to the data association step [21-25]. (2) Batch-based trackers that are able to optimise the association problem by building graphical model [26-31]. (3) The deep learning-based MOTs [32-35], which is the main motivation of this study. With a brief description and comparison, different kinds of backbone networks appeared in some representative MOTs are discussed. 2.1 Multiple object tracking frameworks To directly incorporate the deep learning–based detection into the online tracking framework, Refs [6, 36-40] replace the original detector via separating object detection and re-ID as two tasks. During initialisation, the object detector is utilised to localise all the objects in the current frames, and the obtained objects are cropped into the bounding box to extract re-ID features using an identity embedding network, which are used to link the boxes over time. The main advantage of such a separation mechanism is that it can acquire the most suitable model for each step, as well as to easily replace a better detector without the pre-training of the detection model. This is also the reason why those tracking strategies have a huge leap in performance compared with the previous online trackers without deep learning, but the real-time declamation is not practical in reality due to the pre-requisite of the detection results. 2.2 Backbone network Another type of one-shot multiple object tracking strategies have been proposed to solve the real-time problem. One promising way inspired by [9, 10] is to simultaneously conduct the object detection and extract re-ID features in a single network to achieve significant reduction of inference time, JDE [9], FairMOT [10], which have been discussed in the section of introduction. More recent studies in MOT have shown that the backbone network plays an important role in the tracking framework for performance improvement. Especially to the one-shot trackers, such as FairMOT [10], the designing of the backbone network has been taken into more considerations as an inspiration of the proposed work. 2.2.1 ResNet The structure investigations have shown that the depth of the network is a critical factor which substantially influences the model performance. Usually, the deeper the network, the more information can be obtained [8, 41]. However, gradient dispersion or explosion becomes an obstacle for training deep network, which leads to the failure of convergence. A normal operation to solve this problem is to normalise the input data and the data in middle layer. This way enables the network to adopt stochastic gradient descent [42] in the back propagation for network convergence. But when network goes deeper, the effect of this method gradually declines. To get better recognition accuracy with deeper network structure, ResNet [7] proposed a residual block module to learn the residual between input and output using multiple network layers, shown in Figure 2(a), and it has been widely used in the downstream vision tasks owing to its remarkable progress in image recognition. FIGURE 2Open in figure viewerPowerPoint Three typical backbone network. (a) The ResNet34 structure, the right part shows the residual block module which is the core of ResNet; (b) the Darknet53 structure, which is consisted of 53 convolution layers and has a residual block module as ResNet; (c) DLANet is a deep layer aggregation network 2.2.2 DarkNet As the backbone network of Yolov3 [3], Darknet53 has a big difference in the residual module of ResNet. In order to make better use of the feature information, Darknet53 deepens to 53 layers in comparison to the 19 layers used in Darknet19 [43]. With such a modification, Yolov3 is able to extract the output feature of C0, C1, C2 layers by striding 32, 16, 8 across those layers as shown in Figure 2(b). The most important contribution of this backbone network is the beneficial in small object detection. 2.2.3 DLANet Both of ResNet and DarkNet employed an aggregation approach of skipping connection to aggregate within the network. But such an aggregation has only been limited within the block and simple superposition. To overcome this limitation, a new structure was proposed by DLANet [44] which contains two core modules, IDA (Iterative Deep Aggregation) and HDA (Hierarchical Deep Aggregation). As shown in Figure 2(c), the red box represents the hierarchical deep aggregation module, it is linked by a tree structure which has a better propagative capability of feature and gradient information. The yellow link represents iterative deep aggregation module, which is responsible for linking the features of two adjacent stages, so that the high-level and low-level features can be fused more effectively. The blue line represents the process of downsampling. Same as ResNet, the DLANet also performs the fast downsampling in the beginning, but fewer parameters bring a faster inference speed than ResNet. Therefore, the proposed work is motivated by DLANet in model structure improvement for MOT in challenging pedestrian crowded scenarios. 3 PROPOSED METHOD In this section, the overview of the proposed framework for multiple object tracking is firstly introduced. Then, we deliberatively present a more efficient backbone network based on DLANet, and followed by the detection branch and re-ID branch as well as jointly training loss. Finally how to make data association with detections and re-ID features is explained. 3.1 Framework overview Motivated by FairMOT, the proposed framework is based on one-shot tracking strategy with multi-task learning, which is consisted of two homogeneous branches of object detection and re-ID. As shown in Figure 3, a strip attention module is incorporated into the DLA60 network as the backbone network. The detection branch employs the anchor-free mechanism to represent the object as a single point (the centre of its bounding box), which can be performed more efficiently compared with anchor-based detectors. In parallel with detection branch, the re-ID branch directly utilises features extracted from backbone network for objects distinction. FIGURE 3Open in figure viewerPowerPoint The overview of the proposed tracking framework. The D2LA60-based backbone network is improved with an encoder–decoder scheme, which is shown in the bottom-left of the figure. In the top right of the figure, the object detection branch is presented to show the combined heatmap, box size, and centre offset. How the Re-ID embedding branch perform feature extraction is shown in the bottom right of the figure 3.2 Backbone network 3.2.1 DLA60 network With the development of deep learning network with residual module and batch normalisation, such as ResNet and DLANet, we can find that when the depth of network is deeper, the discrimination ability of the overall model will also be improved. Meanwhile, if the classification ability of backbone network is improved, its downstream detection and tracking tasks will be improved accordingly. To achieve more efficient feature extraction, the proposed backbone network is based on DLA60 instead of DLA34, especially the bottleneck in DLA60 is proposed to design the network flexibly and reduce the amount of computation. Each basic block has two 3 × 3 convolutional layers while bottleneck has three convolutional layers 1 × 1 , 3 × 3 , and 1 × 1 , respectively. The first 1 × 1 layer is for dimensions reduction, the second 1 × 1 layer is responsible for dimension restoration. These two 1 × 1 convolutional layers make the 3 × 3 layer bottleneck with smaller input and output dimensions. The basic block–based DLA34 gain accuracy by increasing depth, but is not as economical as the bottleneck DLANet. Another improvement from DLA34 to DLA60 is the channels in each stage. As shown in Table 1, the channels of DLA60 on stages 3–6 enable the DLA60 to achieve deeper layer and wilder channel without more extra parameters, this makes this bottleneck design to be more practical. TABLE 1. Comparison between DLA34 and DLA60, the second column represents the different types of block. The last six columns represent the number of blocks and the output dimension in each stage of the network Name Block Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 DLA-34 Basicblock 16 32 1–64 2-128 2–256 1–512 DLA-60 Bottleneck 16 32 1–128 2-256 3–512 1–1024 3.2.2 D2LA network As described in Section 2, the efficiency of multi-scale ability was subsequently improved by using residual modules (e.g. ResNet and DarkNet), and hierarchical layer aggregation (e.g. DLA). These methods only enhanced the representative strength of network with layer-wise multi-scale analysis, which also need to be performed at a more granular level due to the subtle difference between pedestrians in crowded scenarios. Inspired by Res2Net [20], as shown in Figure 4, we replace the 3 × 3 filters which is the middle layer of bottleneck of n channels, with s smaller filter groups, each with w ( w = n ÷ s ) channels. These smaller filter groups are connected in a hierarchical and residual style to increase the number of scales that the output features can represent. This design improves the ability of feature extraction without increasing parameters of model, which is a very cost-effective method. FIGURE 4Open in figure viewerPowerPoint The structure of D2LA 3.2.3 Strip attention module More aggregated and deeper backbone network is supposed to enhance framework performance, especially for tracking the pedestrians in the congest crowds. However, the occlusions that frequently occur between the pedestrians or backgrounds always challenge the robustness of the trackers. By taking full advantage of long-range context, feature extraction can be performed more effectively when pedestrians were blocked. In this work, the strip pooling [45] is introduced to encode the long-range context efficiently along either the horizontal or vertical spatial dimension. To elaborate the strip pooling, more commonly used mean pooling and max pooling need to be addressed. As in the convolutional neural networks, mean pooling can preserve the background well, but may cause the unpredicted picture blur. Max pooling can keep the information of the main features and maintain a certain invariance(translation, rotation and scale), but may lost the features of certain locations. In both, the pooling layer get x ∈ R H × W to be a two-dimensional input tensor, where H and W are the height and width. Besides, a spatial extent of the pooling ( h × w , h > 0 , w > 0 ) is required. Consider a simple case where H ′ = H h , ( h > 0 ) and W ′ = W w , ( w > 0 ) , the output y ∈ R H ′ × W ′ after pooling can be obtained by following that the mean pooling takes the average value and max pooling takes the max value within each ( h × w , h > 0 , w > 0 ) extend. Differently, the strip pooling uses a band shape pooling window to perform pooling along either the horizontal or the vertical dimension, which is able to average all the feature values in row or a column. Thus, the spatial extent of pooling become ( H , 1 ) or ( 1 , W ) , and the output y h ∈ R H after horizontal strip pooling can be written as e.g. (1) y i h = 1 W ∑ 0 ≤ j < W x i , j (1) The output y w ∈ R W after vertical strip pooling can be written as e.g. (2) y i w = 1 H ∑ 0 ≤ j < H x i , j (2) To avoid the structural expanding and time increasing of the model training, strip pooling is encapsulated with an attention mechanism. This strip attention module enables the backbone networks to pay more attention to the long-range information when pedestrians are occluded, which also can make the features extraction ability more robust. Figure 5 shows that in the strip attention module, x ∈ R C × H × W is set as an input tensor, where C represent the number of channels. x is fed into two paralleled pathways by a horizontal or vertical strip pooling layer. Then the output of pooling layer is followed by a one-dimensional convolutional layer with a kernel size of 3, in order to modulate the current position and its adjacent features. This operation gives two strip feature y h ∈ R C × H and y w ∈ R C × W . In addition, we further combined y h and y w using (3) to obtain y ∈ R C × H × W for capturing more useful global information. y c , i , j = y c , i h + y c , j w (3) FIGURE 5Open in figure viewerPowerPoint The principle of strip attention module and the output of strip attention module z ∈ R C × H × W is calculated as z = Scale ( x , σ ( f ( y ) ) ) (4)where Scale ( · , · ) is defined as the element-wise multiplication, σ is the sigmoid function and f is a 1 × 1 convolution. The proposed module of strip pooling has been demonstrated to be light weighted and efficient, and specifically effective after conv3 layer of each bottleneck in stage 5. The input tensor of strip attention is R 1024 × H × W , which is used to perform an element-wise multiplication between the output of strip attention and the output of bottleneck. Compared to the other attention module such as [46-54], the proposed module can save a large amount of computation to build the relationships between each pair of location, and easily embedded into backbone networks as well. 3.3 Detection and Re-ID branch Consider to balance the accuracy and efficiency of the one-stage anchor-free approaches [55-61], our detection branch is designed based on the CenterNet instead of only taking care of the accuracy like FCOS [59] and RepPoints [61]. Particularly, three parallel heads are appended to the proposed backbone network for the estimation of heatmaps, object centre offsets and bounding box sizes, respectively. 3.3.1 Heatmap head In the proposed work, the heatmap head is used to estimate the centre of object. Elaboratively, when the response at some location of the heatmap is overlapping with object centre of ground truth, the score of this location is 1. Otherwise, the response declines exponentially in accordance with the distance between the heatmap locations and the object centre. For each ground truth box b i = ( x 1 i , y 1 i , x 2 i , y 2 i ) , the object centre is formulated as c y i = c x i 2 and c y i = c y i 2 by ( x c i , c x i ) . In the feature map, the coordination of object centre is divided by the stride ( c x i ∼ , c y i ∼ ) = ( ⌊ c x i R ⌋ ⌊ c y i R ⌋ ) to downsample the output prediction. Then, each bounding-box in the ground truth can be described as a distribution on the heatmap by Gaussian kernel Y x y = exp ( − ( x − c ∼ x i ) 2 + ( y − c ∼ y i ) 2 2 σ c 2 ) , where the σ c represents an object size-adaptive standard deviation [55]. The process is a penalty-reduced pixel-wise logistic regression with focal loss [62] as L h m = − 1 N ∑ x y ( 1 − Y ̂ x y ) α log ( Y ̂ x y ) if Y x y = 1 ( 1 − Y x y ) β ( Y ̂ x y ) α log ( 1 − Y ̂ x y ) otherwise (5)where Y ̂ x y is the predicted heatmap, and parameters α = 2 and β = 4 are fixed through all the experiments. 3.3.2 Box offset The box offset is used for recovering the error of discretisation caused by the output stride. And the ground truth offset is computed as o i = ( c x i R , c y i R ) − ( ⌊ c x i R ⌋ , ⌊ c y i R ⌋ ) , and o ̂ i denotes the predicted offset, which is learned with an L1 loss: L off = ∑ i = 1 N o i − o ̂ i 1 . (6) 3.3.3 Box size The head of box size is designed to estimate height and width of the box as s i = ( x 2 i − x 1 i , y 2 i − y 1 i ) , which is also learned with L1 loss as L size = ∑ i = 1 N s i − s ̂ i 1 . (7) 3.3.4 Re-identification The Re-ID branch is for distinguishing the objects by feature extraction. By applying a convolutional layer with 128 kernels on the top of backbone features, the Re-ID feature is the centre of object extracted from the feature map, and is mapped to a class distribution vector P = p ( k ) , k ∈ [ 1 , K ] . Suppose the class label of ground truth is represented using the one-hot coding as L i , the loss of Re-ID can be defined: L i d = − ∑ i = 1 N ∑ k = 1 K L i ( k ) log ( p ( k ) ) . (8) The branches are jointly trained with uncertainty loss [63] to automatically balance the detection and the Re-ID tasks: L detection = L h m + L o f f + L size (9) L total = 1 2 1 e ω 1 L detection + 1 e ω 2 L i d + ω 1 + ω 1 (10) The ω 1 and ω 2 are the learnable parameters that balance the two branch. 3.4 Data association When the joint training model is ready, the data association between the detection a