# Multiobject Tracking in Videos Based on LSTM and Deep Reinforcement Learning

Authors: Ming-xin Jiang; Chao Deng; Zhi-geng Pan; Lan-fang Wang; Xing Sun
Journal: Complexity
 (2018)

---

## Abstract

Multiple-object tracking is a challenging issue in the computer vision community. In this paper, we propose a multiobject tracking algorithm in videos based on long short-term memory (LSTM) and deep reinforcement learning. Firstly, the multiple objects are detected by the object detector YOLO V2. Secondly, the problem of single-object tracking is considered as a Markov decision process (MDP) since this setting provides a formal strategy to model an agent that makes sequence decisions. The single-object tracker is composed of a network that includes a CNN followed by an LSTM unit. Each tracker, regarded as an agent, is trained by utilizing deep reinforcement learning. Finally, we conduct a data association using LSTM for each frame between the results of the object detector and the results of single-object trackers. From the experimental results, we can see that our tracker achieves better performance than the other state-of-the-art methods. Multiple targets can be steadily tracked even when frequent occlusions, similar appearances, and scale changes happened.


---

## Body

## 1. Introduction

Multiobject tracking in videos plays an important role in a wide range of applications, for example, video surveillance, robot navigation, intelligent transportation systems, and video analysis, to name a few [1, 2]. Despite that the field has made tremendous progress since early work, visual multiobject tracking is still regarded as a challenging problem due to frequent occlusions, appearance similarity between objects, varying number of objects, and environmental noise within measurements [3, 4].


### 1.1. Related Work

Tracking-by-detection methods [5–7] have appeared as one of the most successful strategies due to recent advances in methods for object detection [8–10]. Most of the recent tracking-by-detection algorithms aim at decomposing multiobject tracking into two stages: object detection and data association. These algorithms apply the object detector in each frame and associate the results of the detector continuously. Therefore, this kind of multiobject tracking method can recognize emerging or disappearing objects in the sequences of a video more easily, and the search space of object hypothesis can be greatly reduced.Tracking-by-detection methods are frequently classified roughly into two categories: offline approaches and online approaches. Offline approaches often use the detections of all the frames of the video sequence together to build long trajectories against false detections and occlusion. A crowded or cluttered scene usually causes some detection failures, which will decrease the accuracy of data association in turn. To compensate for these problems, many multiobject tracking algorithms using the global data association have been proposed [11–14]. However, the performance of the offline approaches is still limited, and it is hard to apply the offline approaches to real-time applications. As data associations between detections and trackers for each frame are performed in an online manner, we can apply the online methods to real-time applications. Bae and Yoon [15] proposed a novel online visual multiobject tracking approach that can handle the similarity between multiple objects.Data association is the major issue of tracking-by-detection methods [16]. Classical data association approaches include the joint probabilistic data association filter (JPDAF) and multihypotheses tracking (MHT) [17]. JPDAFs consider all possible associations between objects to make the best assignment in each time step. MHT considers multiple possible associations over several time steps, but its application can be usually limited due to its complexity. Many recent multiobject tracking algorithms have concentrated on enhancing the performance of the object detector or designing better data association schemes [18–20].In recent years, LSTM has attracted increasing attention in modeling sequential data. The applications cover feature selection [21], machine translation [22], action recognition [23], video captioning [24], and human trajectory prediction [25]. The main advantages of LSTMs for modeling sequential data is that they allow end-to-end fine-tuning and they are not confined to fixed-length inputs to outputs. Inspired by the successful works that have applied LSTM in computer vision fields, we adopt a data association method based on LSTM in this paper. LSTM includes nonlinear transformations and memory cells, which makes it effective for data association.Most previous multiobject tracking methods represent objects using raw pixel and low-level handcrafted features, such as histograms of oriented gradients (HOG) [26], Harr-like features [27], and local binary patterns (LBP) [28]. Although they achieve computational efficiency, they have many limits because handcrafted features cannot capture more complex characteristics of the objects. Recently, deep learning has received much attention with state-of-the-art results in complicated tasks such as object detection [29], image classification [30], object recognition [31], and object tracking [32]. A deep-learning tracker (DLT) was proposed in [33], which uses a stacked denoising autoencoder to learn the generic features from a large number of auxiliary images offline. However, the DLT tracker cannot describe the temporal invariance of deep features, which is important for visual object tracking. In [34], a deep-learning tracking method was developed that uses a two-layer convolutional neural network (CNN) to learn hierarchical features from auxiliary video sequences; in the visual tracking method, appearance variations and complicated motion transformations of objects are taken into account. In [35], the authors present a visual tracking algorithm, which includes a specific feature extractor with CNNs from an offline training set; both spatial and temporal features can be learned by the CNNs jointly from image pairs of two adjacent frames. These deep-learning trackers often overlook how to search the interesting region of objects and select the best candidate as the tracking result.With the recent exciting achievements of deep learning, integrating deep-learning methods with RL has recently shown very promising results on decision-making problems, that is, deep reinforcement learning (DRL). Deep neural networks are able to make reinforcement learning algorithms perform more effectively because they can provide deep feature representations. DRL algorithms have achieved unequalled success in many challenging domains, for example, Atari games [36] and playing board game GO [37]. In the computer vision community, there are also many attempts of applying DRL to solve traditional tasks, such as action recognition [38], object localization [39], object tracking [40], and region proposal [41]. Yun et al. propose an end-to-end active object tracking algorithm via reinforcement learning, which addresses tracking and camera control simultaneously [42]. In [43], the authors present action-decision networks for visual tracking with deep reinforcement learning. However, these tracking methods based on deep reinforcement learning usually focus on a single object; there is little work related to multiobject tracking. Unlike the aforementioned methods, our method exploits how to apply deep reinforcement learning to solve the online multiobject tracking problem.


### 1.2. Summary of Contributions

Our motivation is to design a real-time multiple-object tracker via LSTM and DRL, which can incorporate appearance by DRL and learning a more effective association strategy by LSTM to improve the performance of tracking. The key contributions of this paper can be summarized as follows:(i)
We propose a novel visual multiobject tracking algorithm based on LSTM and deep reinforcement learning to solve the problems in the existing methods, which is model-free and requires no prior knowledge. To the best of our knowledge, we are the first to combine such concepts to overcome problems in the process of the visual multiobject tracking.
(ii)
The proposed multiobject tracker includes three modules: an object detection module, a number of single-object trackers, and a data association module. We adopt YOLO V2 as an object detector as it is a real-time detection system. Each single-object tracker is treated as an agent, which is trained using DRL. An LSTM-based architecture is adopted to solve the joint data association problem.
(iii)
To compare our multiobject tracker with other state-of-the-art methods qualitatively and quantitatively, we conducted extensive experiments on publicly available challenge benchmark datasets.The rest of our paper is structured as follows: Section2 reviews the background. Section 3 introduces the proposed multiobject tracking framework. Section 4 demonstrates the experimental results and analysis. Finally, we draw conclusions in Section 5.


## 1.1. Related Work

Tracking-by-detection methods [5–7] have appeared as one of the most successful strategies due to recent advances in methods for object detection [8–10]. Most of the recent tracking-by-detection algorithms aim at decomposing multiobject tracking into two stages: object detection and data association. These algorithms apply the object detector in each frame and associate the results of the detector continuously. Therefore, this kind of multiobject tracking method can recognize emerging or disappearing objects in the sequences of a video more easily, and the search space of object hypothesis can be greatly reduced.Tracking-by-detection methods are frequently classified roughly into two categories: offline approaches and online approaches. Offline approaches often use the detections of all the frames of the video sequence together to build long trajectories against false detections and occlusion. A crowded or cluttered scene usually causes some detection failures, which will decrease the accuracy of data association in turn. To compensate for these problems, many multiobject tracking algorithms using the global data association have been proposed [11–14]. However, the performance of the offline approaches is still limited, and it is hard to apply the offline approaches to real-time applications. As data associations between detections and trackers for each frame are performed in an online manner, we can apply the online methods to real-time applications. Bae and Yoon [15] proposed a novel online visual multiobject tracking approach that can handle the similarity between multiple objects.Data association is the major issue of tracking-by-detection methods [16]. Classical data association approaches include the joint probabilistic data association filter (JPDAF) and multihypotheses tracking (MHT) [17]. JPDAFs consider all possible associations between objects to make the best assignment in each time step. MHT considers multiple possible associations over several time steps, but its application can be usually limited due to its complexity. Many recent multiobject tracking algorithms have concentrated on enhancing the performance of the object detector or designing better data association schemes [18–20].In recent years, LSTM has attracted increasing attention in modeling sequential data. The applications cover feature selection [21], machine translation [22], action recognition [23], video captioning [24], and human trajectory prediction [25]. The main advantages of LSTMs for modeling sequential data is that they allow end-to-end fine-tuning and they are not confined to fixed-length inputs to outputs. Inspired by the successful works that have applied LSTM in computer vision fields, we adopt a data association method based on LSTM in this paper. LSTM includes nonlinear transformations and memory cells, which makes it effective for data association.Most previous multiobject tracking methods represent objects using raw pixel and low-level handcrafted features, such as histograms of oriented gradients (HOG) [26], Harr-like features [27], and local binary patterns (LBP) [28]. Although they achieve computational efficiency, they have many limits because handcrafted features cannot capture more complex characteristics of the objects. Recently, deep learning has received much attention with state-of-the-art results in complicated tasks such as object detection [29], image classification [30], object recognition [31], and object tracking [32]. A deep-learning tracker (DLT) was proposed in [33], which uses a stacked denoising autoencoder to learn the generic features from a large number of auxiliary images offline. However, the DLT tracker cannot describe the temporal invariance of deep features, which is important for visual object tracking. In [34], a deep-learning tracking method was developed that uses a two-layer convolutional neural network (CNN) to learn hierarchical features from auxiliary video sequences; in the visual tracking method, appearance variations and complicated motion transformations of objects are taken into account. In [35], the authors present a visual tracking algorithm, which includes a specific feature extractor with CNNs from an offline training set; both spatial and temporal features can be learned by the CNNs jointly from image pairs of two adjacent frames. These deep-learning trackers often overlook how to search the interesting region of objects and select the best candidate as the tracking result.With the recent exciting achievements of deep learning, integrating deep-learning methods with RL has recently shown very promising results on decision-making problems, that is, deep reinforcement learning (DRL). Deep neural networks are able to make reinforcement learning algorithms perform more effectively because they can provide deep feature representations. DRL algorithms have achieved unequalled success in many challenging domains, for example, Atari games [36] and playing board game GO [37]. In the computer vision community, there are also many attempts of applying DRL to solve traditional tasks, such as action recognition [38], object localization [39], object tracking [40], and region proposal [41]. Yun et al. propose an end-to-end active object tracking algorithm via reinforcement learning, which addresses tracking and camera control simultaneously [42]. In [43], the authors present action-decision networks for visual tracking with deep reinforcement learning. However, these tracking methods based on deep reinforcement learning usually focus on a single object; there is little work related to multiobject tracking. Unlike the aforementioned methods, our method exploits how to apply deep reinforcement learning to solve the online multiobject tracking problem.


## 1.2. Summary of Contributions

Our motivation is to design a real-time multiple-object tracker via LSTM and DRL, which can incorporate appearance by DRL and learning a more effective association strategy by LSTM to improve the performance of tracking. The key contributions of this paper can be summarized as follows:(i)
We propose a novel visual multiobject tracking algorithm based on LSTM and deep reinforcement learning to solve the problems in the existing methods, which is model-free and requires no prior knowledge. To the best of our knowledge, we are the first to combine such concepts to overcome problems in the process of the visual multiobject tracking.
(ii)
The proposed multiobject tracker includes three modules: an object detection module, a number of single-object trackers, and a data association module. We adopt YOLO V2 as an object detector as it is a real-time detection system. Each single-object tracker is treated as an agent, which is trained using DRL. An LSTM-based architecture is adopted to solve the joint data association problem.
(iii)
To compare our multiobject tracker with other state-of-the-art methods qualitatively and quantitatively, we conducted extensive experiments on publicly available challenge benchmark datasets.The rest of our paper is structured as follows: Section2 reviews the background. Section 3 introduces the proposed multiobject tracking framework. Section 4 demonstrates the experimental results and analysis. Finally, we draw conclusions in Section 5.


## 2. Background

### 2.1. Long Short-Term Memory (LSTM)

Traditional recurrent neural networks (RNNs) contain cyclic connections that make them a powerful tool to learn complex temporal dynamics, as shown in Figure1. The formulas that govern the computation happening in a RNN are as follows:
(1)ht=fUxt+Wht−1,yt=softmaxVht,where f is an element-wise nonlinearity function, xt and yt represent the input vector and the output vector at time step t, and ht∈ℝN is the hidden-layer vector with N hidden units at time step t. U, W, and V are the weight matrices of the connection from input nodes to hidden nodes, hidden nodes to hidden nodes, and hidden nodes to output nodes.Figure 1
Recurrent neural networks.
(a)
RNN architecture
(b)
RNN unitThough RNNs have been successfully used for sequence modeling tasks, they can only model the data within a fixed-size window. At the same time, training conventional RNNs is difficult due to the problem of exploding and vanishing gradients. These problems limit the capability of RNNs to learn long-term dynamics. LSTM was proposed in [44] to solve these problems. The LSTM unit is used in this paper as described in [45], as shown in Figure 2.Figure 2
Long short-term memory unit.In this subsection, we provide the equations of LSTM for a single memory unit only. Letx=x1,…,xT be an input sequence and y=y1,…,yT represent an output sequence; an LSTM network computes a mapping iteratively between x=x1,…,xT and y=y1,…,yT using the following equations:
(2)it=σWxixt+Whiht−1+Wcict−1+bi,ft=σWxfxt+Whfht−1+Wcfct−1+bf,ct=ft⊙ct−1+it⊙tanhWxcxt+Whcht−1+bc,ot=σWxoxt+Whoht−1+Wcoct+bo,ht=ot⊙tanhct,where σx=1/1+exp−x is the logistic sigmoid function, ct is the cell input activation vector, it describes the input gate, ft represents the forget gate, and ot output gate. All of the above are the same size as the hidden vector ht. That is, in addition to a hidden vector ht∈ℝN, the LSTM includes an input gate it∈ℝN, forget gate ft∈ℝN, output gate ot∈ℝN, and memory cell ct∈ℝN. We can find the meaning of the weight matrix; for example, Whi represents the hidden to input gate matrix and Wxo represents the input to output gate matrix. bi, bf, bo, and bc are the bias terms which are added to i, f, o, and c.


### 2.2. Deep Reinforcement Learning (DRL)

Reinforcement learning (RL) can usually be used to solve sequential decision-making problems. The process of reinforcement learning is shown in Figure3. Recently, significant progress has been made by combining reinforcement learning with the ability for learning feature representations in deep learning. Deep Q network (DQN) and policy gradient are two popular methods in DRL algorithms. DQN is a form of Q-learning with function approximation using a neural network, which means it tries to learn a state-action value function Q given by a neural network in DQN by minimizing temporal-difference errors. To improve performance and keep stability, various network architectures are based on the DQN algorithm such as dueling DQN [46] and double DQN [47].Figure 3
The typical framing of a reinforcement learning.A policy gradient approach is a type of reinforcement learning method that directly optimizes parametrized policies by using gradient descent [48]. Policy gradient methods have many advantages compared to traditional reinforcement learning approaches. For example, they need fewer parameters to represent the optimal policy than the corresponding value function and they do not suffer from the difficult problem caused by uncertain state information.


## 2.1. Long Short-Term Memory (LSTM)

Traditional recurrent neural networks (RNNs) contain cyclic connections that make them a powerful tool to learn complex temporal dynamics, as shown in Figure1. The formulas that govern the computation happening in a RNN are as follows:
(1)ht=fUxt+Wht−1,yt=softmaxVht,where f is an element-wise nonlinearity function, xt and yt represent the input vector and the output vector at time step t, and ht∈ℝN is the hidden-layer vector with N hidden units at time step t. U, W, and V are the weight matrices of the connection from input nodes to hidden nodes, hidden nodes to hidden nodes, and hidden nodes to output nodes.Figure 1
Recurrent neural networks.
(a)
RNN architecture
(b)
RNN unitThough RNNs have been successfully used for sequence modeling tasks, they can only model the data within a fixed-size window. At the same time, training conventional RNNs is difficult due to the problem of exploding and vanishing gradients. These problems limit the capability of RNNs to learn long-term dynamics. LSTM was proposed in [44] to solve these problems. The LSTM unit is used in this paper as described in [45], as shown in Figure 2.Figure 2
Long short-term memory unit.In this subsection, we provide the equations of LSTM for a single memory unit only. Letx=x1,…,xT be an input sequence and y=y1,…,yT represent an output sequence; an LSTM network computes a mapping iteratively between x=x1,…,xT and y=y1,…,yT using the following equations:
(2)it=σWxixt+Whiht−1+Wcict−1+bi,ft=σWxfxt+Whfht−1+Wcfct−1+bf,ct=ft⊙ct−1+it⊙tanhWxcxt+Whcht−1+bc,ot=σWxoxt+Whoht−1+Wcoct+bo,ht=ot⊙tanhct,where σx=1/1+exp−x is the logistic sigmoid function, ct is the cell input activation vector, it describes the input gate, ft represents the forget gate, and ot output gate. All of the above are the same size as the hidden vector ht. That is, in addition to a hidden vector ht∈ℝN, the LSTM includes an input gate it∈ℝN, forget gate ft∈ℝN, output gate ot∈ℝN, and memory cell ct∈ℝN. We can find the meaning of the weight matrix; for example, Whi represents the hidden to input gate matrix and Wxo represents the input to output gate matrix. bi, bf, bo, and bc are the bias terms which are added to i, f, o, and c.


## 2.2. Deep Reinforcement Learning (DRL)

Reinforcement learning (RL) can usually be used to solve sequential decision-making problems. The process of reinforcement learning is shown in Figure3. Recently, significant progress has been made by combining reinforcement learning with the ability for learning feature representations in deep learning. Deep Q network (DQN) and policy gradient are two popular methods in DRL algorithms. DQN is a form of Q-learning with function approximation using a neural network, which means it tries to learn a state-action value function Q given by a neural network in DQN by minimizing temporal-difference errors. To improve performance and keep stability, various network architectures are based on the DQN algorithm such as dueling DQN [46] and double DQN [47].Figure 3
The typical framing of a reinforcement learning.A policy gradient approach is a type of reinforcement learning method that directly optimizes parametrized policies by using gradient descent [48]. Policy gradient methods have many advantages compared to traditional reinforcement learning approaches. For example, they need fewer parameters to represent the optimal policy than the corresponding value function and they do not suffer from the difficult problem caused by uncertain state information.


## 3. Proposed Visual Multiobject Tracking Algorithm

In Subsections3.1–3.3, we show a brief architecture of our proposed multiobject tracking algorithm firstly. The details of our method are described in the following content.


### 3.1. Architecture of the Proposed Multiobject Tracking Algorithm

Our method consists of three major components: an object detection module, many single-object trackers, and a data association module, which are shown in Figure4. In the first place, as demonstrated in Figure 4, we choose YOLO V2 [49] as an object detector because it is a state-of-the-art, real-time object detection system. YOLO V2 is applied on every frame and outputs a set of detections Dt at time step t. In each frame, YOLO V2 may output many kinds of detections. To obtain the correct detections to the tracking objects, the intersection-over-union (IoU) distance is computed between the ground truth and the detections at the first frame. The IoU distance between the mean of its short-term history of validated detections and the current detections is also computed to obtain the correct detections at the other frame. Secondly, the single-object tracker is composed of a network that includes a CNN followed by an LSTM unit. Each tracker, regarded as an agent, is trained by utilizing deep reinforcement learning. Finally, inspired by [50], we adopt an LSTM-based architecture that can learn to solve the joint data association problem from training data.Figure 4
Overview of our proposed multiobject tracking algorithm.


### 3.2. Single-Object Tracker via Deep Reinforcement Learning

We cast the problem of object tracking as a Markov decision process (MDP) since this setting provides a formal strategy to model an agent that makes a sequence of decisions. In our formulation, a single-frame image is considered as the environment, in which the agent transforms a bounding box using a set of actions. The MDP includes a set of actionsa∈A, a set of states s∈S, a state transition function fs,a, and a reward signal r. Our single-object tracking framework is illustrated in Figure 5. This section presents details of these components.Figure 5
The pipeline of the single-object tracker.In our paper, the set of actionA is composed of six actions that can be applied to the bounding box and one action to terminate the search process, as shown in Figure 6. Each action is encoded by the 7-dimensional vector. These actions are organized in three subsets: horizontal moves {right, left}, vertical moves {up, down}, and scale changes {scale up, scale down}.Figure 6
LSTM-based architecture for data association.The state definition is a tuplest=pt,vt, where pt is the image patch (which is pointed by a 4-dimensional vector pt=xt,yt,wt,ht) within the bounding box of the object and vt is a vector with the history of taken actions. The history vector stores the past 10 actions, which means vt has 70 dimensions as each action vector has 7 dimensions. At time step t + 1, the state st+1=pt+1,vt+1 is decided by st=pt,vt and the state transition functions, where pt+1=ftpt,at and vt+1=fvvt,at.The agent will receive a reward signalrt from the environment during the training process. In our method, reward rt is given at the end of a tracking episode when the object is tracked successfully. More specifically, the reward signal rt=0 during iteration in MDP in a time step. When the “stop” action is selected at termination step T, the reward signal rT is a thresholding function of IoU as follows:
(3)rT=1,ifIoUpT,g>τ,rT=−1,otherwise,where IoUpT,g=areapT∩g/areapT∪g represents the overlap ratio of pT and the ground truth of the object.We adopt policy-based reinforcement learning methods as they have a better capability of learning random policies and convergence properties. Our whole network is parameterized byW, the policy-based method models, the policy function πa∣s;W, and the value function Va∣s;W′; the aim of training this network is to maximize the overall tracking performance by policy gradient approximation. At each time step t, the goal of the agent is to learn a policy function πat∣st;W. Approximation of the policy function can be obtained by a stochastic gradient ascent algorithm. As there are very limited amounts of labelled data for multiobject tracking, we use synthetic data as a supplementary to the real data in the training. The parameters W and W′ can be learned according to the following equations:
(4)W←W+λRt−Vatst;W′∇Wlogπatst;W+ε∇WHπ⋅st;W,W′←W′−λRt−Vatst;W′,where Rt=∑t′=tt+T−1αt′−trt′ is the sum of future rewards up to T time steps, 0<α≤1, λ is the learning rate, H⋅ is an entropy regularizer, and ε is the regularizer factor.Our deep CNN is conducted on the VGG-16 network, which includes five pooling stages, that is, Conv1-2, Conv2-2, Conv3-3, Conv4-3, and Conv5-3. The gradual decrease in the spatial resolution occurs when the depth of layers increases, because all convolutional layers have a2×2 kernel size and a stride of 2 in the VGG-16 model. For example, when inputting an image with size M×N, the output feature maps of pooling 5 have a size M/25×N/25. In our model, we use the feature maps from Conv3-3, Conv4-3, and Conv5-3, which have been elevated to the same size by using bilinear interpolation.


### 3.3. Data Association

LetPt=ptii=1M represent the set of all outputs of single-object trackers at time step t, pti refers to the state of the ith output of a single-object tracker, and M is the number of objects that can be tracked simultaneously in one time step. The state of the ith object is represented by the 4-dimensional vector pti=xti,yti,wti,hti. We define Qt=qtjj=1N as the set of detections from the object detector with qtj the jth detection and N the number of detections. Let Dt∈ℝM×N denote the similarity matrix for data association that measures the relation between an output of single-object tracker pti and a detection mtj, where Dtij=pti−qtj2 is the Euclidean distance between pti and qtj. Data association based on LSTM for object i is illustrated in Figure 6.The task of data association is to predict the assignment for each object using the temporal step-by-step functionality of LSTM. The inputs at each stepi are the hidden state hi, the cell state ci, and the similarity matrix Dt. The output are the hidden state hi+1, the cell state ci+1, and the assignment probability vector Ati. Ati is a vector of assignment probabilities for object i and all available measurements, which is obtained by applying a softmax layer with normalization to the predicted values. Atij=a (object i assigned to the jth detection) and ∑jAtij=1. Let ε be the correct assignment; we adapt the negative log-likelihood loss as the cost function to measure the misassignment cost:
(5)CAti,ε=−logAtiε,The data association requires more representation power, so it is a more complex task. The data-association-module-based LSTM include two layers and 512 hidden units. It takes approximately 40 hours to train all the modules in our tracker on a CPU. The training can be sped up significantly by using GPUs.


## 3.1. Architecture of the Proposed Multiobject Tracking Algorithm

Our method consists of three major components: an object detection module, many single-object trackers, and a data association module, which are shown in Figure4. In the first place, as demonstrated in Figure 4, we choose YOLO V2 [49] as an object detector because it is a state-of-the-art, real-time object detection system. YOLO V2 is applied on every frame and outputs a set of detections Dt at time step t. In each frame, YOLO V2 may output many kinds of detections. To obtain the correct detections to the tracking objects, the intersection-over-union (IoU) distance is computed between the ground truth and the detections at the first frame. The IoU distance between the mean of its short-term history of validated detections and the current detections is also computed to obtain the correct detections at the other frame. Secondly, the single-object tracker is composed of a network that includes a CNN followed by an LSTM unit. Each tracker, regarded as an agent, is trained by utilizing deep reinforcement learning. Finally, inspired by [50], we adopt an LSTM-based architecture that can learn to solve the joint data association problem from training data.Figure 4
Overview of our proposed multiobject tracking algorithm.


## 3.2. Single-Object Tracker via Deep Reinforcement Learning

We cast the problem of object tracking as a Markov decision process (MDP) since this setting provides a formal strategy to model an agent that makes a sequence of decisions. In our formulation, a single-frame image is considered as the environment, in which the agent transforms a bounding box using a set of actions. The MDP includes a set of actionsa∈A, a set of states s∈S, a state transition function fs,a, and a reward signal r. Our single-object tracking framework is illustrated in Figure 5. This section presents details of these components.Figure 5
The pipeline of the single-object tracker.In our paper, the set of actionA is composed of six actions that can be applied to the bounding box and one action to terminate the search process, as shown in Figure 6. Each action is encoded by the 7-dimensional vector. These actions are organized in three subsets: horizontal moves {right, left}, vertical moves {up, down}, and scale changes {scale up, scale down}.Figure 6
LSTM-based architecture for data association.The state definition is a tuplest=pt,vt, where pt is the image patch (which is pointed by a 4-dimensional vector pt=xt,yt,wt,ht) within the bounding box of the object and vt is a vector with the history of taken actions. The history vector stores the past 10 actions, which means vt has 70 dimensions as each action vector has 7 dimensions. At time step t + 1, the state st+1=pt+1,vt+1 is decided by st=pt,vt and the state transition functions, where pt+1=ftpt,at and vt+1=fvvt,at.The agent will receive a reward signalrt from the environment during the training process. In our method, reward rt is given at the end of a tracking episode when the object is tracked successfully. More specifically, the reward signal rt=0 during iteration in MDP in a time step. When the “stop” action is selected at termination step T, the reward signal rT is a thresholding function of IoU as follows:
(3)rT=1,ifIoUpT,g>τ,rT=−1,otherwise,where IoUpT,g=areapT∩g/areapT∪g represents the overlap ratio of pT and the ground truth of the object.We adopt policy-based reinforcement learning methods as they have a better capability of learning random policies and convergence properties. Our whole network is parameterized byW, the policy-based method models, the policy function πa∣s;W, and the value function Va∣s;W′; the aim of training this network is to maximize the overall tracking performance by policy gradient approximation. At each time step t, the goal of the agent is to learn a policy function πat∣st;W. Approximation of the policy function can be obtained by a stochastic gradient ascent algorithm. As there are very limited amounts of labelled data for multiobject tracking, we use synthetic data as a supplementary to the real data in the training. The parameters W and W′ can be learned according to the following equations:
(4)W←W+λRt−Vatst;W′∇Wlogπatst;W+ε∇WHπ⋅st;W,W′←W′−λRt−Vatst;W′,where Rt=∑t′=tt+T−1αt′−trt′ is the sum of future rewards up to T time steps, 0<α≤1, λ is the learning rate, H⋅ is an entropy regularizer, and ε is the regularizer factor.Our deep CNN is conducted on the VGG-16 network, which includes five pooling stages, that is, Conv1-2, Conv2-2, Conv3-3, Conv4-3, and Conv5-3. The gradual decrease in the spatial resolution occurs when the depth of layers increases, because all convolutional layers have a2×2 kernel size and a stride of 2 in the VGG-16 model. For example, when inputting an image with size M×N, the output feature maps of pooling 5 have a size M/25×N/25. In our model, we use the feature maps from Conv3-3, Conv4-3, and Conv5-3, which have been elevated to the same size by using bilinear interpolation.


## 3.3. Data Association

LetPt=ptii=1M represent the set of all outputs of single-object trackers at time step t, pti refers to the state of the ith output of a single-object tracker, and M is the number of objects that can be tracked simultaneously in one time step. The state of the ith object is represented by the 4-dimensional vector pti=xti,yti,wti,hti. We define Qt=qtjj=1N as the set of detections from the object detector with qtj the jth detection and N the number of detections. Let Dt∈ℝM×N denote the similarity matrix for data association that measures the relation between an output of single-object tracker pti and a detection mtj, where Dtij=pti−qtj2 is the Euclidean distance between pti and qtj. Data association based on LSTM for object i is illustrated in Figure 6.The task of data association is to predict the assignment for each object using the temporal step-by-step functionality of LSTM. The inputs at each stepi are the hidden state hi, the cell state ci, and the similarity matrix Dt. The output are the hidden state hi+1, the cell state ci+1, and the assignment probability vector Ati. Ati is a vector of assignment probabilities for object i and all available measurements, which is obtained by applying a softmax layer with normalization to the predicted values. Atij=a (object i assigned to the jth detection) and ∑jAtij=1. Let ε be the correct assignment; we adapt the negative log-likelihood loss as the cost function to measure the misassignment cost:
(5)CAti,ε=−logAtiε,The data association requires more representation power, so it is a more complex task. The data-association-module-based LSTM include two layers and 512 hidden units. It takes approximately 40 hours to train all the modules in our tracker on a CPU. The training can be sped up significantly by using GPUs.


## 4. Experiments

### 4.1. Qualitative Evaluation

In this section, we compare our visual multiobject tracker with several state-of-the-art methods on the MOT Challenge benchmark [51] in order to show the performance of our algorithm. The synthetic datasets OVVV [52] and virtual KITTI [53] are used as supplementary to the real data in the training. In the single-object tracker, the learning rate for CNN is set to 0.0001, and for fully connected layers it is set to 0.001. In the DRL network, the learning rate λ is set to 0.0001, and the regularizer factor ε is set to 0.01, T=20, α=0.95.The PETS09-S2L2 sequence consists of 436 frames of 768∗576 pixels with heavy crowd density and illumination changes. The pedestrians undergo severe occlusion and scale changes in the sequence. The ADL-Rundle-3 sequence consists of 625 frames of 1920∗1080 pixels. It shows a crowded pedestrian street captured from a stationary camera. Frequent occlusions, missed detections, and illumination variation happen among the multiple objects. The TUD-Crossing sequence shows a road crossing from a side view. It consists of 201 frames of 640∗480 pixels and includes the nonlinear motion, objects in close proximity, and occlusions. The AVG-Town Center contains 450 frames of 1920∗1080 pixels. It shows a busy town center street from a single elevated camera. The sequence contains medium crowd density, frequent dynamic occlusions, and scale changes.We compare our method (LSTM_DRL) with other state-of-the-art trackers including RNN-LSTM [50], LP_SSVM [54], MDPSubCNN [55], and SiameseCNN [56]. Figures 7, 8, 9, and 10 demonstrate the qualitative tracking results of our tracker on PETS09-S2L2, ADL-Rundle-3, TUD-Crossing, and AVG-Town Center. Figures 11, 12, 13, and 14 show the sample tracking results of other trackers on PETS09-S2L2, ADL-Rundle-3, TUD-Crossing, and AVG-Town Center.Figure 7
Qualitative tracking results of our tracker on PETS09-S2L2.
(a)
(b)
(c)
(d)Figure 8
Qualitative tracking results of our tracker on ADL-Rundle-3.
(a)
(b)
(c)
(d)Figure 9
Qualitative tracking results of our tracker on TUD-Crossing.
(a)
(b)
(c)
(d)Figure 10
Qualitative tracking results of our tracker on AVG-TownCentre.
(a)
(b)
(c)
(d)Figure 11
Sample tracking results of other trackers on frame number 15 of PETS09-S2L2.
(a)
(b)
(c)
(d)Figure 12
Sample tracking results of other trackers on frame number 240 of ADL-Rundle-3.
(a)
(b)
(c)
(d)Figure 13
Sample tracking results of other trackers on frame number 45 of TUD-Crossing.
(a)
(b)
(c)
(d)Figure 14
Sample tracking results of other trackers on frame number 405 of AVG-TownCentre.
(a)
(b)
(c)
(d)From these experimental results, we can see that our tracker performs well most of the time despite frequent occlusions, similarity among objects, scale changes, and illumination changes. Nevertheless, there are still some examples of unavoidable tracking failures as illustrated in Figure15. For example, the brightness of the environment results in the failure of object detection in frame number 285 from the PETS09-S2L2 dataset and there are some missing detections in frame number 255 from the AVG-TownCentre dataset.Figure 15
Some selected failure results of our tracker.
(a)
PETS09-S2L2-LSTM-DRL frame number 285
(b)
AVG-TownCentre-LSTM-DRL frame number 255To illustrate the contribution of each component, the detection result and the tracking result of single-object trackers are shown in Figure16. Limited to the space, we only list the results on ADL-Rundle-3.Figure 16
The results of the detection and single-object tracker on ADL-Rundle-3.
(a)
The detection result of YOLO V2 on frame number 30
(b)
The tracking result of the single-object tracker on frame number 30From the results, we can see that the object is missed in the detector, while he is tracked in the single-object tracker according DRL.


### 4.2. Quantitative Evaluation

The CLEAR MOT performance metrics are used in this section for quantitative evaluation: the multiple-object tracking accuracy (MOTA), the multiple-object tracking precision (MOTP), false positive (FP), and identity switches (IDSW). MOTA evaluates the accuracy composed of false negatives, false positives, and identity switches.(6)MOTA=1−∑tfnt+fpt+IDSWt∑tgtt,where fnt, fpt, IDSWt, and gtt are false negatives, false positives, identity switching, and ground truth at frame t.MOTP is the average dissimilarity between all true positives and their corresponding ground truth objects, which calculates the intersection area over the union area of bounding boxes. This is computed as(7)MOTP=1−∑t,idt,i∑tct,where dt,i denotes the bounding box overlap of object i with its assigned ground truth object and ct is the number of matches in frame t.Table1 reports the quantitative comparison results of our tracker (LSTM_DRL) with other state-of-the-art trackers on the 11 sequences of the MOT Challenge dataset.Table 1
The quantitative comparison results of our tracker with other state-of-the-art trackers.
Method
Sequence
MOTA%
MOTP%
FP
IDSW
RNN-LSTM
PETS09-S2L2
38.3
71.6
1016
320
SiameseCNN
47.5
72.6
341
796
MDPSubCNN
34.5
69.7
672
282
LP_SSVM
41.5
70.5
629
212
LSTM_DRL (ours)
45.8
72.9
354
255
RNN-LSTM
ADL-Rundle-3
23.7
72.0
2193
158
SiameseCNN
39.7
72.9
191
33
MDPSubCNN
44.9
79.6
793
56
LP_SSVM
28.0
72.9
1855
81
LSTM_DRL (ours)
45.2
75.1
651
30
RNN-LSTM
TUD-Crossing
57.2
71.1
81
43
SiameseCNN
73.7
73.0
85
8
MDPSubCNN
78.9
76.7
32
6
LP_SSVM
60.0
74.2
48
18
LSTM_DRL (ours)
79.1
75.9
30
11
RNN-LSTM
AVG-Town Center
13.4
68.8
1206
299
SiameseCNN
19.3
69.0
698
142
MDPSubCNN
49.5
70.1
1381
121
LP_SSVM
14.7
70.1
459
123
LSTM_DRL (ours)
38.6
71.0
598
101
RNN-LSTM
Venice-1
12.7
71.7
686
56
SiameseCNN
22.3
73.0
322
4
MDPSubCNN
15.9
72.4
843
47
LP_SSVM
17.8
73.0
696
23
LSTM_DRL (ours)
23.4
73.8
302
6
RNN-LSTM
ETH-Jelmoli
34.8
73.3
314
59
SiameseCNN
42.3
72.8
315
30
MDPSubCNN
32.9
73.6
639
22
LP_SSVM
39.5
74.4
224
17
LSTM_DRL (ours)
43.9
75.1
213
13
RNN-LSTM
ETH-Linthescher
12.4
74.7
164
49
SiameseCNN
16.7
74.2
93
27
MDPSubCNN
27.2
74.7
191
48
LP_SSVM
15.6
75.6
41
11
LSTM_DRL (ours)
27.1
75.4
52
14
eRNN-LSTM
ETH-Crossing
21.1
75.5
27
7
SiameseCNN
27.5
74.1
20
4
MDPSubCNN
28.8
74.7
59
0
LP_SSVM
24.9
75.6
10
2
LSTM_DRL (ours)
29.6
77.9
12
4
RNN-LSTM
ADL-Rundle-1
−2.2
69.9
4213
241
SiameseCNN
25.6
71.6
1999
33
MDPSubCNN
16.2
71.5
3157
49
LP_SSVM
14.0
71.9
3507
69
LSTM_DRL (ours)
27.9
72.8
1846
39From the results of Table1, we can see that our proposed method provides the highest MOTP values and the lowest FN values on the PETS09-S2L2 dataset, provides the highest MOTA values and the lowest FN and IDSW values on the ADL-Rundle-3 dataset, provides the highest MOTA values and the lowest FP and FN values on the TUD-Crossing dataset, and provides the highest MOTP values and the lowest IDSW values on the AVG-Town Center dataset. The proposed method obtains a better performance that can mainly be attributed to the three parts of the tracker: YOLO V2 is a state-of-the-art object detector, the data association strategy based on LSTM can find a global optimal assignment, and the single-object trackers are able to find the location of the object via deep reinforcement learning.We implement the experiments of our proposed multiobject tracking algorithm based on the Windows 10 operating system and using MATLAB R2016b as the software platform. The configuration of the computer is Intel® Core™ i7-4712MQ and GeForce GTXTITAN X GPU, 12.00 GB VRAM.The results of running time on the MOT Challenge test dataset are shown in Table2, where they are compared to some state-of-the-art trackers. Our method is a real-time tracking system and although the speed is slower than RNN-LSTM, which does not incorporate appearance, the other performance of our method is better than it.Table 2
The running time comparison results of our tracker with other state-of-the-art trackers.
Method
FPS
RNN-LSTM
166.8
TC-ODAL [8]
2.4
JPDA-m [56]
35.6
MDPSubCNN
2.1
LSTM_DRL (ours)
108.0


## 4.1. Qualitative Evaluation

In this section, we compare our visual multiobject tracker with several state-of-the-art methods on the MOT Challenge benchmark [51] in order to show the performance of our algorithm. The synthetic datasets OVVV [52] and virtual KITTI [53] are used as supplementary to the real data in the training. In the single-object tracker, the learning rate for CNN is set to 0.0001, and for fully connected layers it is set to 0.001. In the DRL network, the learning rate λ is set to 0.0001, and the regularizer factor ε is set to 0.01, T=20, α=0.95.The PETS09-S2L2 sequence consists of 436 frames of 768∗576 pixels with heavy crowd density and illumination changes. The pedestrians undergo severe occlusion and scale changes in the sequence. The ADL-Rundle-3 sequence consists of 625 frames of 1920∗1080 pixels. It shows a crowded pedestrian street captured from a stationary camera. Frequent occlusions, missed detections, and illumination variation happen among the multiple objects. The TUD-Crossing sequence shows a road crossing from a side view. It consists of 201 frames of 640∗480 pixels and includes the nonlinear motion, objects in close proximity, and occlusions. The AVG-Town Center contains 450 frames of 1920∗1080 pixels. It shows a busy town center street from a single elevated camera. The sequence contains medium crowd density, frequent dynamic occlusions, and scale changes.We compare our method (LSTM_DRL) with other state-of-the-art trackers including RNN-LSTM [50], LP_SSVM [54], MDPSubCNN [55], and SiameseCNN [56]. Figures 7, 8, 9, and 10 demonstrate the qualitative tracking results of our tracker on PETS09-S2L2, ADL-Rundle-3, TUD-Crossing, and AVG-Town Center. Figures 11, 12, 13, and 14 show the sample tracking results of other trackers on PETS09-S2L2, ADL-Rundle-3, TUD-Crossing, and AVG-Town Center.Figure 7
Qualitative tracking results of our tracker on PETS09-S2L2.
(a)
(b)
(c)
(d)Figure 8
Qualitative tracking results of our tracker on ADL-Rundle-3.
(a)
(b)
(c)
(d)Figure 9
Qualitative tracking results of our tracker on TUD-Crossing.
(a)
(b)
(c)
(d)Figure 10
Qualitative tracking results of our tracker on AVG-TownCentre.
(a)
(b)
(c)
(d)Figure 11
Sample tracking results of other trackers on frame number 15 of PETS09-S2L2.
(a)
(b)
(c)
(d)Figure 12
Sample tracking results of other trackers on frame number 240 of ADL-Rundle-3.
(a)
(b)
(c)
(d)Figure 13
Sample tracking results of other trackers on frame number 45 of TUD-Crossing.
(a)
(b)
(c)
(d)Figure 14
Sample tracking results of other trackers on frame number 405 of AVG-TownCentre.
(a)
(b)
(c)
(d)From these experimental results, we can see that our tracker performs well most of the time despite frequent occlusions, similarity among objects, scale changes, and illumination changes. Nevertheless, there are still some examples of unavoidable tracking failures as illustrated in Figure15. For example, the brightness of the environment results in the failure of object detection in frame number 285 from the PETS09-S2L2 dataset and there are some missing detections in frame number 255 from the AVG-TownCentre dataset.Figure 15
Some selected failure results of our tracker.
(a)
PETS09-S2L2-LSTM-DRL frame number 285
(b)
AVG-TownCentre-LSTM-DRL frame number 255To illustrate the contribution of each component, the detection result and the tracking result of single-object trackers are shown in Figure16. Limited to the space, we only list the results on ADL-Rundle-3.Figure 16
The results of the detection and single-object tracker on ADL-Rundle-3.
(a)
The detection result of YOLO V2 on frame number 30
(b)
The tracking result of the single-object tracker on frame number 30From the results, we can see that the object is missed in the detector, while he is tracked in the single-object tracker according DRL.


## 4.2. Quantitative Evaluation

The CLEAR MOT performance metrics are used in this section for quantitative evaluation: the multiple-object tracking accuracy (MOTA), the multiple-object tracking precision (MOTP), false positive (FP), and identity switches (IDSW). MOTA evaluates the accuracy composed of false negatives, false positives, and identity switches.(6)MOTA=1−∑tfnt+fpt+IDSWt∑tgtt,where fnt, fpt, IDSWt, and gtt are false negatives, false positives, identity switching, and ground truth at frame t.MOTP is the average dissimilarity between all true positives and their corresponding ground truth objects, which calculates the intersection area over the union area of bounding boxes. This is computed as(7)MOTP=1−∑t,idt,i∑tct,where dt,i denotes the bounding box overlap of object i with its assigned ground truth object and ct is the number of matches in frame t.Table1 reports the quantitative comparison results of our tracker (LSTM_DRL) with other state-of-the-art trackers on the 11 sequences of the MOT Challenge dataset.Table 1
The quantitative comparison results of our tracker with other state-of-the-art trackers.
Method
Sequence
MOTA%
MOTP%
FP
IDSW
RNN-LSTM
PETS09-S2L2
38.3
71.6
1016
320
SiameseCNN
47.5
72.6
341
796
MDPSubCNN
34.5
69.7
672
282
LP_SSVM
41.5
70.5
629
212
LSTM_DRL (ours)
45.8
72.9
354
255
RNN-LSTM
ADL-Rundle-3
23.7
72.0
2193
158
SiameseCNN
39.7
72.9
191
33
MDPSubCNN
44.9
79.6
793
56
LP_SSVM
28.0
72.9
1855
81
LSTM_DRL (ours)
45.2
75.1
651
30
RNN-LSTM
TUD-Crossing
57.2
71.1
81
43
SiameseCNN
73.7
73.0
85
8
MDPSubCNN
78.9
76.7
32
6
LP_SSVM
60.0
74.2
48
18
LSTM_DRL (ours)
79.1
75.9
30
11
RNN-LSTM
AVG-Town Center
13.4
68.8
1206
299
SiameseCNN
19.3
69.0
698
142
MDPSubCNN
49.5
70.1
1381
121
LP_SSVM
14.7
70.1
459
123
LSTM_DRL (ours)
38.6
71.0
598
101
RNN-LSTM
Venice-1
12.7
71.7
686
56
SiameseCNN
22.3
73.0
322
4
MDPSubCNN
15.9
72.4
843
47
LP_SSVM
17.8
73.0
696
23
LSTM_DRL (ours)
23.4
73.8
302
6
RNN-LSTM
ETH-Jelmoli
34.8
73.3
314
59
SiameseCNN
42.3
72.8
315
30
MDPSubCNN
32.9
73.6
639
22
LP_SSVM
39.5
74.4
224
17
LSTM_DRL (ours)
43.9
75.1
213
13
RNN-LSTM
ETH-Linthescher
12.4
74.7
164
49
SiameseCNN
16.7
74.2
93
27
MDPSubCNN
27.2
74.7
191
48
LP_SSVM
15.6
75.6
41
11
LSTM_DRL (ours)
27.1
75.4
52
14
eRNN-LSTM
ETH-Crossing
21.1
75.5
27
7
SiameseCNN
27.5
74.1
20
4
MDPSubCNN
28.8
74.7
59
0
LP_SSVM
24.9
75.6
10
2
LSTM_DRL (ours)
29.6
77.9
12
4
RNN-LSTM
ADL-Rundle-1
−2.2
69.9
4213
241
SiameseCNN
25.6
71.6
1999
33
MDPSubCNN
16.2
71.5
3157
49
LP_SSVM
14.0
71.9
3507
69
LSTM_DRL (ours)
27.9
72.8
1846
39From the results of Table1, we can see that our proposed method provides the highest MOTP values and the lowest FN values on the PETS09-S2L2 dataset, provides the highest MOTA values and the lowest FN and IDSW values on the ADL-Rundle-3 dataset, provides the highest MOTA values and the lowest FP and FN values on the TUD-Crossing dataset, and provides the highest MOTP values and the lowest IDSW values on the AVG-Town Center dataset. The proposed method obtains a better performance that can mainly be attributed to the three parts of the tracker: YOLO V2 is a state-of-the-art object detector, the data association strategy based on LSTM can find a global optimal assignment, and the single-object trackers are able to find the location of the object via deep reinforcement learning.We implement the experiments of our proposed multiobject tracking algorithm based on the Windows 10 operating system and using MATLAB R2016b as the software platform. The configuration of the computer is Intel® Core™ i7-4712MQ and GeForce GTXTITAN X GPU, 12.00 GB VRAM.The results of running time on the MOT Challenge test dataset are shown in Table2, where they are compared to some state-of-the-art trackers. Our method is a real-time tracking system and although the speed is slower than RNN-LSTM, which does not incorporate appearance, the other performance of our method is better than it.Table 2
The running time comparison results of our tracker with other state-of-the-art trackers.
Method
FPS
RNN-LSTM
166.8
TC-ODAL [8]
2.4
JPDA-m [56]
35.6
MDPSubCNN
2.1
LSTM_DRL (ours)
108.0


## 5. Conclusion

This paper proposes a visual multiobject tracking algorithm based on LSTM and deep reinforcement learning to overcome the problems of the existing algorithms: they have many limits because handcrafted features cannot capture more complex characteristics of the objects, tracking fails when the number of objects vary, and so on. We adopted the object detector YOLO V2 to detect the multiple objects. The single-object tracker is composed of a network that includes a CNN followed by an LSTM unit. Each tracker, regarded as an agent, is trained by utilizing deep reinforcement learning. We conduct data association using LSTM for each frame between a pretrained object detector and a number of single-object trackers. From the experimental results, we can see that the proposed multiobject tracking method improves the robustness and accuracy of the algorithm.



---

*Source: 4695890-2018-11-19.xml*