# Real-Time Pedestrian Tracking and Counting with TLD

Authors: Jiawei Shi; Xianmei Wang; Huer Xiao
Journal: Journal of Advanced Transportation
 (2018)

---

## Abstract

This paper describes a solution to solve the issue of automatic multipedestrian tracking and counting. First, background modeling algorithm is applied to actively obtain multipedestrian candidates, followed by a confirmation step with classification. Then each pedestrian patch is handled by real-time TLD (Tracking-Learning-Detection) to get a new predication position according to similarity measure. Further TLD results are compared with classification list to determine a new, disappeared, or existing pedestrian. Finally single line counting with buffer zone is employed to count pedestrians. Experiments results on the public database, PETS, demonstrate the validity of our solution.


---

## Body

## 1. Introduction

Pedestrian tracking and counting is an extremely significant research in the field of computer vision. It plays a crucial role in many applications including the intelligent monitoring and traffic safety. Nevertheless, some challenges such as great variation of the pedestrian posture, background clutter, partial occlusions, and illumination changing complicate the issue.Current state-of-the-art tracking algorithms can be roughly divided into two categories: generative and discriminative pattern. Generative methods [1, 2], generally, describe the characteristics performance via the generated model and then minimize the reconstruction error by searching the candidate targets. Compared with generative methods, discriminative approaches [3–5] distinguish target and background by trained classifiers to find a decision boundary between object and background. By adequately using both of the target and background information, this approach achieves higher tracking accuracy. In recent years, discriminative approaches have been obtaining vigorous developments.TLD is a popular discriminative method [5]. It integrates an online detector to obtain a good ability of target redetect, which is important to realize continuous tracking in the case of target disappearance in tracking process. In addition, TLD can solve long-term accurate tracking by online learning.TLD has been used in different tracking tasks. Zhang et al. [6] use TLD to realize dynamic gesture tracking. To initialize TLD algorithm, a specific gesture is manually marked by a bounding box in the first frame. Chen et al. [7] improved TLD to track a human in unconstrained environments, and the object needs to be initialized manually as well. Crane tracking and monitoring with stable TLD also show strong robustness and accuracy [8]. However all of these studies are under single target tracking conditions.Pedestrian flow statistics is used to track pedestrians and count their number in the video [9], such as the surveillance of crossroads. Compared with above studies, many people may appear at the same time, leading to the request of multiple object tracking. Moreover, any pedestrian may randomly appear. As a result, the targets cannot be extracted manually and automatic searching for tracking objects is also a key for multipedestrian tracking and counting.We propose a new method called improved TLD (ITLD) by introducing background subtraction to automatically obtain multiobject and then use the updating mechanism of tracking list to realize multiobject management. Figure1 shows the framework of our system, integrating with counting module.Figure 1
The framework of our proposed system.


## 2. Multipedestrian Patch Obtaining

There are several ways to realize multiple objects obtaining. Cao et al. [10] use Haar features and AdaBoost algorithm to automatically detect human faces. Zhou et al. [11] proposed PE-TLD, after the process of ViBe and variance filter, hog features, and SVM classifier were used for automatic targets recognition. Besides, S. Sharma et al. [12] let users manually select the desired tracking objects in the initialization process.In our system a coarse to fine strategy is employed to obtain multitarget. Firstly, a dynamic average background model followed by Ostu algorithm is used to extract candidate pedestrian patches. Next, we utilize pedestrian detection combining Haar-AdaBoost [13, 14] and Hog-SVM [15] to further exclude those nonpedestrian patches.


### 2.1. Extraction of Candidate Pedestrian Patch

Our dynamic average background modeling is based on Gauss statistics, considering the distinction of three color channel variances. In order to reduce modeling error, we make the different statistics in RGB space for all pixels. After the construction of background model, a simple subtract operation between current frame and latest background is employed to get a rough foreground part. In this step Ostu algorithm is used to obtain segmentation threshold. Then morphological processing is used to further extract those small regions and fill the gaps among foreground part. Finally, minimum circumscribed rectangle is used for getting possible pedestrian targets. The detailed process is shown in [16].Given a pixel, supposen is the serial number of frame in video stream, (
x
,
y
) is its position, v
r
(
n
)
(
x
,
y
)
,
v
g
(
n
)
(
x
,
y
), and v
b
(
n
)
(
x
,
y
) are the values of its three color components; then this pixel can be represented by p
(
n
)
(
x
,
y
)
=
(
v
r
(
n
)
(
x
,
y
)
,
v
g
(
n
)
(
x
,
y
)
,
v
b
(
n
)
(
x
,
y
)
). The construct process of background modelM
c
(
x
,
y
) can be expressed by (1)-(3).(1)
M
c
x
,
y
=
∑
n
=
0
N
-
1
v
c
n
x
,
y
F
n
x
,
y
∑
n
=
0
N
-
1
F
n
x
,
y
,
c
∈
r
,
g
,
bwhere F
(
n
)
(
x
,
y
) denotes the binary mask, which is calculated by (2)
F
c
n
x
,
y
=
1
i
f
v
c
n
i
,
j
-
μ
c
0
i
,
j
≤
σ
c
0
i
,
j
∀
c
∈
r
,
g
,
b
0
o
t
h
e
r
w
i
s
e
n
=
0,1
…
N
-
1
F
n
x
,
y
=
∨
F
c
n
x
,
ywhere μ
c
0
(
x
,
y
) and σ
c
0
(
x
,
y
) represent the component mean and standard deviation of pixel p
(
n
)
(
x
,
y
) from frame n-N+1 to current frame n. In our work, N is set as 300.The recurrences of background model updating are shown in the following:(3)
M
c
k
x
,
y
=
M
c
k
-
1
x
,
y
∑
n
=
0
N
-
1
F
n
+
k
-
1
x
,
y
+
v
c
k
x
,
y
F
k
x
,
y
-
v
c
k
-
N
x
,
y
F
k
-
N
x
,
y
∑
n
=
0
N
-
1
F
n
+
k
-
1
x
,
y
+
F
k
x
,
y
-
F
k
-
N
x
,
y
c
∈
r
,
g
,
b
,
k
≥
N


### 2.2. Pedestrian Confirmation

Haar and HOG are two excellent features to describe pedestrians. In each frame, to obtain higher performance, we integrate HOG feature with SVM classifier and Haar feature with AdaBoost classifier (shown in Figure2) to determine if an image patch includes a pedestrian or not. An image patch will be thought to include a pedestrian, if either classifiers output yes.Figure 2
Classifier combination for pedestrian confirmation.In this step, a patch setD
P
(
j
)
=
{
d
p
i
(
j
)
(
x
,
y
,
w
,
h
)
,
i
=
1,2
,
,
,
,
.
N
d
} is used to record the pedestrian patches in frame j, wherex,y,w, andh are the position and size of d
p
i
(
j
).


## 2.1. Extraction of Candidate Pedestrian Patch

Our dynamic average background modeling is based on Gauss statistics, considering the distinction of three color channel variances. In order to reduce modeling error, we make the different statistics in RGB space for all pixels. After the construction of background model, a simple subtract operation between current frame and latest background is employed to get a rough foreground part. In this step Ostu algorithm is used to obtain segmentation threshold. Then morphological processing is used to further extract those small regions and fill the gaps among foreground part. Finally, minimum circumscribed rectangle is used for getting possible pedestrian targets. The detailed process is shown in [16].Given a pixel, supposen is the serial number of frame in video stream, (
x
,
y
) is its position, v
r
(
n
)
(
x
,
y
)
,
v
g
(
n
)
(
x
,
y
), and v
b
(
n
)
(
x
,
y
) are the values of its three color components; then this pixel can be represented by p
(
n
)
(
x
,
y
)
=
(
v
r
(
n
)
(
x
,
y
)
,
v
g
(
n
)
(
x
,
y
)
,
v
b
(
n
)
(
x
,
y
)
). The construct process of background modelM
c
(
x
,
y
) can be expressed by (1)-(3).(1)
M
c
x
,
y
=
∑
n
=
0
N
-
1
v
c
n
x
,
y
F
n
x
,
y
∑
n
=
0
N
-
1
F
n
x
,
y
,
c
∈
r
,
g
,
bwhere F
(
n
)
(
x
,
y
) denotes the binary mask, which is calculated by (2)
F
c
n
x
,
y
=
1
i
f
v
c
n
i
,
j
-
μ
c
0
i
,
j
≤
σ
c
0
i
,
j
∀
c
∈
r
,
g
,
b
0
o
t
h
e
r
w
i
s
e
n
=
0,1
…
N
-
1
F
n
x
,
y
=
∨
F
c
n
x
,
ywhere μ
c
0
(
x
,
y
) and σ
c
0
(
x
,
y
) represent the component mean and standard deviation of pixel p
(
n
)
(
x
,
y
) from frame n-N+1 to current frame n. In our work, N is set as 300.The recurrences of background model updating are shown in the following:(3)
M
c
k
x
,
y
=
M
c
k
-
1
x
,
y
∑
n
=
0
N
-
1
F
n
+
k
-
1
x
,
y
+
v
c
k
x
,
y
F
k
x
,
y
-
v
c
k
-
N
x
,
y
F
k
-
N
x
,
y
∑
n
=
0
N
-
1
F
n
+
k
-
1
x
,
y
+
F
k
x
,
y
-
F
k
-
N
x
,
y
c
∈
r
,
g
,
b
,
k
≥
N


## 2.2. Pedestrian Confirmation

Haar and HOG are two excellent features to describe pedestrians. In each frame, to obtain higher performance, we integrate HOG feature with SVM classifier and Haar feature with AdaBoost classifier (shown in Figure2) to determine if an image patch includes a pedestrian or not. An image patch will be thought to include a pedestrian, if either classifiers output yes.Figure 2
Classifier combination for pedestrian confirmation.In this step, a patch setD
P
(
j
)
=
{
d
p
i
(
j
)
(
x
,
y
,
w
,
h
)
,
i
=
1,2
,
,
,
,
.
N
d
} is used to record the pedestrian patches in frame j, wherex,y,w, andh are the position and size of d
p
i
(
j
).


## 3. Multipedestrian Tracking with TLD

The outputD
P
(
j
) of pedestrian classification servers as the input of TLD algorithm, which can solve the automatic selection of multiobject. Since pedestrian presence or disappearance from the camera is random, the management of multiobject list, including inserting, deleting, or maintaining, becomes important.


### 3.1. Single Pedestrian Tracking with TLD

Single pedestrian tracking with TLD include three components, namely, tracking, learning, and detection [5]. Data preparation is the first step to implement TLD framework. Given a target d
p
i
(
j
), use overlap area A
ounder different scales to choose positive and negative samples. Within each scale, we can get a collection of patches from top to bottom, left to right. Then calculate the overlap ratio r
=
A
0
/
(
d
p
i
(
j
)
.
w
∗
d
p
i
(
j
)
.
h
+
A
p
a
t
c
h
-
A
0
) between a patch and d
p
i
(
j
). Add the best 10 patches with maximum overlap ration (larger than 0.6) into the positive sample set. For those patches with overlap ration smaller than 0.1, we put them into the negative samples set. Before adding, each patch is resized into normalized size.Thusi
t
h object model in frame j can be expressed as M
i
(
j
)
=
{
M
i
(
j
)
+
,
M
i
(
j
)
-
}
=
{
p
i
1
(
j
)
+
,
p
i
2
(
j
)
+
,
.
.
.
,
p
i
m
(
j
)
+
,
p
i
1
(
j
)
-
,
p
i
2
(
j
)
-
,
.
.
.
,
p
i
n
(
j
)
-
}. p
i
1
(
j
)
+ is the first positive patch added to the set of M
i
(
j
). And p
i
m
(
j
)
+ is the positive sample added last so far. After learning initialization, we can use the TLD framework shown in [5]. In model update process, relative similarity S(a,b) is used to measure the similarity between objectsa andb. The similarity between the image patch P
n
e
wand setM
i
(
j
)is calculated as (4)-(7). (4)
S
r
p
n
e
w
,
M
i
j
=
S
+
p
n
e
w
,
M
i
j
+
S
+
p
n
e
w
,
M
i
j
+
+
S
-
p
n
e
w
,
M
i
j
-
(5)
S
+
p
n
e
w
,
M
i
j
+
=
m
a
x
p
i
k
j
+
∈
M
i
j
+
S
p
n
e
w
,
p
i
k
j
+
(6)
S
-
p
n
e
w
,
M
i
j
-
=
m
a
x
p
i
k
j
-
∈
M
i
j
-
S
p
n
e
w
,
p
i
k
j
-where S
+
(
p
n
e
w
,
M
i
(
j
)
+
) and S
-
(
p
n
e
w
,
M
i
(
j
)
-
) are the similarity between P
n
e
w and positive training set M
i
(
j
)
+ and negative training set M
i
(
j
)
-.Expression (7) is used to decide whether a new patch P
n
e
w is a positive one. (7)
i
f
S
r
p
n
e
w
,
M
i
j
>
θ
i
j
t
h
e
n
p
n
e
w
∈
M
i
j
+
e
l
s
e
p
n
e
w
∈
M
i
j
-The output of integrator is measured by conservative similarity shown in the following.(8)
S
c
p
n
e
w
,
M
i
j
=
S
50
%
+
p
n
e
w
,
M
i
j
S
50
%
+
p
n
e
w
,
M
i
j
+
S
-
p
n
e
w
,
M
i
jwhere S
50
%
+
(
p
n
e
w
,
M
i
(
j
)
) is the similarity of the first 50% of the positive patches.For eachd
p
i
(
j
-
1
), we use single TLD framework to get its new position if it is visible in framej. Ifn objects are visible in current frame, we can get a new list and record it as T
L
D
(
j
)
=
{
t
b
i
(
x
,
y
,
w
,
h
)
,
i
=
1,2
,
.
.
.
.
,
N
}. Herex,y,w, andh are the position and size of the bounding box given by TLD.


### 3.2. List Updating for Multiobject Tracking

Because of the random of pedestrian’s appearance and disappearance, the dynamical maintenance of pedestrian list becomes a key in multipedestrian tracking and counting. For example, how to decide a target is a new or an existing one, especially for the case of occlusion or out of vision and then back again. Suppose every target is independent; we design a mechanism for the tracking list updating.We record the information such as position and size and lifespan from the first appearance to current framej as the trajectory of a tracking object. Tracking listTis the set of all tracking object. For current trajectory T, we note its i
t
h tracking target asT
i
=
{
t
b
, tbold, len, vlen, ivlen, rd,
l
d
}. Here, t
b is the position and size of current output by TLD in frame j. t
b
o
l
d is the last record of position and size before current. Andlen,vlen, andivlen are, respectively, the length of total frames (from first appearance to current frame), visible frames, and consecutive invisible frames. r
d and l
d are the labels to describe if a pedestrian passes the left or right border of buffer zone (introduced in Section 4). By comparing current D
P
(
j
) and the existing object trajectory listT, a new tracking list can be updated.Our tracking list updating can be divided into two steps: correlation matching and list adjustment. For each trajectory, correlation matching step is to find the most similar patch in the detection setD
P
(
j
). The similarity s
(
a
,
b
) between patchesa andb is measured by the Euclidean distance between their centroids. If the similarity s
(
T
i
.
t
b
,
d
p
h
(
j
)
) satisfies (9), patch d
p
h
(
j
) is accepted as the most resemble appearance of T
i.(9)
s
T
i
.
t
b
,
d
p
h
j
=
m
i
n
d
p
l
j
∈
D
P
j
s
T
i
.
t
b
,
d
p
l
j
a
n
d
s
T
i
.
t
b
,
d
p
h
j
<
T
0List adjustment process is to updateT. For those matched trajectories we update their records with new parameters. For those unmatched patches in D
P
(
j
), we regard them as new pedestrian targets. Then we add them into T as new trajectory elements. As the trajectories disappeared or unmatched for requirement threshold, we incline to view them as vanishing from the video and then remove them from the set T. The detail updating flow of tracking list is shown in Algorithm 1.Algorithm 1:The flow for multiobject tracking.
Input: Detected targetsD
P
(
j
)
=
{
d
p
1
(
j
)
,
.
.
.
d
p
N
d
(
j
)
} in current framej; Tracking list T;
Current bounding box setT
L
D
(
j
)
=
{
t
b
1
,
t
b
2
,
.
.
.
,
t
b
N
} given by TLD
Output: updated tracking listT
1 Trajectory Initialization. Initialize T according T
L
D
(
j
). If frame number is 1 then return
after initialization.
i
f
j
=
1
t
h
e
n
T
i
.
t
b
=
d
p
i
(
1
)
,
T
i
.
l
e
n
=
1
,
T
i
.
v
l
e
n
=
1
,
T
i
.
i
v
l
e
n
=
0
,
T
i
.
t
b
o
l
d
=
{
0
}
e
l
s
e
T
i
.
t
b
=
T
L
D
i
(
j
)
2 Correlation computation. For each trajectory T
i, compute the distance similarity to each
patch in the detection setD
P
(
j
), then find its most similar patch d
p
h
(
j
) according (9).
3 Trajectory Updating. If d
p
h
(
j
) is found, then update the trajectory T
i based on d
p
h
(
j
)
according to (10).
T
i
.
t
b
o
l
d
=
T
i
.
t
b
,
T
i
.
l
e
n
=
T
i
.
l
e
n
+
1
,
T
i
.
v
l
e
n
=
T
i
.
v
l
e
n
+
1
,
T
i
.
i
v
l
e
n
=
0
,
T
i
.
t
b
=
d
p
h
(
j
) 10
Else, update some parameters of trajectoryT
i according to 11.
T
i
.
t
b
o
l
d
=
T
i
.
t
b
,
T
i
.
l
e
n
=
T
i
.
l
e
n
+
1
,
T
i
.
i
v
l
e
n
=
T
i
.
i
v
l
e
n
+
1 11
4 Trajectory Deletion. Search throughout the updated trajectory set T. If a trajectory T
k
satisfies following condition, then delete it fromT.
T
k
.
l
e
n
<
8
&
&
T
k
.
v
l
e
n
/
T
k
.
l
e
n
<
0.6
o
r
T
k
.
i
v
l
e
n
>
200 12
5 Trajectory Insertion. For those unmatched patches {
d
p
l
(
j
)
} in D
P
(
j
), insert them into T
to form new trajectory records. Suppose the index number is m inT for a new record. Then the
parameters forT
m are set as following.
T
m
.
l
e
n
=
1
,
T
m
v
l
e
n
=
1
,
T
m
i
v
l
e
n
=
0
,
T
m
.
t
b
o
l
d
=
{
0
}
,
T
m
.
t
b
=
d
p
l
(
j
) 13


## 3.1. Single Pedestrian Tracking with TLD

Single pedestrian tracking with TLD include three components, namely, tracking, learning, and detection [5]. Data preparation is the first step to implement TLD framework. Given a target d
p
i
(
j
), use overlap area A
ounder different scales to choose positive and negative samples. Within each scale, we can get a collection of patches from top to bottom, left to right. Then calculate the overlap ratio r
=
A
0
/
(
d
p
i
(
j
)
.
w
∗
d
p
i
(
j
)
.
h
+
A
p
a
t
c
h
-
A
0
) between a patch and d
p
i
(
j
). Add the best 10 patches with maximum overlap ration (larger than 0.6) into the positive sample set. For those patches with overlap ration smaller than 0.1, we put them into the negative samples set. Before adding, each patch is resized into normalized size.Thusi
t
h object model in frame j can be expressed as M
i
(
j
)
=
{
M
i
(
j
)
+
,
M
i
(
j
)
-
}
=
{
p
i
1
(
j
)
+
,
p
i
2
(
j
)
+
,
.
.
.
,
p
i
m
(
j
)
+
,
p
i
1
(
j
)
-
,
p
i
2
(
j
)
-
,
.
.
.
,
p
i
n
(
j
)
-
}. p
i
1
(
j
)
+ is the first positive patch added to the set of M
i
(
j
). And p
i
m
(
j
)
+ is the positive sample added last so far. After learning initialization, we can use the TLD framework shown in [5]. In model update process, relative similarity S(a,b) is used to measure the similarity between objectsa andb. The similarity between the image patch P
n
e
wand setM
i
(
j
)is calculated as (4)-(7). (4)
S
r
p
n
e
w
,
M
i
j
=
S
+
p
n
e
w
,
M
i
j
+
S
+
p
n
e
w
,
M
i
j
+
+
S
-
p
n
e
w
,
M
i
j
-
(5)
S
+
p
n
e
w
,
M
i
j
+
=
m
a
x
p
i
k
j
+
∈
M
i
j
+
S
p
n
e
w
,
p
i
k
j
+
(6)
S
-
p
n
e
w
,
M
i
j
-
=
m
a
x
p
i
k
j
-
∈
M
i
j
-
S
p
n
e
w
,
p
i
k
j
-where S
+
(
p
n
e
w
,
M
i
(
j
)
+
) and S
-
(
p
n
e
w
,
M
i
(
j
)
-
) are the similarity between P
n
e
w and positive training set M
i
(
j
)
+ and negative training set M
i
(
j
)
-.Expression (7) is used to decide whether a new patch P
n
e
w is a positive one. (7)
i
f
S
r
p
n
e
w
,
M
i
j
>
θ
i
j
t
h
e
n
p
n
e
w
∈
M
i
j
+
e
l
s
e
p
n
e
w
∈
M
i
j
-The output of integrator is measured by conservative similarity shown in the following.(8)
S
c
p
n
e
w
,
M
i
j
=
S
50
%
+
p
n
e
w
,
M
i
j
S
50
%
+
p
n
e
w
,
M
i
j
+
S
-
p
n
e
w
,
M
i
jwhere S
50
%
+
(
p
n
e
w
,
M
i
(
j
)
) is the similarity of the first 50% of the positive patches.For eachd
p
i
(
j
-
1
), we use single TLD framework to get its new position if it is visible in framej. Ifn objects are visible in current frame, we can get a new list and record it as T
L
D
(
j
)
=
{
t
b
i
(
x
,
y
,
w
,
h
)
,
i
=
1,2
,
.
.
.
.
,
N
}. Herex,y,w, andh are the position and size of the bounding box given by TLD.


## 3.2. List Updating for Multiobject Tracking

Because of the random of pedestrian’s appearance and disappearance, the dynamical maintenance of pedestrian list becomes a key in multipedestrian tracking and counting. For example, how to decide a target is a new or an existing one, especially for the case of occlusion or out of vision and then back again. Suppose every target is independent; we design a mechanism for the tracking list updating.We record the information such as position and size and lifespan from the first appearance to current framej as the trajectory of a tracking object. Tracking listTis the set of all tracking object. For current trajectory T, we note its i
t
h tracking target asT
i
=
{
t
b
, tbold, len, vlen, ivlen, rd,
l
d
}. Here, t
b is the position and size of current output by TLD in frame j. t
b
o
l
d is the last record of position and size before current. Andlen,vlen, andivlen are, respectively, the length of total frames (from first appearance to current frame), visible frames, and consecutive invisible frames. r
d and l
d are the labels to describe if a pedestrian passes the left or right border of buffer zone (introduced in Section 4). By comparing current D
P
(
j
) and the existing object trajectory listT, a new tracking list can be updated.Our tracking list updating can be divided into two steps: correlation matching and list adjustment. For each trajectory, correlation matching step is to find the most similar patch in the detection setD
P
(
j
). The similarity s
(
a
,
b
) between patchesa andb is measured by the Euclidean distance between their centroids. If the similarity s
(
T
i
.
t
b
,
d
p
h
(
j
)
) satisfies (9), patch d
p
h
(
j
) is accepted as the most resemble appearance of T
i.(9)
s
T
i
.
t
b
,
d
p
h
j
=
m
i
n
d
p
l
j
∈
D
P
j
s
T
i
.
t
b
,
d
p
l
j
a
n
d
s
T
i
.
t
b
,
d
p
h
j
<
T
0List adjustment process is to updateT. For those matched trajectories we update their records with new parameters. For those unmatched patches in D
P
(
j
), we regard them as new pedestrian targets. Then we add them into T as new trajectory elements. As the trajectories disappeared or unmatched for requirement threshold, we incline to view them as vanishing from the video and then remove them from the set T. The detail updating flow of tracking list is shown in Algorithm 1.Algorithm 1:The flow for multiobject tracking.
Input: Detected targetsD
P
(
j
)
=
{
d
p
1
(
j
)
,
.
.
.
d
p
N
d
(
j
)
} in current framej; Tracking list T;
Current bounding box setT
L
D
(
j
)
=
{
t
b
1
,
t
b
2
,
.
.
.
,
t
b
N
} given by TLD
Output: updated tracking listT
1 Trajectory Initialization. Initialize T according T
L
D
(
j
). If frame number is 1 then return
after initialization.
i
f
j
=
1
t
h
e
n
T
i
.
t
b
=
d
p
i
(
1
)
,
T
i
.
l
e
n
=
1
,
T
i
.
v
l
e
n
=
1
,
T
i
.
i
v
l
e
n
=
0
,
T
i
.
t
b
o
l
d
=
{
0
}
e
l
s
e
T
i
.
t
b
=
T
L
D
i
(
j
)
2 Correlation computation. For each trajectory T
i, compute the distance similarity to each
patch in the detection setD
P
(
j
), then find its most similar patch d
p
h
(
j
) according (9).
3 Trajectory Updating. If d
p
h
(
j
) is found, then update the trajectory T
i based on d
p
h
(
j
)
according to (10).
T
i
.
t
b
o
l
d
=
T
i
.
t
b
,
T
i
.
l
e
n
=
T
i
.
l
e
n
+
1
,
T
i
.
v
l
e
n
=
T
i
.
v
l
e
n
+
1
,
T
i
.
i
v
l
e
n
=
0
,
T
i
.
t
b
=
d
p
h
(
j
) 10
Else, update some parameters of trajectoryT
i according to 11.
T
i
.
t
b
o
l
d
=
T
i
.
t
b
,
T
i
.
l
e
n
=
T
i
.
l
e
n
+
1
,
T
i
.
i
v
l
e
n
=
T
i
.
i
v
l
e
n
+
1 11
4 Trajectory Deletion. Search throughout the updated trajectory set T. If a trajectory T
k
satisfies following condition, then delete it fromT.
T
k
.
l
e
n
<
8
&
&
T
k
.
v
l
e
n
/
T
k
.
l
e
n
<
0.6
o
r
T
k
.
i
v
l
e
n
>
200 12
5 Trajectory Insertion. For those unmatched patches {
d
p
l
(
j
)
} in D
P
(
j
), insert them into T
to form new trajectory records. Suppose the index number is m inT for a new record. Then the
parameters forT
m are set as following.
T
m
.
l
e
n
=
1
,
T
m
v
l
e
n
=
1
,
T
m
i
v
l
e
n
=
0
,
T
m
.
t
b
o
l
d
=
{
0
}
,
T
m
.
t
b
=
d
p
l
(
j
) 13


## 4. Pedestrian Flow Counting

Tracking trajectory is the source of pedestrian counting. So far, single or double line counting is the popular way to count pedestrians (moving left or right). Considering the acquirement of real-time, stability, and accuracy, we choose single line counting with buffer zone shown in Figure3 to count pedestrians.Figure 3
Single line counting method with buffer zone.To avoid the direction statistical error caused by the wandering pedestrian near the counting line, buffer zone is introduced. An image frame only has one buffer which is a small range centered on the counting line (the center line of the video). And only when a target crosses the whole zone, does the direction counting continue.Let the margins of the buffer range beL
L and L
R. The way to count right or left moving pedestrians is shown in the following.14
i
f
T
i
.
r
d
=
1
&
T
i
.
l
d
=
1
t
h
e
n
N
u
m
R
=
N
u
m
R
+
1
i
f
T
i
.
r
d
=
-
1
&
T
i
.
l
d
=
-
1
t
h
e
n
N
u
m
L
=
N
u
m
L
+
1where15
T
i
.
r
d
=
1
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
>
L
R
&
&
T
i
.
t
b
.
x
<
L
R
&
&
T
i
.
t
b
.
x
>
T
i
.
t
b
o
l
d
.
x
-
1
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
>
L
R
&
&
T
i
.
t
b
.
x
<
L
R
&
&
T
i
.
t
b
.
x
<
T
i
.
t
b
o
l
d
.
x
0
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
<
L
L
∥
T
i
.
t
b
.
x
>
L
R
T
i
.
l
d
=
1
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
>
L
L
&
&
T
i
.
t
b
.
x
<
L
L
&
&
T
i
.
t
b
.
x
>
T
i
.
t
b
o
l
d
.
x
-
1
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
>
L
L
&
&
T
i
.
t
b
.
x
<
L
L
&
&
T
i
.
t
b
.
x
<
T
i
.
t
b
o
l
d
.
x
0
T
i
.
t
b
.
x
+
T
i
.
t
b
.
w
<
L
L
∥
T
i
.
t
b
.
x
>
L
ROnce targetT
i has been counted, moving label r
d and l
d will be reset to zero immediately.


## 5. Experiments and Analysis

Two experiments are designed to test the performance of our multiobject tracking and pedestrian counting on the database of PETS [21]. PETS is a public dataset which consists of multisensor sequences containing different crowd activities. It has been widely used to test the performance of new or existing systems of pedestrian detection and tracking within a real-world environment. We select ten clips for pedestrian counting from the PETS. The longest clip has 1189 frames and the shortest is 276 frames.


### 5.1. The Evaluation of Multitarget Tracking

MOTA (multiple object tracking accuracy) and MOTP (multiple object tracking precision) [22] are adopted to evaluate tracking process. MOTP evaluates the alignment of tracks with the ground truth. It measures the precision with which objects are located using the intersection of the estimated region with the ground truth region. And MOTA combines all missed targets, false positives, and identity mismatches and is normalized with the total number of targets (100% corresponds to no errors) [17]. Table 1 presents the comparison results of our method and some of the state-of-the-art approaches for multiobject tracking.Table 1
The comparison results of multitarget tracking.
Algorithm
MOTP
MOTA
Andriyenko [17]
69.0%
63.7%
Berclaz [18]
63.0%
77.0%
Milan [19]
67.2%
67.0%
Jin [20]
72.4%
72.1%
Our method
69.9%
71.4%From Table1, we can observe that our tracking approach can achieve a competitive result, especially the metric value of MOTP. For [17, 18], MOTP and MOTA cannot achieve good values simultaneously. When one measure increases, the other metric decreases obviously. Moreover, the MOTA of Berclaz approach is the highest, because it uses probabilistic occupancy map (POM) to create background and detection data. Compared with our basic foreground segmentation process without considering any prior real data, POM model uses some empty background images in PETS.As [19], it can obtain a good balance between MOTA and MOTP, but its metric value is a little lower than that of our method. The reason may be attributed to the performance of TLD algorithm. TLD combines learning, detection, and tracking together for a tracking task with parameters online updating. To achieve high detection accuracy, P-N learning paradigm is used to exploit the temporal and spatial structure in data, leading to the mutual compensation of missed detection and false alarms and the promotion of tracking accuracy and precision.We can also see that [20] can get a bit higher performance than our method. To use the holography information among multiple view, [20] integrates the crowd simulation into traditional single camera method. However, the two metric values in [20] are computed by manually assigning the size and position of the initial patch for each tracking. In contrast, our method extracts foreground and initializes each tracker automatically. This fully automatic step may decrease the performance of our tracking system to some extent. In order to improve the performance, in the initialization stage, we can use more proper detection method and better background modeling method in our future work.


### 5.2. Statistics of Pedestrian Flow

Table2 displays the results of our statistics of pedestrian flow. L and R mean the number of pedestrians walking toward left or right. In our 10 clips, the illumination of 1 and 2 is inadequate, clips 7 and 8 have many pedestrians, and 9 and 10 have serious occlusion problem. The other clips are in normal condition.Table 2
The statistics result of pedestrian flow counting.
Video sequence
Authentic pedestrian number
Our Statistics result
Statistical errors
Accuracy rate
1
L:12 R:6
L:10 R:6
L:-2 R:0
88.88%
2
L:15 R:7
L:13 R:6
L:-2 R:-1
86.36%
3
L:6 R:6
L:6 R:6
L:0 R:0
100%
4
L:8 R:15
L:10 R:14
L:+2 R:-1
86.96%
5
L:17 R:14
L:16 R:13
L:-1 R:-1
93.55%
6
L:9 R:11
L:8 R:12
L:-1 R:+1
90%
7
L:42 R:33
L:34 R:28
L:-8 R:-5
81.58%
8
L:39 R:25
L:32 R:20
L:-7 R:-5
81.25%
9
L:18 R:26
L:14 R:23
L:-4 R:-3
84.09%
10
L:21 R:12
L:17 R:10
L:-4 R:-2
81.82%From Table2, it is easy to be observed that our system obtains higher accuracy during normal condition (especially 3 and 5). For those clips with crowed pedestrians or frequent occlusion (clips 7, 8, 9, and 10), the accuracy is about 81% to 84%. For those clips (clips 1 and 2) with inadequate illumination but less objects and occlusion, the accuracy is over 86%. And for the all clips, our average statistical accuracy is 87.4%.


## 5.1. The Evaluation of Multitarget Tracking

MOTA (multiple object tracking accuracy) and MOTP (multiple object tracking precision) [22] are adopted to evaluate tracking process. MOTP evaluates the alignment of tracks with the ground truth. It measures the precision with which objects are located using the intersection of the estimated region with the ground truth region. And MOTA combines all missed targets, false positives, and identity mismatches and is normalized with the total number of targets (100% corresponds to no errors) [17]. Table 1 presents the comparison results of our method and some of the state-of-the-art approaches for multiobject tracking.Table 1
The comparison results of multitarget tracking.
Algorithm
MOTP
MOTA
Andriyenko [17]
69.0%
63.7%
Berclaz [18]
63.0%
77.0%
Milan [19]
67.2%
67.0%
Jin [20]
72.4%
72.1%
Our method
69.9%
71.4%From Table1, we can observe that our tracking approach can achieve a competitive result, especially the metric value of MOTP. For [17, 18], MOTP and MOTA cannot achieve good values simultaneously. When one measure increases, the other metric decreases obviously. Moreover, the MOTA of Berclaz approach is the highest, because it uses probabilistic occupancy map (POM) to create background and detection data. Compared with our basic foreground segmentation process without considering any prior real data, POM model uses some empty background images in PETS.As [19], it can obtain a good balance between MOTA and MOTP, but its metric value is a little lower than that of our method. The reason may be attributed to the performance of TLD algorithm. TLD combines learning, detection, and tracking together for a tracking task with parameters online updating. To achieve high detection accuracy, P-N learning paradigm is used to exploit the temporal and spatial structure in data, leading to the mutual compensation of missed detection and false alarms and the promotion of tracking accuracy and precision.We can also see that [20] can get a bit higher performance than our method. To use the holography information among multiple view, [20] integrates the crowd simulation into traditional single camera method. However, the two metric values in [20] are computed by manually assigning the size and position of the initial patch for each tracking. In contrast, our method extracts foreground and initializes each tracker automatically. This fully automatic step may decrease the performance of our tracking system to some extent. In order to improve the performance, in the initialization stage, we can use more proper detection method and better background modeling method in our future work.


## 5.2. Statistics of Pedestrian Flow

Table2 displays the results of our statistics of pedestrian flow. L and R mean the number of pedestrians walking toward left or right. In our 10 clips, the illumination of 1 and 2 is inadequate, clips 7 and 8 have many pedestrians, and 9 and 10 have serious occlusion problem. The other clips are in normal condition.Table 2
The statistics result of pedestrian flow counting.
Video sequence
Authentic pedestrian number
Our Statistics result
Statistical errors
Accuracy rate
1
L:12 R:6
L:10 R:6
L:-2 R:0
88.88%
2
L:15 R:7
L:13 R:6
L:-2 R:-1
86.36%
3
L:6 R:6
L:6 R:6
L:0 R:0
100%
4
L:8 R:15
L:10 R:14
L:+2 R:-1
86.96%
5
L:17 R:14
L:16 R:13
L:-1 R:-1
93.55%
6
L:9 R:11
L:8 R:12
L:-1 R:+1
90%
7
L:42 R:33
L:34 R:28
L:-8 R:-5
81.58%
8
L:39 R:25
L:32 R:20
L:-7 R:-5
81.25%
9
L:18 R:26
L:14 R:23
L:-4 R:-3
84.09%
10
L:21 R:12
L:17 R:10
L:-4 R:-2
81.82%From Table2, it is easy to be observed that our system obtains higher accuracy during normal condition (especially 3 and 5). For those clips with crowed pedestrians or frequent occlusion (clips 7, 8, 9, and 10), the accuracy is about 81% to 84%. For those clips (clips 1 and 2) with inadequate illumination but less objects and occlusion, the accuracy is over 86%. And for the all clips, our average statistical accuracy is 87.4%.


## 6. Conclusions

To realize multiobject tracking, our system combines TLD tracking algorithm and dynamic average background modeling. The former can track objects with long term and robustness in real-time. The latter with confirmation module can automatically localize candidate multiobject, which are further tested by pedestrian detection. By comparing the TLD output and pedestrian detection results, we can manage pedestrian records easily, such as updating parameters, inserting new objects, and deleting those disappeared ones for long time. Counting with buffer zone can decrease the influence of the wandering of pedestrians around counting line. The accuracy and stability of our system have been proved by several experiments and analysis. Future work will focus on how to improve the tracking accuracy for high-density crowd and better robustness.



---

*Source: 8486906-2018-10-29.xml*