USING PANORAMIC VIDEOS FOR MULTI-PERSON LOCALIZATION AND TRACKING
IN A 3D PANORAMIC COORDINATEFan Yang?‡ , Feiran Li∗∗ , Yang Wu† , Sakriani Sakti?‡ , and Satoshi Nakamura?‡

arXiv:1911.10535v5 [cs.CV] 8 Mar 2020?

Nara Institute of Science and Technology, Japan; ∗∗ Osaka University, Japan; † Kyoto University, Japan‡RIKEN, Center for Advanced Intelligence Project, Japan
ABSTRACT

3D panoramic multi-person localization and tracking areprominent in many applications, however, conventional methods using LiDAR equipment could be economically expensive and also computationally inefficient due to the processing of point cloud data. In this work, we propose an effective and efficient approach at a low cost. First, we utilize
RGB panoramic videos instead of LiDAR data. Then, we
transform human locations from a 2D panoramic image coordinate to a 3D panoramic camera coordinate using camera geometry and human bio-metric property (i.e., height).
Finally, we generate 3D tracklets by associating human appearance and 3D trajectory. We verify the effectiveness of
our method on three datasets including a new one built by
us, in terms of 3D single-view multi-person localization, 3D
single-view multi-person tracking, and 3D panoramic multiperson localization and tracking. Our code is available athttps://github.com/fandulu/MPLT.
Index Terms— 3D localization, multi-target tracking,panoramic videos.
1. INTRODUCTION
In daily life, we understand surrounding visual scenes in 3D.
For instance, we usually decide how to interact with surrounding people by first localizing and tracking them in a 3D egocentric coordinate: when we are walking down the street, weplan our path to avoid collisions by analyzing the trajectoriesof the surrounding people; when we see friends walking towards us, we might also walk to them and have a greeting.
For applications (e.g., social robotics) that also require visual scene understanding, performing multi-person localization and tracking in a 3D coordinate is strongly desired.
Typical single-view 3D coordinate localization methods
fall into two categories: using depth sensors, or, using object
size and camera geometry. Previous studies [1, 2] relied on
depth sensors (e.g., LiDAR) and instance segmentation to obtain the target location in a 3D camera coordinate. In practice,
however, the instance person segmentation algorithm is imperfect in crowd scenes, resulting in the assigning of incorrect
locations to a target person. To some extent, these methods
are more suitable for multiple vehicle tracking [3], since they

are rigid objects with known shapes and the distance between
them is generally larger. In contrast, other methods [4, 5] infer 3D camera-coordinate locations by object bounding box
size and camera geometry. However, there is a scale variance between standing persons and sitting persons in terms ofbounding box height. Moreover, when a person is near thecamera, only the upper body can be observed. Consequently,
simply taking the bounding box height as a reference is inappropriate. Recently, a study [6] demonstrated that using theskeleton length can obtain more accurate locations than using
bounding boxes. We embrace this idea into our framework to
obtain target locations in a single-view 3D camera coordinate.
Conventional multi-person tracking takes two stages.
The first detects each person by an object detector, and thesecond associates the cross-frame identities by considering
their appearance similarity and trajectory trend [7, 8]. Most
existing studies work on 2D/3D single-view [2, 5], or, 2D
panoramic multi-target tracking [9, 10]. However, previous
works have limitations: the 2D tracking results could not bedirectly used in some applications (i.e., robotics) since thereal-world coordinate is 3D; it is easy to lose the tracking target in a 3D narrow-angle-view coordinate since it only covers
a part of the surrounding environment. Therefore, we propose
3D panoramic multi-target tracking to address the aforementioned issues.
We propose a novel framework for multi-person localization and tracking in a 3D panoramic coordinate withpanoramic RGB videos1 . In our framework, 2D humanposes are estimated for each person to obtain the 2D location and body height. Utilizing single-view intrinsic camera
parameters, a person’s 3D location can be approximated byassuming the body height is a constant. We further transform locations from a 3D single-view camera coordinate to a
3D panoramic coordinate using extrinsic camera parameters.
Unlike in a 2D image coordinate, the real-scale location andmotion are preserved in a 3D coordinate. As a benefit, it iseasier to harness the power of Kalman filter to model human
trajectories. To further address issues like occlusion and missdetection, we associate the appearance similarity and the
trajectory trend together to approach multi-person tracking.
1 Yang Wu is the corresponding author of this work.

Upper body poses

2D image coordinateswith θ=90°v
u
3D coordinateswith θ=90°
Z+
X

Frame kab
Crop upper
body patches

Geometry
Transformation
ModuleTransform topanoramic
0°$
cameracenteredcoordinates
270°$

Pose Detection Modulec0°$
Frame k+1
90°$

90°$

270°$

Appearance
Re-identification
ModuledComputeappearance
distance

a

b

c

0.2

0.8

d

0.7

0.3

3D Panoramic
coordinates

Assign tracking IDs
and update tracking records

+
180°$

Tracking
Module
a

b

c

0.1

0.7

d

0.6

0.1

180°$
Frame kFrame k+1

Compute distance between
observed locations and estimatedlocations by Kalman-filter

Fig. 1. Our framework for 3D panoramic multi-person localization and tracking.
We annotated a Multi-person Panoramic Localization
and Tracking (MPLT) dataset to evaluate our framework.
We also compared our framework with others on the KITTI
dataset [11] and the 3D MOT dataset [12], where only singleview 3D localization and tracking results are provided.
2. METHODOLOGYOur framework includes four modules: Pose Detection Module, Geometry Transformation Module, Appearance Reidentification Module, and Tracking Module (see Fig. 1).
They work seamlessly together to achieve the target goal.
2.1. Obtain 2D Person Poses

capture panoramic videos. By removing the overlapping areas, we obtain four single-view images with a 90◦ Horizontal
Field of View at each frame. Following a clockwise path, we
can assign each single-view image with view angle θ, whereθ ∈ {0◦Y , 90◦Y , 180◦Y , 270◦Y }.
Let [uθ , vθ ]T be a point in the 2D image coordinate andlet [Xθ , Yθ , Zθ ]T be the corresponding point in the 3D camera
coordinates of each single view. Then, we have
 
 
uθ
Xθ
 vθ  = K  Yθ  ,1Zθ

where K is the intrinsic matrix.
To transform locations from 3D camera coordinates to a
3D panoramic coordinate, we construct an extrinsic matrix:
Similar to previous work [6], we use off-the-shelf PifPaf [13]as our Pose Detection Module to estimate 2D human poses.
Depends on the need, 2D person poses can be obtained either by a top-down approach or a bottom-up approach. In theformer, an object detector (e.g., YOLO [14]) is used to acquire the 2D bounding box for each person, and then PifPaf
estimates 2D poses within each single bounding box. Alternatively, PifPaf can simultaneously estimate 2D poses for all
persons and assign them to each person, which is a bottomup approach. Compared with the top-down approach, thebottom-up approach is faster but less accurate.
2.2. Coordinate Transformation
We build a Geometry Transformation Module to map person
locations from 2D image coordinates to a 3D panoramic coordinate. In our setting, four single-view cameras are used to

(1)

R
0T


t
≡ [R|t] ∈ R4×4 |R ∈ SO(3), t ∈ R3 ,1

(2)

where SO(n) denotes a Special Orthogonal Group with dimension n; 0 indicates a zero vector; R and t are the 3D
rotation matrix and the translation matrix. In our settings, all
single-view coordinate centers are close to each other, so thatt can be approximated by a zero vector and R only containY-axis rotation.
Accordingly, for location [X, Y, Z]T in a 3D panoramic
coordinate, the complete projection matrix can be defined:

and we havePθ = K[R(θ)|t],

(3)

 
 
X
uθ
Y 
 vθ  = P θ   .
Z 
1
1

(4)

At first glance, [X, Y, Z]T cannot be determined by[uθ , vθ ]T in the above equation. However, we assume that
real-world body height Hbody is a constant value. Since thecorresponding body height in a 2D image coordinate (i.e.,
hbody ) can be obtained by a pose estimator, for each person,
the corresponding X and Z can be calculated by solvingXhbody  = Pθ Hbody  , ∃Hbody ≈ constant.
 Z 
1
1


uθ





where Ctraj
i,j is the trajectory cost of instance i of the previous
frame to instance j of current frame. Additionally, [X̂i , Ẑi ]denotes the estimated location of instance i at current frame
by Kalman filter, while Lj,: = [Xj , Zj ] presents the detected
location of instance j at current frame, where L denotes thelocation values of all the detected instances.
traj
We can simply associate Cappi,j and Ci,j by letting(5)
app
Ci,j = Ctraj
i,j + Ci,j ,Hence, we can transform a target from a 2D single-view
image coordinate to a 3D panoramic coordinate. Since mostreal applications focus on the ground plane scenario, we treat
Y = 0 for all the persons in the 3D panoramic coordinate.

where Ci,j is the associate cost of matching instance i of theprevious frame to instance j of the current frame. Then optimal assignment M∗ is obtained by minimizing the total costM∗ = arg min

2.3. Matching Cost

M

The appearance of people can be utilized as an important
tracking cue to alleviate the occlusion issue in tracking. Although existing works exploit the entire body appearance [7,8], we suppose that only using the upper body appearance
can alleviate occlusion problems in crowd scenes. We further
demonstrate this point in our experimental results of Table 4.
Since 2D body poses are estimated in this work, the upper
body image patches can be cropped accordingly. We use an
off-the-shelf model [15] as our Appearance Re-identification
Module. Given an upper-body image patch, it extracts thecorrespondent appearance embedding vector.
In the tracking processes, appearance similarly is used to
re-identify each person in the spatio-temporal domain. Morespecifically, the appearance cost between two consecutive
frames is formulated asCappi,j = 1 −ai aj
, i ∈ {1, . . . , Ni }, j ∈ {1, . . . , Nj }kai k kaj k(X̂i − Xj )2 + (Ẑi − Zj )2
Ctraj
i,j = 1 − exp −
2
Hbody

(7)

i

Ci,j Mi,j ,jAlgorithm 1: Tracking algorithm
Input : k (current tracked frame number), C (association cost
matrix), L (instance location matrix), T (active instanceset), ε (matching cost threshold)1 if k = 1 then2
Initialize active instance set T ← ∅.
3
for j ← 1 to Nj do
4
Tj [location] ← Tj [location] ∪ {Lj,: }.
5
Tj [lif espan] = 10.
78where Cappi,j is the appearance cost of instance i of the previous frame to instance j of the current frame; Ni and Nj
are the corresponding number of instances; ai and aj are theappearance embedding vectors with dimension 2048.
Apart from the appearance cue, the trajectory trend isalso a critical cue to track targets. With regard to previous
works [7, 8], Kalman filter [? ] is commonly used to model
the trajectory trend. In contrast with modeling the trajectory trend in a 2D image coordinate, modeling it in a 3D
coordinate can alleviate the position and motion distortions,which simplifies the procedure of applying Kalman filter tomodel trajectories. To be consistent with Capp at value range
0 − 1, we apply an exponential kernel to calculate the distance between detected locations and Kalman filter estimatedlocations that are normalized by Hbody . The trajectory cost
between two consecutive frames is defined by

XX

(9)

where M is a Boolean matrix. When row i is assigned tocolumn j, we have Mi,j = 1. Note that, each row can be assigned to at most one column and each column to at most one
row. The optimization can be done by the Hungarian method.

6

(6)

(8)

9
10
11
12
13
14
15

16
17
18
19
20

21
22

else
Obtain M∗ by optimizing Eq. (9) with the Hungarian method.
Initialize the Nj × 1 dimensional matching indicator vectorm = 0 for current frame k.
for i ← 1 to Ni do
Ti [lif espan] = Ti [lif espan] − 1.
for j ← 1 to Nj do
if M∗i,j = 1 and Ci,j < ε thenTi [location] ← Ti [location] ∪ {Lj,: }.
Ti [lif espan] = 10.
mj = 1.
for j ← 1 to Nj do
if mj = 0 (i.e., instance j is unmatched) thenAdd one more active instance to T:
T|T|+1 [location] ← T|T|+1 [location] ∪ {Lj,: }.
T|T|+1 [lif espan] = 10.
if Ti [lif espan] = 0 thenRemove Ti from T.

for l ← 1 to |T| do
Update Kalman filter with Tl [location].
25
Tl [location estimated] = [X̂l , Ẑl ], estimated using the
updated Kalman filter.
Output: T
23
24

2.4. Multi-person Tracking
In the Tracking Module, we create a tracking set T to store
and update tracked instances. At the k-th tracked frame, we
obtain a set of 3D panoramic locations L by Eq. (5) and thecost matrix C by Eq. (8). In the first frame, all the observed
locations are assigned to a tracking set. After that, the acrossframe connections are determined by Mi,j and Ci,j . When
Mi,j = 1 and Ci,j is smaller than a threshold ε, the instancei of frame k − 1 is likely to be the instance j of frame k.
However, across-frame instances may not always be perfectly
matched. For unmatched instance j, we assign it to T as a new
instance. For unmatched instance i, which is already recorded
in T, we reduce its lifespan by 1. While new instances comeinto the tracking area, old instances may also leave. Therefore, we delete unseen instances in the tracking set after 10
frames. We summarize this process in Algorithm 1.

which outperforms the second place method by 1.5. For our
proposal dataset MPLT, we list the performance of our framework and make it as a baseline (see Table 4). Furthermore,
we also show, due to the occlusion, selecting the whole body
appearance may impair the model performance. The qualitative evaluation results are available on our project page 3 .
Table 2. Monocular-camera-based localization precision
on KITTI Dataset. If distance from predicted locations toground-truth location is within a threshold, it is correctly predicted.
Methods
Mono3D [17](training on KITTI)
SAMono[18](training on KITTI)
MonoDepth [19](training on KITTI)
MonoLoco [6] (training on KITTI)
Ours (w/o KITTI)

Localization precision by threshold< 0.5m < 1.0m
< 2.0m
13.2%
23.3%
39.0%
19.8%
33.9%
48.5%
20.6%
35.4%
50.7%
29.0%
49.6%
71.2%
22.0%
39.4%
63.5%

Table 3. 3D MOT Benchmark. ↑(↓) indicates that thelarger(smaller) the value is, the better the performance. Mul3. EXPERIMENTS
tiple Object Tracking Accuracy (MOTA) is the dominant criExperimental Datasets. We annotate a Multi-person Panoramic terion. The details of the evaluation metrics were previously
explained in [16].
Localization and Tracking (MPLT) Dataset to enable model
Methods
MOTA↑
MT↑
ML↓
FP↓
FN↓
evaluation on 3D panoramic multi-person localization and
AMIR3D
[20]
25.0
3.0%
27.6%
2,038
9,084
tracking. It represents real-world scenarios and contains a
MCFPHD [21]39.9
25.7% 16.8% 3,029 6,700
crowd of people in each frame. And, over 1.8K frames and
GPDBN [22]49.8
25.7% 17.2% 1,813 6,300
densely annotated 3D trajectories are included. For comparMOANA [23]52.7
28.4% 22.0% 2,226 5,551
Ours
54.2
30.6% 20.9% 2,385 4,930
ison with related works, we also evaluate our framework on
the KITTI [11] and 3D MOT [12] datasets. The properties ofTable 4. Performance of our framework on MPLT dataset.
three experimental datasets are listed as follows:
We evaluate localization and tracking performance within 10
Table 1. Properties of experimental datasets.
m of the coordinate center.
DatasetKITTI [11]3D MOT [12]MPLT

3D single-view
localization

3D single-view
localization&
tracking

3D panoramic
localization&
tracking

44Appearance Selection

Threshold

MOTA↑

Whole body
Whole body
Upper body
Upper body

< 0.5m
< 1.0m
< 0.5m
< 1.0m

62.4
70.2
65.2
74.9

4

Experimental Setup. Since off-the-shelf pose detector andappearance extractor are applied, we do not train any models
in this work. For the KITTI and MPLT datasets, we apply thebottom-up pose estimation approach. For 3D MOT dataset,
we apply the top-down pose estimation with the given public
bounding boxes. Based on the properties of each dataset, weevaluate the model performance from different perspectives.
Experimental Results. In Table 2, we report the localization precision under three thresholds for the KITTI Dataset. It
shows good generalization property of our method. Although
without any training on KITTI Dataset, its performance canreach the second place in terms of 3D single-view localization
without any extra training. In Table 3, we compare our framework with others on the 3D MOT Benchmark2 , which targets at 3D single-view localization and tracking. We achieve
the state-of-the-art performance (i.e., 1st place of the public leaderboard) on the dominant criterion (i.e., MOTA [16]),
2 https://motchallenge.net/results/3D_MOT_2015/

4. CONCLUSION
We proposed a simple yet effective solution for 3D panoramic
multi-person localization and tracking with panoramic videos.
On two existing datasets, the effectiveness of our method isdemonstrated by the promising performance. Meanwhile,a strong baseline is offered for our new benchmark dataset.
Since our method can faithfully keep the realistic locations
and motions for tracking targets in a 3D panoramic coordinate, it can help human-related video understanding applications. As future work, we plan to integrate our frameworkwith a previous work [? ] for automatically detecting humanactivities in a 3D panoramic coordinate.
ACKNOWLEDGEMENTS. Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101
and JP17K00237, and a MSRA Collaborative Research 2019
Grant by Microsoft Research Asia.
3 https://github.com/fandulu/MPLT

References[1] Aljoša Osep, Wolfgang Mehner, Markus Mathias, and Bastian
Leibe, “Combined image-and world-space tracking in traffic
scenes,” in 2017 IEEE International Conference on Robotics
and Automation (ICRA). IEEE, 2017.
[2] Marina Kollmitz, Andreas Eitel, Andres Vasquez, and Wolfram
Burgard, “Deep 3d perception of people and their mobility
aids,” Robotics and Autonomous Systems, vol. 114, 2019.
[3] Hyunggi Cho, Young-Woo Seo, BVK Vijaya Kumar, and Ragunathan Raj Rajkumar, “A multi-sensor fusion system formoving object detection and tracking in urban driving environments,” in 2014 IEEE International Conference on Robotics
and Automation (ICRA). IEEE, 2014.
[4] Samuel Scheidegger, Joachim Benjaminsson, Emil Rosenberg,
Amrit Krishnan, and Karl Granström, “Mono-camera 3d multiobject tracking using deep learning detections and pmbm filtering,” in 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE,
2018.
[5] Sarthak Sharma, Junaid Ahmed Ansari, J Krishna Murthy, andK Madhava Krishna, “Beyond pixels: Leveraging geometry
and shape cues for online multi-object tracking,” in 2018 IEEE
International Conference on Robotics and Automation (ICRA).
IEEE, 2018.
[6] Lorenzo Bertoni, Sven Kreiss, and Alexandre Alahi,“Monoloco: Monocular 3d pedestrian localization and uncertainty estimation,” arXiv preprint arXiv:1906.06059, 2019.
[7] Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu, Xiaohua Shi,and Junjie Yan, “Poi: Multiple object tracking with high performance detection and appearance feature,” in European Conference on Computer Vision. Springer, 2016.
[8] Nicolai Wojke, Alex Bewley, and Dietrich Paulus, “Simple
online and realtime tracking with a deep association metric,”
in 2017 IEEE International Conference on Image Processing(ICIP). IEEE, 2017.
[9] Ahmad Delforouzi, David Holighaus, and Marcin Grzegorzek,“Deep learning for object tracking in 360 degree videos,” inInternational Conference on Computer Recognition Systems.
Springer, 2019.
[10] Ahmad Delforouzi, Seyed Amir Hossein Tabatabaei, Kimiaki
Shirahama, and Marcin Grzegorzek, “A polar model for fast
object tracking in 360-degree camera images,” Multimedia
Tools and Applications, vol. 78, no. 7, 2019.
[11] Andreas Geiger, Philip Lenz, and Raquel Urtasun, “Are we
ready for autonomous driving? the kitti vision benchmark
suite,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[12] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele,“Monocular 3d pose estimation and tracking by detection,” in2010 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition. IEEE, 2010.
[13] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi, “Pifpaf:
Composite fields for human pose estimation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2019.
[14] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali
Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2016.

[15] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei
Jiang, “Bag of tricks and a strong baseline for deep person
re-identification,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
[16] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara,
and Carlo Tomasi, “Performance measures and a data set for
multi-target, multi-camera tracking,” in European Conference
on Computer Vision. Springer, 2016.
[17] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja
Fidler, and Raquel Urtasun, “Monocular 3d object detection
for autonomous driving,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[18] Dan Xu, Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, andElisa Ricci, “Structured attention guided convolutional neural fields for monocular depth estimation,” in Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[19] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow,“Unsupervised monocular depth estimation with left-right consistency,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2017.
[20] Amir Sadeghian, Alexandre Alahi, and Silvio Savarese,“Tracking the untrackable: Learning to track multiple cueswith long-term dependencies,” in Proceedings of the IEEE International Conference on Computer Vision, 2017.
[21] Nicolai Wojke and Dietrich Paulus, “Global data association for the probability hypothesis density filter using network
flows,” in 2016 IEEE International Conference on Robotics
and Automation (ICRA). IEEE, 2016.
[22] Tobias Klinger, Frank Rottensteiner, and Christian Heipke,“Probabilistic multi-person localisation and tracking in image
sequences,” ISPRS Journal of Photogrammetry and Remote
Sensing, vol. 127, 2017.
[23] Zheng Tang and Jenq-Neng Hwang, “Moana: An online
learned adaptive appearance model for robust multiple object
tracking in 3d,” IEEE Access, vol. 7, 2019.