Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?
Authors: Patrick Dendorfer
Vladimir Yugay
Aljosa Osep
Laura Leal-Taixé
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-object tracking
tracking
trajectory prediction
computer vision

Abstract: Recent developments in monocular multi-object tracking have been very successful in tracking visible objects and bridging short occlusion gaps, mainly relying on data-driven appearance models. 
While significant advancements have been made in short-term tracking performance, bridging longer occlusion gaps remains elusive: state-of-the-art object trackers only bridge less than 10% of occlusions longer than three seconds. 
We suggest that the missing key is reasoning about future trajectories over a longer time horizon. Intuitively, the longer the occlusion gap, the larger the search space for possible associations. 
In this paper, we show that even a small yet diverse set of trajectory predictions for moving agents will significantly reduce this search space and thus improve long-term tracking robustness. Our experiments suggest that the crucial components of our approach are reasoning in a bird's-eye view space and generating a small yet diverse set of forecasts while accounting for their localization uncertainty. This way, we can advance state-of-the-art trackers on the MOTChallenge dataset and significantly improve their long-term tracking performance. This paper's source code and experimental data are available at https://github.com/dendorferpatrick/QuoVadis.

Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?
Patrick Dendorfer
Vladimir Yugay
Aljoša Ošep
Laura Leal-Taixe
Technical University of Munich
{patrick. Shatrickkdendorfer, dendorfer,
vladimir.yugan aljosa. osep, leal. taixe) taixe>]tum de
Abstract
Recent developments in monocular multi-object tracking have been very successful in tracking visible objects and bridging short occlusion gaps, mainly relying on data-driven appearance models. While significant advancements have been made in short-term tracking performance. bridging longer occlusion gaps remains elusive: state-of-th--an object trackers only bridge less than 10% of occlusions longer than three seconds. We suggest that the missing key is reasoning about future trajectories over a longer time horizon. Intuitively, the longer the occlusion gap, the larger the search space for possible associations. In this paper, we show that even a small yet diverse set of trajectory predictions for moving agents will significantly reduce this search space and thus improve long-term tracking robustness. Our experiments suggest that the crucial components of our approach are reasoning in bird's s-eye view space and generating a small yet diverse set forecasts while accounting for their localization uncertainty. This way, we can advance state-of-the-ar trackers on the MOT Challleee dataset and significantly improve their long-term tracking performance. This paper's 's source code and experimental data available at htyps:////ttubb com http//dddooffppeee wawpndorferpatricc//uooooo
1 Introduction
Multi-object tracking (MOT) a long-standing research problem with wide range of applications,
including real-time dynamic situational awareness for robot navigation 221 15
77
44
62
10] traffic
monitoring [69], studying animal behavior 5 and monitoring biological phenomena
State-of-the-art MOT methods
75 [ in conjunction with identity re- re-identification (ReID) models 32 59
55
combine regression 75 and combinatorial
optimization
75 66
track objects in the image space. Such approaches have been very successful for tracking visible objects and bridging short-term occlusions. However, can be seen in Figure 1b long-term tracking remains an open challenge: state-of-ther -fttherrt methods successfully bridge 50% of occlusions within one second, falling below 10% when the occlusion extends for more than 3 seconds. This is often not reflected standard benchmarks [5| 21
69 77], as long-tenm perg-term& occlusions are statistically rare.
In the past, combining RelD models with simple motion models has been immensely helpful [TS] for short-term tracking. Nonetheless, as the occlusion time becomes longer, the set of possible associations grows exponentially with the increasing gap length (5 This combinatorial complexity hinders the ability of visual-basee ReID models disambiguate between objects. Consequently, we believe that ReID models are insufficient to resolve long-term occlusions. However, continued efforts to develop stronger appearance models will remain an important research direction in vision-basee MOT. Tracking moving pedestrians during occlusions challenging, and simple linear motion models fail since human motion is complex and driven by non-observable factors such goals, intent, or simply preferences. Therefore, we propose an alternative in this work: using long-term trajectory forecasting order to prune down the combinatorial search space of feasible trajectory continuations. As the main contribution of this paper, we carefully study what is needed to leverage trajectory forecasting for multi-object tracking, as we have recently witnessed rapid progress learning-based 36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0.7
SOTA
SOTA
Traj. Forecasting
0.5
1D Recall
0.4
0.1
Observation
Prediction
Matching
Occlusion Time [s]
Illustration of method.
(b) ID recall.
Figure 1: State-of-thc-as methods for dision-based MOT can successfully track visible objects and bridge short occlusion gaps; however, they fail at long-term tracking. (a) To bridge occlusion gaps, we lift monocular 2D detections to 3D world space, in which we reason about their possible future locations. This transformation allows us to reconnect detections that undergo long occlusions. As yellow track becomes occluded, our method predicts a small set of plausible future locations in 3D. In turn, we correctly associate red detection the yellow track by accounting for forecast uncertainty area. (b) As can be seen from the ratio of correct track association after different occlusion time lengths for the prior work and our method, this approach allows us to improve long-term tracking capabilities and gap longer occlusion gaps significantly. Best seen in color.
However, these methods operate in fully-observed, metric bird's-eye view (BEV) space, effectively disentangling the effect of the perspective projection reasoning about motion. By contrast, monocular MOT methods only observe a projection of the visible portion of our 3D space. Our analysis reveals that trajectories in BEV-space, but crucially, the localization of 2D bounding boxes in BEV must be temporally coherent. We achieve this by estimating a single homography per sequence in a data-driven manner.
trajectory forecasting 5 38
bridge this gap by localizing
Forecasting methods can reason beyond simple linear extrapolations, predict multiple possible future outcomes, and account for social interactions. But are these all necessary ingredients for bridging complex and long-term occlusions? Our study suggests that the key ingredient to estimate a set of forecasts that can possibly cover several diverging future paths with only handful of samples and account for prediction uncertainty.
Our trajectory forecasting approach can be applied to improve the long-term tracking capabilities of existing object tracking methods. In particular, by applying framework top of the state-of- the-art art method [8]] of the MOTChallenge benchmark, we improve the performance on HOTA on MOT17 by 0.09pp and MOT20 by 0.10pp and further decrease the number of IDSW by 93 and 36, respectively. We hope our conclusions will encourage the community to continue investigating how 3D) reconstruction and trajectory forecasting improve single-camera long-term tracking.
We summarize main contributions as follows: (i) present a study on how two related fields of research on vision-based trajectory forecasting and monocular multi-object tracking. Our study reveals that the core component of this interplay is temporally coherent reasoning about motion in 3D space. We (ii) utilize a synthetic MOT dataset to study how to localize objects in 3D BEV space in manner that facilitates robust reasoning about plausible future motion and which are the core forecasting components needed to bridge longer occlusion gaps; Finally, (iii) we demonstrate that can generalize our conclusions from synthetic sandbox real-world monocular MOTChallenge sequences and demonstrate that our recipe can be used to improve long-term tracking performance for several object trackers.
reconcile
2 Preliminaries
This section discusses the fundamentals of vision- based multi-objes tracking and trajectory forecast- ing, the current state-of-theeart, and analyzes failure cases.

---

2.1 Multi-object Tracking
Monocular multi-object tracking (MOT) is the task of localizing objects as bounding boxes image sequences and assigning them an dientity-preserving unique ID. State-of-the-art methods decompose the problem into object detection and detection association.
Quantifying tracking errors. The detection aspect of the task is commonly quantified by counting per-frame detection errors over the sequence. To quantify association errors, count identity switches (IDSW) (i.e., wrong ID swaps or re- re-initializing a ground-truth track with a different tracking ID) and identity transfers (IDTR) (i.e., incorrectly linking two different objects with the same tracking ID). While a successful association over occlusion gaps decreases the number of IDSW, a wrong association between tracklets leads to an IDTR instead. Recently introduced HOTA 4 metric separately evaluates object detection and temporal association aspects of the tracking task. Temporal association is quantified via association accuracy (AssA) term, that quantifies association recall (AssRe) and association precision (AssPr). The AssRe term accounts for IDSW errors, while AssPr accounts for IDTR errors.
Are all identity errors created equal? Correct track association of objects that undergo longer occlusion gaps is especially challenging because the appearance and position of an object may drastically change. In MOT datasets [5] 21 the majority of occlusions are short occlusions ($ 2s).
Hence, solving rare long occlusions (> 2s) does not significantly impact the model performance. As a result, long-term tracking is commonly overlooked in the literature. This can be seen in Figure 1b
state-of-the-arr methods bridge less than 10% gaps beyond Siree-second-long occlusions.
Prior work. Early tracking methods focus on combinatorial optimization 79 36
40 and
hand-crafting visual and motion-based descriptors 48 33 [1]] especially beneficial in the era of unreliable object detectors. State-of thetherrrt methods for monocular visual MOT are data-driven and primarily rely on appearance. Regression- methods [ 75 82 can localize objects even when object detections are missing, often used in conjunction with RelD models to bridge short occlusions. However, regression models fail when an occluded person appears at a distant image position. For solving long-term occlusion, discrete optimization methods, combined with end-to-end learning based on graph neural networks [S] 70 78 construct large graphs stretching over multiple seconds leading to high computational costs and complexity. Motion has always played an essential role 4. especially beneficial in 3D where it is dis-entangled from projective distortion [34 50 26 The interplay between reasoning in 3D for monocular pedestrian tracking and
in visual tracking 1
20
linear motion models was first investigated in [29]
Identity preservation is important in several applications, ranging from video editing, safety camera analysis, and social robots interacting with humans to autonomous driving. We only have access to a single RGB camera in several application scenarios. Exceptions are autonomous driving datasets 221 ITO that generally provide 3D sensory data, together with 3D track information. However, only handful of object tracks contain occlusion gaps longer than 2s: 0.6% in BDD100K [777 and 4% in widely-use used KITTI tracking [1] dataset. Therefore, autonomous driving datasets are, at the moment, not well suited for studying long-ter tracking. Instead we conduct our experiments and analysis using MOTChallenge [13] dataset, where 19.4% of tracks undergo long (> 2s) occlusions gaps.
We hypothesize that bridging long-term gaps requires understanding the projection geometry and motion models that can reason about plausible diverging future paths and non-linear motion.
2.2 Trajectory Forecasting
Pedestrian trajectory forecasting has been studied independently of the closely related task of object tracking. Forecasting is challenging because (i) human behavior and, therefore, future motion underlies the effect of complex social and scene interactions and latent navigating intent. Moreover, (ii) entire scene geometry is usually not directly visible to the observer, and in general, it is difficult to localize past trajectories precisely. To this end, existing models use standard datasets (53) 655 38 55
and study forecasting in idealized conditions: given an accurate bird's-cye view of the scene and perfectly-localized past trajectories to predict trajectory continuations in metric space.
Quantifying forecasting accuracy. Forecasting performance is measured in metric space as La distance between the prediction and ground-truth trajectory (as final displacement error, FDE, or average displacement error, ADE) wrt. top-k forecasts (commonly 1 20). We note that this approach mainly incentivizes high forecasting recall and neglects forecasting precision which is important for the application of forecasting methods [3]
3

---

a) e-ee view of delections and tracks
b) Trajectory forecasting for lost tracks
c) Filtering incorrect predictions
d) Matching predictions with new detections
Soeee Image
Scene Eye View
Vialake
Frecgpacc
Figure 2: Our method: we bridge long-term occlusions by (a) localizing object tracks in bird's bird's-eye view via the estimated homography and (b) forecasting future trajectories for lost tracks. We (d) continually aim match these inactive track predictions with new object detections and remove incorrect predictions under a visibility constraint (c).
Prior work. Early forecasting methods were deterministic, firstly based on physical models 25
and later on data-driven LSTM-based encoder-decoder networks I methods, focusing on modeling social 2 2 and scene 55 [1] interactions. The forecasting task is inherently uncertain, and we need to express the stochasticity in the model. With learning distribution of possible future trajectories, generative models
have emerged as state-of-the-ar prediction methods. Recent efforts have been explicitly focusing on conditioning forecasting on estimated pedestrian goal/intent [14 3 42 and estimating multimodal posterior distributions [38 [ that yield
56
diverse trajectories that cover different plausible directions. These deep neural network approaches can model complex and non-linear trajectories beyond simple linear models.
Can bring the two worlds together, and if so, how? Furthermore, which of the aforementioned aspects of forecasting methods (i.e.. stochasticity, non-linearity, multimodality, diversity, accounting for interactions) are important in the context of multi-object tracking? These are the questions we discuss in the following sections.
Methodology
In this section, we present our method for long-term multi-object tracking based on trajectory forecasting in bird's-eye view (BEV) scene representation. Simply applying trajectory prediction to multi-object tracking is not trivially possible, as object trajectories observed in the image space break multiple assumptions of real-world trajectory prediction. While trajectory prediction works bird's s- eye view coordinates, the motion and size of objects in image space depend on the camera's 's intrinsic parameters, orientation, and position. In addition, we face temporal (limited length of observation), association (association errors along with observation). and measurement (imprecise localization of objects) uncertainties of the trajectories. Contrarily, objects are represented as bounding boxes in the image instead of single 2D positions for the object tracking task. To bridge the gap between prediction and tracking, we must find a transformation from the image to the real space. We assume objects move planar ground to formulate such a transformation. Thus, the bottom-center points of detection bounding boxes p can be mapped to 2D BEV coordinate x via an initially unknown homography transformation H that relates the homogeneous coordinates
H p.
Overview. Given a monocular video sequence captured from stationary camera from arbitrary viewpoint, we first estimate the homography H, which maps the image plane to the 3D world ground- plane for the whole sequence (Section B.I. Then, we incorporate our model into an online tracker that takes monocular tracker output and localizes tracks and detections in BEV space (Figure : using the estimated homography. Next, we forecast lost tracks in BEV space (Figure 2b) using our trajectory forecasting network (described in Section 3.2) Finally, integrate forecasts into the online tracker (Section 3.3 while accounting for the uncertainty in estimated forecasts, and match new detections to existing tracks to resolve short- and long-term occlusions (Figure 2
3.1 Data-driven Homography Estimation
For combining monocular object tracking and forecasting, we first need transform object detections and tracks from the image sequence into points and trajectories a bird's bird's-eye view representation.

---

RGB
Image
Dapth Network
H
Image Depth
Homography
Transformation
Seg.
3D point cloud
Detections
Segmentation
Bird's-Eye View Representation
Detociir
Figure 3: We estimate the homography for a sequence by reconstructing 3D point cloud using monocular depth estimator. We obtain ground image-to toe-ttooo-n--ooudd correspondences using semantic segmentation model that masks ground pixels as needed to estimate the homography matrix. With the estimated homography matrix, we transform the bottom points of bounding boxes to 2D BEV coordinates.
Given set of 2D object detections represented bounding boxes localized in the image plane, aim to find a homography
that maps their bottom-center positions to their corresponding 2D BEV coordinates. In Figure El outline homography estimation method. We first train monocular depth estimator
[6] on synthetic dataset 19 to reconstruct a 3D point cloud (with estimated or known intrinsics) of the first frame of a static sequence. Then, we leverage the semantic segmentation network A to mask, select, and fit a plane to the ground pixels. We estimate the normal vector of the ground plane in 3D and align the plane the XY plane. Then, we project ground points along the z-axis, leaving us with pairwise correspondence between ground pixels in the image and 2D position in BEV, needed estimate the homography between the planes. We also linearize the homography transformation for pixel positions close to the plane's ' horizon to prevent the transformation from diverging (for more information, see Appendix [
Static camera. We compute homography only for the first frame the sequence and use throughout the sequence, making our pedestrian localization robust to temporal fluctuations of the depth estimator.
Moving camera. For moving camera sequences, we also need to account for the egomotion of the camera, which we estimate between consecutive frames as follows. First, compute frame- dependent homography He for each frame. Then, we compute pairwise pixel-correspondencs between (masked) ground pixels using optical flow 1 and compute translation vector between the two point sets (lifted to 3D via H2).
Empirically, we observe that estimating only translation (without rotation) yields more robust egomo- tion estimates.
3.2 Forecasting
Localization of object tracks in BEV enables leverage data-driven forecasting models beyond simple linear motion to reason for future trajectories. However, these models expect ground-truth fixed-size past trajectory observations, while our projected trajectories are noisy and of varying lengths. As discussed in Section 2.2 prediction models are optimized for metrics that incentivize large number of predictions and minimize L2 distance to ground-truth trajectories. It is thus unclear how different proposed concepts translate into real-world tracking scenarios. We, therefore, identify the main design patterns proposed in the forecasting community and verify their impact directly in the context of forecasting to track.
Preprocessing. Forecasting models encode trajectories using LSTM encoder-decoder [I] archi- tecture, which takes fixed-size observed trajectory as input and predicts a future trajectory. We construct input trajectories from temporally consecutive detections of the same track ID localized in BEV. To account for localization noise, we smooth the noisy observations using the Kalman filter and linearly extrapolate trajectories into the past get trajectories of the required fixed-size input length. Trajectory forecasting design patterns. our experimental setup, we build on the LSTM encoder- decoder architecture [I] and include the following key design patterns recently emerging in the forecasting community.
Stochasticity. Stochastic trajectory predictors enable us to sample multiple plausible future trajectories to account for the uncertainty in future positions. We follow the approach by (23) and learn generative GAN 22 model and train it with a best-of-many loss. As result, the network
5

---

IeV
1.0
0.74
Proels
Preels
0.50
0.43
12
30
5.
Tragectury Reseweeniicee
Oedasia Time
(a) Overall recall (BEV and pixel-space).
(b) Recall wyt. different occlusion lengths.
Figure 4: Endpoint matching recall of predictions and GT trajectories using linear motion predictor. A prediction is successfully matched when Abo > 0.5 or Arg distance < 2m. We also project the prediction back to the image for forecasts in the bird's s-eye view. The model Pixel (L2) predicts motion in pixel space and projects the endpoint into BEV for matching.
internally learns an ieservation-conditionne distribution of future trajectories, from which we can sample.
Social Interactions. Social interactions impact future motion: pedestrians adapt their trajectories mh-the-fly to avoid collisions. Several methods 23 II
BI account for interactions in the forecasting literature. These methods leverage pooling 23 attention [56 graph neural netwchecklorks BI
to provide social context (i.e., trajectories of surrounding agents) to the trajectory decoder. To answer whether modeling social interactions matters for tracking, implement Social GAN (S-GAN) (2), which uses pairwise interaction features between neighboring pedestrians by using max-pooling before they are passed to the decoder.
Multimodality and Diversity. While the aforementioned generative models learn a distribution over trajectories, they need to sample many trajectories to cover all modes present in the scene, as learning a multimodal posterior with a single GAN is difficult with as few samples possible, we implement multi generator GAN network [13], extending the presented GAN architecture by training multiple decoder heads, where each decoder learns focus on particular model. As a result, we get set of plausible but maximally separated predictions by sampling from these different generators.
To predict the scene's main modes
3.3 Tracking via Forecasting
We assume we have an online object tracker capable of tracking visible objects (e.g.. bounding box regression-basee tracker 4). As long tracks are being updated with new detections, we consider them active and keep them in the active set S.. Once we cannot associate a detection, a track becomes inactive and is stored in the set of inactive tracks S₁. For each frame i the tracker outputs set of tracks O = (o)..... оM) with o; (ID,, b;, fi) where ID E N+ represents the track identity, b E R² denotes bounding box in pixel space (see 2. ) and f E IR represents D-dim feature vector encoding the appearance information obtained from a pre-trained convolutional network 24]. We
localize bounding boxes in BEV coordinates R² using our estimated homography H.
Quo Vadis? If object track becomes inactive (i.e., temporally lost), we move the active track into the memory bank and predict & trajectories of length Tmaa in BEV space using the trajectory forecasting model as described in Section 3.2 As long as the track is inactive and not yet matched, we move along the predicted trajectory and do not predict an entirely new trajectory in each frame. Filtering and removing predictions. No prediction can live forever. When we stochastic trajectory predictors with multiple predictions, we need to limit the number of inaccurate or obsolete predictions decrease the chance of false re-association. In practice, we limit the lifetime of prediction to maximal lifetime of Tmaa and try to filter out unlikely forecasts. We use spatial and social context determine the freespace [29] in which objects should be visible the camera. We assume that visible objects eventually are detected and, therefore, remove prediction branches that should be visible for more than Teis frames.
We consider an object as visible if neither scene other pedestrians occlude the object. In particular, this means that the predicted BEV position lies in an area of the projected ground mask (shown in Figure 2 and has bounding box overlap > 0.25 with any other object detection, closer the camera. Relative order can be determined based on bottom bounding box coordinates for amodal detections. If all predictions from an inactive track are removed even before Tmaz, we also remove the entire track.

---

Matching predictions with new defections. Given the trajectory forecasts, we match them with new detections via bi- partite matching, following the standard practice in tracking [1 68 This boils down to computing association costs Cij between the predictions of an inactive track and new un-associated detections j:
39
[
Cij = (Aiou max (TL2 - A2:: 0)) App 2 TApp and DoU 2 Tlou);
(1)
where ALs is the IoU score hetween the two bounding boxes, L, is the Fuclidean distance between the prediction i and a detection j in BEV, and A App represents the cosine distance between visual features fi and fz. T ToU, and TApp denote thresholds for the matching. Therefore, we determine an association in BEV metric space and in the image domain using IoU bounding box overlap between the forecast and detected box.
Matching tracks based on spatial distance in real space leads to high recall and reduces the number of ID switches (IDSW) but may also lead to an increase in ID transfer errors (IDTR), especially in crowded scenes with many new detections close each other. While forecasting significantly narrows combinatorial search space for associations, verifying potential associations with an appearance model is still beneficial in practice. To decrease the number of wrong associations, we require minimal visual similarity TApp and minimum IoU overlap of the bounding boxes ToU for close objects. These thresholds serve filter of visually incompatible matches but do not add the value of the cost function for the matching. In essence, we obtain a pre-selection of potential matching candidates by using the trajectory forecast and filter those if the appearance drastically deviates between the last observation and the new detection.
Experimental Evaluation
In this section, we first discuss our evaluation test-bed, followed by an experimental study on bird's- eye-view (BEV) trajectory reconstruction (Section 4.T). Then, we analyze different forecasting design patterns applied to the domain of object tracking in BEV space and discuss the relevance of different model modules for tracking (Section 4.2) Afterward, we demonstrate how our approach can be used to improve several vision-based MOT methods on static sequences and to justify our design decisions. Finally, we show that our forecasting model can be used to establish new state-of-the on the real-world MOT17 and MOT20 datasets (Section For visualization of our tracking method, we refer the reader to Appendix C
Datasets. We evaluate our trajectory prediction framework on different publicly available pedes- trian tracking datasets, namely synthetic MOTSynth [9] and two real-world MOT17 and MOT20 datasets [5| MOTSyuth is large synthetic dataset for multi-object tracking. It provides 764 diverse sequences with various viewpoints, lighting, and weather conditions. Importantly, it provides ground-truth depth information and 3D key points for pedestrians, allowing us to study the suitability of different methods for BEV trajectory reconstruction. MOT17 147 and MOT20 [16] are real- world tracking datasets commonly used to benchmark pedestrian tracking models. We use these datasets to evaluate our method on real-world recordings. For our experiments, we utilize the commonly used split of the MOT17 training set, where the first half of each sequence is used for training and the second half for the evaluation Y7
XC 72
Metrics. For measuring the quality of the bird's-eye view reconstruction, we indirectly evaluate the quality of the reconstruction by evaluating the forecasting and tracking performance.
To compare different models for trajectory forecasting, we report the standard Ly final displacement error (FDE) for top-k predictions for 2s and 4s prediction horizons (see Section 2.2).
For multi-object tracking evaluation, we report higher-order tracking accuracy (HOTA) 41 with a focus on the association aspect of the task. To this end, we also report AssA, AssPr, and the number of ID switches IDSWs. Additionally, we report IDSW when the tracker loses an object and re-initiates a new track for the same object when it re-appears. We call these errors as ID lost, For metric discussion, we refer to Section 2.1 Metrics labeled with either S (short) or (long) only consider prediction or occlusion lengths shorter or longer than 2s, respectively
Hyperparameters. For all experiments, we the following parameters for the matching of detections with inactive tracks: TLa = 2.5m, Tap = 0.8, T = 0.2 The maximal lifetime of prediction is Tmaz = 6s and maximal visibility Tvis 1s before is removed. We refer the reader to Appendix B for further information on implementation details.
Object trackers. We study and ablate our method on eight high-ranked state-of-the-art trackers of MOTChallenge and refer to them as baseline. We use BYTE [X)] JDE (8). CSTrack (37] FairMOT [81 TraDes
72), QDTrack [5]]. CenterTrack [82 and TransTrack
[ for an evaluation
7

---

Table 1: Which forecasting modules matter for tracking? Evaluated on MOTI7 validation set.
Model
Nr. Samples
Deser- ministic
Stoch-
Social
Multi- modal
Prediction
Tracking
PDE
AssA
AssPy
1055"
Sttie
-14.77
Kalman (pixel)
54.08
54,02
72.83
-22.37
1:23
54.04
71:73
-19.50
-16.77
GAN
1.26
-17.99
GAN
0.0
0.98
53.81
53:40
60.45
71.31
-18.00 %
-15.33
S-GAN
0.87
1.21
54.52
34.78
61.22
73.28
-16.92
.8.7 %
MG-GAN
0.67
1.03
54.52
54.80
61.35
73.13
-21.19 19
-17 43/4%%%
on the MOTI7 validation set and BYTE and CenterTrack on the MOT20 training dataset. These trackers use RelD similarity and/or simple motion cues for bridging short-term occlusions.
4.1 B-rd's-Eye View Estimation
This section discusses different approaches obtaining scene BEV representations of detected objects in the image for forecasting. We work with static sequences of the MOTSynth dataset (that provides depth maps used for evaluating and training monocular depth estimator). With this, test a linear motion model to gap occlusions of different durations, which we obtain by running a CenterTrack [82] baseline tracker. We evaluate the ratio of successful matches between the target and the linear prediction and count match to be successful if the IoU of the predicted bounding box is larger than 0.5 the L2 distance in metric space is lower than 2m. We get the predicted bounding box by translating the last observed bounding box by the predicted displacement in the image.
Baselines. We compare motion in (i) BEV and (ii) pixel space. We evaluate different approaches to localize trajectories: (a) using ground truth (GT) 3D coordinates orthographically projected to BEV (oracle), (b) the proposed homography estimation as described in Section 3.1 and (c) directly using learned monocular depth estimates and resulting point clouds, followed by orthogonal projection of points these representing
object.
Conclusions. As seen in Figure GT 3D (oracle) based motion estimates solve 89.3% of the gaps, suggesting that the motion in the synthetic dataset is dominantly linear. Our proposed data- driven homography estimation approach only drops by 15% compared using ground-truth 3D keypoints. By contrast, estimating linear motion in pixels space only resolves 50.2%, and using per-frame monocular depth estimates 43.1% of the occlusion gaps. This is likely because such depth estimates are not temporally stable. As can be seen in Figure 4b this performance is especially apparent for longer occlusion gaps. Furthermore, we forecast motion in pixel space but transform the prediction into BEV for matching. While increasing performance compared to exclusive forecasting and matching pixel space, we find that the results are inferior to predictions in BEV due to the distortion of projecting real motion into the image plane. We conclude that our proposed homography transformation is suitable for forecasting.
4.2 Trajectory Prediction Models
In this section, we evaluate different forecasting models and components (as discussed in Section 3.2) We compare conssanttvelociit model (Kalman filter) in BEV and pixel space, identity (static) model, and a stochastic GAN predictor generating predicting social interactions with GAN and the multimodal trajectories with MG-GAN in BEV space.
and = 20 samples. Furthermore, we test
Scures
ToU
Threshold
HOTA 1 AwAt
AssRe
AssPy
ID
53.56
60143
72.21
%
%
60.43
36
55555
60.97
72.00
Sart
5555
61.17
02%
$3.97
53.75
61.08
70.73
%
36.05
53.92
61,07
71.01
21.40
64.27
54.29
61.08
72.36
200.599
Figure 5: Visualization of BEV reconstruction for moving camera sequence and egomotion estimation.
Table 2: Ablation of matching prediction and ef- fect of different thresholds 7 on different track- ing metries.
Forecasting. We observe in Table [ that the linear model performs well for short-term windows (0.69 FDEs), suggesting that linear motion is suitable for short occlusions. We also do not find significant difference between GAN w/o social module (S-GAN). While FDE error suggests vanilla GAN (k 20) yields the best forecasting results (0.65 FDEs and 0.99 FDE,), but this configuration
8

---

Table 3: We improve tracking results of all top-8 state-of- tr- models (MOT17 validation set and MOT20 training set). Differences to the baseline performance are shown in (-).
MUTIT yval sodie scenes)
MOT2O (inin)
SYTE bul
Come:Track
CSirack 11
FaiMOT L1
TraDeS
TransTookk X3
OUTaak 5
BYTE
Centerfook
X
HOTT
71.36 (-0.2))
61.78 (+1.56
61.00 (+0.43)
58.22 (-0.09
06 (40.20)
62.45(40.67
50.6 (40.55
58.87 (40.54 54
(6655
52.71 (+0,6)
AMA
73.56 (-0.40)
06. (+7.54)
05.54 (40.5
59.21 (+0.37)
54.26 (41.05
67.41
(5,47100.99
08.14 (-) 223
53.90 (+0.20)
2656 (+1.34)
AeRi
79.21 (-0,6))
60 65 (+5.58)
69 (43.07
64.31 (40.5)
50.82 (40.89)
73.1 (62.43
69.19640.0)
55.31 (42.06)
59.89 (40.4)
54.34 (65, 12)
Andlt
85.31 (0.67)
a1.33 (:547)
260
74.45 (3:31
0851-2335)
830--1991
18.31--1.00
77.0-244)
08:00 (5524)
52.57 (:21 (0)
IDSW
(:30
(-16)
200 (20)
(-133
516
106
112
2191-34)
1815 (:78)
92.92
73.93 (x0.59)
75.36 (11.21
72.47 (10.37)
IDR
78.11 (-0.39)
65.25 (+5.25)
67.53 (+0.67)
66.3 (-0.51
56.08 (+1.09
67. (-3 (4.000
61.59 (-0000)
62.07 (-0.65)
66. 04 (40,34)
55.77 (-3.23
IDP
$7.72 (-0.44
56.71 (+5.3)
52.23 (+1.05)
K3.65 (+0,67)
77.33 (+1.53
85.55 (11.4)
85.47 (+0002)
81.17 (H0.99)
(904))
03.53 (15
leads to the lowest association precision (71.31) and HOTA score (53.81) wtz. tracking performance. This result suggests a misalignment of evaluation metrics used in forecasting and tracking; a better forecaster in ferms of ADE/DDE does not necessarily lead better tracker. This is known drawback of ADE/DDE metrics, which essentially measure only recall and not the precision of the forecasting output. Furthermore, this shows the careful trade-off between the number of predictions k and the recall/precision of the predictions and tracking results.
Tracking. To investigate the effect on long-term occlusions, we foeus the change of ID for short (tocel S 2s) and long (foed >> 2s) occlusion gaps. As can be seen in Table I even the static motion model solves 8.4% (ID) as many occluded objects do not move. By modeling linear motion (Kalman filter in pixel space), we can improve short-term re-association for 0.59pp (long-term IDSW) over the static model. We focus the discussion on long occlusion gaps, In terms of the generative model, we observe that interaction-awee GAN (8.57% ID for kt- = 3; interestingly, both are below linear Kalman filter (BEV) performance, suggesting that these models suffer from low precision. Only MG-GAN explicitly trained to generate multimodal trajectories, outperforms the linear model (17. 43% ID(x"*) and significantly outperforms vanilla GAN with only three samples. These conclusions generalize to tracking metrics.
S-GAN (8.64% =f is on-par with vanilla
4.3 Tracking Evaluation
In this section, we study the impact of forecasting models on the valuation set's tracking performance. First, discuss the impact of different design decisions on matching strategy, explained in Section 5
Trajectory matching. First, we ablate the matching cost function (Equation (I). As can be seen in Table 2 we find that combination of L2 and IoU without any threshold T leads to the highest decrease in terms of ID (-28.02%) and overall highest association recall (AssRe) (61.17). However, this is at the cost of decreasing association precision (AssPr) (-4.09). We obtain the highest AssPr (73.36) by only relying on L2 matching and thresholding, however, at the loss of AssRe (-0.74). Adding appearance-based Tapp and IoU ToU thresholds provide the best trade-off and overall highest AssA (54.29) and HOTA score (54.27) while still recovering 21% of lost trajectories. Validation results. In Table 3 we present the performance of different state-of-the-art trackers on the static sequences of 170/7-val and M0120-train (trained on MOT17), equipped with our trajectory forecasting model. As can be seen, our model brings stable improvements over all the key metrics: HOTA, AssA, and IDSW. Our trajectory prediction model consistently reduces IDSW for all models. This is also shown in Figure lb where we demonstrate that our forecasting model improves ID recall significantly for occlusion times > 18.
While our focus was on sequences with stationary viewpoints, we show that our model is also applicable to sequences with moving cameras by estimating the camera's 's egomotion described in Section 3.1 In Table 4 we present results on the moving sequences of the MOT17 validation set (excluding sequence 17-07 for which the quality and consistency of our depth estimator was too low to construct time-consistent homographies). As can be seen, we improve 6 out of 8 trackers wrt. HOTA score and even improve CenterTrack [ by 3.07 pp. We visualize the traversed BEV map of sequence MOT17-07 in Figure 5
4.4 Benchmark Evaluation
In this section, we apply our method to state-of-the-an tracker Byte 30 and, by improving its long-term tracking capabilities, establish a new state-of-the-art on the MOT17 MO120 benchmarks. We evaluate our method in the private detection regime, as these trackers use private detectors. In Table 5] we compare our VudVadis to the base tracker ByteTTrack (80) and compare both to MOT17 benchmark published per-orrrormrrs. We improve performance on key metries overall top methods. Notably, we reduce the number of identity switches by 93 compared to 80 and establish new
9

---

Table 4: Results of top-8 state-of-the the-t---ttrtttt art models on dynamic scenes of the MOT17 validation set excluding MOT17-05. Differences in the baseline performance are shown in (+)
MOF17 (val, moving secnes)
BYTE Mi
CenterTinck [R]
CSTrack E
FairMOT
IDE [6
TraDeS PP1
TrancTrack X1
QiTTrack S1
HOTA
60.08 (:0.02)
(+3.07)
(0.0)
56.1 (0.0)
(2.14()1.47)
(11227
52.7 (:0.38)
(+0.76)
AssA
60.44 (+0.05)
53 (+6.49)
(40.0)
61.15 (0.0)
(-3.00)
54.08 (+2,44)
51.99 (40.54)
53.71 (+1.6)
AssRc
66.53
58.21 (+8.32)
63.35 (+0.0)
65.87 (0.0)
(-.72
59,86 (+3.1)
99.13 (+0.79)
61.8 (+3.82)
AscPr
78.29 (+0.09)
76.98 (-4.66)
(-0.6
(0.0)
(-0.9)
(-3.08)
72.45 (-0.32)
72.53 (-4.76)
54 (+1)
131 (-62)
97 (-5)
86 (0)
122 (-10)
(-1))
120
71 (-7)
MOTA
72.54(-0.00)
59.46 (+0-16)
60.68 (40.04)
63.78 (0.0)
60.52 (-0.07)
64.13 (+0.08)
(+0.04)
60.21 (+0.05)
IDFI
73.11 (0.0)
(+5.76)
(0.0)
73.1 (0.0)
68.23 (+1.85
67.72 (+2.29)
(+0.79)
(+2.86)
Table 5: Comparison under the "private detector"
Table 6: Comparison under the "private detector" protocol on MOT20 test set.
protocol MOT17 test set.
Tracker
HOTA
IDFI
MOTA
IDSW
AsxA
Tracker
HOTA
IDFI
MOTA
IDSW
AssA
ReMOT
10
59.73
71.99
77.01
2853
$7.08
FairMOT S1
54.42
68.44
59.57
1881
56.6
CrowdTrack
6
60.26
73.62
75.61
2544
59.26
CrowdTrack 61
54.95
68.24
70.68
3198
52.57
TLR
177
60.72
73.58
76.48
3369
58.88
57.28
71.15
73.90
MAA
20
61.98
75.88
79.36
1452
60. 16
ReMOT 726
61.15
73.14
77.42
1789
58.68
ByteTTack 80
63.05
77.30
2196
61.97
BytcTrack
61.34
75.20
77.76
1223
59.55
Quo Vadis (Ours)
63.14
77.71
80.27
2103
62.07
Quo (Ouns)
61.48
75.70
77.77
1187
59.87
state-of-the-art in terms of HOTA (63.14). We observe similar trends on MOT20, where we improve over the base tracker Byte [80] by +0.5 in terms of IDFI and reduce the number of identity switches by 36, similarly establishing a new state-of-the-art (61.48 HOTA).
Remarks and Limitations
The paper primarily focuses on the conceptual work of building an entire pipeline from video to tracks studying different forecasting paradigms, and showing the benefit of leveraging trajectory forecasting in BEV for the tracking task. Nevertheless, we want to outline further remarks and limitations of our work.
Model complexity. Our model is complex, consisting of multiple sub-modules, as constructing object trajectories in BEV space based monocular video and trajectory forecasting are challenging problems. We foresee that future work will improve the end-to-end integration and efficiency of the algorithms.
Bird's-eye view reconstruction. A vital part of our approach is an accurate homography trans- formation that allows us to project the objects in the image into the 3D ground plane. However, the homography transformation depends on the quality of depth estimates, which makes the overall approach sensitive to errors in 3D localization. Future work will benefit from further development of time-consistent depth estimators. Furthermore, the presented trajectory prediction models do not yet account for the BEV localization uncertainty, which results from errors in the transformation or the simplified assumption of the ground plane. These limitations show the need to develop trajectory forecasting models that account for the localization uncertainties of the upstream tasks.
6 Conclusion
This paper presents study how bridge the gap between real-world trajectory prediction and single-camera tracking. Throughout our paper, we identified challenges and solutions to leveraging real-world trajectory prediction to benefit single-camera tracking. In particular, we focus on resolving the re-identification of objects after long-term occlusions. Here, we start from the first principles, questioning motion representation in pixel space and using a combination of models to construct a more accurate BEV representation of the scene. We find that the key component is a forecasting approach reasoning about multiple feasible future directions with small set of multimodal forecasts. We can substantiate our conclusion by achieving new statc-of-the-arr performance on the MOT17 and MOT20 datasets.
Ultimately, we have showcased novel way of combining state-of-the-art trajectory prediction models and multi-object tracking task. We have outlined a new way of thinking about motion prediction in tracking and motivating the beneficial symbiosis of both tasks. We hope that both fields start moving towards each other and incorporate the requirements and needs of each other.
Acknowledgments. This research was partially funded by the Humboldt Foundation through the Sofja Kovalevskaja Award.
10

---

References
[1] Alexandre Alahi, Kratarth Goel. Vignesh Ramanathan, Alexandre Robiequet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In Conference on Computer Vision and Pottern Recognition, 2016.
[2] Javad Amirian, Jean-Bemard Hayet, and Julien Peuré. Social Ways: Learning multi- distributions of pedestrian trajectories with GANs, Conference on Computer Vision and Pattern Recognition Workshops, 2019.
[3] Samreen Anjum and Danna Gurari. CTMC: Cell tracking with mitosis detection dataset challenge. In Conference on Computer Vision and Pattern Recognition Workshops, 2020.
[4] Philipp Bergmann, Tim Meinhardt, and Laura Taixé. Tracking without bells and whistles. In International Conference on Computer Vision, 2019.
Alex Bewley, Zongyuan Ge. Lionel Oit, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In International Conference on Image Processing, 2016.
[6] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Conference on Computer Vision and Puttern Recognition, 2021.
[7] Apratim Bhattacharyya, Bernt Schicle, and Mario Fritz. Accurate and diverse sampling of sequences based on a "Best of Many" sample objective. In Conference on Computer Vision and Pattern Recognition, 2018.
[8] Guillem Braso and Laura Leal-Taixe. Learning a neural solver for multiple object tracking. In Conference Computer Vision and Pattern Recognition, 2020.
[9] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, 2012.
[10] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Conference on Computer Vision and Pattern Recognition, 2020.
[11] Wongun Choi. Near-online multi-target tracking with aggregated local flow descriptor. In International Conference on Computer Vision, 2015.
[12] MMFlow Contributors. MMFlow: Openmmlab optical flow toolbox and benchmark. https://github com/ http///onmmmmaaaaaa hati-mmmmmammmmm/o/ooo 2021.
[13] Patrick Dendorfer, Sven Elfiein, and Laura Leal-Taixé, MG-GAN: A multi generator model preventing out-of-distribution samples in pedestrian trajectory prediction. In International Conference on Computer Vision, 2021.
[14] Patrick Dendorfer, Aljoka Ošep, and Laura Leal-Taixé. Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation. In Asian Conference on Computer Vision, 2020.
[15] Patrick Dendorfer, Aljoša Ošep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. MOTChallenge: A benchmark for single-camera multiple target tracking. In International Journal of Computer Vision, 2020.
[16] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixt MOT20: benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.0900. 2020.
[17] A. Dosovitskiy, P. Fischer, E. Ilg. P. Häusser, C. Hazurba, V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In International Conference on Computer Vision, 2015.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
[19] Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljoša OSep. Simone Calderara. Laura Taixé and Rita Cucchiara. MOTSynth: How can synthetic data help pedestrian detection and tracking? In International Conference Computer Vision, 2021.

---

[20] Andreas Geiger, Martin Lauer, Christian Wojek, Christoph Stiller, and Raquel Urtasun. 3D traffic scene understanding from movable platforms. In Transactions Pattern Analysis and Machine Intelligence. 2014.
[21] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. Conference on Computer Vision and Pottern Recognition, 2012.
[22] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Conference on Neural Information Processing Systems, 2014.
[23] Agrim Gupta, Justin Johnson. Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Conference Computer Vision and Pattern Recognition, 2018.
[24] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
[25] Dirk Helbing and Péter Molnár. Social force model for pedestrian dynamics. Physical Review 1995.
[26] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Lin, Min Sun, Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu. Joint Monocular 3D Vehicle Detection and Tracking. In International Conference on Computer Vision, 2018.
[27] Chang Huang, Bo Wu, and Ramakant Nevatia. Robust object tracking by hierarchical association of detection responses. In European Conference on Computer Vision, 2008.
[28] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard I. Hartley, Learning to estimate hidden motions with global motion aggregation. In International Conference Computer Vision, 2021.
[29] Tarasha Khurana, Achal Dave, and Deva Ramanan. Detecting invisible people. In International Conference Computer Vision, 2021.
[30] Daniel Kondermann, Rahul Nair, Stephan Meister, Wolfgang Mischler, Burkhard Güssefeld, Katrin Honauer, Sabine Hofmann, Claus Brenner, Bernd Jähne. Stereo ground truth with error bars. In Asian Conference on Computer Vision, 2014.
[31] Vincet Kosaraju, Amir Sadeghian, Roberto Martin-Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-BiGAT: Multimodal trajectory forecasting using Bicycle-GAN and graph attention networks. In Conference on Neural Information Processing Systems, 2019.
[32] Laura Taixé. Cristian Canton-Ferred and Konrad Schindler. Learning by tracking: Siamese cnn for robust target association. In Conference on Computer Vision and Pattern Recognition Workshops, 2016.
[33] Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learning an image-based motion context for multiple people tracking. Conference on Computer Vision and Pottern Recognition, 2014.
[34] Bastian Leibe, Konrad Schindler, Nico Cornelis, and Luc Van Gool. Coupled object detection and tracking from static cameras and moving vehicles. In Transactions on Pattern Analysis and Machine Intelligence. 2008.
[35] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by Example. In Comput. Graph. Forum, 2007.
[36] Yuan Li, Chang Huang, and Ramkat Nevatia. Learning to associate: Hybridboosted multi target tracker for crowded scene. In Conference on Computer Vision and Potiern Recognition, 2009.
[37] Chao Liang, Zhipeng Zhang, Xue Zhou, Bing Li. Shuyuan Zhu, and Weiming Hu. Rethinking the competition between detection and rReID multiobject tracking. In Transactions on Image Processing. 2022.
[38] Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, and Alexander Hauptmann. The Garden Forking Paths: Towards Multi-Future Trajectory Prediction. In Conference on Computer Vision and Puttern Recognition, 2020.
[39] Qiankun Liu, Qi Chu, Bin Liu, and Nenghai Yu. GSM: Graph similarity model for multi-object tracking. In International Joint Conferences on Artificial Intelligence, 2020.
12

---

[40] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019.
[41] Jonathon Luiten, Aljoša OSep. Patrick Dendorfer, Philip Torr Andreas Geiger, Laura Leal- Taixé, and Bastian Leibe. HOTA: higher order metric for evaluating multi-object tracking. In International Journal of Computer Vision, 2020.
[42] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In International Conference on Computer Vision, 2021.
[43] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision, 2020.
[44] Roberto Martin-Martin Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, Jun Young Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese. JRDB: A dataset and benchmark egocentric robot visual perception of humans in built environments. In Transactions on Puttern Analysis and Machine Intelligence, 2021.
[45] N. Mayer, Ilg, P. Häusser, Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Conference on Computer Vision and Pottern Recognition, 2016.
[46] Moritz Menze, Christian Heipke, and Andreas Geiger. Object scene flow. In Journal of Photogrammetry and Remote Sensing, 2018.
[47] A. Milan, L. Leix- I. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-objeet tracking. In arXiv preprint arXiv: 1603 0003,0 2016.
[48] Anton Milan, Stefan Roth, and Konrad Schindler. Continuous energy minimization for multitarget tracking. In Transactions on Pattern Analysis and Machine Intelligence, 2014.
[49] Anton Milan, Konrad Schindler, and Stefan Roth. Detection- and trajectory- exclusion multiple object tracking. In Conference on Computer Vision and Pattern Recognition, 2013.
[50] Aljoša Ošep. Wolfgang Mehner, Markus Mathias, and Bastian Leibe. Combined image- and world-space tracking in traffic scenes. In International Conference on Robotics and Automation, 2017.
[51] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Conference on Computer Vision and Pattern Recognition, 2020.
[52] Malte Pedersen, Joakim Bruslund Haurum, Stefan Hein Bengtson, and Thomas B Moeslund. 3D-ZeF: 3D zebrafish tracking benchmark dataset. Conference on Computer Vision and Pattern Recognition, 2020.
[53] S. Pellegrini, Andreas Ess, and L. Gool. Improving data association by joint modeling of pedestrian trajectories and groupings. In European Conference on Computer Vision, 2010.
[54] Donald B Reid. An algorithm for tracking multiple targets. In Transactions on Automatic Control, 1979.
[55] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social ctiquette: Human trajectory understanding in crowded scenes. In European Conference Computer Vision, 2016.
[56] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive GAN for predicting paths compliant to social and physical constraints. In Conference on Computer Vision and Pattern Recognition. 2019.
[57] Leslic N. Smith and Nicholay Topin. Super-convergence:: Very fast training of neural networks using large leamnigg rates. In arXiv preprint arxiv: 2018
[58] Samuel I.. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
[59] Jeany Son, Mooyeol Back, Minsu Cho, and Bohyung Han. Multi-object tracking with quadruplet convolu- tional neural networks. In Conference on Computer Vision and Puttern Recognition, 2017.
[60] Daniel Stadler and Jürgen Beyerer. Modelling ambiguous assignments for In Winter Conference on Applications of Computer Vision, 2022.
tracking in crowds.
13

---

[61] Daniel Stadler and Jürgen Beyerer. On the performance of crowd-specifie detectors in multi pedti-pedesttian tracking. In International Conference on Advanced Video and Signal-Based Surveillance, 2021.
[62] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Conference on Computer Vision and Pattern Recognition, 2020.
[63] Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. In arXiv preprint arXiv: 20/2. 15460, 2020.
[64] Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: no gans land. Proceedings of Machine Learning and Systems, 2020.
[65] Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon. Learning to track with object permanence. In Conference on Computer Vision and Pattern Recognition, 2021.
[66] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, B.B.G Sekar, Andreas Geiger, and Bastian Leibe. MOTS: Multi-object tracking and segmentation. In Conference on Computer Vision and Pattern Recognition, 2019.
[67] Qiang Wang, Yun Zheng, Pan Pan, and Yinghui Xu. Multiple object tracking with correlation learning. In Conference on Computer Vision and Pattern Recognition, 2021.
[68] Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi-object tracking. In European Conference on Computer Vision, 2020.
[69] Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei, Ming-CChing Chang, Honggang Qi, Jongwoo Lim, Ming- Hsuan Yang, and Siwei Lyu. UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking. Computer Vision and Image Understanding, 2015.
[70] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris Kitani. GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning. In Conference on Computer Vision and Pattern Recognition, 2020.
[71] Bo Wu and Ramkat Nevatia. Detection and segmentation of multiple, partially occluded objects by grouping, merging, assigning part detection responses. International Journal of Computer Vision, 2009.
[72] Jialian Wu, Jiale Cao. Liangchen Song, Yu Wang, Ming Yang. and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In Conference on Computer Vision and Pattern Recognition, 2021.
[73] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https 2019.
//github. //github.com//ffceee http:///.cccccooooeeeee///ettttt
[74] Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Mameda-Pinede Tran- scenter: Transformers with dense queries for multiple-object tracking. arXiv preprint arXiv:2103.1111 151.55 2021.
[75] Yihong Xu, Aljoša OKep, Yutong Ban, Radu Horaud, Laura Leal- Taixé, and Xavier Alameda-Pineee How to train your deep multi-object tracker. In Conference on Computer Vision and Pattern Recognition, 2020.
[76] Fan Yang. Xin Chang, Sakriani Sakti, Yang Wu, and Satoshi Nakamura. ReMOT: A model-agnostic
refinement for multiple object tracking. In Image and Vision Computing, 2020.
[77] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Conference on Computer Vision and Pattern Recognition, 2020.
[78] Jan-Nico 7aech, Alexander Liniger. Dengxin Dai, Martin Danelljan, and Luc Van Gool, I Learnale Online Graph Representations for 3D Multi-Object Tracking. In Robotics and Automation Society, 2022.
[79] Li Zhang, Li Yuan, and Ramakant Nevatia. Global data association for multi-object tracking using network flows. In Conference on Computer Vision and Pattern Recognition, 2008
[80] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng. Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. ByteTrack: Multi-object tracking by associating every detection box. In European Conference on Computer Vision, 2022.
14

---

[81] Yifu Zhang. Chunyu Wang, Xinggang Wang. Wenjun Zeng, and Wenyu Liu. FairMOT: On the fairness of detection and re-identification in multiple object tracking. In International Journal of Computer Vision, 2021.
[82] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects points. In European Conference on Computer Vision, 2020.
15


Reviews:
[{"review_id": "vbEJoxftums", "reviewer": "Reviewer_Tgjz", "summary": "This paper discovers that trajectory predictions for moving agents will significantly reduce this search space and explores the trajectory prediction can improve long-term tracking robustness of MOT. Furthermore, they show that their proposed method reasons MOT in the bird-eye space and generates a small yet diverse set of forecasts while accounting for their localization uncertainty. Therefore, they manage to advance state-of-the-art trackers on the public benchmarks.\n\n", "questions": "- The proposed model estimates the depth and segmentation masks of the video frame, and thus project to BEV space. The quality of BEV is the basis of trajectory prediction. This paper indeed investigated how good the view projection (in Fig. 4) in synthetic data. I wonder if is it possible to conduct experiments to assess the impact of different view projection methods on MOT? \n- The pointers to tables are missing, which makes reading a bit difficult.\n- MOTA seems to be a more popular metric in MOT challenge. In previous MOT works (e.g., Bytetrack), they provide MOTA metrics. For better reference, it would be better to provide MOTA as well.\n- The implementation details on how to cope with the moving camera are not clearly provided. ", "limitations": "- The major limitation of this work on the uncertainty of homography transformation has been discussed. Their proposed method partially account for these uncertainties via simple strategies. Considering this is a novel exploration along this direction, I think the limitations have been sufficiently addressed. \n\n- The potential negative societal impact of their work was not dicussed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This idea is interesting and it will benefit the development of MOT community. Considering occlusions are one of the main challenges in tracking, this work proposes a novel way to overcome this concern. \n- This paper identifies several hurdles in the integration of trajectory prediction and MOT. Besides, there are several interesting and inspiring conclusions. \n- The presentation of this paper is fine and the organization is clear.\n- They achieve the state-of-the-art performance on public benchmarks. Maybe, it may inspire more following MOT methods on this path. \n\n### Weaknesses\n- The framework seems too complex. It includes five independent sub-models, e.g., depth estimation network and segmentation network. All of these sub-models require separate training, which may degrades the robustness of the framework.\n- Before the deep learning era, there are a few works that already attempted to incorporate crowd motion prediction or crowd motion models, e.g., social force, to MOT, some of which are listed below. It'd be better to refer to these works.\n> G Antonini，SV Martinez，M Bierlaire，JP Thiran: _Behavioral Priors for Detection and Tracking of Pedestrians in Video\nSequences_ IJCV 2006  \n> Stefano Pellegrini, Andreas Ess, Luc Van Gool: _You'll Never Walk Alone: Modeling Social Behavior for Multi-target Tracking_ ICCV 2009  \n> Kota Yamaguchi, Alexander C. Berg, Luis E. Ortiz, Tamara L. Berg: _Who are you with and Where are you going?_ CVPR 2011  \n> Wenxi Liu, Antoni B. Chan, Rynson W. H. Lau, Dinesh Manocha: _Leveraging Long-Term Predictions and Online Learning in Agent-Based Multiple Person Tracking_ TCSVT 2015  \n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "r-nQzm78JhF", "reviewer": "Reviewer_yuJE", "summary": "The paper investigates the exploitability of trajectory forecasting in multi target tracking. First, unmatched trajectories and unmatched detections are projected in bird-eye view (BEV), trajectories are then extended according to some model and tested for association. Projection in BEV also introduces visibility constraints that further reduces the search space of true matchings. The author validate their projection method against the true 3D position of pedestrians (in MOTSynth), then study the applicability and benefits of different trajectory forecasting models. They show that the proper combination can improve the HOTA score and reduce ID switches of 7 SOTA methods on MOT17 validation set. In the supplementary, the authors detail how their projection method can be extended to moving sequences.", "questions": "- Can the authors add the ID recall from [A] to the ablation studies in Tab.1 (the MOTChallenge evaluation kit does provide the means to compute it as it is required to compute the final ID F1) or, in alternative, justify why they don't find the metric appropriate and instead use AssA, AssPR and IDS.\n- Can the authors help the reader putting in perspective the impact of the different contributions, i.e. working in bird eye view vs trajectory forecasting?\n- Can the authors motivate the choice of 2m as a threshold for BEV matching and try to justify the difference between pixels and pixels 2D in Fig.4b?\n- Can the authors help the reader understanding the end of line 322 \"tracking results suggest otherwise\"? How do the authors explain this unexpected result?", "limitations": "nothing else to add", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- (+) paper is well written and nicely structured, the reading is fluent; figures and tables are also helpful in the understanding of the key steps\n- (+) the analysis of the contributions is carried out with an appropriate level of rigour\n- (+) results are in favor of the proposed approach \n\n- (-) HOTA is a complex and composed measure that tries to unify/project many aspects of tracking to a single scalar, so the authors have to compensate by also using AssA, AssPR as well as ID switches. Why did the authors thought these metrics to be more appropriate than ID recall [A] which measures the ability of a tracker to associate the same identity to a trajectory despite occlusions and interruptions? Isn't this exactly what the authors are trying to improve? Is the \"ID Recall\" in Fig.1 the one from [A]?\n\n[A] Performance measures and a data set for multi-target, multi-camera tracking\n\n- (-) From what I can see from Tab.1 most of the gain comes from a linear prediction method in 3D space. Other than that there is only a 0.5% HOTA left to gain from other trajectory forecasting methods. In this perspective I think the paper is overstating the importance of trajectory forecasting methods in multi target tracking and not helping the reader draw the correct conclusions. 3D tracking has been around since MOTChallenge 15 and kalman filtering even earlier and this seems to be the thing that explains 90% of the performance improvement (54.11-50.71)/(54.52-50.71).\n\n- (-) For the same reason as above, I would have liked to see a comparison on 3D MOT 2015\n\n- (-) In Fig.4 B we see a small difference between H and pixels L2, and a large difference between pixels L2 and pixels. The only way the reviewer can explain the difference between pixels L2 and pixels is through the use of a threshold in different domain, which seems to indicate that the threshold in BEV is less tight than the threshold in pixel space. As a matter of fact 2m seems like a very large margin to associate a trajectory and a detection, how did the authors choose this threshold? I think to remember that for both MOTA and IDF1 the threshold for 3D tracking was 1m instead of 2m (can be checked in the MOTChallenge evaluation kit).\n\n- (-) lines 127-129 \"simply applying trajectory prediction to MOT is not trivially possible\" are misleading, the community has been doing this for years and it's also reported in Tab.1 Kalman Filter (pixel) with decent results.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r-jJL5Ca4vd", "reviewer": "Reviewer_cbmW", "summary": "This paper explores the long-term occlusion problem in multi-object tracking, and proposes to using trajectory forecasting methods to compensate the tracking losts. The forecasting module is conducted in bird-eye-view. The method can advance state-of-the-art trackers on the MOT Challenge dataset. ", "questions": "See weaknesses. ", "limitations": "The authors discussed the limitations in the paper. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I like the idea in this paper. I agree that most current multi-object trackers do not tackle long-term occlusions. The solution in this paper is sound and makes sense to me. However, I feel that the work of this paper hasn’t been finished yet. It doesn’t have supportive experiments. I hope the authors can improve the draft with the following comments. \n\n- Strengths\n1. The paper is well-written and easy to follow. \n2. The paper tackled an existing problem in multi-object tracking and the solution is reasonable. \n\n- Weaknesses and Suggestions \n1. It makes sense that long-term occlusion rarely occurs in MOT Challenge datasets. However, the paper should have supportive experimental results. Current results, boost tracker by ~0.1 HOTA cannot prove the effectiveness of the method. I suggest the authors try to test the oracle of forecasting matching and show the advanced percentage of the method in this part of errors. \n2. How about doing experiment in autonomous driving datasets? The dataset has camera parameters. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3Mxa-OCb_Xt", "reviewer": "Reviewer_XrjC", "summary": "This paper investigates the problem of long-term multi-object tracking. This is a relatively under-explored problem that worths investigation since most existing tracking methods focus on short-term tracklets (usually shorter than 2s). The main contribution of this paper is that they project the scene into bird-eye views (BEV) using homography transformation, and then apply trajectory forecasting and tracking jointly in the BEV space. The original version described in the main text supports only videos captured from fixed views, and an extending version described in the supplementary material steps further to support videos captured from moving cameras. Results on standard test sets suggest the proposed method improves tracking accuracy when combined existing trackers, and refreshes state-of-the-art when combined with the best performant tracker ByteTrack.", "questions": "1. As mentioned in the paper, forecasting multiple possible trajectories for lost tracklets bring more computation cost when perform association. It would be good to see how this affect the tracking accuracy/efficiency and how to do the trade-off.\n\n2.  When dealing with moving cameras, homography is estimated for each single frame. This seems contradict conclusions in Figure 4(a), where it is shown that per-frame depth estimation is not stable so that the esitmated homography is largely affected. Is it possible to stablize the  depth estimation along the temporal dimension? \n\nTypo:\nL#309 0.43% -> 43.0%", "limitations": "The authors carefully discussed the limitation and and potential negative societal impact of their work in the main text.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nGood. Existing MOT datasets and evluation metics focus more on short-time tracking accuracy. While many methods already perform fairly well on these metrics, they always fail when targets lost for relatively long time.  This paper introduces trajectory forecasting in the BEV space to handle long-term lost tracklets and shows promising results. To my knowledge, this is not well-explored in previous literature so the originality is good.\n\nQuality:\n\nGood. The entire pipeline is simple, easy to understand, and seems work well for long-term tracklets.  \nMy biggest concern is about the forecasting module. From Table 1 it can be seen that applying Kalman Filter in the BEV space already performs well in terms of both prediction and tracking, and using advanced learning based methods seems do not bring too much gain. This is somehow below my expectation since Kalman Filter is such a simple linear model. Perhaps the problem lies in that most motion in the considered dataset is linear, and in this case, comparing different forecasting modules in a dataset with more complicated (non-linear) motion partterns may be helpful to validate the effectiveness of GAN. \n\nClarity:\n\nGood but can be improved. In general, the presentation is clear and easy to follow. However, it would be better if key experimental conclusions are highlighted more clearly, especially for Table 1. \n\nSignificance:\n\nGood. The long-term tracking problem is absolutely an imporant problem that is not well-explored due to limitation of existing datasets and metrics. This paper sheds new light on a promising direction towards solving this problem:  combine forecasting and tracking in BEV space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept