HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

arXiv:2009.07736v2 [cs.CV] 29 Sep 2020

Jonathon Luiten · Aljos̆a Os̆ep · Patrick Dendorfer · Philip Torr · Andreas Geiger ·Laura Leal-Taixé · Bastian Leibe

Abstract Multi-Object Tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize
the importance of either detection or association. To address this, we present a novel MOT evaluation metric, HOTA(Higher Order Tracking Accuracy), which explicitly balances
the effect of performing accurate detection, association andlocalization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which
are able to evaluate each of five basic error types separately,
which enables clear analysis of tracking performance. We
evaluate the effectiveness of HOTA on the MOTChallenge
benchmark, and show that it is able to capture important
aspects of MOT performance not previously taken into account by established metrics. Furthermore, we show HOTA
scores better align with human visual evaluation of tracking
performance.1

gt:
A:

B:

DetA MOTA HOTA IDF1 AssA

100

50% 50% 50% 67% 50%

50

70% 69% 50% 52% 35%

35

35

C:
25

25

25

25

100% 97% 50% 25% 25%
Increasingly
Measuring
Detection

Increasingly
Measuring
Association

Fig. 1 A simple tracking example highlighting one of the main differences between evaluation metrics. Three different trackers are shown inorder of increasing detection accuracy and decreasing association accuracy. MOTA and IDF1 overemphasize the effect of accurate detection
and association respectively. HOTA balances both of these by being an
explicit combination of a detection score DetA and an association score
AssA.

1 Introduction
Keywords Multi-Object Tracking · Evaluation Metrics ·Visual Tracking

J. Luiten, B. Leibe
RWTH Aachen University, Germany
E-mail: {luiten,leibe}@vision.rwth-aachen.de
A. Os̆ep, P. Dendorfer, L. Leal-Taixé
Technical University Munich, Germany
E-mail: {aljosa.osep,patrick.dendorfer,leal.taixe}@tum.de
P. Torr
University of Oxford, UK
E-mail: phst@robots.ox.ac.uk
A. Geiger
Max Planck Institute for Intelligent Systems, Tübingen, and University
of Tübingen, Germany
E-mail: andreas.geiger@tue.mpg.de
1 Pre-print. Accepted for Publication in the International Journal ofComputer Vision, 19 August 2020.
Code is available athttps://github.com/JonathonLuiten/HOTA-metrics.

Multi-Object Tracking (MOT) is the task of detecting thepresence of multiple objects in video, and associating thesedetections over time according to object identities. The MOT
task is one of the key pillars of computer vision research,and is essential for many scene understanding tasks such as
surveillance, robotics or self-driving vehicles. Unfortunately,
the evaluation of MOT algorithms has proven to be very difficult. MOT is a complex task, requiring accurate detection,
localisation, and association over time.
This paper defines a metric, called HOTA (Higher Order
Tracking Accuracy), which is able to evaluate all of these
aspects of tracking. We provide extended analysis as to why
HOTA is often preferable to current alternatives for evaluating MOT algorithms. As can be seen in Fig. 1, currently
used metrics MOTA [8] and IDF1 [60] overemphasize detection and association respectively. HOTA explicitly measuresboth types of errors and combines these in a balanced way.
HOTA also incorporates measuring the localisation accuracy of tracking results which isn’t present in either MOTA
or IDF1.

2

HOTA can be used as a single unified metric for ranking
trackers, while also decomposing into a family of sub-metrics
which are able to evaluate different aspects of tracking separately. This enables clear understanding of the different typesof errors that trackers are making and enables trackers to be
tuned for different requirements.
The HOTA metric is also intuitive to understand. Thiscan be seen clearly in Fig. 1. The detection accuracy, DetA,
is simply the percentage of aligning detections. The association accuracy, AssA, is simply the average alignment between matched trajectories, averaged over all detections. The
final HOTA score is the geometric mean of these two scores
averaged over different localisation thresholds.
In this paper we make four major novel contributions: (i)We propose HOTA as a novel metric for evaluating multiobject tracking (Sec. 5); (ii) We provide thorough theoretical analysis of HOTA as well as previously used metrics
MOTA, IDF1 and Track-mAP, highlighting the benefits andshortcomings of each metric (Sec. 7 and 9); (iii) We evaluate HOTA on the MOTChallenge benchmark and analyse its
properties compared to other metrics for evaluating current
state-of-the-art trackers (Sec. 10); (iv) We perform a thorough user-study comparing how different metrics align withhuman judgment of tracking accuracy and show that HOTA
aligns closer with the desired evaluation properties of users
compared to previous metrics (Sec. 11).

2 Related Work
Early History of MOT Metrics. Multi-Object tracking has
a long history dating back to at least the 70s [59, 66, 71,68]. Early work tended to evaluate using their own simple
evaluation metrics, such that comparison wasn’t possible between groups. In the early 2000s a number of different groups
sought to define standard MOT evaluation metrics. This included the PETS (Performance Evaluation of Tracking and
Surveillance) workshop series [86], the VACE (Video Analysis and Content Extraction) program [32], and a numberof other groups [75, 13, 57]. In 2006, the CLEAR (CLassification of Events, Activities and Relationships) workshop[72] brought together all of the above groups and sought
to define a common and unified framework for evaluating
MOT algorithms. This became the CLEAR MOT metrics[8] which positions the MOTA metric as the main metric for
tracking evaluation alongside other metrics such as MOTP.
MOTA was adopted for evaluation in the PETS workshop
series [23] and remains, to this day, the most commonly usedmetric for evaluating MOT algorithms, although it has often been highly criticised [65, 6, 43, 40, 55, 60, 18, 48, 51, 76,
52,88, 20, 50] for its bias toward overemphasizing detection
over association (see Fig. 1), as well as a number of otherissues (see Sec. 9).

Jonathon Luiten et al.

Benchmarks’ use of Metrics. In the last five years thetwo most commonly used benchmarks for evaluating MOT
have been the MOTChallenge [39, 54, 19] and KITTI [28]benchmarks. Both of these have ranked trackers using the
MOTA metric, contributing to the general widespread use of
MOTA in the community.
Within the last few years the multi-object tracking community has grown enormously due in part to large investment from the autonomous vehicle industry. This has resulted in a large number of new MOT benchmarks being
proposed. Many of these rank trackers using the MOTA metric (PANDA [77], BDD100k [87], Waymo [73], ArgoVerse[14], PoseTrack [1], MOTS [74]), or a variation of MOTA
(nuScenes [12], UA-DETRAC [78]).
Two other metrics have recently been adopted by some
MOT benchmarks. The IDF1 metric [60] which was proposed specifically for tracking objects through multiple cameras has been used by ‘multi-camera MOT’ benchmarks suchas Duke-MTMC [60], AI City Challenge [56] and LIMA[38]. IDF1 has also recently been implemented as a secondary metric on the MOTChallenge benchmark, and has
become preferred over MOTA for evaluation by a numberof single camera tracking methods [51, 52, 76] due to its focus on measuring association accuracy over detection accuracy (see Fig.1). IDF1 however exhibits unintuitive andnon-monotonic behaviour in regards to detection (see Sec.
9).
The Track-mAP metric (also called 3D-IoU) was introduced for tracking evaluation on the ImageNet-Video benchmark [61]. Recently it has been adapted by a number ofbenchmarks such as YouTube-VIS [83], TAO [18] and VisDrone [90]. Track-mAP differs from previously described
metrics in that it doesn’t operate on a set of fixed tracks,
but rather requires a set of tracks ranked by the tracker’s
confidence that each track exists. This makes Track-mAP incompatible with many of the commonly used benchmarks(such as MOTChallenge and KITTI) which do not require
trackers to output a confidence score with their predictions.
Track-mAP suffers from many of the same drawbacks as
IDF1 due to its use of global track-based matching, while
also having a number of other drawbacks related to the useof ranking-based matching (See Sec. 9).
A number of extensions to the MOTA metric, such as
PR-MOTA [78] and AMOTA [79], have previously been
proposed to adapt the MOTA metric to handle confidence
ranked tracking results as is done in Track-mAP. We present
a simple extension to our HOTA metric in Sec. 8 whichsimilarly extends HOTA to confidence ranked results, andwhich reduces to the standard HOTA metric when taking afixed set of detections above a certain confidence threshold.
A number of other metrics [65, 6, 58, 21, 67, 81] have been
been proposed for MOT evaluation, but to the best of ourknowledge none of them have been adopted by any MOT

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

3benchmarks and thus have not become widely used for evaluation.
Other metrics such as the trajectory-based metrics(Mostly-Tracked, Partially-Tracked, Mostly-Lost, Fragmentation) [80, 44], and the metrics of Leichter and Krupka(False Negative Rate, False Positive Rate, Fragmentation
Index, Merger Index, Mean Deviation) [43] are commonly
shown as secondary metrics on benchmarks but are neverused to comprehensively rank trackers as they are too
simple, often focusing on only a single type of error each,and easy to be gamed if desired.
Meta-Evaluation of Metrics. PETS vs VACE [53] discusses
the trade-off between presenting multiple evaluation metrics
vs a single unifying metric. Their conclusion is that multiple
metrics are useful for researchers to debug algorithms andidentify failure components, while a unified metric is usefulfor end-users wishing to easily choose highly performant
trackers from a large number of options. We resolve thisconflict by presenting both a unified metric, HOTA, and its
decomposition into simple components which can be used
to analyse different aspects of tracking behaviour (See Sec.
6).
Milan et al. [55] analyse the tracking metrics available in2013 (CLEAR MOT [8] and Trajectory-based Metrics [80])and identify a number of deficits present in these metrics,
particularly in MOTA. Based on this analysis they find noneof the metrics that were analysed to be suitable as a single
unified metric and recommend to present results over all
available metrics. We present HOTA as a solution to such
issues, as a metric suitable for unified comparison between
trackers.
Leichter and Krupka [43] present a theoretical framework for analysing MOT evaluation metrics, which involves
two components. The first is a characterisation of five error
types that can occur in MOT (False negatives, False positives, Fragmentation, Mergers and Deviation). The second
component is the description of two fundamental properties
that MOT evaluation metrics should have: monotonicity anderror type differentiability. In [43], they show that all previous
metrics (including MOTA) don’t have either of these properties. They propose a set of five separate simple metrics, one
for each error type, however they make no effort to combine
these into one unified metric, and thus the usefulness of these
metrics in comparing trackers is limited. In Sec. 6 we show
how HOTA can be decomposed into components which correspond to each of these five error types (Detection Recall,
Detection Precision, Association Recall, Association Precision and Localisation Accuracy, respectively), and as such
HOTA has the property of error type differentiability which
requires that the metrics are informative about the tracker’s
performance with respect to each of the different basic error
types. In Sec. 7 we show how the combined HOTA metric isstrictly monotonic with regards to each of these five types oferrors, thus having the second required property. In Sec. 9 weshow that no other recently used metric has these desirable
properties.
Tracking the Trackers [40] sought to analyse different
evaluation metrics using human evaluators, in order to findout how well different metrics reflect human perception of
the quality of tracking results. They only evaluated the setof CLEAR MOT [8] and trajectory-based [80, 44] metrics.
From these metrics they found that MOTA remains the mostrepresentative measure that coincides to the highest degreewith human visual assessment, despite pointing out its many
limitations. In Sec. 11 we seek to repeat this study and compare our HOTA metric with MOTA and IDF1 using humanvisual evaluation. We find that HOTA performs much betterin this user-study than both MOTA and IDF1, particularly
among MOT researchers.

3 Preliminaries
In this section, we lay the framework needed for understanding the content of this paper. This includes describing themulti-object task, the role of evaluation metrics, the groundtruth and prediction representation and the notation we will
be using, as well as other fundamental concepts that will be
used throughout the paper.
What is Multi-Object Tracking? Multi-Object Tracking(MOT) is one of the core tasks for understanding objects invideo. The input to the MOT task is a single continuous video(although we present extensions for our metric for multicamera tracking in Sec. 8), split up into discrete frames at a
certain frame rate. Each discrete frame could be an image,or could be another representation such as point cloud froma laser scanner.
The output of the MOT task is a representation that encodes the information about: (a) what objects are present ineach frame (detection), where they are in each frame (localisation) and whether objects in different frames belong to thesame or different objects (association).
Evaluation Metrics and Ground-Truth. In order to evaluate how well a tracker performs, we need to compare its
output to a ground-truth set of tracking results. The purposeof an evaluation metric is to evaluate the similarity between
the given ground-truth and the tracking results. This is nota well defined problem, as there are many different ways ofscoring such a similarity (especially between complex representations such as sets of trajectories). However, the choiceof evaluation metric is extremely important, as the properties of the metric determine how different errors contributeto a final score, as such it is favourable that metrics havecertain properties. The choice of metric also has the ability
to heavily influence the direction of research within the research community. In the age of competitive benchmarks, a

4

lot of research (for better or for worse) is evaluated on its
ability to improve the scores on these benchmarks. If benchmarks are using metrics to evaluate these scores which arebiased towards only certain aspects of a task, this will also
bias research and methods towards focusing more on these
aspects.
MOT Ground-Truth and Prediction Format. The setof ground-truth tracks is represented as a set of detections(gtDets) in each video frame, where each gtDet is assigned
an id (gtID), such that the gtIDs are unique within each frame
and the gtIDs are consistent over time for detections from thesame ground-truth object trajectory (gtTraj).
For most evaluation metrics (MOTA, IDF1 and HOTA) a
tracker’s prediction is in the same format as the ground-truth
data. It consists of a set of predicted detections (prDets) ineach frame, each assigned a predicted id (prID), such thatthe prIDs are unique within each frame and consistent overtime for detections from the same predicted object trajectory(prTraj).
For the Track-mAP metric, in addition to the prDets with
PrIDs as above, each prTraj is assigned a confidence score
estimating how likely it is that this trajectory exists. If each
prDet is assigned a confidence score instead of each prTraj,
the confidence score for the prTraj is simply the average of theconfidence scores over the prDets that belong to the prTraj.
For tracking multiple object classes, each gtTraj andprTraj may also be assigned a class id (gtCl / prCl). Previous metrics have all been applied per class and averaged
over classes. Thus for simplicity we can ignore this class
id when defining metrics and assume that metrics are calculated only over a single class of objects at a time. However, we
also present an extension to HOTA in Sec. 8 which explicitly
deals with multi-class tracking.
Types of Tracking Errors. The potential errors between a
set of predicted and ground-truth tracks can be classified into
three categories: detection errors, localisation errors and association errors [43]. Detection errors occur when a tracker
predicts detections that don’t exist in the ground-truth, orfails to predict detections that are in the ground-truth. Association errors occur when trackers assign the same prID
to two detections which have different gtIDs, or when they
assign different prIDs to two detections which should havethe same gtID. Localisation errors occur when prDets are
not perfectly spatially aligned with gtDets.
There are other ways of defining basic error types for
MOT, such as identification errors [60]. However detection,
association and localisation errors are the most commonlyused error types [43] and are widely applicable for evaluating
tracking for a wide range of different tracking scenarios (seeSec. 6 for examples). As such we only consider these error
types when analysing and comparing evaluation metrics.
Different Detection Representations. Object detections
within each frame (gtDets and prDets) may have a num-

Jonathon Luiten et al.

ber of different representations depending on the domain
and application. Commonly used representations include 2D
bounding boxes [39, 28], 3D bounding boxes [73, 14], segmentation masks [74, 83], point estimates in 2D or 3D [23],and human pose skeletons [1]. In general tracking evaluation
metrics are agnostic to the specific representation except for
the need to define measures of similarity for each representation.
Measures of Similarity. Most metrics require the definitionof a similarity score between two detections, S. Track-mAP
requires such a similarity score between trajectories, Str .
This similarity score should be chosen based on the detection representation used, but should be constrained to be
between 0 and 1, such that when S is 1 the prDet and gtDetperfectly align, and when S is 0 there is no overlap between
detections. The most commonly used similarity metric for 2D
boxes, 3D boxes and segmentation masks is IoULoc , which isthe spatial intersection of two regions divided by the unionof the regions. For point representations (and human joint
locations), a score of one minus the Euclidean distance is often used, such that points are said to have zero overlap when
they are more than one meter apart.
Bijective Matching. A common procedure in MOT evaluation metrics (MOTA, IDF1, HOTA) is to perform a bijective(one-to-one) matching between gtDets and prDets. This ensures that all gtDets are matched to at most one prDet andvice versa, and that any extra or missed predictions are penalised. Such a bijective mapping can be easily calculated
by calculating a matching score between all pairs of gtDetand prDet and using the Hungarian algorithm for finding thematching that optimises the sum of the matching score. Usually there is a minimum similarity S ≥ α requirement for amatch to occur. After matching has occurred, we have some
gtDets and prDets that are matched together. We call thesepairs as true positives (TP). These are considered correct
predictions. Any gtDets that are not matched (missed) arefalse negatives (FN). Any prDets that are not matched (extra
predictions) are false positives (FP). FNs and FPs are two
types of incorrect predictions.
Matching vs Association. The words match and association
are used throughout this paper to refer to two different things.
We refer to a match as a pair consisting of a matching groundtruth detection and a predicted detection. On the other handassociation refers to a number of detections with the same
ID such that they are associated to the same trajectory.
Jaccard Index and IoU. The Jaccard Index is a measure of
the similarity between two sets. It is defined as follows:
Jaccard Index =|TP||TP| + |FN| + |FP|(1)

The Jaccard index is commonly called the IoU (intersection
over union) because it is calculated as the intersection of
the two discrete sets divided by their union. However, inHOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

5

tracking IoU is also often used for describing the spatial
overlap between two spatial regions (e.g. boxes, masks). In
order to not confuse these terms we use ‘Jaccard index’ torefer to the operation over discrete sets, and IoULoc to referto the operation over spatial regions.
Mathematical Metric Definition. The term metric has astrict mathematical definition. For a distance measure (suchas HOTA) between two sets (e.g., ground-truth tracks andpredicted tracks) to be strictly a metric in the mathematicalsense, it must satisfy three conditions, (i) identity of indiscernibles, (ii) symmetry, and (iii) subadditivity (the triangle
inequality).
Within the computer vision community, the term metricis commonly used for functions which calculate a score bywhich algorithms can be ranked, without requiring the three
conditions above. We use this definition throughout this paper, and for example, refer to MOTA as an evaluation metriceven though it doesn’t meet the last two requirements.
Note that for the purpose of evaluating the MOT task,
it is not strictly necessary for metrics to be symmetric or
subadditive. However, as we show in Sec. 7 these are both
useful properties to have. Of the commonly used metrics,
HOTA is the only one to have these properties and thus
technically be mathematically a metric.

allowed to match when S ≥ α. In practice, multiple matches
could occur, the actual matching is performed such that the
final MOTA and MOTP scores are optimised (see below).
Measuring Association. In MOTA, association is measured
with the concept of an Identity Switch (IDSW). An IDSW
occurs when a tracker wrongfully swaps object identities
or when a track was lost and reinitialised with a different
identity. Formally, an IDSW is a TP which has a prID that isdifferent from the prID of the previous TP (that has the samegtID). IDSWs only measure association errors compared tothe single previous TP, and don’t count errors where the sameprID swaps to a different gtID (ID Transfer).
Scoring Function. MOTA measures three types of tracking
errors. The detection errors of FNs and FPs, as well as the association error of IDSW. The final MOTA score is calculated
by summing up the total number of these errors, dividing by
the number of gtDets, and subtracting from one.
MOTA = 1 −|FN| + |FP| + |IDSW||gtDet|(2)

MOTP. Note that MOTA doesn’t include a measure of
localisation error. Instead the CLEAR MOT metrics define asecondary metric, MOTP (Multi-Object Tracking Precision),
which solely measures localisation accuracy. It is simply theaverage similarity score, S, over the set of TPs.

4 Overview of Previous Metrics
In this section we provide a brief overview of the calculationof the three metrics which are currently used by MOT benchmarks (see Sec. 2). Our HOTA metric builds upon many of
the ideas of previous metrics (especially MOTA), and as suchit is important for the reader to have an overview of how theywork. An analysis comparing the properties and deficits ofall of the methods is presented in Sec. 9, and Tab. 1 gives
an overview of the different design decisions between themetrics.

4.1 CLEARMOT: MOTA and MOTP
Matching Predictions and Ground-Truth. In MOTA(Multi-Object Tracking Accuracy), matching is done at
a detection level. A bijective (one-to-one) mapping isconstructed between prDets and gtDets in each frame. Any
prDets and gtDets that are matched (correct predictions)become true positives (TPs). Any remaining prDets thatare not matched (extra predictions) become false positives(FPs). Any gtDets that are not matched (missing predictions)become false negative (FNs). prDets and gtDets can only
be matched if they are adequately spatially similar. MOTA
thus requires the definition of a similarity score, S, between
detections (e.g. IoULoc for 2D bounding boxes), and thedefinition of a threshold, α, such that detections are only

MOTP =1 Õ
S|TP| TP(3)

Matching to Optimise MOTA and MOTP. The matchingof prDets to gtDets is performed so that the final MOTA and
MOTP scores are maximised. This is implemented, in each
new frame, by first fixing matches in the current frame whichhave S ≥ α and don’t cause an IDSW. For the remaining
potential matches the Hungarian algorithm is run to selectthe set of matches that as a first objective maximises thenumber of TPs, and as a secondary objective maximises themean of S across the set of TPs.

4.2 The Identification Metrics: IDF1
Matching Predictions and Ground-Truth. IDF1 calculatesa bijective (one-to-one) mapping between the sets of gtTrajs
and prTrajs (unlike MOTA which matches at a detectionlevel). This defines new types of detection matches. IDTPs(identity true positives) are matches on the overlapping part(where S ≥ α) of trajectories that are matched together.
IDFNs (identity false negatives) and IDFPs (identity false
positives) are the remaining gtDets and prDets respectively,from both non-overlapping sections of matched trajectories,
and from the remaining trajectories that are not matched.

6

Jonathon Luiten et al.

Scoring Function. The ID-Recall, ID-Precision and IDF1
scores are calculated as follows:|IDTP||IDTP| + |IDFN||IDTP|
ID-Precision =|IDTP| + |IDFP||IDTP|
IDF1 =|IDTP| + 0.5 |IDFN| + 0.5 |IDFP|
ID-Recall =(4)number of TPTrs in this list up to index n be |TPTr|n . For
each value of n, the precision (Prn ) and recall (Ren ) can becalculated as:|TPTr|nn|TPTr|nRen =|gtTraj|
Prn =

(5)
(6)

Matching to Optimise IDF1. The matching is performed
such that IDF1 is optimised. This is performed by enumerating the number of IDFPs and FDFNs that would result
from each match (non-overlapping sections), and from nonmatched trajectories (number of prDet and gtDet in these
trajectories, respectively). The Hungarian algorithm is used
to select which trajectories to match so that the sum of thenumber of IDFPs and IDFNs is minimised. Note that the localisation accuracy is not minimised during IDF1 matchingunlike in MOTA.

4.3 Track-mAP
Matching Predictions and Ground-Truth. Track-mAP(mean average precision) matches predictions and groundtruth at a trajectory level. It requires the definition of a trajectory similarity score, Str , between trajectories (in contrast
to MOTA and IDF1 which use a detection similarity score,S), as well as a threshold αtr such that trajectories are only
allowed to match if Str ≥ αtr . A prTraj is matched with a
gtTraj if it has the highest confidence score of all prTrajs
with Str ≥ αtr . If one prTraj should match with more than
one gtTraj, it is matched with the one for which it has thehighest Str , and the other gtTrajs can be matched by the
prTraj with the next highest confidence score. We define thematched prTrajs as true positive trajectories (TPTr), and theremaining prTrajs as false positive trajectories (FPTr).
Trajectory Similarity Scores. Str is commonly defined for
2D bounding box tracking in two different ways.
In [61, 90], the set of TPs is defined as pairs of detectionsin the trajectories where S ≥ α (with S being IoULoc , and α
being 0.5). FNs and FPs are the remaining gtDets and prDets
respectively. Str is then equal to |TP|/(|TP| + |FN| + |FP|).
In [83, 18], Str is defined as the sum of the spatial intersection of the boxes across the whole trajectories, divided by
the sum of the spatial union of the boxes across the whole
trajectories.
Scoring Function. Track-mAP follows the calculation of
the average precision metric [24, 61, 45] over trajectories (instead of detections as commonly used for evaluating object
detection).
PrTrajs are ordered by decreasing confidence score. Let
the index of this ordering (starting at one) be n. Let the

(7)
(8)

The precision values are interpolated (InterpPrn ) so that they
are monotonically decreasing.
InterpPrn = max(Prm )
m≥n

(9)

The Track mAP score is then the integral under the interpolated precision-recall curve created by plotting InterpPrn
against Ren for all values of n. This integral is approximatedby averaging over a number of fixed recall values.
Threshold and Class Averaging. Track-mAP is sometimes
calculated at a fixed value of αtr [18], and sometimes averaged over a range different αtr values [83].
When multiple classes are to be tracked, Track-mAP isusually calculated per class separately and then the final scoreis averaged over the classes.

5 The HOTA Evaluation Metric
The main contribution of this paper is a novel evaluation
metric for evaluating Multi-Object Tracking (MOT) performance. We term this evaluation metric HOTA (Higher Order
Tracking Accuracy). HOTA builds upon the previously used
MOTA metric (Multi-Object Tracking Accuracy) [8], while
addressing many of its deficits.
HOTA is designed to: (i) provide a single score fortracker evaluation which fairly combines all different aspectsof tracking evaluation, (ii) evaluate long-term higher-order
tracking association, and finally, (iii) decompose into submetrics which allow analysis of the different components oftracker’s performance.
In this section, we provide a definition of the HOTA metric. In Sec. 6 we show how HOTA can be decomposed into
a set of sub-metrics which can be used to analyse different
aspects of tracking performance. In Sec. 7 we analyse different properties of HOTA, and examine the design decisions
inherent to the HOTA formulation. In Sec. 9 we present a
comparison of HOTA to MOTA, IDF1 and Track-mAP, andshow how HOTA addresses many of the deficits of previous
metrics.
Matching Predictions and Ground-Truth. In HOTA,
matching occurs at a detection level (similar to MOTA).
A true positive (TP) is a pair consisting of a gtDet and a
prDet, for which the localisation similarity S is greater thanor equal to the threshold α. A false negative (FN) is a gtDetthat is not matched to any prDet. A false positive (FP) isHOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

7Fig. 2 A visual explanation of the concepts of TPA, FPA and FNA. The different TPAs, FPAs and FNAs are highlighted for the TP of interest c.
The TPAs (green) for c (red) are the matches which have the same prID and the same gtID. The FPAs have the same prID but either a different or
no gtID. The FNAs have the same gtID but either a different or no prID. In the diagram, c has five TPAs, four FPAs and three FNAs. Conceptuallythese concepts are trying to answer the question: For the matched TP c, how accurate is the alignment between the gtTraj for this TP (large dark
blue circles) and the prTraj for this TP (small black circles).

a prDet that is not matched to any gtDet. The matching
between gtDets and prDets is bijective (one-to-one) in eachframe. Multiple different combinations of matches could
occur, the actual matching is performed to maximise thefinal HOTA score (see below).
Measuring Association. The concepts of TPs, FNs and FPs
are commonly used to measure detection accuracy. In orderto evaluate the success of association in a similar way, we
propose the novel concepts of TPAs (True Positive Associations), FNAs (False Negative Associations) and FPAs (False
Positive Associations), which are defined for each TP. For agiven TP, c, the set of TPAs is the set of TPs which have boththe same gtID and the same prID as c:
TPA(c) = {k},k ∈ {TP | prID(k) = prID(c) ∧ gtID(k) = gtID(c)}(10)
For a given TP, c, the set of FNAs is the set of gtDets withthe same gtID as c, but that were either assigned a different
prID as c, or no prID if they were missed:
FNA(c) = {k},k∈

{TP | prID(k) , prID(c) ∧ gtID(k) = gtID(c)}(11)∪ {FN | gtID(k) = gtID(c)}Finally, for a given TP, c, the set of FPAs is the set of prDets
with the same prID as c, but that were either assigned a different gtID as c, or no gtID if they did not actually correspondto an object:
FPA(c) = {k},k∈

{TP | prID(k) = prID(c) ∧ gtID(k) , gtID(c)}(12)∪ {FP | prID(k) = prID(c)}.

A visual example explaining the concept of TPAs, FNAs and
FPAs is shown in Fig. 2.
Note that although TPAs, FPAs, and FNAs are measured
between pairs of detections, these measures can be easily andefficiently calculated by counting the number of matches pereach prID-gtID pair, and there is no need to explicitly iterateover all pairs of detections.
Scoring Function. Now we have defined concepts used
to measure successes and errors in detection (TPs, FPs,
FNs) and association (TPAs, FPAs, FNAs), we can define
the HOTAα score for a particular localisation threshold α:s Íc ∈ {TP} A(c)HOTAα =
(13)
|TP| + |FN| + |FP|
|TPA(c)|A(c) =(14)|TPA(c)| + |FNA(c)| + |FPA(c)|We call this a ‘double Jaccard’ formulation, where atypical Jaccard metric is used over detection concepts of
TPs/FPs/FNs with each of the TPs in the numerator being
weighted by an association score A for that TP, which is equal
to another Jaccard metric, but this time over the association
concepts of TPAs/FPAs/FNAs.

8

Jonathon Luiten et al.

A measures the alignment between the gtTraj and prTraj
which are matched at the TP c. This alignment is calculatedusing the same formulation (Jaccard) as is used to measure
the alignment between the whole set of all gtTrajs and all
prTrajs for detection, but in this case only over the subset of
trajectories that are matched at a TP.
Note that concepts such as A, TP, FN, FP, TPA, FNA and
FPA, are all calculated for a particular value of α. However,the α subscript is omitted for clarity. We will continue toomit this α throughout the paper except where it is needed
for understanding.
Matching to Optimise HOTA. Like in MOTA (and IDF1)
the matching occurs in HOTA to maximise the final HOTA
score. The Hungarian algorithm is run to select the set ofmatches, such that as a first objective the number of TPs ismaximised, as a secondary objective the mean of the association scores A across the set of TPs is maximised, andas a third objective the mean of the localisation similarity
S across the set of TPs is maximised. This is implemented
with the following scoring for potential matches, MS, between each gtDet i and each prDet j.
nMS(i, j) = 1 + Amax (i, j) + S(i, j) if S(i, j) ≥ α (15)where  is small number such that the components have different magnitudes. Amax is the maximum A score if detections are not required to be bijectively matched, e.g. if each
prDet and each gtDet are allowed to match with multiple
others. Amax is optimised as a proxy for A. This is because
A depends upon which matches are selected, and thus cannot be optimised using a linear assignment formulation. Thisapproximation is valid, because A approaches Amax for the
optimal assignment.
Integrating over Localisation Thresholds. The formulation for HOTAα in Eq. 13 accounts for both detection andassociation accuracy, but doesn’t take into account localisation accuracy. In order for HOTA to measure localisation,
the final HOTA score is the integral (area under the curve)of the HOTA score across the valid range of α values between 0 and 1. This is approximated by evaluating HOTA
at a number of different distinct α values (0.05 to 0.95 in
0.05 intervals) and averaging over these values. For each α
value the matching between gtDets and prDets is performed
separately.
∫ 1
Õ1
HOTA =HOTAα dα ≈
HOTAα
19
0
0.05, 0.1, ...
α∈ {0.9, 0.95}(16)
HOTA in One Sentence. If HOTA were to be described in
one sentence it would be:
HOTA measures how well the trajectories of matchingdetections align, and averages this over all matching detections, while also penalising detections that don’t match.

6 Decomposing HOTA into Different Error Types.
A set of evaluation metrics has two main purposes. The first
purpose is to enable simple comparison between methods todetermine which perform better than others. For this purpose
it is important that there exists a single metric by which
methods can be ranked and compared, for this we propose the
HOTA metric (see Sec 5). The second purpose of evaluation
metrics is to enable the analysis and understanding of thedifferent types of errors that algorithms are making, in orderto understand how algorithms can be improved, or where
they are likely to fail when used. In this section, we showthat the HOTA metric naturally decomposes into a familyof sub-metrics which are able to separately measure all of
the different aspects of tracking. Fig. 3 shows each of thesesub-metrics and the relations between them.
HOTA solves the long-held debate in the tracking community [53, 55] about whether it is better to have a single
evaluation metric or multiple different metrics. HOTA simultaneously gives users the benefits of both options, a single metric for ranking trackers, and the decomposition intosub-metrics for understanding different aspects of tracking
behaviour.
Taxonomy of Error Types. We classify potential tracking
errors into three categories: detection errors, association errors and localisation errors. Detection errors can be further
categorised into errors of detection recall (measured by FNs)
and detection precision (measured by FPs). Association errors can be further categorised into errors of association
recall (measured by FNAs) and association precision (measured by FPAs).
Detection recall errors occur when trackers fail to predict
detections that are in the ground-truth (misses). Detection
precision errors occur when trackers predict extra detections
that don’t exist in the ground-truth. Association recall errors
occur when trackers assign two different prIDs to the same
gtTraj. Association precision errors occur when trackers assign the same prID to two different gtTrajs. Localisation
errors occur when prDets are not perfectly spatially alignedwith gtDets.
Our taxonomy aligns with previous work to classify different tracking errors [43], which defines the five basic error types as: false negatives, false positives, fragmentations,mergers and deviations. These are equivalent to detection recall, detection precision, association recall, association precision, and localisation, respectively.
Leichter and Krupka [43] argue that any set of tracking
metrics must be both error type differentiable and monotonic
with respect to these five basic error types. HOTA meets bothcriteria as it naturally decomposes into separate sub-metrics
measuring each basic error type, and is designed to ensure
monotonicity (see Sec. 7).

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

Where S(c) is the spatial similarity score between the prDetand gtDet which make up the TP c. This is similar to MOTP[8], however it is evaluated over multiple localisation thresholds α, similarly to how HOTA is calculated.
Separating Detection and Association. HOTA can be naturally decomposed into a separate detection accuracy score(DetA) and an association accuracy score (AssA) as follows:

HOTA
HOTAα

LocA
DetAα
DetReαAssAαDetPrαAssReαAssPrαFig. 3 Diagrammatic representation of how HOTA can be decomposed
into separate sub-metrics which are able to differentiate between different types of tracking errors.

Metrics for Different Tracking Scenarios with Different
Requirements. The decomposition of HOTA into different
sub-metrics has the further advantage that it enables users
to select algorithms or tune algorithms’ hyper-parameters
based on the nuances of their particular use-case.
For example, in human motion analysis, crowd analysis
or sports analysis [55, 40] it may be far more important topredict correct, identity preserving trajectories (association
recall and precision), than to find all present objects (detection recall). Whereas for a driving assistance system, it
is crucial to detect every pedestrian to avoid collision (detection recall) while also not predicting objects that are notpresent to avoid unnecessary evasive action (detection precision). However, correctly associating these detections overtime may be less crucial (association recall and precision).
In surveillance scenarios, it is typically more important toensure all objects are found (detection recall), whereas extra
detections can easily be ignored by human observers (detection precision). For short-term future motion prediction
it is important to have accurate trajectories of recent object
motion, without mixing trajectories of multiple objects (association precision), in order to extrapolate to future motion.
Whereas it doesn’t matter if trajectories are not correctly
merged into long-term consistent tracks (association recall).
With HOTA, different aspects of tracking can easily beanalysed and optimised for, which was not as easily or intuitively possible with previous metrics.
Measuring Localisation. HOTA is calculated at a numberof different localisation thresholds α, and the final HOTA
score is the average of the HOTAα scores calculated ateach threshold. This formulation ensures that the final HOTA
score takes the actual localisation accuracy into account.
A localisation accuracy score (LocA) can be measured
separately from other aspects of tracking, as follows:
LocA =∫ 101|TPα |

Õc ∈ {TPα }S(c) dα

9

(17)

|TP|
|TP| + |FN| + |FP|
1 ÕA(c)AssAα =|TP|

DetAα =

(18)
(19)

c ∈ {TP}|TPA(c)||TPA(c)| + |FNA(c)| + |FPA(c)|s Íc ∈ {TP} A(c)HOTAα =
|TP| + |FN| + |FP|p
= DetAα · AssAαA(c) =(20)(21)

We see that HOTA is equal to the geometric mean of a detection score and an association score. This formulation ensuresthat both detection and association are evenly balanced, unlike many other tracking metrics, and that the final score issomewhere between the two.
It also ensures that both the detection score and association score have the same structure. Both are calculatedusing a Jaccard index formulation, and both are calculated
to ensure that each detection contributes equally to the finalscore.
The detection score is simply the standard Jaccard index, which is commonly used for evaluating detection. The
association score is a sum of Jaccard indices over the TPs,
where each is equal to a standard Jaccard index evaluatedonly between the trajectories which are a part of that TP.
In practice (see Sec. 10) this often results in trackers
having DetA and AssA scores that are quantitatively similar.
Final DetA and AssA scores can be calculated by integrating over the range of α values in the same way as done
for the HOTA score in Eq. 16. Note that DetA and AssA
should only be combined into HOTA before integrating overa range of α values, not afterwards.
Separating Recall and Precision. HOTA is further decomposable in that each of the detection and association components can be simply decomposed into a recall and precisioncomponent.
Detection Recall/Precision. The detection recall/precision
are defined as follows:|TP|
DetReα =(22)|TP| + |FN|
|TP|
DetPrα =(23)|TP| + |FP|
DetReα · DetPrα
DetAα =
(24)
DetReα + DetPrα − DetReα · DetPrα10Jonathon Luiten et al.

These are equivalent to the concepts commonly used in thefield of object detection [24]. Detection recall is the percentage of ground-truth detections that have been correctlypredicted, and detection precision is the percentage of detection predictions made which are correct. DetRe and DetPr
are easily combined into DetA. Final DetRe and DetPr scorescan be calculated by integrating over the range of α values
in the same way as done for the HOTA score in Eq. 16.
Association Recall/Precision.
The association recall/precision are defined as follows:
AssReα =AssPrα =AssAα =

1
|TP|
1
|TP|

Õ
c ∈ {TP}Õc ∈ {TP}|TPA(c)||TPA(c)| + |FNA(c)|(25)|TPA(c)||TPA(c)| + |FPA(c)|(26)

AssReα · AssPrα
AssReα + AssPrα − AssReα · AssPrα(27)

Unlike the detection equivalent, these are novel concepts.
Association recall measures how well predicted trajectories
cover ground-truth trajectories. E.g. a low AssRe will result
when a tracker splits an object up into multiple predictedtracks. Association precision measures how well predictedtrajectories keep to tracking the same ground-truth trajectories. E.g. a low AssPr will result if a predicted track extends
over multiple objects.
AssRe and AssPr are easily combined into AssA. The
introduction of association precision and recall are very powerful tools for measuring different aspects of MOT performance and are a natural extension to the similar widely used
detection concepts. Final AssRe and AssPr scores can becalculated by integrating over the range of α values in thesame way as done for the HOTA score in Eq. 16.

7 Analysing the Design Space of HOTAIn this section, we analyse a number of different design
choices of the HOTA algorithm, and the effect of thesechoices on the properties of the evaluation metric.
Higher-Order vs First-Order Association. In HOTA (andin MOTA) the concept of an association is measured for eachdetection. The association score in HOTA, and the numberof IDSWs in MOTA seek to answer the question ‘how wellis this detection associated throughout time?’. In MOTA an
IDSW measures this association only one time-step back into
the past. E.g. whether this detection has the correct association compared to the previous detection. Since associationis measured only over one step, we term this first-order association. A metric which considers associations over twotime-steps could be called second order association. In contrast, HOTA measures association over all frames of a video.
We term this concept higher-order association and name our

HOTA metric after it. This property allows HOTA to measure long-term association, which is lacking from the MOTA
metric.
Higher-Order vs First-Order Matching. Just as one of
the main conceptual differences between HOTA and MOTA
is first-order vs higher-order association between detections,one of the main conceptual differences between HOTA and
IDF1 can be thought of as first-order vs higher-order matching between trajectories.
In IDF1, each trajectory is matched only with a single
other trajectory and scored by how well it aligns with thissingle trajectory. We call this first-order matching. This is
enforced by a unique bijective matching between prTrajs andgtTrajs. HOTA in contrast, is able to measure how well eachtrajectory matches with all possible matching trajectories,
which we term higher-order matching. This is done by performing matching at a detection level, which allows eachtrajectory to match to different trajectories in each time-step,and then scoring the alignment between each of these matching trajectories for each matching detection.
Thus HOTA can be thought of as being higher-order interms of both association and matching.
Detection vs Trajectory Matching. Both HOTA and
MOTA create matches between sets of detections, whereas
other metrics like IDF1 and Track-mAP directly match
whole trajectories with one another. Matching detections
has the advantage over matching trajectories that all of thepossible trajectory matches can be measured by the metricsimultaneously. In IDF1, if a gtTraj is split between multiple
prTrajs, only the best matching trajectories are consideredcorrect, while all the remaining trajectories are considered
incorrect. This causes the problem that the association
accuracy of non-matched trajectories are ignored, no matter
how well they are associated it will not affect the score. This
also has the disadvantage that the score actually decreases
as detection accuracy increases for non-matched segments.
This is because such segments are considered negatives anddecrease the score. Matching at a detection level insteadof a trajectory level is required in order to ensure that theassociation accuracy of all segments contributes to thefinal score, and that the metric monotonically increases asdetection improves.
Jaccard vs F1 vs MODA. In HOTA we use the Jaccard
index to measure both detection and association accuracy.
We compare this formulation with two possible alternatives,
the F1 score and the MODA score.
|TP||TP| + |FN| + |FP|
|TP|
F1 =
|TP| + 0.5|FN| + 0.5|FP||TP| − |FP|
MODA =|TP| + |FN|

Jaccard =

(28)
(29)
(30)

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

A: 2
1
gt

B: 2
1
gt

Fig. 4 A simple tracking example showing why the Jaccard formulationfor HOTA is preferable to a number of others such as F1, MODA anda formulation which excludes FNs and FPs from the association score(see text). For tracking a single object present in all frames (bold line),
two tracking results, A and B (thin lines) are presented. The x-axis of theplots is the ratio of the len(traj 1)/len(gt). The tracking result of A should
always have a higher score than B for the metric to be monotonic indetection, over all possible ratios of the len(traj 1)/len(gt), and also if thepredictions and ground-truth are swapped. This is only valid for Jaccard
based HOTA. Note that the MODA formulation is non-symmetric so
the results when swapping the ground-truth and tracks are shown asdashed lines and labeled with an asterisk. The other formulations are
symmetric.

The F1 score (also called the Dice coefficient) is the harmonicmean of recall and precision and is used in IDF1 as well asother metrics such as PQ for panoptic segmentation [36].
The MODA score (Multi-Object Detection Accuracy) is the
MOTA score without considering IDSWs and is often used
for measuring detection accuracy in video.
Of the three, the Jaccard formulation is the only one
that meets all three requirements to mathematically be a
metric, and the only one that obeys the triangle inequality(MODA also isn’t symmetric). Note that when using theJaccard formulation the entire HOTA metric also obeys thetriangle inequality, and is mathematically a metric, because
the mean of metrics over non-empty finite subsets of a metricspace is also a metric [27].
Fig. 4 shows a simple tracking example designed to show
the monotonicity of different formulations. The HOTA scoreis evaluated using the three different formulations for both
detection and association scores. If the evaluation measure ismonotone, the tracking results in (A) should always be higher
than (B) because (A) contains more correct detections. Thisshould also be true when swapping which set is the groundtruth and which is the prediction. As can be seen of the three
formulations, Jaccard is the only one to exhibit this monotone

11

property. The F1 formulation scores B higher when the tracklabeled (1) is adequately long. The MODA formulation is
non-symmetric. It acts the same as Jaccard in the absenceof TPs and TPAs, but exhibits very undesirable behaviour
when the ground-truth and predictions are switched. Boththe monotone and symmetric properties result in the Jaccard
formulation being preferable to the other two.
Including vs Excluding Detection Errors in the Association Score. In HOTA, the association score for a TP is
the Jaccard index score between the gtTraj and the prTraj
that have the same gtID and prID as the TP, respectively.
These trajectories could include FNs and FPs which are notmatched. This can be seen in the definitions of TPA and TNA
in Sec. 5. A potential alternative formulation would only calculate this Jaccard index over TPs such that any FNs or FPs
in these trajectories are ignored, and do not count toward
the count of FNAs and FPAs. This formulation may seemadvantageous in that association is now calculated only overTPs, and detection errors would no longer decrease the association score. However, as seen in the fourth panel of Fig. 4
this results in non-monotonic results where adding in correct
detections decreases the overall score, as the AssA decreases
faster than the DetA increases. By including all detectionsof matching trajectories in the association score calculation,
we ensure that DetA and AssA are perfectly balanced suchthat an improvement in either one cannot result in a larger
decrease in the other, thus ensuring the monotonicity of themetric.
Note that in HOTA, the presence of these FNs and FPs
in the association score does not influence the error type
differentiability of the metric. The AssA still measures only
association and the DetA still measures only detection. These
terms in the association score correspond to measuring associations to unmatched detections.
With vs Without the Square Root. The HOTA formulation
contains a square root operation after the double Jaccard
formulation. This square root has three effects. The first isthat it increases the magnitude and spread of trackers’ scores.
While the magnitude of the scores is in itself not important,
it is nice to see that both the magnitude and spread of HOTA
scores is in the same ball-park range as previous metrics(see Sec. 10 Fig. 13). This means that researchers’ current
intuitive understanding of how good certain scores are, stillroughly holds.
The second effect of the square root is its interpretation
as the geometric mean of a detection score and an association
score. This is natural and intuitive as we wish for HOTA toevenly balance both detection and association, thus having
HOTA as the geometric mean of these two scores is a good
choice. The geometric mean has the advantage over otherformulations such as an arithmetic mean that the score approaches 0 as either of the two sub-scores approach zero.

12

Jonathon Luiten et al.

Thus when a tracker completely fails in either detection or
association, a very low score will effectively represent this.
The third effect is that it accounts for double countingof similar error types. As discussed above, the association
score also includes FP and FN detection errors. The use of
the square root prevents double counting of these errors. Thiscan be illustrated by a simple example. In cases where there
is only one gtTraj and one prTraj, such as in single object
tracking (SOT), the HOTA score reduces to the following
equation:
HOTAα {SOT}√= DetA.AssA
v
t
|TP|
1 Õ
|TP|
=
.
|TP| + |FN| + |FP| |TP|
|TP| + |FN| + |FP|
|TP |

|TP|
=
|TP| + |FN| + |FP|
(31)
In the single object tracking case, the association score andthe detection score are the same. This makes sense because
the association score can be thought of as the average detection score between matched trajectories, and where there
is only one possible matching trajectory these scores are thesame. The square root thus cancels out these same errors
being counted twice, and results in an intuitive metric for
SOT, which is just a Jaccard metric measuring the ratio ofcorrect to incorrect predictions.
With vs Without the Detection Accuracy. As we haveseen above, the AssA takes into account FP and FN detection
errors for matched trajectories. A natural question to ask then,is whether we even need the DetA, or if the AssA is adequate
by itself. The DetA is critical for two reasons, the first is inaccounting for non-matched trajectories. If we have gtTrajs
or prTraj that are not matched at all, then these FNs and FPs
will not at all be taken into account in the AssA. The DetA
is needed to penalise these errors. The second reason is that
the AssA by itself is non-monotonic. Since it is the averageassociation over detections, if we add in one extra matchingdetection (TP), if this TP has a low association score, thenthe overall AssA will decrease. By including the DetA in
the HOTA score, the DetA and AssA are perfectly balancedsuch that adding any correct match (TP) will always increase
the overall score and thus HOTA is monotonic.
Averaging over Detections vs Averaging over Associations. In order to calculate the AssA we average the association scores between matching trajectories over matchingdetections. This is a natural formulation because it ensuresthat each detection contributes equally to the AssA, and it
results in the intuitive understanding that the contribution foreach detection is weighted by how accurately that detection
is associated across time.

An alternative formulation would be to calculate AssA by
averaging over all possible associations rather than averaging
over detections. This would result in a quadratic dependence
on the number of TPs such that longer matching trajectories
would contribute quadratically more than shorter matching
trajectories, which is undesirable. It would also lose the interpretation that the contribution of each TP is weighted
by its association accuracy, as well as making HOTA nonmonotonic.
Final Tracks vs Potential Tracks with Confidence
Scores. Taking into account the confidence of predictions
has both advantages and disadvantages for metrics. Oneof the main disadvantages is that your metric is no longer
evaluating an actual final tracking result, but rather aselection of potential tracking results which are ranked by
how likely each one is. Such an approach makes sensewhen the ground-truth is inherently ambiguous (such as
trajectory forecasting) as we can’t expect algorithms topredict the correct results and as such it is only fair to allow
multiple ranked predictions to be evaluated. It also makes
sense when the task is too difficult for current algorithms
to accurately predict the correct ground-truth (such as
monocular 3D detection and tracking), and again it is fair toallow algorithms to predict multiple ranked results.
However, as algorithms become better at a task, it isbetter to evaluate an algorithm’s ability to actually predictthe correct ground-truth values. This also has other benefits
such as enabling constraints between detection representations such as commonly used in segmentation mask tracking[74] where segmentation masks are not allowed to overlap.
For these reasons, HOTA is designed to operate on final
tracking results. We also present an extension to HOTA in
Sec. 8 in which we present a confidence ranked version of
HOTA, which reduces to the default HOTA when taking afixed set of detections above a certain confidence threshold.
Drawbacks of HOTA. HOTA has two main potential drawbacks, which could make using it less than ideal in somesituations. The first is that it may not be ideal for evaluating online tracking. This is because association is measured
over the entire video, and the association score for each TP
depends on how well it is associated in the future, which is
not a desirable feature for online evaluation.
The second potential drawback is that it doesn’t takefragmentation of tracking results into account (see Fig. 5).
This is by design, as we wish for the metric to measure longterm global tracking. However, for applications where this isimportant, this could be a drawback of HOTA.
In Sec. 8 we present both an online version of HOTA
and a fragmentation-aware version of HOTA, which address
both of these issues.

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

8 HOTA Extensions
In this section we provide a number of different extensions
to HOTA for use in different tracking scenarios.
Variety of MOT scenarios. HOTA is designed for a variety of MOT scenarios, from 2D box tracking, segmentation
mask tracking, 3D tracking, human pose tracking or point
tracking or beyond. HOTA can easily be adapted to any such
representation, all the is required is measure of similarity S
between objects in whichever representation is chosen (seeSec. 3).
Multi-Camera HOTA. HOTA also extends trivially toevaluating Multi-Camera MOT. Both the ground-truth andpredictions could contain trajectories in multiple different
cameras with consistent ids across cameras. HOTA could
then be applied without any changes. We recommend theuse of HOTA for multi-camera tracking over the currently
used IDF1 metric due to the many advantages of HOTA anddrawbacks of IDF1 (see Sec. 9).
HOTA for Single Object Tracking. As can be seen in
Eq. 31, HOTA simplifies trivially in the single object tracking (SOT) case to being a Jaccard index (integrated over α
values). Currently in SOT most evaluation procedures don’t
penalise FPs and only evaluate over frames where a gtDetis present. We believe that the HOTA formulation is also
perfectly suited for SOT evaluation, both for effectively penalising FPs, as well as for promoting unification between
the MOT and SOT communities.
Online HOTA. By default, HOTA scores association globally over the whole video sequence. Thus the association
score for a particular TP depends on how well it is associated both forward and backward in time. For online tracking
scenarios (such as autonomous vehicles) this is not ideal
evaluation behaviour as the results from the tracker are usedonline in each time-step (for decision making) and should be
evaluated in a similar way.
Thus we propose a simple extension of HOTA to the online case which we call Online HOTA (OHOTA). OHOTA
is calculated in the same way as HOTA but only time-steps
up to the current time-step are used for calculating the association accuracy for each TP.
This is a natural way to extend HOTA to online scenarios,and has the further benefit that it can also be used to evaluate
online tracking where trackers are able to update previous
predictions in each new time-step. In this case, for each new
time-step the TPs for this time-step will have their association
scores calculated with the most up to date predictions fromall previous time-steps.
Fragmentation-Aware HOTA. HOTA is designed to evaluate global association alignment between gtTrajs and prTrajs.
However, in some cases it is important to measure shortrange alignment, which we call fragmentation. Fig. 5 clearlyshows the difference between association and fragmentation.

13
A: 4
3
2
1
gt
AssA:
FragA:HOTA:
FA-HOTA:

B: 4
3
2
1
gt

0.5
0.5
0.71
0.71

C: 4
3
2
1
gt
0.5
0.25
0.71
0.59

0.25
0.25
0.5
0.5

Fig. 5 An example showing the difference between fragmentation andassociation. One gtTraj is present in all frames (bold line), and three
tracking results, A, B and C (sets of thin lines) are presented. A and B
have equal association (global alignment), whereas they have different
fragmentation (short-range alignment). B and C have equal fragmentation but different association. HOTA only measures association by
design. FA-HOTA measures both association and fragmentation.

In most tracking scenarios the default version of HOTA
is preferable, however, for when measuring fragmentations
is important, we present an extension which we callfragmentation-aware HOTA (FA-HOTA).
v
u

p
tÍ
A(c) · F(c)c ∈ {TP}FA-HOTAα =(32)|TP| + |FN| + |FP|
|FrA(c)|F(c) =(33)|TPA(c)| + |FNA(c)| + |FPA(c)|where the set of fragment associations of c, FrA(c), is thesubset of TPA(c) which belongs to the same fragment as c.
A fragment is a set of TPAs for which there are no FNAs or
FPAs between them.
We compute the geometric mean of fragmentation andassociation for each TP in order to concurrently measure both
short-term fragmentation alignment and long-term association. As the fragmentation score is bounded by the association score, FA-HOTA equals HOTA when no fragmentation
occurs. This formulation also allows us to compute the overall fragmentation accuracy FragA:
1 Õ
FragA =
Frag(c).
(34)|TP|
c ∈ {TP}

Importance Weighted HOTA. As detailed in Sec. 6 different tracking applications can assign different importanceto different aspects of tracking (detection/association recall/precision). We present an extension of HOTA, Weighted
HOTA (W-HOTA) which allows users to apply different
weights to each aspect depending on their requirements.
sÍc ∈ {TP} Aw (c)W-HOTAα =(35)|TP| + wFN |FN| + wFP |FP|
|TPA(c)|Aw (c) =|TPA(c)| + wFNA |FNA(c)| + wFPA |FPA(c)|(36)
Where each of the weightings, wFN , wFP , wFNA , wFNA are values between 0 and 1. When all weightings are 1 we have theoriginal HOTA. When any single weight is 0 that component
no longer contributes to the score.

14

Jonathon Luiten et al.

The default weighting provides a balanced weighting between the different components, and should be used for tracking evaluation unless there is a strong reason to use a different
weighting for a particular desired outcome.
Classification-Aware HOTA. Traditionally, there are two
ways to deal with evaluation for tracking multiple classes.
The first option is to require trackers to assign each object
to a class and then evaluate over each class separately before
averaging the results over classes. A second option is toignore the effect of classification all together and simply
evaluate all classes together in a class-agnostic way as though
they were all the same class.
We propose a third option for dealing with evaluating
multiple classes which we call classification-aware HOTA
(CA-HOTA). We require that each prediction is assigned
a probability that it belongs to each class such that theseprobabilities sum to one over all classes. CA-HOTA thenbecomes:vutÍc ∈ {TP} A(c) · C(c)CA-HOTAα =(37)|TP| + |FN| + |FP|
where C(c) is probability that the prDet of c assigned to theclass of the gtDet of c. This effectively weights the contribution of each TP by the classification score, in the same waythat it is weighted by an association score. In this setting wehave to include the classification score in the matching procedure so that the matching still maximises the final score.
Eq. 15 now becomes:
MS(i, j)(=1
 + Amax (i, j) · C(i, j) + S(i, j)

if S(i, j) ≥ α0,otherwise(38)

Each detection, even those belonging to the same prTraj canhave different class probabilities.
We can also compute the overall classification accuracy
ClaA, in order to evaluate the success of classification separate from other aspects of tracking.
1 Õ
ClaA =
C(c).
(39)
|TP|
c ∈ {TP}=

A(c) · C(c, Cls)Í
|TPCls | + |FNCls | + f ∈ {FP} C( f , Cls)c ∈ {TPCls }Fed-HOTAα {Cls}
v
u
u
Í
t
=


A(c)
·
C(c,
Cls)
c ∈ {TPCls }Í|TPCls | + |FNCls | + f ∈ {FP} I( f , Cls) · C( f , Cls)
(41)

(40)

where TPCls and FNCls are those which have a ground-truthclass Cls, and the notation C(c, Cls) is the classification score

where I( f , Cls) is 1 if f is from an image where the class
Cls should be counted as false positive, and 0 otherwise.
Confidence-Ranked HOTA. HOTA operates on final
tracking predictions rather than confidence-ranked potential
tracks. However, for certain tracking scenarios such as
monocular 3D tracking where it is difficult for trackers toaccurately localise detections a confidence-ranked version(CR-HOTA) is more suitable. This follows other metrics
such as Track mAP [61] or sAMOTA [79].
When using CR-HOTA trackers must output a confidence
score for each detection, k. Detections over the whole benchmark are ordered by decreasing confidence. Looping over theordered detections, the detection recall score is calculated foreach one considering all detections with a higher confidence.
For 19 fixed recall values (0.05 to 0.95 in 0.05 intervals), the
HOTA score is calculated using Eq. 13 and Eq. 16, by taking into account all detections with a confidence score higher
than the maximum confidence score needed to obtain a recall
at that threshold. The final CR-HOTA score is given by:
CR-HOTA =Class-Averaged Classification-Aware HOTA. We can also
calculate a class-averaged classification-aware HOTA (CA2 HOTA), by calculating a score for each class, Cls, as follows:
CA2 -HOTAα {Cls}
v
u
u
Í
t

which prediction c has assigned to the ground-truth class Cls.
The final score is calculated by averaging over all classes
before averaging over α thresholds.
For datasets with many classes [18, 83] we recommend
the use of CA2 -HOTA as it adjusts for class bias during
evaluation.
Federated HOTA. TAO [18] uses a federated evaluation
strategy. Not all objects are annotated in all images. Instead,each image is labeled with the set of classes for which there
is confirmed no unannotated objects (for which FPs can beevaluated). Extra predictions of other classes should be ignored as they could be present but unannotated.
We propose a version of HOTA which adapts Eq. 40 tofederated evaluation (Fed HOTA).

∫ 1
0

HOTAk
1
dk ≈
DetRek19

HOTAk
DetRek
0.05, 0.1, ...
Õk ∈{0.9, 0.95}(42)
By integrating the value of HOTA/DetRe over a range of
DetRe scores, we obtain a formulation which reduces to theoriginal HOTA score when only evaluating detections above
a given threshold. In this case the HOTA score would be thesame for all values recall value from 0 to DetRe and would
be zero afterward. Note that this formulation is the same as
how MOTA is adapted to sAMOTA in [79].

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking
MOTA

IDF1

Track-mAP

HOTA

Potential Tracks
Final Tracks
Final Tracks
with Conf. Score

Representation

Final Tracks

Matching Mechanism

Bijective

Bijective

Highest Conf.

Bijective

Matching Domain

Detection

Trajectory

Trajectory

Detection

Matched DetsAll Dets

Association Domain Prev. One Det Matched Dets
Í

Scoring FunctionErr
1 − |GTDet|

F1 Score

Av. PrecisionDoub. Jaccard

Bias Toward

Detection

Association

Association

Balanced

Table 1 An overview of different design choices and properties foreach of the previously used three metrics and HOTA.

9 Analysing Previous Evaluation Metrics.
In this section, we analyse the previous evaluation metrics
MOTA [8], IDF1 [60] and Track-mAP [61], identifying a
number of drawbacks for each one and drawing comparisons
to our HOTA metric. See Sec. 4 for descriptions of each of
the previous metrics. Note that our analysis of each metricis with respect to the properties of detection, association andlocalization. This is not the only framework in which results
can be analysed, but one that is common within the tracking
community [43].
High-level Comparison Between Metrics. Table 1 shows
an overview of the key differences between the four metrics.
HOTA can be thought of as a middle ground between
MOTA and IDF1. MOTA performs both matching and association scoring at a local detection level which biases scores
toward measuring detection, while IDF1 performs both at a
trajectory level which ends up being biased towards association. HOTA matches at the detection level while scoring
association globally over trajectories. This results in HOTA
being balanced between measuring detection and association, exhibiting many of the benefits of each method without
the drawbacks.
Track-mAP is similar to IDF1 in many ways in that it
performs both matching and association at a trajectory level
and as such is biased toward measuring association. However,Track-mAP differentiates itself in that it operates on potential
tracks with confidence scores rather than final tracks anddoesn’t perform bijective mapping but matches based on thehighest confidence valid matches.

9.1 Problems with MOTA
MOTA has been the main MOT evaluation metric since2006. It has served the community over the years, however
we believe its drawbacks have restricted tracking research.
Now that we are equipped with better tools and a better understanding of the tracking task, we are able to analyse all of
the problems with MOTA, and ensure that new metrics, suchas HOTA, don’t have the same issues. We hope HOTA will

15

quickly replace MOTA as the default standard for evaluating
MOT algorithms.
Below we highlight 9 separate problems of the MOTA
metric, and show how these problems are addressed in
HOTA.
Problem with MOTA 1 Detection performance significantly outweighs association performance.
MOTA measures detection errors as FNs and FPs, and association errors as IDSWs. The ratio of the effect of detection
errors to association errors on the final score is given by|FN|+|FP| : |IDSW|.
For real trackers this ratio is extremely high. For the
top ten trackers on the MOT17 benchmark [54] on the 1st
April 2020 this ratio varies between 42.3 and 186.4, with anaverage of 98.6. This is not because trackers are 100 times
better at detection than association, but rather that MOTA
is heavily biased towards measuring detection. In fact, on
average the effect of detection on the final score is 100 times
as large as the effect of association.
We also compared the MOTA and MODA (MOTA without IDSWs, which only measures detection) scores for 175
trackers on the MOT17 benchmark. When fitting a linear
regression model between MOTA and MODA the R2 valueis 99.4 indicating that the detection only score MODA explains more than 99% of the variation in final MOTA score.
In contrast, the R2 value between MOTA and the number of
IDSWs is only 23.7.
This can potentially have significant negative effects on
the tracking community. If researchers are tuning their trackers to optimise MOTA to increase scores on benchmarks,
then such trackers will be tuned toward performing well for
detection while mostly ignoring the requirement of performing successful association.
HOTA is designed to not have this problem as it is composed of a detection and association score which both contribute equally. Tab. 2 and Fig. 13 show that the numericalvalues for each of these scores are similar to one another forreal trackers.
Problem with MOTA 2 Detection Precision significantly
outweighs the effect of the Detection Recall.
Let us consider MOTA without considering IDSWs. Thisis MODA (multi-object detection accuracy). The equation
for MOTA/MODA as shown in Eq. 2 can be rearranged asfollows:|FN| + |FP|
|gtDet||TP| − |FP|=
|TP| + |FN|
1
= DetRe · (2 −)DetPr

MODA = 1 −(43)

Jonathon Luiten et al.

MODA

16

1.0
0.5
0.0
0.5
1.0
1.5
2.0

How MOTA/MODA varies
with Recall and Precision

A: 3
2
1
gt
MOTA:
HOTA:

Varying Re (with Pr==1)
Varying Pr (with Re==1)
0.0

0.2

0.4 0.6
Re/Pr

0.8

1.0

0.17
0.67

1-2/L
0.85

B: 3
2
1
gt

0.17

0.33
0.67

1-1/L
0.75

C: 3
2
1
gt

0.17
0.17
0.67

1-2/L
0.71

Fig. 7 An example showing how MOTA does not correctly evaluate
tracking situations when a tracker corrects itself after making a mistake.
The thick line is the gtTraj. Thin lines are prTrajs. All detections are
TPs.

Fig. 6 Figure showing how MOTA and MODA vary with the Recall
and Precision.

It can be seen that this is not symmetric in terms of recall andprecision. The score increases linearly with increasing recall
and hyperbolically with increasing precision. This relation
can be seen visually in Fig. 6. Poor precision has a muchgreater effect on the final score than poor recall.
Again this promotes researchers to tune trackers to optimise precision at the cost of recall, because poor precision
values are penalised extremely heavily by MOTA. This canbe seen starkly in Fig. 13 where precision values for all trackers are much higher than recall values.
As described in Sec. 6, for different tracking applications
the importance of recall vs precision varies. For applications
such as surveillance, recall is often much more important
than precision, and as such, the bias imparted by MOTA
is particularly harmful. Ideally a benchmark designed forevaluating trackers for a range of applications would evenly
weight precision and recall. HOTA solves this issue by using a symmetric Jaccard formulation and thus ensuring thatprecision and recall are weighted evenly.
Problem with MOTA 3 Association errors in MOTA, measured in the form of IDSWs, only take into account short-term(or first order) association.
In MOTA, an IDSW is a TP which has a prID that is different
from the prID of the previous TP (that has the same gtID).
Formally a TP, c, is an IDSW under the following criteria:c ∈ IDSW

prev(c) , ∅if
andprID(c) , prID(prev(c))(

prev(c) =argmaxk (t(k)),

if k , ∅∅,otherwisek ∈ {TP | t(k) < t(c) ∧ gtID(k) = gtID(c)}(44)
An IDSW only measures if there is an association tracking error to the previous gtDet. This is equal to a single FNA
for each TP comparing to the previous TP from the same
gtTraj. This can be thought of as a first-order approximation
to the global association score over the whole trajectory.
This is only able to capture algorithms’ ability to perform ‘short-term tracking’, and is unable to evaluate global

long-term tracking over a whole video. HOTA in contrast isable to evaluate higher-order global tracking by measuring
FNAs and FPAs compared to all other detections in matchedtrajectories.
Problem with MOTA 4 MOTA doesn’t take into accountassociation precision (ID transfers).
The transpose of an IDSW is called an ID transfer (IDTR).
An IDTR is a TP which has a gtID that is different from thegtID of the previous TP that has the same prID. Whereas
IDSWs compare to the previous gtDet, IDTRs compare to
the previous prDet. This is a first-order FPA, whereas an
IDSW is a first-order FNA. ID transfers commonly occur
when a predicted track spreads over two ground truth tracks.
MOTA doesn’t at all penalise such errors.
As an example consider a short video with only twoframes. In scenario A, there is one ground-truth object
present in both frames, the tracker correctly detects it ineach frame but splits it into two tracks. In this case an IDSW
occurs and the MOTA score is 0.5. In scenario B, there are
two ground-truth objects which are present only in one of
the frames each, the tracker again detects both correctly but
predicts that they are the same object. In scenario B an ID
transfer has occurred, but this is not an IDSW and is not
penalised and the MOTA score is a perfect 1.0.
Earlier we saw how MOTA wasn’t symmetric between
detection precision and recall. Now we see that it also isn’tsymmetric between association precision and recall. In fact
it doesn’t measure association precision (ID transfer) errors
at all.
This is again potentially extremely undesirable behaviour. Trackers can take advantage of this fact to ‘hackthe metric’ to improve their score while performing worse
tracking by artificially merging their trajectories overmultiple ground-truth objects.
HOTA solves this error by measuring both FPAs and
FNAs when measuring association accuracy.
Problem with MOTA 5 MOTA does not reward trackers
that correct their own association mistakes. In fact, it unfairly
penalises such corrections.
MOTA is not able to successfully evaluate tracking when a
tracker corrects itself after making an association mistake.

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

A: 2

0.5

1
gt
MOTA:
HOTA:

B: 2
1
gt

0.5

1-1/L
0.71

0.33
0.67

1-1/L
0.75

C: 2
1
gt

0.17
0.83

1-1/L
0.85

Fig. 8 An example showing how MOTA does not reward trackers
for having a greater alignment between predicted and ground-truth
trajectories. The thick line is the gtTraj. Thin lines are prTrajs. All
detections are TPs.

17
A: 2
1
gt
Duration:
Frame Rate:
Num. Frames:
MOTA AssErr:
MOTA:

B: 2
1
gt
2.5 sec
40 fps
100
1/100
0.99

2.5 sec
4 fps
10
1/10
0.90

Fig. 9 An example on MOTAs variability with the frame rate. The
thick line is the GT trajectory. Thin lines are predicted trajectories. All
detections are TPs.

In this case, MOTA will penalise the tracker twice, first formaking a mistake, and then for correcting it. An example ofthis can be seen in Fig. 7. In this example the ordering of thescores should be (A)>(B)>(C). This is because (A) corrects
itself when making a tracking mistake. (B) does not correct
itself and continues to track the object with the wrong ID.
(C) is even worse making a further tracking mistake. However, the MOTA score does not follow this intuitive ranking.
MOTA is unable to account for the tracker correcting itself
and counts the correction as a further mistake. This meansthat under MOTA (A) and (C) are equal even though (A) is
clearly much better than (C). This property of MOTA heavily
disincentivises research into long-term trackers that are able
to correct from mistakes.
HOTA solves this issue by measuring the association
globally over the whole sequence. In the example HOTA
ranks the trackers with the intuitive ranking of (A)>(B)>(C).

same tracker. In (A), however, it is evaluated at the originalvideo frame rate of 40 fps, in (B) it is evaluated at a reduced
frame rate of 4 fps. Different scenarios call for different frame
rates, e.g., in a surveillance scenario the frame rate of thecameras tends to be low. In Figure 9 we show how the tracker
is performing identically in both scenarios. However, as canbe seen, the MOTA score reflects a poorer performance ofthe tracker when used at a low frame rate, 0.90 MOTA vs0.99. This occurs because MOTA registers only a single
error in both cases (one IDSW), and yet this is normalisedby the total number of GT detections over the video, which
increases with the number of frames.
Our HOTA does not have this problem, because it measures the association error for each TP and averages this over
the TPs. Thus HOTA is independent of the frame rate, andthe HOTA value will be the same (0.707) for both (A) and(B) in Figure 9.

Problem with MOTA 6 MOTA does not reward trackers
for having a greater alignment between predicted andground-truth trajectories.

Problem with MOTA 8 MOTA does not take into accountlocalisation accuracy.

Another problem that arises because MOTA only evaluates
first-order short-term tracking, is that it does not reward trackers based on how well predicted trajectories and ground-truth
trajectories align. This can be seen clearly in Figure 8, where
it is clear that (C)>(B)>(A) due to the fact that in (C) oneof the prTrajs is 83% similar to the gtTraj, whereas in (A) at
best one of the trajectories is only 50% similar to the gtTraj.
However, because MOTA only evaluates short-term association, it is unable to differentiate between these cases andcorrectly measure the trajectory alignment.
This property is important, because as prTrajs and gtTrajs
become more similar to one another the evaluation score
should increase.
HOTA solves this problem by measuring association
globally across the whole sequence. Thus HOTA correctly
ranks these trackers, and is able to take into account the levelof alignment between trajectories.
Problem with MOTA 7 In MOTA, the influence of association (IDSWs) on the final score is highly dependent on thevideo frame rate.
This is easily seen with an example, as shown in Figure 9,
where we show exactly the same sequence processed by the

MOTA is calculated at a pre-set value of α for determining
the matching between prDets and gtDets, but the value of
MOTA is the same regardless of how correct these matches
are as long as they are over a minimum localisation threshold. Thus, MOTA was proposed as one of two metrics thatshould be used in combination for measuring MOT accuracy. The second of these is MOTP (Multi-Object Tracking
Precision), which is simply the average localisation similarity over the TP matches (ignoring detection and association
errors). Since MOTA was designed to be used together with
MOTP, it is not able to measure localisation together in asingle metric with detection and association.
HOTA solves this issue by including the localisation accuracy into its calculation. By calculating the score over arange of α values, HOTA is able to include localisation alongwith detection and association together into one score.
Problem with MOTA 9 MOTA scores can be negative andare unbounded.
The final, and perhaps most frustrating, problem of MOTA isthat scores are not between 0 and 1, as is typically expected
for evaluation metrics. Although the maximum MOTA scoreis 1 for perfect tracking, there is no lower limit to the MOTA

18

score, and it can go down to negative infinite. This is causedby the score decreasing linearly with the number of FPs,
which can be continuously added forever. This leads to ascore that is hard to interpret: how can we understand anegative MOTA score? Furthermore, a MOTA close to zeromight not be as bad as it seems. HOTA does not have thisproblem by conveniently being a score between 0 and 1.

Jonathon Luiten et al.

Predicted
Trajectories

IDF1 Matches

Ground-truth
Trajectories

9.2 Problems with IDF1
The IDF1 metric [60] was originally designed for evaluating
tracking in a multi-camera setting, but is trivial to apply to a
standard single camera setting. An overview of how IDF1 iscalculated can be found in Sec. 4.
IDF1 was designed to measure the concept of ‘identification’. This concept is related to, but distinct from theconcepts of detection and association which we analyse inthis paper. Identification is more about determining which
trajectories are present, rather than detecting objects and associating them throughout time.
In recent years IDF1 has been adopted by a numberof MOT papers [51, 52, 76] for MOT evaluation instead of
MOTA, as these papers wish to adequately account for association in evaluation, which is lacking from MOTA. However, IDF1 produces counter-intuitive and non-monotonic results for measuring detection. Due to this, no single-camera
MOT benchmarks have adopted IDF1 as the main metric forevaluating trackers and new benchmarks are still choosing
to use MOTA instead of IDF1 despite all of MOTAs drawbacks [74, 73, 87, 77]. This is because detection is such animportant part of tracking evaluation, and IDF1 isn’t able toadequately measure it.
In IDF1, each gtTraj can be matched with a single prTraj
and vice versa. This contrasts to HOTA where gtTrajs andprTrajs are evaluated as being matched if they are matchedat any point in time at any detections.
Just like MOTA can be thought of as only measuring
first-order association (a single previous association for eachdetection) compared to HOTA measuring higher-order association (all possible associations for each detection), IDF1
can be thought of as measuring first-order matching (a single possible match for each trajectory), compared to HOTA
measuring higher-order matching (all possible matches foreach trajectory).
Because IDF1 only allows a single best set of matching
trajectories to be evaluated, any trajectory that doesn’t end
up in this matching set is counted as a negative and decreases
the score, even if it contributes correct detections and associations.
Below we highlight 5 separate problems of the IDF1 metric, and show how these problems are addressed in HOTA.

Fig. 10 A tracking example which shows how the single best trajectory
matching, as performed by IDF1, can result in unintuitive matches
between trajectories.

A: 2
1
gt
IDF1:
HOTA:

0.5
0.5

B: 2
1
gt

0.5
0.71

0.5

0.66
0.5

Fig. 11 A simple tracking example showing how IDF1 fails to correctly rank tracking performance because the score can decrease withimproving detection. The thick line is the gtTraj. Thin lines are prTrajs.
All detections are TPs, except the second half of the gt in B which are
FNs.

Problem with IDF1 1 The best unique bijective mappingof whole trajectories is often not representative of the actualalignment between ground-truth and predicted trajectories.
This can be best understood with an example, as shownin Fig 10. In this example the best bijective mapping matches
the blue gtTraj with the grey prTraj. This occurs even though
this gtTraj is better matched with both other prTraj, andthis prTraj is better matched with both other gtTraj. Thusin this case, matching these trajectories is the worst possible
matching for both the gtTraj and the prTraj. However they arestill matched here, as when considering all trajectories, all of
the better matching options are better matched elsewhere andthus this worst fitting match is what ends up being evaluated.
This is a perfect example to show why a unique bijective
mapping of trajectories does not make intuitive sense for
evaluating tracking, which often results in complex overlaps
between ground-truth and predicted trajectories.
HOTA avoids this problem by not forcing a single global
matching between trajectories but rather evaluating over all
combinations of ground-truth and predicted trajectories thatoverlap at any point.
Problem with IDF1 2 IDF1
improving detection.

actually

decreases

with

In Fig. 10 it can be seen that in non-matched regions there
are many correct detection results. These correct detections
don’t add positively to the final score. In fact, each one ofthese correct detections decreases the final IDF1 score. Thus
IDF1 is non-monotonic in detection.
This can be seen more clearly in Fig. 11, where the scorefor (A) should be higher than for (B), but IDF1 ranks the two

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

tracking results in the other order because it penalises thecorrect detections in the second trajectory.
This property is extremely counter productive for many
tracking scenarios such as in autonomous driving where it iscritical to correctly detect objects.
HOTA does not have this problem and is strictly monotonic in detection such that improving detection always improves the HOTA score.
Problem with IDF1 3 IDF1 ignores the effect of how good
association is outside of matched sections.
Due to the fact that only the matched sections counttowards the score, the association can be trivially bad in nonmatched regions and this will have no effect on the IDF1
score. Thus creating better or worse association does notnecessarily correlate to an increase or decrease in the IDF1
score. This can be seen most clearly in Fig. 12 where the
IDF1 scores for (A), (B) and (C) are identical where it isclear that the trackers should be evaluated such that (A) >(B) > (C).
HOTA does not have the same problem as all detections
are evaluated, not just the best matching ones. Thus HOTA
ranks these trackers correctly.
Problem with IDF1 4 Scoring highly on IDF1 is more
about estimating the total number of unique objects in ascene than it is about good detection or association.
Due to the fact that any extra trajectories that are notmatched as one of the best matching trajectories are automatically counted as negatives, one of the key design goals
for trackers that are optimizing for IDF1 becomes to estimate the total number of unique objects in the scene andonly produce that many tracks, rather than performing good
detection or good association. This is because if there are
more (or less) predicted trajectories than ground-truth trajectories, the extra (missing) trajectories are automaticallycounted as negatives and severely decrease the score. Thisis a very different objective than the objective defined inthis paper for multi-object tracking, which is to detect thepresence of objects and to associate these consistently overtime.
In contrast, optimizing for HOTA directly optimises forboth accurate detection and accurate association as the final
HOTA score is a combination of scores for each of thesecomponents.
Problem with IDF1 5 IDF1 does not evaluate the localisation accuracy of trackers.
Like MOTA, IDF1 is also evaluated at a fixed α thresholdfor how accurate localisation needs to be for detections tomatch, however the actual localisation of the detections isignored as long as they are beyond the threshold.
HOTA is evaluated over a range of localisation thresholds
α and as such HOTA increases as the localisation of trackers
increases.

19

A: 4
3
2
1
gt
IDF1:
HOTA:

0.5

0.5

0.50
0.71

B:4
3
2
1
gt

0.25
0.25
0.5

0.50
0.61

C: 4
3
2
1
gt

0.17
0.17
0.17
0.5

0.50
0.58

Fig. 12 A simple tracking example showing how IDF1 fails to correctly rank tracking performance because it ignores the effect of any
association that is not included in the best matching trajectories. The
thick line is the gtTraj. Thin lines are prTrajs. All detections are TPs.

9.3 Problems with Track-mAP
Track-mAP is an extension of the mAP metric, commonlyused for evaluating detection [24, 61, 45], to the video domain. An overview of Track-mAP can be found in Sec. 4.
Track-mAP doesn’t operate on final tracking results but on
confidence-ranked potential tracking results. This results init being non-trivial to compare to other metrics as it operates
on a different, metric specific tracking output format. TrackmAP is similar to IDF1 in that it also performs matching at a
trajectory level. This results in it also being non-monotonic
in detection. Below we highlight 5 problems of Track-mAP,and show how these are addressed in HOTA.
Problem with Track-mAP 1 The interpretation of tracking
outputs is not trivial, nor easily visualisable.
With other metrics, when one wishes to understand a tracker,
all they have to do is visualise the tracking results. One
can easily identify each of the error types defined for eachof the other metrics. This is not the case for Track-mAP.
Here the output is likely many overlapping outputs, many ofwhich have low confidence scores, with the actual influenceof each trajectory on the final score being hidden behind theimplicit confidence ranking. This makes developing trackers
that optimise Track-mAP a potentially frustrating experience.
It also makes user-comparison, like the type we performin Sec. 11 impossible, as the representation doesn’t allow
meaningful analysis of the visual results.
Problem with Track-mAP 2 It is possible to game the metric by tuning trackers for quirks of the metric, which do notnecessarily correspond to better tracking.
There are a number of ways in which a trackers output
can be tuned to increase Track-mAP without actually improving the tracking result. One of these, as discussed in [49], isthat by producing many different predictions with low confidence scores, it greatly increases the chances of obtaining
a correct trajectory match and thus improving the score. In[49] they do this by replicating trajectories for each class forall other classes. Since this is possible to tune algorithms
in this way, it becomes a requirement for methods to adequately compete. Trackers that don’t do this will be heavily
penalised. Some benchmarks [18] have attempted to mitigate this issue by restricting results to a maximum number of20

trajectories per video. However [18] still allows 300 trajectories per frame which still enables significant gaming. HOTA
and other metrics don’t have this issue because trackers are
required to produce final tracking results.
Problem with Track-mAP 3 The threshold for being
counted as a positive match is so high that a lot of improvement in detection, association and localisation is ignored by
the metric.
In the Track-mAP version used in [61, 90] for a trajectory to
be counted as a positive it must be successfully detected andassociated such that the Jaccard over detections is at least
50%. This is a very high threshold for many tracking scenarios. We can see in Fig. 13 that for all published trackers on
the MOTChallenge leader-board the average Jaccard association alignment between trajectories (AssA) ranges between
0.25 and 0.5. This means that even for the very best tracker,
more than half of its best guess predictions will be counted
as errors in Track-mAP. This has the effect that as trackers
improve significantly in terms of detection and association
this is not shown by an improvement in metric scores. E.g.
trajectories that align 5% or 45% are given the same score.
For the Track-mAP version used in [18, 83] this threshold iseven harder to reach because although the threshold is still
50%, the score that must be above this threshold is effectively a multiplication of both the trajectory alignment andthe average localisation score across the trajectory which isusually much lower (see Sec. 4)
HOTA doesn’t have this issue as it measures the alignment between all trajectory pairs, not just those over a certain
threshold.
Problem with Track-mAP 4 Improving detection (adding
correctly matching trajectories) can decrease the score.
Track-mAP is non-monotonic in detection. This is because
it matches at a trajectory level (like IDF1) such that extra
trajectories are counted as negatives, even if they containcorrect, previously unaccounted for detections. As described
previously this is a non-intuitive and undesirable property
for many applications such as autonomous driving wheredetection is critical. HOTA on the other hand is monotonic
in detection.
Problem with Track-mAP 5 Track-mAP mixes associationand detection (and localisation) in a way that is not error
type differentiable.
In the Track-mAP version used in [61, 90] the score used
to measure whether trajectories match is a combination ofboth detection scores and association scores in a way that is
not separable or interpretable. Thus trajectories can matchand add positively to the score if they have high detection
accuracy and medium association accuracy, or medium detection accuracy and high association accuracy. The TrackmAP metric doesn’t give any indication as to which of these

Jonathon Luiten et al.

situations is occurring and as such has very limited use forunderstanding and optimizing the behaviour of trackers. The
Track-mAP version used in [18, 83] is even worse in that thematching score also mixes localisation accuracy with detection and association. Furthermore the effect of localisation
accuracy of the score can vary based on differences in object
scale over time, which is even more unintuitive. Note thatthe detection version of mAP [24, 61, 45] which the tracking
version is based upon doesn’t have this problem because thematching score only measures a single type of error, which
is detection errors. The problem arises when this metric is
extended to video to measure multiple error types simultaneously.
HOTA doesn’t have this problem because it is designed
to be decomposable into separate scores for each error type,such that the effect of each error type on the final metric isclear and the overall metric is error type differentiable.

10 Evaluating Trackers with HOTA on MOTChallenge
In order to see how HOTA compares to other metrics forreal state-of-the-art trackers, we evaluated HOTA on trackers submitted to the MOTChallenge MOT17 benchmark, andcompared these HOTA scores with the MOTA and IDF1
scores. We cannot compare to Track-mAP because this metric requires trackers to supply confidence scores which is not
the case for the MOTChallenge benchmark.
We restrict our evaluation to only those methods thatare published in peer reviewed journals and conferences.
We evaluate 37 different trackers [11, 76, 7, 64, 51, 84, 89, 33,
16, 82, 30, 47, 15, 35, 42, 85, 26, 29, 17, 34, 46, 41, 70, 25,31, 2,
62, 63, 9, 10, 3, 69, 4, 22, 37, 5] on MOT17 [54]. This includes
all of the trackers for which the relevant bibliographic
information was available when this analysis was performedon the 1st April 2020.
Ranking Methods by HOTA. In Table 2 we show all of
the published trackers, ranked according to our proposed
HOTA metric. For fine-grained analysis, we also show thedetection accuracy DetA, the association accuracy AssA,
the detection recall DetRe, the detection precision DetPr, theassociation recall AssRe and the association precision AssPr.
For a definition of these metrics, see Sec. 6. We also show
the scores for MOTA and IDF1 metrics, and add an indicator
to how the rankings change in compared to HOTA ranking.
Comparing Trends across Metrics. In Fig. 13 we show
how the scores for different metrics and sub-metrics varyacross all of the trackers that we evaluated on MOT17.
Fig. 13 (left) shows results for HOTA along with thetwo previously used metrics MOTA and IDF1. We observe,(i) that although these metrics do not always agree, they dofollow a similar loose trend. (ii) it is clear according to all
three metrics which trackers are performing well and which

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

21

Ranking

ScoresSub-Scores

HOTA

MOTA

IDF1

HOTA

MOTA

IDF1

DetA

AssA

DetRe

DetPr

AssRe

AssPr

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

2 ( ↑ 1)6 ( ↑ 4)1 ( ↓ 2)7 ( ↑ 3)31 ( ↑ 26)
8 ( ↑ 2)21 ( ↑ 14)
11 ( ↑ 3)
10 ( ↑ 1)
3 ( ↓ 7)12 ( ↑ 1)
4 ( ↓ 8)13 (-)
15 ( ↑ 1)
24 ( ↑ 9)
22 ( ↑ 6)
20 ( ↑ 3)
23 ( ↑ 5)
9 ( ↓ 10)5 ( ↓ 15)14 ( ↓ 7)
19 ( ↓ 3)
28 ( ↑ 5)
16 ( ↓ 8)
17 ( ↓ 8)
25 ( ↓ 1)
27 (-)
18 ( ↓ 10)
35 ( ↑ 6)
34 ( ↑ 4)
26 ( ↓ 5)
29 ( ↓ 3)
33 (-)
30 ( ↓ 4)
32 ( ↓ 3)
37 ( ↑ 1)
36 ( ↓ 1)

1 (-)
2 (-)
6 ( ↑ 3)7 ( ↑ 3)3 ( ↓ 2)10 ( ↑ 4)
5 ( ↓ 2)
8 (-)
9 (-)
12 ( ↑ 2)
4 ( ↓ 7)17 ( ↑ 5)
11 ( ↓ 2)
15 ( ↑ 1)
14 ( ↓ 1)
13 ( ↓ 3)
16 ( ↓ 1)
18 (-)
22 ( ↑ 3)
19 ( ↓ 1)
23 ( ↑ 2)
21 ( ↓ 1)
20 ( ↓ 3)
24 (-)
26 ( ↑ 1)
25 ( ↓ 1)
27 (-)
28 (-)
29 (-)
30 (-)
31 (-)
33 ( ↑ 1)
32 ( ↓ 1)
35 ( ↑ 1)
36 ( ↑ 1)
34 ( ↓ 2)
37 (-)

46.6
45.1
45.1
43.6
43
42.9
42.7
42.6
42.6
42.5
42.4
41.5
41.5
41.4
41.1
40.7
40.5
39.4
39.2
39
38.9
38.6
38.2
38.2
37.8
36.9
36.6
36.2
34.7
34.1
33.7
33.4
33.2
31.5
30.7
30.4
30.4

55.7
51.9
56.3
51.8
44.2
51.4
48.2
51.2
42.6
53.5
50.9
52.6
50.9
50
47.5
48.1
48.3
48
51.3
52
50.7
48.6
44.9
49.9
49.6
46.9
45.1
48.8
42.6
43.1
45.5
44.6
43.7
44.4
44
39.6
42.1

59.1
58.1
55.1
54.7
57.2
54
55.7
54.5
54.5
52.3
56
50.8
52.7
51.3
51.9
52.1
51.1
49.6
47.6
48.7
47.2
47.9
48.4
47.1
45.2
46
43.2
43.2
41.8
39.8
39.4
38.8
39.2
36.2
34.2
36.6
33.9

46.2
44.1
45.3
43.6
37.5
43
39.9
41.6
41.9
43.4
41.6
44
41.9
42.3
40
39.9
39.8
40.1
42.6
41.7
41.9
40
38.8
41.7
42.5
39
37.2
41.1
36.8
37.4
38
38.7
37
38
38
35.9
36.4

47.3
46.4
45
44
49.6
43.1
46
44.1
43.5
42
43.5
39.5
41.3
40.8
42.5
41.8
41.4
39
36.4
36.8
36.5
37.5
38
35.2
34.2
35.2
36.4
32.1
33.2
31.4
30.2
29.2
30.2
26.4
25.1
26.1
25.7

49.2
47.5
47.3
46.9
40
46
42.4
44.3
44.2
45.3
44.2
47.3
44.5
45.4
42.5
42.4
42.1
42.5
45.2
43.7
44.5
42.1
41.8
44.2
45.8
41.6
38.7
43.7
39.2
39.8
40.1
41.1
39.3
40
40
39.4
38.2

75.9
72.8
79.1
73.4
72.8
74.1
73.2
73.1
75.1
77.8
73.3
73.5
74.4
73.4
74.4
73.9
75
74.6
74.9
75.9
75.1
75.5
71.9
74.9
71.8
73.1
78.3
74.2
72.2
73.7
74.8
74
73.2
75.3
75.3
67.2
75.7

52.8
51.1
48.1
49.3
53.2
48
49.6
46.4
47.4
45.4
47
42.7
45.3
43.1
46.9
44.9
44.1
43.1
39.4
40.6
38
40.2
41.3
40.8
36.1
38.4
37.9
34.9
35.5
32.8
32.6
30.7
31.8
28.9
26.3
27.6
27

70.2
71.6
78.2
69.8
75.8
70.4
72.7
77.6
72.7
75.7
72.3
73.6
70.2
77.5
73.5
73.8
75.7
70.2
71.9
69.2
80
73.8
70.2
61.5
75.9
70.2
81.2
69.5
72.7
79.4
73.7
76.1
76.9
69.6
75.9
72.4
76.1

MPNTrack17 [11]eTC17 [76]Tracktor++v2 [7]eHAF17 [64]SAS_MOT17 [51]YOONKJ17 [84]
DMAN [89]jCC [33]
NOTA [16]Tracktor++ [7]
STRN_MOT17 [82]
JBNOT [30]
MOTDT17 [47]
EDMT17 [15]MHT_bLSTM [35]AM_ADM17 [42]HAM_SADF17 [85]PHD_GSDL17 [26]FWT_17 [29]FAMNet [17]MHT_DAM_17 [34]OTCD_1_17 [46]FPSN [41]
GMPHDOGM17 [70]
MTDF17 [25]MASS [31]LM_NN_17 [2]PHD_GM [62]
EAMTT_17 [63]SORT17 [9]IOU17 [10]HISP_T17 [3]
GMPHD_SHA [69]
GMPHD_DAL [4]GM_PHD_D [22]GMPHD_KCF [37]
GMPHD_N1Tr [5]Table 2 Results on MOT17 for all published (peer reviewed) trackers.

1.0
0.60

HOTA
MOTA
IDF1

0.55
0.8
0.50
0.6
0.45

0.50

HOTA
DetA
AssA

0.45

0.8
0.7
0.6

0.4
0.40

0.35

0.5

0.35
0.2
0.30
0.0
0.00

0.30

Score

0.40

10

20 0.2 30

0.25

DetRe
DetPr
AssRe
AssPr

0.4
0.3
0 0.4 1020
Rank by HOTA

300.6

0

100.8

20

30

1.0

Fig. 13 Trends between various metrics and sub-metrics on MOT17. All 37 trackers from Table 2 are shown in order of decreasing HOTA.

do not. This is not surprising, as all three metrics are aiming
to quantify the quality of tracking. This can also be seen in
Tab. 2 where the top-performing methods according to each
metric is always within the top four methods over all othermetrics. (iii) there are some significant differences in ranking
between HOTA and both MOTA and IDF1. (iv) in general,
HOTA aligns better in ranking with IDF1 than MOTA. This
is not surprising, as HOTA and IDF1 both aim to measure
long-term tracking quality, whereas MOTA is only able tocapture short-term tracking success. This is also reflected in
the table, where the change in rankings for MOTA is larger
compared to IDF1.
In Fig. 13 (middle), the HOTA score is compared with
its two major components, the DetA and AssA scores whichmeasure detection success and association success, respectively. HOTA is computed as the geometric mean of thetwo and thus is always between the two values. We makethe following observations. (i) both detection accuracy andassociation accuracy improve as trackers get better; (ii) topperformers are better at association than detection, while
poor performing trackers are better at detection; (iii) thereis larger variability over the association score over different
trackers compared to the detection scores. This is expected
as all trackers used the given public detections as input proposals.
In Fig. 13 (right) we compare four different components
of HOTA: the detection precision and recall, and association
precision and recall. As can be seen, (i) precision values arehigher than recall for both detection and association; (ii)
precision values are mostly within a similar range acrossall trackers, whereas recall values show an obvious trend to
decrease as the tracker performance is dropping.
Analysing the State-of-the-Art in Multiple Dimensions. HOTA combines the different aspects of tracking

22Jonathon Luiten et al.

DetA

HOTA

50

80

72.5
70.0
67.5

30

65.0
25

30

35

40

DetA

45

50

46.2 - MPNTrack17
44.1 - eTC1745.3 - Tracktor++v2
43.6 - eHAF17
37.5 - SAS_MOT17
32.5 35.0 37.5 40.0 42.5 45.0 47.5 50.0

DetRe

40

HOTA

35

AssPr

46.6 - MPNTrack17
45.1 - eTC1745.1 - Tracktor++v2
43.6 - eHAF17
43.0 - SAS_MOT17

40

50
75

75.0

DetPr

AssA

60

77.5

45

25

HOTA

AssA

80.0

70
65

55

20

47.3 - MPNTrack17
46.4 - eTC1745.0 - Tracktor++v2
44.0 - eHAF17
49.6 - SAS_MOT17

60
25

30

35

30

10

40

AssRe

45

50

46.6 - MPNTrack17
45.1 - eTC1745.1 - Tracktor++v2
43.6 - eHAF17
43.0 - SAS_MOT17
20

40

60

80

Fig. 14 Comparison between sub-metrics showing results for all peer reviewed trackers on MOT17. Each of the four plots shows a different
decomposition of metrics into their corresponding sub-metrics for evaluating different aspects of tracking. The grey curves are level sets contoursof constant score. The red staircase function shows the Pareto front. Only the top-5 trackers by HOTA are shown in the legend and far right plot forclarity.

in a balanced way suitable for ranking trackers. However
it is also informative to compare trackers along all of thedifferent dimensions of tracking. In Fig 14 we compare
trackers along a number of different dimensions within the
sub-metric space of the HOTA family of metrics.
This analysis allows one to clearly see the benefits andpitfalls of certain trackers, and allows for the selection of
top performing trackers for different applications that may
have different requirements. Any tracker along the multidimensional Pareto front can be considered to be state-ofthe-art in at least one aspect of tracking performance.
The fourth subplot shows how the HOTA score varies
over the localisation threshold α for the top five ranked trackers. By showing performance over the range of all thresholds
we are able to analyse and compare different properties oftrackers that are not otherwise apparent by using a single
evaluation score, such as which trackers perform very wellwhen matches are allowed to be loosely localized and thosethat still perform well when a higher standard of localizationis required.
What is also clear from this analysis is that the set oflowest-level sub-metrics (DetRe, DePr, AssRe, AssPr) arenot enough on their own to tell the whole story about the results between different trackers. One is able to gain a greaterlevel of understanding by examining the higher-level metrics
which are combinations of these sub-metrics (DetA, AssA
and HOTA). This highlights one of the key benefits of HOTA
compared to previous evaluation approaches, that it simultaneously is able to measure different aspects of tracking
performance, while being able to combine these together
into unified representative scores.
Analyzing Metrics across Detectors. The MOT17 benchmark requires methods to produce results using the same
tracking method with a set of three different input detections.
Thus it is possible to analyze how different performance metrics behave when using different detectors. Fig. 15 showssuch an analysis for HOTA, MOTA and IDF1, as well as forall of the sub-metrics of HOTA. For all main metrics, using
a better input detector improves the score. Of the three main

HOTA

0.5

MOTA

0.6

IDF1

0.6

0.4
SDP
FRCNN
DPM

0.3
0.2
0
0.5

0.4

0.4

20

0

DetA

20

DetRe

0.5

0.4

0.3
0

20

0.6
0

AssA

0.8

20

DetPr

0.7

0.4

0.3

0

20

0

AssRe

20

AssPr

0.8
0.4

0.4

0.2

0.2
0

20

0.7
0

20

0.6

0

20

Fig. 15 Trends for various metrics when using different detectors as
input on MOT17. All 37 trackers from Table 2 are shown, and areordered separately for each plot by the metric used.

metrics, MOTA is by the most affected by the choice of input
detector. On the other hand, HOTA and IDF1 exhibit similar trends when using different detectors as input. In fact,
MOTA exhibits similar trends to DetA. This is because, as
discussed in Sec. 9.1, MOTA is mostly a proxy for detection accuracy and thus is highly correlated with DetA. As
expected, the association scores are far less dependent on thedetector input, although it can be seen that better detectors
still aid better association. This is not surprising – having
more correct detections allows for more correct associations
to be made. Precision values for both detection and association are less affected by the choice of detector compared torecall values.
Do the Metrics Disagree where we expect them too? In
Sec. 9 we laid out a number of theoretical problems of both

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

MOTA
0.96

0.58

0.67

0.46

0.97

0.94

HOTA

IDF1

DetA

AssA

Fig. 16 Plotting each of the 3 main metrics against both the detection
score and the association score for trackers on MOT17. The line of best
fit is plotted in red, and the coefficient of determination (R2 ) is shownin the top left of each plot.

the MOTA and IDF1 metrics and discussed how HOTA
addresses these issues. In that analysis, we argued that MOTA
and IDF1 are two ends of the metric spectrum, with HOTA
being the middle ground. One of the main issues with MOTA
is that it does not adequately score association and mostly
only depends on detection accuracy, while IDF1 is exactly
the opposite – heavily relying on accurate association while
exhibiting non-intuitive behavior with regards to detection
quality.
In Fig. 16 we plot the MOTA, IDF1 and HOTA scoresfor all trackers on MOT17 against the DetA and AssA subscores, which measure the detection and association accuracy, respectively. We obtain the coefficient of determination (R2 ) by fitting the line-of-best-fit and determining thestrength of correlation between these metrics. As can beseen, the theoretical analysis of the weaknesses of MOTA
and IDF1 is reflected in these results. Our observations are
the following. (i) MOTA highly correlates with the detection score (0.96) while exhibiting low correlation with theassociation score (0.46). (ii) IDF1 exhibits almost the opposite behaviour, correlating strongly with the association
score (0.97), but shows low correlation with the detection
score (0.58). (iii) HOTA is between these two extremes, correlating reasonably strongly with both detection (0.67) andassociation (0.94). This explains why in many cases in Tab. 2
the HOTA score causes trackers to increase in rank comparedto MOTA while simultaneously decreasing in rank comparedto IDF1 (and vice versa).
Note that the correlation is stronger for association thandetection. This is not because association is assigned a higher
weight by HOTA (both are weighted equally). The reason forthis is that there is a wider range of association scores among
the trackers, compared to the range of detection scores. This
is to be expected, when all trackers are using a set of given
public detections as input. Thus the variation in tracking
scores is more likely to come from association than detection.

23

Methods which HOTA ranks higher than MOTA but
lower than IDF1 are those for which the association ismore accurate than the detection. An example of this is
SAS_MOT17 [51] which rises 26 places in HOTA comparedto MOTA but falls 2 places compared to IDF1. This method
specifically focuses on performing accurate association (andthey also analyse how IDF1 is better correlated with better
association than MOTA), at the cost of detection accuracy.
Thus this method performs poorly on MOTA and very wellaccording to IDF1, while performing somewhere in between
according to HOTA.
Methods which HOTA ranks higher than IDF1 but lower
than MOTA are those for which the detection is more accurate than the association. An example of this is JBNOT [30]which drops 8 places compared to MOTA, but rises 5 places
w.r.t. to IDF1. This method focuses on improving detection
recall, particularly during occlusions by using body joint detectors. However this does not perform association as wellas many other methods. Therefore, this method ranks highly
according to MOTA, but poorly according to IDF1. HOTA
again places it in-between, taking both detection and association evenly into account.
11 Human Visual Assessment Study
In previous sections we have provided theoretical analysis as
to why HOTA is preferable to other metrics, as well as experimental analysis when using these metrics to evaluate real
trackers. In this section we take this analysis one step further
and perform a large-scale user-study in order to determine
how these metrics align with human judgment of tracking
quality. Our study follows many aspects of the design of [40]which previously attempted to evaluate tracking metrics using human evaluators. This study evaluated MOTA against
a number of simple metrics such as Mostly Tracked, Detection Recall and MOTP. We wish to extend their analysis byrunning a study comparing HOTA, MOTA and IDF1.
Each of the different metrics has its own set of assumptions about what is important for tracking and evaluatesagainst the best practices according to its own assumptions.
Here we seek to answer the question of whether the assumptions for each metric align with the assumptions that humans
make when viewing objects tracked through a video. While
this is not a perfect proxy for the usefulness of these assumptions for any particular tracking application, it is nonethelessuseful to know how each of these sets of assumptions aligns
with human ranking. Furthermore, by specifically recruiting
MOT researchers to participate in our study, we are able toevaluate how the assumptions of each metric align with theassumptions of the community of people who would be using
these metrics. We believe it is a useful property for metrics
to evaluate tracking results in a way that is similar to how
experts would rank the results when viewing them.

24

Jonathon Luiten et al.

Fig. 17 An example of user-interface used for the user-experiment and the tracking visualisation.

Such an experiment needs to be carefully designed, suchthat the experiment imposes as little bias to the results as
possible. In order to conduct this study, we split the 7 sequences of the test-set of MOT17 up into 6 seconds chunks,
which gives us 36 short clips. We performed trial evaluations across video lengths of 3, 6 and 9 seconds and foundthat the optimal span for this study was 6-second clips. At
9 seconds the video was too long for the human attention
span to adequately judge tracking quality accurately. At 3
seconds the videos were too short for the tracking results tobe representative of meaningful tracking scenarios. We thenevaluated all trackers on the MOT17 benchmark, across these
36 six second video clips, and evaluated the MOTA, IDF1
and HOTA scores. In order to directly compare between metrics in the most user efficient way possible, we designed ourexperiment as a head-to-head comparison between metrics,
such that the user will be presented with a pair of videos
showing the tracking results of two different trackers. Eachvideo pair consists of the results for two trackers where twoof the metrics significantly disagree on which of the trackers
performs better. Users are then to select which tracker performs better, thus agreeing with one of the two metrics in thehead-to-head comparison. Users were also given the option
to select that both trackers performed equally, or to skip avideo pair.

Determining the pairs. We determined which pairs to showby first determining pairs that met our head-to-head requirements, and then by sub-selecting valid pairs based on diversity of sequences, detectors and trackers shown. Two trackers,
A and B, are are a valid pair to compare for the two metrics M1 and M2 (for a particular sub-sequence and detector

input) if they meet the following conditions:S1 · S2 < 0,

|S1| > 0.05,|S2| > 0.05

(45)

where S1 and S2 are defined as:
S1 = M1(A) − M1(B),S2 = M2(A) − M2(B)(46)

The first constraint ensures that the two metrics disagree
about which tracker is better (e.g., that the difference between
scores for the trackers on one metric should have a different
sign than for the other metric). The second two constraints
ensure that for both metrics there is a significant difference
between the trackers (at least 0.05) so that any difference inranking between the metrics is significant.
We evaluated 175 trackers on 108 unique combinationsof sub-sequence and input detections (36 sub-sequences andthree detection inputs). When comparing all pairs of trackers
which met the above constraints there were hundreds of valid
pairs per combination of metric. However we were aiming
for a smaller set of videos for the user study, as we wished foreach pair to be evaluated by a number of different users. We
sub-sampled the pairs in such a way that we took at most one
pair from each sub-sequence/detection combo and at mostone pair that contained each tracker. We did this by greedily
taking the pairs which minimise S1×S2, thus finding pairs forwhich the metrics maximally disagree, and for each chosen
pair removing all pairs which contain the same tracker or thesame sub-sequence/detection combination from the pool ofvalid pairs, and iterating greedily until there is no more pairs
to choose from. This gave us 67 pairs for HOTA vs MOTA,
51 pairs for HOTA vs IDF1, and 69 pairs for MOTA vs IDF1.
Result Visualisation. The way tracking results are displayed
to users is critically important for such a user study. Depending on how they are displayed, different aspects of tracking

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

could be emphasised. As we have seen in Sec. 9 and 10,
MOTA scores depend more on the quality of detection while
IDF1 scores depend more on the quality of association, thus
it is important that the visualisation method makes both typesof potential errors as obvious as possible to users.
If the visualisation fairly balances the visual saliency
of detection and association, then MOTA and IDF1 should
perform equally well when compared head-to-head in theuser-study, as each is representative of these two different
error types. Fig. 18 shows that this ends up being the case.
An example of our tracking visualisations (along with theuser interface for rating trackers) can be seen in Figure 17.
As can be seen we have made detection as obvious as
possible by showing bounding boxes with both a thick colored border and a slightly transparent fill. We have also madeassociation as obvious as possible by showing a tracking history of the bottom of each bounding box (in 2D pixel space)
as coloured points with lines joining them. This history remains shown for the whole history of a track, and only disappears when that object is no longer present in the current
frame. Such a visualisation style allows for both a quick understanding of the properties of trackers, as well as allowing
for a conscientious user to take their time and understand allof the complex detections and associations present.
We play videos to users at half the natural frame-rate
for easier video clarity. Users are also able to move aroundframes of the video by either clicking or dragging with themouse, or by using the arrow keys. Pairs of trackers wereshown to users in a random order, and users could evaluate
as many pairs as they desired. The videos within each pair
were shuffled so that the placement had no impact on which
metric was under evaluation.
Results of the User Study. We obtained user study results from 230 participants, 62 of which are multi-object
tracking researchers, and 122 of which are computer visionresearchers. On average each user evaluated 9.02 pairs of
trackers, for a total of 2075 unique tracker comparisons. On
average users took 2 minutes and 13 seconds to evaluate eachtracking pair, spending on average 20 minutes evaluating
trackers. This is the equivalent of 80 hours spent evaluating
tracking results.
A visualisation of results of the head-to-head comparisons between trackers can be seen in Figure 18. Note thatsome pairs were used in multiple metric head-to-heads. As
can be seen, looking at all participants, HOTA outperforms
MOTA by agreeing with human evaluators 61.6% of thetime compared to 38.4% for MOTA (when excluding thosethat voted for both). In the comparison of HOTA and IDF1,
human evaluators agree even more that HOTA is a better
tracking metric, agreeing with HOTA 72.0% of the timecompared with only 28.0% for IDF1. In the head-to-head for
MOTA vs IDF1 each metric agreed with users around 50%of the time.

25

HOTA vs MOTAHOTA MOTA
32.7%
38.4%
52.5% 61.6%

MOT

14.8%
both
n = 1269

18.5%

NOT MOT

20.3% 9.1%45.6% 45.2%37.7%
54.8%
79.7%

72.4%

n = 319CV

16.7%
n = 940

NOT CV

HOTA vs IDF1

MOTA vs IDF1

HOTA

MOTA IDF1

IDF1
23.8%

28.0%
72.0%
61.4%

42.3%

n = 1014

14.1%
85.9%
79.5%
n = 300

CV

41.8%

14.8%
both

MOT
13.0%

50.3%49.7%

NOT MOT

15.9%
both n = 1234

MOT

NOT MOT

28.1%
35.8%
39.3%
44.0%
34.3%
40.6%
59.4%52.3% 52.8%47.2%
53.9%65.7%

7.5%

18.0%
n = 709

NOT CV

11.9%
n = 218CV

16.7%
n = 1005

NOT CV

19.9%
28.8%
30.1%
36.0%
23.6%
33.5% 42.0%51.2%48.8%40.0%43.0%49.9%50.1%43.1%
35.7% 50.2% 41.8%
58.2%
54.3%64.3%
76.4% 15.6%57.3%66.5%
64.5%
13.9%
15.6%
13.8%
18.0%
13.9%
n = 687n = 572n = 588n = 421n = 588n = 635

Fig. 18 Results of our user study showing which metrics better agreewith human judgment. Experiments are set up in a head-to-head mannerwhere for a pair of videos the two metrics in the pair disagree on which
is better. Users selected the better tracking results, or alternatively rated
both the same. Inner circles are the results excluding those that ranked
both equally, while outer circles include these results. ‘n=’ shows thenumber of video pairs evaluated for each comparison. Results from allusers are shown in the big circles. Results are also shown by user selfreported domain, using information about whether the user is a multiobject tracking researcher (MOT) and whether they are a computer
vision researcher (CV).

For researchers who work in multi-object tracking thelevels of agreement with the HOTA metric compared to bothalternatives are much higher. Compared to MOTA, users
agreed with HOTA 79.3% of the time. Compared to IDF1,
users agreed with HOTA 85.9% of the time.
These results show that HOTA better aligns with humanjudgment of the accuracy of tracking results than previous
metrics. The fact that MOT researchers agree even more consistently with HOTA is a strong indication that HOTA is able
to successfully evaluate trackers in a way that is relevant forthe multi-object tracking community. Evaluating trackers is
a difficult task for humans, with often many objects present
and extremely complex scenes. The ‘correct’ answer is usually not obvious (as shown by users taking on average morethan 2 minutes per pair). However researchers in this field
have experience working with such data and know what tolook for in good tracking results. As such, the fact that HOTA
agrees so strongly with the judgment of MOT researchers is
a strong indication of the usefulness for the HOTA metric.

12 Conclusion
In this paper, we introduce HOTA (Higher Order Tracking
Accuracy), a novel metric for evaluating multi-object tracking. Previously used metrics only capture part of what is
important for tracking. MOTA is unable to capture association accurately. On the other hand, IDF1 and Track-mAP

26

perform non-intuitively in regards to detection. HOTA tackles these problems with a simple, elegant formulation that
equally weights detection and association accuracy.
We argue analytically and experimentally why our
proposed metric is preferable over the alternatives, testing
HOTA using state-of-the-art trackers on the MOTChallenge
benchmark. Furthermore, we perform a large-scale user
study and demonstrate that human visual assessment of
tracking accuracy aligns better with HOTA compared toboth MOTA and IDF1.
We believe that HOTA will change the nature of tracking
research, laying the groundwork for new algorithms to be
designed and benchmarked against a metric that measuresboth detection and association quality.
Acknowledgements The authors would like to thank Tobias Fischer,Achal Dave, Pavel Tokmakov, Jack Valmadre, Alex Bewley, Joao Henriques and Fisher Yu, as well as the anonymous reviewers, for their helpful comments on our manuscript. For partial funding of this project, JL
and BL would like to acknowledge the ERC Consolidator Grant DeeViSe (ERC-2017-COG-773161) and a Google Faculty Research Award.
AO, PD and LLT would like to acknowledge the Humboldt Foundation
through the Sofja Kovalevskaja Award. PT would like to acknowledge
CCAV project Streetwise and EPSRC/MURI grant EP/N019474/1.

References
1. Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L., Milan,A., Gall, J., Schiele, B.: Posetrack: A benchmark for human pose
estimation and tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5167–5176(2018)
2. Babaee, M., Li, Z., Rigoll, G.: A dual cnn–rnn for multiple people
tracking. Neurocomputing 368, 69–83 (2019)
3. Baisa, N.L.: Online multi-target visual tracking using a hisp filter. In: Proceedings of the 13th International Joint Conference on
Computer Vision, Imaging and Computer Graphics Theory andApplications - Volume 5: VISAPP„ pp. 429–438 (2018)
4. Baisa, N.L.: Online multi-object visual tracking using a gm-phd
filter with deep appearance learning. In: 2019 22th International
Conference on Information Fusion (FUSION), pp. 1–8 (2019)
5. Baisa, N.L., Wallace, A.: Development of a n-type gm-phd filter
for multiple target, multiple type visual tracking. Journal of Visual
Communication and Image Representation 59, 257 – 271 (2019)
6. Bento, J., Zhu, J.J.: A metric for sets of trajectories that is practicaland mathematically consistent. arXiv preprint arXiv:1601.03094(2016)
7. Bergmann, P., Meinhardt, T., Leal-Taixé, L.: Tracking without bells
and whistles. In: ICCV (2019)
8. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking
performance: the clear mot metrics. EURASIP Journal on Imageand Video Processing 2008, 1–10 (2008)
9. Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online
and realtime tracking. In: 2016 IEEE International Conference on
Image Processing (ICIP), pp. 3464–3468 (2016)10. Bochinski, E., Eiselein, V., Sikora, T.: High-speed tracking-bydetection without using image information. In: AVSS (2017)11. Brasó, G., Leal-Taixé, L.: Learning a neural solver for multiple
object tracking. In: CVPR (2020)
12. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu,
Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes:

Jonathon Luiten et al.
A multimodal dataset for autonomous driving. In: Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 11621–11631 (2020)
13. Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain,
T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., et al.:The ami meeting corpus: A pre-announcement. In: International
workshop on machine learning for multimodal interaction, pp. 28–
39. Springer (2005)
14. Chang, M.F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett,
A., Wang, D., Carr, P., Lucey, S., Ramanan, D., Hays, J.: Argoverse: 3d tracking and forecasting with rich maps. In: The IEEE
Conference on Computer Vision and Pattern Recognition (CVPR)(2019)
15. Chen, J., Sheng, H., Zhang, Y., Xiong, Z.: Enhancing detection
model for multiple hypothesis tracking. In: BMTT Workshop(2017)
16. Chen, L., Ai, H., Chen, R., Zhuang, Z.: Aggregate tracklet appearance features for multi-object tracking. IEEE Signal Processing
Letters 26(11), 1613–1617 (2019)17. Chu, P., Ling, H.: Famnet: Joint learning of feature, affinity andmulti-dimensional assignment for online multiple object tracking.
In: ICCV (2019)18. Dave, A., Khurana, T., Tokmakov, P., Schmid, C., Ramanan, D.:
Tao: A large-scale benchmark for tracking any object. In: ECCV(2020)
19. Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid,
I., Roth, S., Schindler, K., Leal-Taixé, L.: CVPR19 tracking anddetection challenge: How crowded can it get? arXiv:1906.04567[cs] (2019)20. Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid,
I., Roth, S., Schindler, K., Leal-Taixé, L.: Mot20: A benchmark for
multi object tracking in crowded scenes. arXiv:2003.09003[cs](2020). URL http://arxiv.org/abs/1906.04567. ArXiv:
2003.09003
21. Edward, K.K., Matthew, P.D., Michael, B.H.: An information theoretic approach for tracker performance evaluation. In: 2009 IEEE
12th International Conference on Computer Vision, pp. 1523–
1529. IEEE (2009)22. Eiselein, V., Arp, D., Pätzold, M., Sikora, T.: Real-time multihuman tracking using a probability hypothesis density filter andmultiple detectors. In: 2012 IEEE Ninth International Conference
on Advanced Video and Signal-Based Surveillance (2012)
23. Ellis, A., Ferryman, J.: Pets2010 and pets2009 evaluation of results
using individual ground truthed single views. In: 2010 7th IEEE
International Conference on Advanced Video and Signal Based
Surveillance, pp. 135–142. IEEE (2010)
24. Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman,
A.: The pascal visual object classes (VOC) challenge. IJCV 88(2),303–338 (2010)
25. Fu, Z., Angelini, F., Chambers, J., Naqvi, S.M.: Multi-level cooperative fusion of gm-phd filters for online multiple human tracking.
IEEE Transactions on Multimedia 21(9), 2277–2291 (2019)
26. Fu, Z., Feng, P., Angelini, F., Chambers, J., Naqvi, S.: Particle phd
filter based multiple human tracking using online group-structured
dictionary learning. In: IEEE Access (2018)
27. Fujita, O.: Metrics based on average distance between sets. Japan
Journal of Industrial and Applied Mathematics 30(1), 1–19 (2013)
28. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous
driving? the kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)
29. Henschel, R., Leal-Taixé, L., Cremers, D., Rosenhahn, B.: Fusionof head and full-body detectors for multi-object tracking. Trajnet
CVPRW (2018)30. Henschel, R., Zou, Y., Rosenhahn, B.: Multiple people tracking
using body and joint detections. CVPRW (2019)
31. Karunasekera, H., Wang, H., Zhang, H.: Multiple object tracking
with attention to appearance, structure, motion and size. IEEE
Access 7, 104423–104434 (2019)

HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking

27

32. Kasturi, R., Goldgof, D., Soundararajan, P., Manohar, V., Boonstra,
M., Korzhova, V.: Performance evaluation protocol for face, person
and vehicle detection & tracking in video analysis and content
extraction (vace-ii). Computer Science & Engineering Universityof South Florida, Tampa (2006)
33. Keuper, M., Tang, S., Andres, B., Brox, T., Schiele, B.: Motion
segmentation amp; multiple object tracking by correlation coclustering. IEEE Transactions on Pattern Analysis and MachineIntelligence pp. 1–1 (2018)
34. Kim, C., Li, F., Ciptadi, A., Rehg, J.M.: Multiple hypothesis tracking revisited. In: ICCV (2015)
35. Kim, C., Li, F., Rehg, J.M.: Multi-object tracking with neural gating
using bilinear lstm. In: ECCV (2018)
36. Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic
segmentation. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp. 9404–9413 (2019)
37. Kutschbach, T., Bochinski, E., Eiselein, V., Sikora, T.: Sequential
sensor fusion combining probability hypothesis density and kernelized correlation filters for multi-object tracking in video data.
In: AVSS (2017)
38. Layne, R., Hannuna, S., Camplani, M., Hall, J., Hospedales, T.M.,
Xiang, T., Mirmehdi, M., Damen, D.: A dataset for persistent multitarget multi-camera tracking in rgb-d. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition Workshops, pp. 47–55 (2017)
39. Leal-Taixé, L., Milan, A., Reid, I., Roth, S., Schindler, K.: MOTChallenge 2015: Towards a benchmark for multi-target tracking.
arXiv:1504.01942 [cs] (2015)
40. Leal-Taixé, L., Milan, A., Schindler, K., Cremers, D., Reid, I.,
Roth, S.: Tracking the trackers: an analysis of the state of the art in
multiple object tracking. arXiv preprint arXiv:1704.02781 (2017)
41. Lee, S., Kim, E.: Multiple object tracking via feature pyramid
siamese networks. IEEE access 7 (2018)
42. Lee, S.H., Kim, M.Y., Bae, S.H.: Learning discriminative appearance models for online multi-object tracking with appearance discriminability measures. IEEE Access 6, 67316–67328 (2018)
43. Leichter, I., Krupka, E.: Monotonicity and error type differentiability in performance measures for target detection and tracking invideo. IEEE transactions on pattern analysis and machine intelligence 35(10), 2553–2560 (2013)
44. Li, Y., Huang, C., Nevatia, R.: Learning to associate: Hybridboosted multi-target tracker for crowded scene. In: 2009 IEEE
Conference on Computer Vision and Pattern Recognition, pp.
2953–2960. IEEE (2009)
45. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan,D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects incontext. In: European conference on computer vision, pp. 740–755.
Springer (2014)
46. Liu, Q., Liu, B., Wu, Y., Li, W., Yu, N.: Real-time online multiobject tracking in compressed domain. IEEE Access 7, 76489–
76499 (2019)
47. Long, C., Haizhou, A., Zijie, Z., Chong, S.: Real-time multiple
people tracking with deeply learned candidate selection and person
re-identification. In: ICME (2018)
48. Luiten, J., Fischer, T., Leibe, B.: Track to reconstruct and reconstruct to track. IEEE Robotics and Automation Letters 5(2), 1803–
1810 (2020)
49. Luiten, J., Torr, P., Leibe, B.: Video instance segmentation 2019:
A winning approach for combined detection, segmentation, classification and tracking. In: Proceedings of the IEEE International
Conference on Computer Vision Workshops, pp. 0–0 (2019)
50. Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Zhao, X., Kim,
T.K.: Multiple object tracking: A literature review. arXiv preprint
arXiv:1409.7618 (2014)
51. Maksai, A., Fua, P.: Eliminating exposure bias and metric mismatch in multiple object tracking. In: CVPR (2019)52. Maksai, A., Wang, X., Fleuret, F., Fua, P.: Non-markovian globallyconsistent multi-object tracking. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2544–2554 (2017)53. Manohar, V., Boonstra, M., Korzhova, V., Soundararajan, P., Goldgof, D., Kasturi, R., Prasad, S., Raju, H., Bowers, R., Garofolo, J.:
Pets vs. vace evaluation programs: A comparative study. In: Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, pp. 1–6 (2006)
54. Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: MOT16:
A benchmark for multi-object tracking. arXiv:1603.00831 (2016)
55. Milan, A., Schindler, K., Roth, S.: Challenges of ground truth evaluation of multi-target tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops(2013)
56. Naphade, M., Anastasiu, D.C., Sharma, A., Jagrlamudi, V., Jeon,
H., Liu, K., Chang, M.C., Lyu, S., Gao, Z.: The nvidia ai
city challenge. In: 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pp. 1–6. IEEE
(2017)
57. Nghiem, A.T., Bremond, F., Thonnat, M., Valentin, V.: Etiseo, performance evaluation for video surveillance systems. In: 2007 IEEE
Conference on Advanced Video and Signal Based Surveillance, pp.
476–481. IEEE (2007)
58. Rahmathullah, A.S., García-Fernández, Á.F., Svensson, L.: A metric on the space of finite sets of trajectories for evaluation ofmulti-target tracking algorithms. arXiv preprint arXiv:1605.01177(2016)
59. Reid, D.: An algorithm for tracking multiple targets. IEEE transactions on Automatic Control 24(6), 843–854 (1979)
60. Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera
tracking. In: European Conference on Computer Vision, pp. 17–35.
Springer (2016)
61. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S.,
Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C.,
Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge.
International Journal of Computer Vision (IJCV) (2015)62. Sanchez-Matilla, R., Cavallaro, A.: A predictor of moving objects
for first-person vision. In: IEEE International Conference on Image
Processing (ICIP) (2019)63. Sanchez-Matilla, R., Poiesi, F., Cavallaro, A.: Online multi-target
tracking with strong and weak detections. In: Computer Vision –ECCV 2016 Workshops, pp. 84–99 (2016)
64. Sheng, H., Zhang, Y., Chen, J., Xiong, Z., Zhang, J.: Heterogeneous
association graph fusion for target association in multiple object
tracking. IEEE Transactions on Circuits and Systems for Video
Technology (2018)
65. Shitrit, H.B., Berclaz, J., Fleuret, F., Fua, P.: Tracking multiple
people under global appearance constraints. In: 2011 International
Conference on Computer Vision, pp. 137–144. IEEE (2011)
66. Singer, R., Sea, R., Housewright, K.: Derivation and evaluation of
improved tracking filter for use in dense multitarget environments.
IEEE Transactions on Information Theory 20(4), 423–432 (1974)
67. Smith, K., Gatica-Perez, D., Odobez, J.M., Ba, S.: Evaluating
multi-object tracking. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)Workshops, pp. 36–36. IEEE (2005)
68. Smith, P., Buechler, G.: A branching algorithm for discriminating
and tracking multiple objects. IEEE Transactions on AutomaticControl 20(1), 101–104 (1975)
69. Song, Y., Jeon, M.: Online multiple object tracking with the hierarchically adopted gm-phd filter using motion and appearance.
IEEE/IEIE The International Conference on Consumer Electronics(ICCE) Asia (2016)

28
70. Song, Y., Yoon, K., Yoon, Y., Yow, K., Jeon, M.: Online multiobject tracking with gmphd filter and occlusion group management. IEEE Access (2019)71. Stein, J., Blackman, S.: Generalized correlation of multi-target
track data. IEEE Transactions on Aerospace and Electronic Systems 11(6), 1207–1217 (1975)
72. Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J., Mostefa,D., Soundararajan, P.: The clear 2006 evaluation. In: International
evaluation workshop on classification of events, activities and relationships, pp. 1–44. Springer (2006)
73. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V.,
Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability inperception for autonomous driving: Waymo open dataset. arXiv
pp. arXiv–1912 (2019)74. Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B.B.G.,
Geiger, A., Leibe, B.: Mots: Multi-object tracking and segmentation. In: Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (2019)
75. Waibel, A., Steusloff, H., Stiefelhagen, R., Watson, K.: Computers in the human interaction loop. In: Computers in the Human
Interaction Loop, pp. 3–6. Springer (2009)
76. Wang, G., Wang, Y., Zhang, H., Gu, R., Hwang, J.N.: Exploit
the connectivity: Multi-object tracking with trackletnet. In: ACM
International Conference on Multimedia, pp. 482–490 (2019)
77. Wang, X., Zhang, X., Zhu, Y., Guo, Y., Yuan, X., Xiang, L., Wang,
Z., Ding, G., Brady, D., Dai, Q., et al.: Panda: A gigapixel-level
human-centric video dataset. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp.
3268–3278 (2020)
78. Wen, L., Du, D., Cai, Z., Lei, Z., Chang, M.C., Qi, H., Lim, J.,
Yang, M.H., Lyu, S.: Ua-detrac: A new benchmark and protocol for
multi-object detection and tracking. Computer Vision and Image
Understanding 193, 102907 (2020)
79. Weng, X., Wang, J., Held, D., Kitani, K.: 3d multi-object tracking:A baseline and new evaluation metrics. In: IROS (2020)
80. Wu, B., Nevatia, R.: Tracking of multiple, partially occluded humans based on static body part detection. In: 2006 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition(CVPR’06), vol. 1, pp. 951–958. IEEE (2006)
81. Wu, C.W., Zhong, M.T., Tsao, Y., Yang, S.W., Chen, Y.K., Chien,
S.Y.: Track-clustering error evaluation for track-based multicamera tracking system employing human re-identification. In:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–9 (2017)
82. Xu, J., Cao, Y., Zhang, Z., Hu, H.: Spatial-temporal relation networks for multi-object tracking. ICCV (2019)
83. Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5188–5197 (2019)84. YOON, K., GWAK, J., SONG, Y.M., YOON, Y.C., JEON, M.:
Oneshotda: Online multi-object tracker with one-shot-learningbased data association. IEEE Access 8, 38060–38072 (2020)
85. Yoon, Y., Boragule, A., Song, Y., Yoon, K., Jeon, M.: Online multiobject tracking with historical appearance matching and scene
adaptive detection filtering. In: IEEE AVSS (2018)86. Young, D.P., Ferryman, J.M.: Pets metrics: On-line performance
evaluation service. In: 2005 IEEE International Workshop on
Visual Surveillance and Performance Evaluation of Tracking and
Surveillance, pp. 317–324. IEEE (2005)
87. Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan,V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2636–
2645 (2020)
88. Yu, S.I., Meng, D., Zuo, W., Hauptmann, A.: The solution path
algorithm for identity-aware multi-object tracking. In: Proceedingsof the IEEE conference on computer vision and pattern recognition,
pp. 3871–3879 (2016)

Jonathon Luiten et al.
89. Zhu, J., Yang, H., Liu, N., Kim, M., Zhang, W., Yang, M.H.: Online
multi-object tracking with dual matching attention networks. In:
ECCV (2018)
90. Zhu, P., Wen, L., Du, D., Bian, X., Hu, Q., Ling, H.: Vision meetsdrones: Past, present and future. arXiv preprint arXiv:2001.06303(2020)