Multi-Object Tracking Metrics
By: Amrith Coumaran
Tags: Precision, Accuracy, Metrics, Tracking, Mot

The Evaluation process is one of the most important steps in build a Machine Learning Model. Especially when it comes to real-time detection plus tracking system. Computer Vision applications in tracking are becoming increasingly popular in surveillance, Sports Analytics, Autonomous vehicles, etc. So, evaluating your model would be the most difficult task before your deploy it. Today, let’s go through a set of metrics that evaluates your tracking system and gives you a better understanding of your model.

Photo by Mika Baumeister on Unsplash

Introduction

The Multi-Object tracking system in a real-world environment has always been challenging and there are not a lot of metrics out there to evaluate your tracking system. This post describes the implementation of one such metric called the CLEAR MOT Metrics. The main goal for any tracking evaluation system would be to judge the tracker’s precision and check the trajectory of the object in motion, while there are no multiple routes for a single object.

CLEAR MOT Metrics

Let’s say for every frame in the video feed, the tracker system outputs ’n’ hypothesis and the frame has ‘m’ ground truth objects the the evaluation process of CLEAR MOT metrics is as follows:

To find and pair the best match between the hypothesis and ground truth based on their coordinates and with the help of different matching algorithms. For each matched pair, find the error in the object's position. Calculate the sum of multiple errors such as:

a) Misses, where the tracker failed to produce any hypothesis for the given object.

b)False Positives, where the tracker produced a hypothesis but no object was present.

c) Mismatch errors, where the tracker’s hypothesis of a ground truth changed the present frame.

Figure: Bernardin, Keni, and Rainer Stiefelhagen. “Evaluating multiple object tracking performance: the clear mot metrics.” EURASIP Journal on Image and Video Processing 2008 (2008): 1–10.

Based on the above procedure, the performance can be expressed in two Metrics:

MOTP (Multi-Object Tracking Precision) expresses how well exact positions of the object are estimated. It is the total error in estimated position for matched ground truth-hypothesis pairs over all frames, averaged by the total number of matches made. This metric is not responsible for recognizing object configurations and evaluating object trajectories. MOTA (Multi-Object Tracking Accuracy) shows how many errors the tracker system has made in terms of Misses, False Positives, Mismatch errors, etc. Therefore, it can be derived from three error ratios: the ratio of Misses, the ratio of False positives, and the ratio of Mismatches over all the frames.

Implementation Using Python

If you want a detailed mathematical explanation for the metrics, I suggest you go through their official paper linked below. Now, let’s dive into the implementation of the metrics using python.

The py-motmetrics python library not only gives you MOTA and MOTP but also a wide variety of values through which you can analyze your tracking system.

Install the library using the pip command,

pip install motmetrics

Import the necessary packages and give in the hypothesis and ground truths coordinates and object IDs and get the distance matrix using the IOU method.

import motmetrics as mm

import numpy as np



acc = mm.MOTAccumulator(auto_id=True)

a = np.array([

[121,458,187,551], #coordinates of three hypothesis in the frame

[ 316 ,414 ,373,492] # Format--> X, Y, Width, Height

])

b = np.array([

[600 , 612, 700, 712], #coordinates of three groundtruth objects

[134, 445, 173, 523],

[321 , 436, 366, 481]

])

distance_matrix = mm.distances.iou_matrix(a, b, max_iou=0.7)

print(distance_matrix) OUTPUT --> [[ nan 0.16199685 nan]

[ nan nan 0.08276546]]

Now let’s update the accumulator with object IDs and get the evaluation report.

acc.update(

[1, 2], # Ground truth object ID in this frame

[0,3,4], # Detector hypothesis ID in this frame

[distance_matrix]) mh = mm.metrics.create() report = mh.compute(acc, metrics=['num_frames', 'num_objects','num_matches' ,'mota','motp', 'num_misses','num_false_positives','num_switches','mostly_tracked','partially_tracked','mostly_lost'], name='acc') print(report)

Check the documentation for more parameters to be included in your evaluation report. You can automate your code for multiple frames for your video.

Head on to the MOT Challenge Website to download sample videos with a frame-by-frame annotated dataset for different video scenarios to test your tracking system. Example parsing of ground truth data from MOT17 and MOT20:

def gtparser(current_frame,end_frame,file,split): '''

current_frame and end_frame gives you a range of frames you wanna work with.

file -> location of the gt.txt file.

split-> how are the values in gt seperated(mostly it is a ',' or a space ' ').

'''

dict1 = {}

while current_frame<=end_frame:

f_bb,idarr = [], []

f = open(file,'r')

for line in f.readlines():

bb = []

if int(line.split(split)[0])==current_frame and int(line.split(split)[7])==1:

x_scale = 960/1920

y_scale = 540/1080

(origLeft, origTop, origWidth, oriHeight) = (int(float(line.split(split)[2])),int(float(line.split(split)[3])),int(float(line.split(split)[4])),int(float(line.split(split)[5]))) dict1 = {}

while current_frame<=end_frame:

f_bb,idarr = [], []

f = open(file,'r')

for line in f.readlines():

bb = []

if int(line.split(split)[0])==current_frame and int(line.split(split)[7])==1:

x_scale = 960/1920

y_scale = 540/1080

(origLeft, origTop, origWidth, oriHeight) = (int(float(line.split(split)[2])),int(float(line.split(split)[3])),int(float(line.split(split)[4])),int(float(line.split(split)[5]))) x = int(np.round(origLeft * x_scale))

y = int(np.round(origTop * y_scale)) w = int(np.round(origWidth * x_scale))

h = int(np.round(oriHeight * y_scale))

bb.append(x)

bb.append(y)

bb.append(w)

bb.append(h)

idarr.append(int(line.split(split)[1]))

f_bb.append(bb)

f_bb = np.array(f_bb)

dict1[current_frame] = (idarr,f_bb)

current_frame+=1

f.close()

#print(len(dict),current_frame) return dict1

Interpretation of Results from py-motmetrics

MOTP ranges from 0–1 (multiply with 100 if you needed in percentage) If the MOTP value is 1, then the precision of the system is poor. And if it is close to zero then the precision of the system is good. MOTA ranges from –inf to 1. (multiply with 100 if you need the MOTA in percentage). If the MOTA is 1 then the Accuracy of the system is Good. If the MOTA is around zero or less than zero then the accuracy of the system is poor. num_objects, total numbers of unique object appearances over all frames. num_matches, total number matches between hypothesis and ground truth object IDs. misses, the total number of misses (False negatives). false_positives (FP), the total number of false positives (false alarms). switches, the total number of track switches. mostly_tracked, number of objects tracked for at least 80 percent of lifespan partially_tracked, number of objects tracked between 20 and 80 percent of lifespan. mostly_lost, the number of objects tracked less than 20 percent of lifespan.

Sources