Title: Object tracking with dlib - PyImageSearch

URL Source: https://pyimagesearch.com/2018/10/22/object-tracking-with-dlib/

Published Time: 2018-10-22T14:00:52+00:00

Markdown Content:
![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

This tutorial will teach you how to perform object tracking using dlib and Python. After reading today’s blog post you will be able to track objects in real-time video with dlib.

A couple months ago we discussed [centroid tracking](https://pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/), a simple, yet effective method to (1) _assign unique IDs to each object in an image_ and then (2) _track each of the objects and associated IDs as they move around in a video stream_.

The biggest downside to this object tracking algorithm is that a separate object detector has to be run on _each and every input frame_ — in most situations, this behavior is undesirable as object detectors, including HOG + Linear SVM, Faster R-CNNs, and SSDs can be computationally expensive to run.

An alternative approach would be to:

1.  Perform object detection once (or once every _N_ frames)
2.  And then apply a dedicated tracking algorithm that can keep tracking of the object as it moves in subsequent frames _without_ having to perform object detection

Is such a method possible?

The answer is yes, and in particular, we can use dlib’s implementation of the **correlation tracking algorithm**.

In the remainder of today’s blog post, you will learn how to apply dlib’s correlation tracker to track an object in real-time in a video stream.

**To learn more about dlib’s correlation tracker, _just keep reading_.**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

We’ll start off today’s tutorial with a brief discussion of dlib’s implementation of correlation-based object tracking.

From there I will show you how to utilize dlib’s object tracker in your own applications.

Finally, we’ll wrap up today by discussing some of the limitations and drawbacks of dlib’s object tracker.

### What are correlation trackers?

The [dlib correlation tracker implementation](http://blog.dlib.net/2015/02/dlib-1813-released.html) is based on Danelljan et al.’s 2014 paper, [_Accurate Scale Estimation for Robust Visual Tracking_](http://www.bmva.org/bmvc/2014/papers/paper038/index.html).

Their work, in turn, builds on the popular MOSSE tracker from Bolme et al.’s 2010 work, [_Visual Object Tracking using Adaptive Correlation Filters_](http://www.cs.colostate.edu/~vision/publications/bolme_cvpr10.pdf). While the MOSSE tracker works well for objects that are translated, it often fails for objects that change in scale.

The work of Danelljan et al. proposed utilizing a scale pyramid to accurately estimate the scale of an object after the optimal translation was found. This breakthrough allows us to track objects that change in both (1) translation and (2) scaling throughout a video stream — and furthermore, we can perform this tracking in real-time.

For a detailed review of the algorithm, please refer to the papers linked above.

### Project structure

To see how this project is organized, simply use the `tree` command in your terminal:

$ tree
.
├── input
│   ├── cat.mp4
│   └── race.mp4
├── output
│   ├── cat\_output.avi
│   └── race\_output.avi
├── mobilenet\_ssd
│   ├── MobileNetSSD\_deploy.caffemodel
│   └── MobileNetSSD\_deploy.prototxt
└── track\_object.py

3 directories, 7 files

We have three directories:

*   `input/` : Contains input videos for object tracking.
*   `output/` : Our processed videos. In the processed video, the tracked object is annotated with a box and label.
*   `mobilenet_ssd/` : The Caffe CNN model files are contained within this directory.

Today we’ll be reviewing one Python script: `track_object.py` .

### Implementing our dlib object tracker

Let’s go ahead and get started implementing our object tracker using dlib.

Open up `track_object.py` and insert the following code:

\# import the necessary packages
from imutils.video import FPS
import numpy as np
import argparse
import imutils
import dlib
import cv2

Here we import our required packages. Notably, we’re using [dlib](http://dlib.net/), [imutils](https://github.com/jrosebr1/imutils), and [OpenCV](https://opencv.org/).

From there, let’s parse our [command line arguments](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/):

\# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-p", "--prototxt", required=True,
	help="path to Caffe 'deploy' prototxt file")
ap.add\_argument("-m", "--model", required=True,
	help="path to Caffe pre-trained model")
ap.add\_argument("-v", "--video", required=True,
	help="path to input video file")
ap.add\_argument("-l", "--label", required=True,
	help="class label we are interested in detecting + tracking")
ap.add\_argument("-o", "--output", type=str,
	help="path to optional output video file")
ap.add\_argument("-c", "--confidence", type=float, default=0.2,
	help="minimum probability to filter weak detections")
args = vars(ap.parse\_args())

Our script has four required command line arguments:

*   `--prototxt` : Our path to the Caffe deploy prototxt file.
*   `--model` : The path to the Caffe pre-trained model.
*   `--video` : The path to the input video file. Today’s script works with video files rather than your webcam (but you could easily change it to support a webcam stream).
*   `--label` : A class label that we are interested in detecting and tracking. Review the next code block for the available classes that this model supports.

And two optional ones:

*   `--output` : An optional path to an output video file if you’d like to save the results of the object tracker.
*   `--confidence` : With a `default=0.2` , this is the minimum probability threshold and it allows us to filter weak detections from our Caffe object detector.

Let’s define the classes that this model supports and load our network from disk:

\# initialize the list of class labels MobileNet SSD was trained to
# detect
CLASSES = \["background", "aeroplane", "bicycle", "bird", "boat",
	"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
	"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
	"sofa", "train", "tvmonitor"\]

# load our serialized model from disk
print("\[INFO\] loading model...")
net = cv2.dnn.readNetFromCaffe(args\["prototxt"\], args\["model"\])

We’ll be using a pre-trained MobileNet SSD to perform object detection in a single frame. From there the object location will be handed off to dlib’s correlation tracker for tracking throughout the remaining frames of the video.

The model included with the _**“Downloads”**_ supports 20 object classes (plus 1 for the background class) on **Lines 27-30**.

_**Note:** If you’re using a different Caffe model, you’ll need to redefine this `CLASSES` list. Similarly, don’t modify this list if you’re using the model included with today’s download. If you’re confused about how deep learning object detectors work, [be sure to refer to this getting started guide.](https://pyimagesearch.com/2018/05/14/a-gentle-guide-to-deep-learning-object-detection/)_

Prior to looping over frames, we need to load our model into memory. This is handled on **Line 34** where all that is required to load a Caffe model is the path to the prototxt and model files (both available in our command line `args` dictionary).

Now let’s perform important initializations, notably our video stream:

\# initialize the video stream, dlib correlation tracker, output video
# writer, and predicted class label
print("\[INFO\] starting video stream...")
vs = cv2.VideoCapture(args\["video"\])
tracker = None
writer = None
label = ""

# start the frames per second throughput estimator
fps = FPS().start()

Our video stream, `tracker` , and video `writer` objects are initialized on **Lines 39-41**. We also initialize our textual `label` on **Line 42**.

Our frames-per-second estimator is instantiated on **Line 45**.

Now we’re ready to begin looping over our video frames:

\# loop over frames from the video file stream
while True:
	# grab the next frame from the video file
	(grabbed, frame) = vs.read()

	# check to see if we have reached the end of the video file
	if frame is None:
		break

	# resize the frame for faster processing and then convert the
	# frame from BGR to RGB ordering (dlib needs RGB ordering)
	frame = imutils.resize(frame, width=600)
	rgb = cv2.cvtColor(frame, cv2.COLOR\_BGR2RGB)

	# if we are supposed to be writing a video to disk, initialize
	# the writer
	if args\["output"\] is not None and writer is None:
		fourcc = cv2.VideoWriter\_fourcc(\*"MJPG")
		writer = cv2.VideoWriter(args\["output"\], fourcc, 30,
			(frame.shape\[1\], frame.shape\[0\]), True)

We begin our `while` loop on **Line 48** and proceed to grab a `frame` on **Line 50**.

Our frame is resized and the color channels are swapped on **Lines 58 and 59**. Resizing allows for faster processing — you can experiment with the frame dimensions to achieve higher FPS. Converting to RGB color space is required by dlib (OpenCV stores images in BGR order by default).

Optionally, at runtime, an output video path can be passed via command line arguments. So, if necessary, we’ll initialize our video `writer` on **Lines 63-66**. For more information on writing video to disk with OpenCV, see [this previous post](https://pyimagesearch.com/2016/02/22/writing-to-video-with-opencv/).

Next, we’ll need to detect an object for tracking (if we haven’t already):

	# if our correlation object tracker is None we first need to
	# apply an object detector to seed the tracker with something
	# to actually track
	if tracker is None:
		# grab the frame dimensions and convert the frame to a blob
		(h, w) = frame.shape\[:2\]
		blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)

		# pass the blob through the network and obtain the detections
		# and predictions
		net.setInput(blob)
		detections = net.forward()

If our `tracker` object is `None` (**Line 71**), we first need to _**detect**_ objects in the input `frame` . To do so, we create a `blob` (**Line 74**) and pass it through the network (**Lines 78 and 79**).

Let’s handle the `detections` now:

		# ensure at least one detection is made
		if len(detections) > 0:
			# find the index of the detection with the largest
			# probability -- out of convenience we are only going
			# to track the first object we find with the largest
			# probability; future examples will demonstrate how to
			# detect and extract \*specific\* objects
			i = np.argmax(detections\[0, 0, :, 2\])

			# grab the probability associated with the object along
			# with its class label
			conf = detections\[0, 0, i, 2\]
			label = CLASSES\[int(detections\[0, 0, i, 1\])\]

If our object detector finds any objects (**Line 82**), we’ll grab the one with the largest probability (**Line 88**).

We’re only demonstrating how to use dlib to perform single object tracking in this post, so we need to find the detected object with the highest probability. Next week’s blog post will cover _multi-object tracking with dlib_.

From there, we’ll grab the confidence (`conf` ) and `label` associated with the object (**Lines 92 and 93**).

Now it’s time to filter out the detections. Here we’re trying to ensure we have the right type of object which was passed by command line argument:

			# filter out weak detections by requiring a minimum
			# confidence
			if conf > args\["confidence"\] and label == args\["label"\]:
				# compute the (x, y)-coordinates of the bounding box
				# for the object
				box = detections\[0, 0, i, 3:7\] \* np.array(\[w, h, w, h\])
				(startX, startY, endX, endY) = box.astype("int")

				# construct a dlib rectangle object from the bounding
				# box coordinates and then start the dlib correlation
				# tracker
				tracker = dlib.correlation\_tracker()
				rect = dlib.rectangle(startX, startY, endX, endY)
				tracker.start\_track(rgb, rect)

				# draw the bounding box and text for the object
				cv2.rectangle(frame, (startX, startY), (endX, endY),
					(0, 255, 0), 2)
				cv2.putText(frame, label, (startX, startY - 15),
					cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

On **Line 97** we check to ensure that `conf` exceeds the confidence threshold _**and**_ that the object is actually the class type we’re looking for. When we run the script later, we’ll use “person” or “cat” as examples so you can see how we can filter results.

We determine bounding `box` coordinates of our object on **Lines 100 and 101**.

Then we establish our **dlib object tracker** and provide the bounding box coordinates (**Lines 106-108**). Future tracking updates will be easy from here on.

A bounding box rectangle and object class `label` text is drawn on the `frame` on **Lines 111-114**.

Let’s handle the case where we’ve already established a `tracker` :

	# otherwise, we've already performed detection so let's track
	# the object
	else:
		# update the tracker and grab the position of the tracked
		# object
		tracker.update(rgb)
		pos = tracker.get\_position()

		# unpack the position object
		startX = int(pos.left())
		startY = int(pos.top())
		endX = int(pos.right())
		endY = int(pos.bottom())

		# draw the bounding box from the correlation object tracker
		cv2.rectangle(frame, (startX, startY), (endX, endY),
			(0, 255, 0), 2)
		cv2.putText(frame, label, (startX, startY - 15),
			cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

This `else` block handles the case where we’ve already locked on to an object for tracking.

Think of it like a [dogfight](https://www.youtube.com/watch?v=uvFc0EPRSI4) in the movie, _Top Gun_. Once the enemy aircraft has been locked on by the “guidance system”, it can be tracked via updates.

This requires two main actions on our part:

1.  Update our tracker object (**Line 121**) — the heavy lifting is performed in the backend of this `update` method.
2.  Grab the position (`get_position` ) of our object from the `tracker` (**Line 122**). This would be where a PID control loop would come in handy if, for example, a robot seeks to follow a tracked object. In our case, we’re just going to annotate the object in the frame with a bounding box and label on **Lines 131-134**.

Let’s finish out the loop:

	# check to see if we should write the frame to disk
	if writer is not None:
		writer.write(frame)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the \`q\` key was pressed, break from the loop
	if key == ord("q"):
		break

	# update the FPS counter
	fps.update()

If the `frame` should be written to video, we do so on **Lines 137 and 138**.

We’ll show the `frame` on the screen (**Line 141**).

If the quit key (“q”) is pressed at any point during playback + tracking, we’ll `break` out of the loop (**Lines 142-146**).

Our `fps` estimator is updated on **Line 149**.

Finally, let’s perform print out FPS throughput statistics and release pointers prior to the script exiting:

\# stop the timer and display FPS information
fps.stop()
print("\[INFO\] elapsed time: {:.2f}".format(fps.elapsed()))
print("\[INFO\] approx. FPS: {:.2f}".format(fps.fps()))

# check to see if we need to release the video writer pointer
if writer is not None:
	writer.release()

# do a bit of cleanup
cv2.destroyAllWindows()
vs.release()

Housekeeping for our script includes:

*   Our `fps` counter is stopped and the FPS information is displayed in the terminal (**Lines 152-154**.
*   Then, if we were writing to an output video, we release the video `writer` (**Lines 157 and 158**).
*   Lastly, we close all OpenCV windows and release the video stream (**Lines 161 and 162**).

### Running dlib’s object tracker in real-time

To see our dlib object tracker in action, make sure you use the _**“Downloads”**_ section of this blog post to download the source code.

From there, open up a terminal and execute the following command:

$ python track\_object.py --prototxt mobilenet\_ssd/MobileNetSSD\_deploy.prototxt \\
	--model mobilenet\_ssd/MobileNetSSD\_deploy.caffemodel --video input/race.mp4 \\
	--label person --output output/race\_output.avi
\[INFO\] loading model...
\[INFO\] starting video stream...
\[INFO\] elapsed time: 13.18
\[INFO\] approx. FPS: 25.80

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

[Usain Bolt](https://en.wikipedia.org/wiki/Usain_Bolt) (Olympic World Record holder) was detected initially with highest confidence at the beginning of the video. From there, he is tracked successfully throughout his 100m race.

The full video can be found below:

Below we have a second example of object tracking with dlib:

$ python track\_object.py --prototxt mobilenet\_ssd/MobileNetSSD\_deploy.prototxt \\
	--model mobilenet\_ssd/MobileNetSSD\_deploy.caffemodel --video input/cat.mp4 \\
	--label cat --output output/cat\_output.avi
\[INFO\] loading model...
\[INFO\] starting video stream...
\[INFO\] elapsed time: 6.76
\[INFO\] approx. FPS: 24.12

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

The cat above was part of a [BuzzFeed segment](https://www.youtube.com/watch?v=9C1leq--_wM) cat owners trying to take their cats for a walk (as if they were dogs). Poor cats!

### Drawbacks and potential improvements

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

If you watched the full output video of the demo above, you would have noticed the object tracker behaving strangely towards the end of the demo, as this GIF demonstrates.

So, what’s going on here?

Why is the tracker losing the object?

Keep in mind there is no such thing as a “perfect” object tracker — and furthermore, this object tracking algorithm is not requiring you to run a more expensive object detector on each and every frame of the input image.

Instead, dlib’s correlation tracker is combining both (1) prior information regarding the location of the object bounding box in the _previous_ frame along with (2) data garnered from the _current_ frame to infer where the new location of the object is.

There will _certainly_ be times when the algorithm loses the object.

To remedy this situation, I recommend occasionally running your more expensive object detector to (1) validate the object is still there and (2) reseed the object tracking with the updated (and ideally correct) bounding box coordinates. August’s blog post on [people counting with OpenCV](https://pyimagesearch.com/2018/08/13/opencv-people-counter/) accomplished exactly this, so be sure to check it out.

### What about multi-object tracking?

Undoubtedly, I know there will be PyImageSearch readers wishing to apply this method to _multi-object tracking_ rather than single object tracking.

Is it possible to track multiple objects using dlib’s correlation tracker?

The answer is yes, absolutely!

I’ll be covering multi-object tracking next week, so stay tuned.

### Video credits

To create the examples for this tutorial I needed to use clips from two different videos. A big thank you and credit to [BuzzFeed Video](https://www.youtube.com/watch?v=9C1leq--_wM) and [GERrevolt](https://www.youtube.com/watch?v=3nbjhpcZ9_g).

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post we discussed dlib’s object tracking algorithm.

Unlike July’s tutorial on [centroid tracking](https://pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/), dlib’s object tracking algorithm can _update itself_ utilizing information garnered from the input RGB image — the algorithm _does not_ require that a set of bounding boxes be computed for each and every frame in the input video stream.

As we found out, dlib’s correlation tracking algorithm is quite robust and capable of running in real-time.

However, the biggest drawback is that the correlation tracker can become “confused” and lose the object we wish to track if viewpoint changes substantially or if the object to be tracked becomes occluded.

In those scenarios we can re-run our (computationally expensive) object detector to re-determine the location of our tracked object — be sure to refer to this blog post on people counting for such an implementation.

**In our next blog post we’ll be discussing multi-object tracking with dlib — to be notified when the next blog post goes live (and download the source code to today’s post), just _enter your email address in the form below._**

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!