Title: Multi-object tracking with dlib - PyImageSearch

URL Source: https://pyimagesearch.com/2018/10/29/multi-object-tracking-with-dlib/

Published Time: 2018-10-29T14:00:10+00:00

Markdown Content:
![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

In this tutorial, you will learn how to use the dlib library to efficiently track multiple objects in real-time video.

So far in this series on object tracking we have learned how to:

1.  [Track _single_ objects with OpenCV](https://pyimagesearch.com/2018/07/30/opencv-object-tracking/)
2.  [Track _multiple_ objects utilizing OpenCV](https://pyimagesearch.com/2018/08/06/tracking-multiple-objects-with-opencv/)
3.  [Perform single object tracking with dlib](https://pyimagesearch.com/2018/10/22/object-tracking-with-dlib/)
4.  [Track and count people entering a business/store](https://pyimagesearch.com/2018/08/13/opencv-people-counter/)

We can of course track _multiple_ objects with dlib; however, to obtain the best performance possible, we need to utilize _multiprocessing_ and distribute the object trackers _across multiple cores of our processor._

Correctly utilizing multiprocessing allows us to improve our dlib multi-object tracking frames per second (FPS) throughput rate _by over 45%!_

**To learn how to track multiple objects using dlib, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this guide, I’ll demonstrate how to can implement a simple, naïve dlib multi-object tracking script. This program will track multiple objects in video; however, we’ll notice that the script runs a bit slow.

To increase our FPS throughput rate I’ll show you a _faster, more efficient_ dlib multi-object tracker implementation.

Finally, I’ll discuss some improvements and suggestions you can make to enhance our multi-object tracking implementations as well.

### Project structure

To get started, make sure you use the _**“Downloads”**_ section of this tutorial to download the source code and example video.

From there, you can use the `tree` command to view our project structure:

$ tree
.
├── mobilenet\_ssd
│   ├── MobileNetSSD\_deploy.caffemodel
│   └── MobileNetSSD\_deploy.prototxt
├── multi\_object\_tracking\_slow.py
├── multi\_object\_tracking\_fast.py
├── race.mp4
├── race\_output\_slow.avi
└── race\_output\_fast.avi

1 directory, 7 files

The `mobilenet_ssd/` directory contains our MobileNet + SSD Caffe model files which allow us to detect people (along with other objects).

We’ll review two Python scripts today:

1.  `multi_object_tracking_slow.py` : The simple “naïve” method of dlib multiple object tracking.
2.  `multi_object_tracking_fast.py` : The advanced, fast, method which takes advantage of multiprocessing.

The remaining three files are videos. We have the original `race.mp4` video and two processed output videos.

### The “naïve” dlib multiple object tracking implementation

The first dlib multi-object tracking implementation we are going to cover today is “naïve” in the sense that it will:

1.  Utilize a simple list of tracker objects.
2.  Update each of the trackers sequentially, using only a single core of our processor.

For some object tracking tasks this implementation will be _more than sufficient_; however, to optimize our FPS throughput rate, we should distribute the object trackers across multiple processes.

We’ll start with our simple implementation in this section and then move on to the faster method in the next section.

To get started, open up the `multi_object_tracking_slow.py` script and insert the following code:

\# import the necessary packages
from imutils.video import FPS
import numpy as np
import argparse
import imutils
import dlib
import cv2

We begin by importing necessary packages and modules on **Lines 2-7**. Most importantly we’ll be using dlib and OpenCV. We’ll also use some features from my [imutils](https://github.com/jrosebr1/imutils) package of convenience functions such as the frames per second counter.

To install dlib, [follow this guide](https://pyimagesearch.com/2018/01/22/install-dlib-easy-complete-guide/). I have a number of [OpenCV installation tutorials](https://pyimagesearch.com/opencv-tutorials-resources-guides/) available as well (even for the latest OpenCV 4!). You might even try the fastest way to [install OpenCV on your system via pip](https://pyimagesearch.com/2018/09/19/pip-install-opencv/).

To install `imutils` , simply use pip in your terminal:

$ pip install --upgrade imutils

Now that we (a) have the software installed, and (b) have placed the relevant import statements in our script, let’s parse our command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-p", "--prototxt", required=True,
	help="path to Caffe 'deploy' prototxt file")
ap.add\_argument("-m", "--model", required=True,
	help="path to Caffe pre-trained model")
ap.add\_argument("-v", "--video", required=True,
	help="path to input video file")
ap.add\_argument("-o", "--output", type=str,
	help="path to optional output video file")
ap.add\_argument("-c", "--confidence", type=float, default=0.2,
	help="minimum probability to filter weak detections")
args = vars(ap.parse\_args())

If you aren’t familiar with the terminal and command line arguments, please [give this post a read](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/).

Our script processes the following command line arguments at runtime:

*   `--prototxt` : The path to the Caffe “deploy” prototxt file.
*   `--model` : The path to the model file which accompanies the prototxt.
*   `--video` : The path to the input video file. We’ll perform multi-object tracking with dlib on this video.
*   `--output` : An optional path to an output video file. If no path is specified then no video will be output to disk. I recommend outputting to an .avi or .mp4 file.
*   `--confidence` : An optional override for the object detection confidence threshold of `0.2` . This value represents the minimum probability to filter weak detections from the object detector.

Let’s define our list of `CLASSES` that this model supports as well as load our model from disk:

\# initialize the list of class labels MobileNet SSD was trained to
# detect
CLASSES = \["background", "aeroplane", "bicycle", "bird", "boat",
	"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
	"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
	"sofa", "train", "tvmonitor"\]

# load our serialized model from disk
print("\[INFO\] loading model...")
net = cv2.dnn.readNetFromCaffe(args\["prototxt"\], args\["model"\])

The MobileNet SSD pre-trained Caffe model supports 20 classes and 1 background class. The `CLASSES` are defined on **Lines 25-28** in list form.

_**Note:**_ _Do not modify this list or the ordering of class objects if you’re using the Caffe model provided in the **“Downloads”**. Similarly, if you happen to be loading a different model, you’ll need to define the classes that the model supports here (order does matter). If you’re curious how our object detector works, [be sure to refer to this post.](https://pyimagesearch.com/2018/05/14/a-gentle-guide-to-deep-learning-object-detection/)_

We’re only concerned about the `"person"` class for today’s foot race example, but you could easily modify **Line 95** (covered later in this post) below to track alternate class(es).

On **Line 32**, we load our pre-trained object detector model. We will use our pre-trained SSD to detect the presence of objects in a video. From there we will create a dlib object tracker to track each of the detected objects.

We have a few more initializations to perform:

\# initialize the video stream and output video writer
print("\[INFO\] starting video stream...")
vs = cv2.VideoCapture(args\["video"\])
writer = None

# initialize the list of object trackers and corresponding class
# labels
trackers = \[\]
labels = \[\]

# start the frames per second throughput estimator
fps = FPS().start()

On **Line 36**, we initialize our video stream — we’ll be reading frames from our input video one at a time.

Subsequently, on **Line 37** our video `writer` is initialized to `None` . We’ll work more with the video `writer` in the upcoming `while` loop.

Now let’s initialize our `trackers` and `labels` lists on **Lines 41 and 42**.

And finally, we start our frames per second counter on **Line 45**.

We’re all set to begin processing our video:

\# loop over frames from the video file stream
while True:
	# grab the next frame from the video file
	(grabbed, frame) = vs.read()

	# check to see if we have reached the end of the video file
	if frame is None:
		break

	# resize the frame for faster processing and then convert the
	# frame from BGR to RGB ordering (dlib needs RGB ordering)
	frame = imutils.resize(frame, width=600)
	rgb = cv2.cvtColor(frame, cv2.COLOR\_BGR2RGB)

	# if we are supposed to be writing a video to disk, initialize
	# the writer
	if args\["output"\] is not None and writer is None:
		fourcc = cv2.VideoWriter\_fourcc(\*"MJPG")
		writer = cv2.VideoWriter(args\["output"\], fourcc, 30,
			(frame.shape\[1\], frame.shape\[0\]), True)

On **Line 48** we begin looping over frames, where **Line 50** actually grabs the `frame` itself.

A quick check to see if we’ve reached the end of the video file and need to stop looping is made on **Lines 53 and 54**.

Preprocessing takes place on **Lines 58 and 59**. First, the `frame` is resized to `600` pixels wide, maintaining aspect ratio. Then, the `frame` is converted to the `rgb` color channel ordering for dlib compatibility (OpenCV’s default is BGR and dlib’s default is RGB).

From there we instantiate the video `writer` (if necessary) on **Lines 63-66**. To learn more about writing video to disk with OpenCV, check out my [previous blog post](https://pyimagesearch.com/2016/02/22/writing-to-video-with-opencv/).

Let’s begin the **object _detection_ phase:**

	# if there are no object trackers we first need to detect objects
	# and then create a tracker for each object
	if len(trackers) == 0:
		# grab the frame dimensions and convert the frame to a blob
		(h, w) = frame.shape\[:2\]
		blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)

		# pass the blob through the network and obtain the detections
		# and predictions
		net.setInput(blob)
		detections = net.forward()

In order to perform object tracking we must first perform object detection, either:

1.  **Manually**, by stopping the video stream and hand-selecting the bounding box(es) of each object.
2.  **Programmatically**, using an object detector trained to detect the presence of an object (which is what we are doing here).

If there are no object trackers (**Line 70**), then we know we have yet to perform object detection.

We create and pass a `blob` through the SSD network to detect objects on **Lines 72-78**. To learn about the `cv2.blobFromImage` function, be sure to [refer to my writeup in this article](https://pyimagesearch.com/2017/11/06/deep-learning-opencvs-blobfromimage-works/).

Next, we proceed to loop over the detections to find objects belonging to the `"person"` class since our input video is a human foot race:

		# loop over the detections
		for i in np.arange(0, detections.shape\[2\]):
			# extract the confidence (i.e., probability) associated
			# with the prediction
			confidence = detections\[0, 0, i, 2\]

			# filter out weak detections by requiring a minimum
			# confidence
			if confidence > args\["confidence"\]:
				# extract the index of the class label from the
				# detections list
				idx = int(detections\[0, 0, i, 1\])
				label = CLASSES\[idx\]

				# if the class label is not a person, ignore it
				if CLASSES\[idx\] != "person":
					continue

We begin looping over detections on **Line 81** where we:

1.  Filter out weak detections (**Line 88**).
2.  Ensure each detection is a `"person"` (**Lines 91-96**). You can, of course, remove this line of code or customize it to your own filtering needs.

Now that we’ve located each `"person"` in the frame, let’s instantiate our trackers and draw our initial bounding box(es) + class label(s):

				# compute the (x, y)-coordinates of the bounding box
				# for the object
				box = detections\[0, 0, i, 3:7\] \* np.array(\[w, h, w, h\])
				(startX, startY, endX, endY) = box.astype("int")

				# construct a dlib rectangle object from the bounding
				# box coordinates and start the correlation tracker
				t = dlib.correlation\_tracker()
				rect = dlib.rectangle(startX, startY, endX, endY)
				t.start\_track(rgb, rect)

				# update our set of trackers and corresponding class
				# labels
				labels.append(label)
				trackers.append(t)

				# grab the corresponding class label for the detection
				# and draw the bounding box
				cv2.rectangle(frame, (startX, startY), (endX, endY),
					(0, 255, 0), 2)
				cv2.putText(frame, label, (startX, startY - 15),
					cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

To begin tracking objects we:

*   Compute the bounding `box` of each _detected_ object (**Lines 100 and 101**).
*   Instantiate and pass the bounding box coordinates to the _tracker_ (**Lines 105-107**). The bounding box is especially important here. We need to create a `dlib.rectangle` for the bounding box and pass it to the `start_track` method. From there, dlib can start to track the object.
*   Finally, we populate the `trackers` list with the individual tracker (**Line 112**).

As a result, in the next code block, we’ll handle the case where trackers have already been established and we just need to update positions.

There are two additional tasks we perform in the initial detection step:

*   Append the class label to the `labels` list (**Line 111**). In the event that you’re tracking multiple types of objects (such as `"dog"` + `"person"` ), you may wish to know what the type of each object is.
*   Draw each bounding box `rectangle` around and class `label` above the object (**Lines 116-119**).

If the length of our detections list is greater than zero, we know we are in the **object _tracking_ phase:**

	# otherwise, we've already performed detection so let's track
	# multiple objects
	else:
		# loop over each of the trackers
		for (t, l) in zip(trackers, labels):
			# update the tracker and grab the position of the tracked
			# object
			t.update(rgb)
			pos = t.get\_position()

			# unpack the position object
			startX = int(pos.left())
			startY = int(pos.top())
			endX = int(pos.right())
			endY = int(pos.bottom())

			# draw the bounding box from the correlation object tracker
			cv2.rectangle(frame, (startX, startY), (endX, endY),
				(0, 255, 0), 2)
			cv2.putText(frame, l, (startX, startY - 15),
				cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

In the object _tracking_ phase, we loop over all `trackers` and corresponding `labels` on **Line 125**.

Then we proceed to `update` each object position (**Lines 128-129**). In order to update the position, we simply pass the `rgb` image.

After extracting bounding box coordinates, we can draw a bounding box `rectangle` and `label` for each tracked object (**Lines 138-141**).

The remaining steps in the frame processing loop involve writing to the output video (if necessary) and displaying the results:

	# check to see if we should write the frame to disk
	if writer is not None:
		writer.write(frame)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the \`q\` key was pressed, break from the loop
	if key == ord("q"):
		break

	# update the FPS counter
	fps.update()

Here we:

*   Write the `frame` to video if necessary (**Lines 144 and 145**).
*   Show the output frame and capture keypresses (**Lines 148 and 149**). If the `"q"` key is pressed (“quit”), we `break` out of the loop.
*   Finally, we update our frames per second information for benchmarking purposes (**Line 156**).

The remaining steps are to print FPS throughput information in the terminal and release pointers:

\# stop the timer and display FPS information
fps.stop()
print("\[INFO\] elapsed time: {:.2f}".format(fps.elapsed()))
print("\[INFO\] approx. FPS: {:.2f}".format(fps.fps()))

# check to see if we need to release the video writer pointer
if writer is not None:
	writer.release()

# do a bit of cleanup
cv2.destroyAllWindows()
vs.release()

To close out, our `fps` stats are collected and printed (**Lines 159-161**), the video `writer` is released (**Lines 164 and 165**), and we close all windows + release the video stream.

Let’s assess accuracy and performance.

To follow along and run this script, make sure you use the _**“Downloads”**_ section of this blog post to download the source code + example video.

From there, open up a terminal and execute the following command:

$ python multi\_object\_tracking\_slow.py --prototxt mobilenet\_ssd/MobileNetSSD\_deploy.prototxt \\
	--model mobilenet\_ssd/MobileNetSSD\_deploy.caffemodel \\
	--video race.mp4 --output race\_output\_slow.avi
\[INFO\] loading model...
\[INFO\] starting video stream...
\[INFO\] elapsed time: 24.51
\[INFO\] approx. FPS: 13.87

It appears that our multi-object tracker is working!

But as you can see, we are only obtaining **~13 FPS**.

For some applications, this FPS throughput rate may be sufficient — however, if you need _faster_ FPS, I would suggest taking a look at our _more efficient_ dlib multi-object tracker below.

Secondly, understand that tracking accuracy isn’t perfect. Refer to the third suggestion in the _“Improvements and Suggestions”_ section below as well as [read my first post on dlib object tracking](https://pyimagesearch.com/2018/10/22/object-tracking-with-dlib/) for more information.

### The _fast, efficient_ dlib multi-object tracking implementation

If you run the dlib multi-object tracking script from the previous section and open up your system’s activity monitor at the same time, you’ll notice that only _one_ core of your processor is being utilized.

In order to speed up our object tracking pipeline we can leverage [Python’s multiprocessing module](https://docs.python.org/3.4/library/multiprocessing.html), similar to the [threading module](https://docs.python.org/3.4/library/threading.html#module-threading), but instead used to spawn _processes_ rather than _threads_.

Utilizing processes enables our operating system to perform better process scheduling, mapping the process to a particular processor core on our machine (most modern operating systems are able to efficiently schedule processes that are using a lot of CPU in a parallel manner).

If you are new to Python’s multiprocessing module I would suggest you read [this excellent introduction](https://sebastianraschka.com/Articles/2014_multiprocessing.html) from Sebastian Raschka.

Otherwise, go ahead and open up `mutli_object_tracking_fast.py` and insert the following code:

\# import the necessary packages
from imutils.video import FPS
import multiprocessing
import numpy as np
import argparse
import imutils
import dlib
import cv2

Our packages are imported on **Lines 2-8**. We’re importing the `multiprocessing` library on **Line 3**.

We’ll be using the Python `Process` class to spawn a new process — each new process is _independent_ from the original process.

To spawn this process we need to provide a function that Python can call, which Python will then take and create a brand new process + execute it:

def start\_tracker(box, label, rgb, inputQueue, outputQueue):
	# construct a dlib rectangle object from the bounding box
	# coordinates and then start the correlation tracker
	t = dlib.correlation\_tracker()
	rect = dlib.rectangle(box\[0\], box\[1\], box\[2\], box\[3\])
	t.start\_track(rgb, rect)

The first three parameters to `start_tracker` include:

*   `box` : Bounding box coordinates of the object we are going to track, presumably returned by some sort of object detector, whether manual or programmatic.
*   `label` : Human-readable label of the object.
*   `rgb` : An RGB-ordered image that we’ll be using to start the initial dlib object tracker.

Keep in mind how Python multiprocessing works — Python will call this function and then create a brand new interpreter to execute the code within. Therefore, each `start_tracker` spawned process will be independent from its parent. To communicate with the Python driver script we need to leverage either [Pipes or Queues](https://docs.python.org/3.4/library/multiprocessing.html#pipes-and-queues). Both types of objects are thread/process safe, accomplished using locks and semaphores.

In essence, we are creating a simple producer/consumer relationship:

1.  Our parent process will **produce** new frames and add them to the queue of a particular object tracker.
2.  The child process will then **consume** the frame, apply object tracking, and then return the updated bounding box coordinates.

I decided to use `Queue` objects for this post; however, keep in mind that you could use a `Pipe` if you wish — be sure to refer to the [Python multiprocessing documentation](https://docs.python.org/3.4/library/multiprocessing.html) for more details on these objects.

Now let’s begin an infinite loop which will run in the process:

	# loop indefinitely -- this function will be called as a daemon
	# process so we don't need to worry about joining it
	while True:
		# attempt to grab the next frame from the input queue
		rgb = inputQueue.get()

		# if there was an entry in our queue, process it
		if rgb is not None:
			# update the tracker and grab the position of the tracked
			# object
			t.update(rgb)
			pos = t.get\_position()

			# unpack the position object
			startX = int(pos.left())
			startY = int(pos.top())
			endX = int(pos.right())
			endY = int(pos.bottom())

			# add the label + bounding box coordinates to the output
			# queue
			outputQueue.put((label, (startX, startY, endX, endY)))

We loop indefinitely here — this function will be called as a daemon process, so we don’t need to worry about joining it.

First, we’ll attempt to grab a new frame from the `inputQueue` on **Line 21**.

If the frame is not empty, we’ll grab the frame and then `update` the object tracker, allowing us to obtain the updated bounding box coordinates (**Lines 24-34**).

Finally, we write the `label` and bounding box to the `outputQueue` so the parent process can utilize them in the main loop of our script (**Line 38**).

Back to the parent process, we’ll parse our command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-p", "--prototxt", required=True,
	help="path to Caffe 'deploy' prototxt file")
ap.add\_argument("-m", "--model", required=True,
	help="path to Caffe pre-trained model")
ap.add\_argument("-v", "--video", required=True,
	help="path to input video file")
ap.add\_argument("-o", "--output", type=str,
	help="path to optional output video file")
ap.add\_argument("-c", "--confidence", type=float, default=0.2,
	help="minimum probability to filter weak detections")
args = vars(ap.parse\_args())

The command line arguments for this script are exactly the same as our slower, non-multiprocessing script. If you need a refresher on the arguments, just [click here](#clargs). And furthermore, read my post about [argparse and command line arguments](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/) if you aren’t familiar with them.

Let’s initialize our input and output queues:

\# initialize our lists of queues -- both input queue and output queue
# for \*every\* object that we will be tracking
inputQueues = \[\]
outputQueues = \[\]

These queues will hold the objects we are tracking. Each process spawned will need two `Queue` objects:

1.  One to read input frames from
2.  And a second to write results to

This next block is identical to our previous script:

\# initialize the list of class labels MobileNet SSD was trained to
# detect
CLASSES = \["background", "aeroplane", "bicycle", "bird", "boat",
	"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
	"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
	"sofa", "train", "tvmonitor"\]

# load our serialized model from disk
print("\[INFO\] loading model...")
net = cv2.dnn.readNetFromCaffe(args\["prototxt"\], args\["model"\])

# initialize the video stream and output video writer
print("\[INFO\] starting video stream...")
vs = cv2.VideoCapture(args\["video"\])
writer = None

# start the frames per second throughput estimator
fps = FPS().start()

We define our model’s `CLASSES` and load the model itself (**Lines 61-68**). Remember, these `CLASSES` are static — our MobileNet SSD supports these classes and only these classes. If you want to detect + track other objects you’ll need to find another pretrained model or train one. Furthermore, _the order of this list matters!_ Do not change the ordering of the list unless you enjoy being confused! I would also recommend [reading this tutorial](https://pyimagesearch.com/2018/05/14/a-gentle-guide-to-deep-learning-object-detection/) if you want to further understand how object detectors work.

We initialize our video stream object and set our video `writer` object to `None` (**Lines 72 and 73**).

Our frames per second calculator is instantiated and started on **Line 76**.

Now let’s begin looping over frames from the video stream:

\# loop over frames from the video file stream
while True:
	# grab the next frame from the video file
	(grabbed, frame) = vs.read()

	# check to see if we have reached the end of the video file
	if frame is None:
		break

	# resize the frame for faster processing and then convert the
	# frame from BGR to RGB ordering (dlib needs RGB ordering)
	frame = imutils.resize(frame, width=600)
	rgb = cv2.cvtColor(frame, cv2.COLOR\_BGR2RGB)

	# if we are supposed to be writing a video to disk, initialize
	# the writer
	if args\["output"\] is not None and writer is None:
		fourcc = cv2.VideoWriter\_fourcc(\*"MJPG")
		writer = cv2.VideoWriter(args\["output"\], fourcc, 30,
			(frame.shape\[1\], frame.shape\[0\]), True)

The above code block is, yet again, identical to the one in the previous script. Be sure to [refer above](#while) as needed.

Now let’s handle the case where we have no `inputQueues` :

	# if our list of queues is empty then we know we have yet to
	# create our first object tracker
	if len(inputQueues) == 0:
		# grab the frame dimensions and convert the frame to a blob
		(h, w) = frame.shape\[:2\]
		blob = cv2.dnn.blobFromImage(frame, 0.007843, (w, h), 127.5)

		# pass the blob through the network and obtain the detections
		# and predictions
		net.setInput(blob)
		detections = net.forward()

		# loop over the detections
		for i in np.arange(0, detections.shape\[2\]):
			# extract the confidence (i.e., probability) associated
			# with the prediction
			confidence = detections\[0, 0, i, 2\]

			# filter out weak detections by requiring a minimum
			# confidence
			if confidence > args\["confidence"\]:
				# extract the index of the class label from the
				# detections list
				idx = int(detections\[0, 0, i, 1\])
				label = CLASSES\[idx\]

				# if the class label is not a person, ignore it
				if CLASSES\[idx\] != "person":
					continue

If there are no `inputQueues` (**Line 101**) then we know we need to apply object _detection_ prior to object _tracking_.

We apply object _detection_ on **Lines 103-109** and then proceed to loop over the results on **Line 112**. We grab our `confidence` values and filter out weak `detections` on **Lines 115-119**.

If our `confidence` meets the threshold established by our command line arguments, we consider the detection, but we further filter it out by class `label` . In this case, we’re only looking for `"person"` objects (**Lines 122-127**).

Assuming we have found a `"person"` , we’ll create queues and spawn tracking processes:

				# compute the (x, y)-coordinates of the bounding box
				# for the object
				box = detections\[0, 0, i, 3:7\] \* np.array(\[w, h, w, h\])
				(startX, startY, endX, endY) = box.astype("int")
				bb = (startX, startY, endX, endY)

				# create two brand new input and output queues,
				# respectively
				iq = multiprocessing.Queue()
				oq = multiprocessing.Queue()
				inputQueues.append(iq)
				outputQueues.append(oq)

				# spawn a daemon process for a new object tracker
				p = multiprocessing.Process(
					target=start\_tracker,
					args=(bb, label, rgb, iq, oq))
				p.daemon = True
				p.start()

				# grab the corresponding class label for the detection
				# and draw the bounding box
				cv2.rectangle(frame, (startX, startY), (endX, endY),
					(0, 255, 0), 2)
				cv2.putText(frame, label, (startX, startY - 15),
					cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

We first compute the bounding box coordinates on **Lines 131-133**.

From there we create two new queues, `iq` and `oq` (**Lines 137 and 138**), appending them to `inputQueues` and `outputQueues` respectively (**Lines 139 and 140**).

From there we spawn a new `start_tracker` process, passing the bounding box, `label` , `rgb` image, and the `iq` + `oq` (**Lines 143-147**). Don’t forget to [read more about multiprocessing here](https://sebastianraschka.com/Articles/2014_multiprocessing.html).

We also draw the detected object’s bounding box `rectangle` and class `label` (**Lines 151-154**).

Otherwise, we’ve already performed object detection so we need to apply each of the dlib object trackers to the frame:

	# otherwise, we've already performed detection so let's track
	# multiple objects
	else:
		# loop over each of our input ques and add the input RGB
		# frame to it, enabling us to update each of the respective
		# object trackers running in separate processes
		for iq in inputQueues:
			iq.put(rgb)

		# loop over each of the output queues
		for oq in outputQueues:
			# grab the updated bounding box coordinates for the
			# object -- the .get method is a blocking operation so
			# this will pause our execution until the respective
			# process finishes the tracking update
			(label, (startX, startY, endX, endY)) = oq.get()

			# draw the bounding box from the correlation object
			# tracker
			cv2.rectangle(frame, (startX, startY), (endX, endY),
				(0, 255, 0), 2)
			cv2.putText(frame, label, (startX, startY - 15),
				cv2.FONT\_HERSHEY\_SIMPLEX, 0.45, (0, 255, 0), 2)

Looping over each of the `inputQueues` , we add the `rgb` image to them (**Lines 162 and 163**).

Then we loop over each of the `outputQueues` (**Line 166**), obtaining the bounding box coordinates from each independent object tracker (**Line 171**). Finally, we draw the bounding box + associated class `label` on **Lines 175-178**.

Let’s finish out the loop and script:

	# check to see if we should write the frame to disk
	if writer is not None:
		writer.write(frame)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the \`q\` key was pressed, break from the loop
	if key == ord("q"):
		break

	# update the FPS counter
	fps.update()

# stop the timer and display FPS information
fps.stop()
print("\[INFO\] elapsed time: {:.2f}".format(fps.elapsed()))
print("\[INFO\] approx. FPS: {:.2f}".format(fps.fps()))

# check to see if we need to release the video writer pointer
if writer is not None:
	writer.release()

# do a bit of cleanup
cv2.destroyAllWindows()
vs.release()

We write the frame to the output video if necessary as well as show the `frame` to the screen (**Lines 181-185**).

If the `"q"` key is pressed, we “quit”, breaking out of the loop (**Lines 186-190**).

If we do continue processing frames, our `fps` calculator is updated on **Line 193**, and then we start the process at the beginning of the `while` loop again.

Otherwise, we’re done processing frames, and we display the FPS throughput info + release pointers and close windows.

To execute this script, make sure you use the _**“Downloads”**_ section of the post to download the source code + example video.

From there, open up a terminal and execute the following command:

$ python multi\_object\_tracking\_fast.py --prototxt mobilenet\_ssd/MobileNetSSD\_deploy.prototxt \\
	--model mobilenet\_ssd/MobileNetSSD\_deploy.caffemodel \\
	--video race.mp4 --output race\_output\_fast.avi
\[INFO\] loading model...
\[INFO\] starting video stream...
\[INFO\] elapsed time: 14.01
\[INFO\] approx. FPS: 24.26

As you can see, our faster, more efficient multi-object tracker is running at **24 FPS, _an improvement by over 45% from our previous implementation ?!_**

Furthermore, if you open up your activity monitor while this script is running you will see that more of your system’s overall CPU Is being utilized.

This speedup is obtained by allowing each of the dlib object trackers to run in a separate process which in turn enables your operating system to perform more efficient scheduling of the CPU resources.

### Improvements and suggestions

The dlib multi-object tracking Python scripts I’ve shared with you today will work just fine for processing _shorter video streams_; however, if you intend on utilizing this implementation for _long-running production environments_ (in the order of many hours to days of video) there are two primary improvements I would suggest you make:

The **first improvement** would be to **utilize processing pools** rather than spawning a brand new process for each object to be tracked.

The implementation covered here today constructs a brand new `Queue`  and `Process`  for each object that we need to track.

For today’s purposes that’s fine, but consider if you wanted to track 50 objects in a video — this implies that you would spawn 50 processes, one for each object. At that point, the overhead of your system managing all those processes will destroy any increase in FPS throughput. Instead, you would want to utilize processing pools.

If your system has _N_ processor cores, then you would want to create a pool with _N – 1_ processes, leaving one core to your operating system to perform system operations. Each of these processes should perform multiple object tracking, maintaining a list of object trackers, similar to the first multi-object tracking we covered today.

This improvement will allow you to utilize all cores of your processor without the overhead of having to spawn many independent processes.

The **second improvement** I would make is to **clean up the processes and queues**.

In the event that dlib reports an object as “lost” or “disappeared” we are not returning from the `start_tracker`  function, implying that that process will live for the life of the parent script and only be killed when the parent exits.

Again, that’s fine for our purposes here today, but if you intend on utilizing this code in production environments, you should:

1.  Update the `start_tracker`  function to return once dlib reports the object as lost.
2.  Delete the `inputQueue`  and `outputQueue`  for the corresponding process as well.

Failing to perform this cleanup will lead to needless computational consumption and memory overhead for long-running jobs.

The **third improvement** is to **improve tracking accuracy by running the object detector every _N_** **frames** (rather than just once at the start).

I actually demonstrated this in my previous post on [people counting with OpenCV](https://pyimagesearch.com/2018/08/13/opencv-people-counter/). It requires more logic and thought, but yields a much more accurate tracker.

I elected to forego the implementation for this script so that I could teach you the multiprocessing method concisely.

Ideally, you would use this third improvement in addition to multiprocessing.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this tutorial, we learned how to utilize the dlib library to perform multi-object tracking.

We also learned how to leverage multiprocessing to:

1.  Distribute the actual object tracker instantiations to multiple cores of our processor,
2.  Thereby leading to an **increase in FPS throughput rate by over 45%.**

I would encourage you to utilize the multiprocessing implementation of our dlib multi-object tracker for your own applications as it’s faster and more efficient; however, you should refer to the _“Improvements and suggestions”_ section of this tutorial where I discuss how you can further enhance the multi-object tracking implementation.

**If you enjoyed this series on object tracking, be sure to _enter your email in the form below_ to download today’s source code + videos as well as to be notified of future tutorials here on PyImageSearch.**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!