Title: Tracking multiple objects with OpenCV - PyImageSearch

URL Source: https://pyimagesearch.com/2018/08/06/tracking-multiple-objects-with-opencv/

Published Time: 2018-08-06T14:00:16+00:00

Markdown Content:
![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

Inside today’s tutorial, you will learn how to track multiple objects using OpenCV and Python.

This post was inspired by a question I received from PyImageSearch reader, Ariel.

Ariel writes:

> Hi Adrian, thanks for [last week’s blog post on object tracking.](https://pyimagesearch.com/2018/07/30/opencv-object-tracking/)
> 
> I have a situation where I need to track multiple objects but the code last week didn’t seem to handle that use case. I’m a bit stumped on how I can modify the code to track multiple objects.
> 
> What should I do?

Great question, Ariel — as we’ll find out, tracking multiple objects is fairly similar to tracking _single objects_, with only one or two caveats that we need to keep in mind.

**To learn how to track multiple objects with OpenCV, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the remainder of this tutorial, you will utilize OpenCV and Python to track multiple objects in videos.

**I will be assuming you are using _OpenCV 3.2 (or greater)_ for this tutorial.**

If you are using OpenCV 3.1 or below you should use my [OpenCV install tutorials](https://pyimagesearch.com/opencv-tutorials-resources-guides/) to install an updated version.

From there, let’s get started implementing OpenCV’s multi-object tracker.

### Project structure

Today’s code + video files can be obtained via the _**“Downloads”**_ section of this blog post. Once you’ve downloaded the zip to your computer, you can use the following 3 commands to inspect the project structure:

$ unzip multi-object-tracking.zip
$ cd multi-object-tracking
$ tree
.
├── multi\_object\_tracking.py
└── videos
    ├── los\_angeles.mp4
    ├── nascar.mp4
    ├── soccer\_01.mp4
    └── soccer\_02.mp4

1 directory, 5 files

The output of `tree` shows our project structure.

*   We’ll be discussing one Python script — `multi_object_tracking.py` .
*   I’ve supplied 4 example videos for you to experiment with. Credits for these videos are given later in this blog post.

### Implementing the OpenCV multi-object tracker

Let’s get started coding our multi-object tracker.

Create a new file named `multi_object_tracking.py` and insert the following code:

\# import the necessary packages
from imutils.video import VideoStream
import argparse
import imutils
import time
import cv2

To begin, we import our required packages. You’ll need [OpenCV](https://pyimagesearch.com/opencv-tutorials-resources-guides/) and [imutils](https://github.com/jrosebr1/imutils) installed in your environment.

To install `imutils` , simply use pip:

$ pip install --upgrade imutils

From there we’ll parse command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-v", "--video", type=str,
	help="path to input video file")
ap.add\_argument("-t", "--tracker", type=str, default="kcf",
	help="OpenCV object tracker type")
args = vars(ap.parse\_args())

Our two command line arguments consist of:

*   `--video` : The path to our input video file.
*   `--tracker` : The OpenCV object tracker to use. There are 7 trackers listed in the next code block to choose from and by default `kcf` is used.

From there we’ll initialize our multi-object tracker:

\# initialize a dictionary that maps strings to their corresponding
# OpenCV object tracker implementations
OPENCV\_OBJECT\_TRACKERS = {
	"csrt": cv2.TrackerCSRT\_create,
	"kcf": cv2.TrackerKCF\_create,
	"boosting": cv2.TrackerBoosting\_create,
	"mil": cv2.TrackerMIL\_create,
	"tld": cv2.TrackerTLD\_create,
	"medianflow": cv2.TrackerMedianFlow\_create,
	"mosse": cv2.TrackerMOSSE\_create
}

# initialize OpenCV's special multi-object tracker
trackers = cv2.MultiTracker\_create()

Please refer to the previous post on **[_OpenCV Object Trackers_](https://pyimagesearch.com/2018/07/30/opencv-object-tracking/)** for the full explanation of the 7 available tracker algorithms defined on **Lines 18-26**.

I recommend the following three algorithms:

*   **KCF:** Fast and accurate
*   **CSRT:** More accurate than KCF but slower
*   **MOSSE:** Extremely fast but not as accurate as either KCF or CSRT

On **Line 29** we initialize the **multi-object tracker** via the `cv2.MultiTracker_create` method. The class allows us to:

1.  Add _new_ object trackers to the `MultiTracker`
2.  Update all object trackers _inside_ the `MultiTracker` with a _single function call_

Let’s move on to initializing our video stream:

\# if a video path was not supplied, grab the reference to the web cam
if not args.get("video", False):
	print("\[INFO\] starting video stream...")
	vs = VideoStream(src=0).start()
	time.sleep(1.0)

# otherwise, grab a reference to the video file
else:
	vs = cv2.VideoCapture(args\["video"\])

**Lines 32-35** handle creating a video stream object for a webcam. Otherwise, a video path must have been supplied, so we’ll create a video capture object which will read frames from a video file on disk (**Lines 38 and 39**).

It’s now time to loop over frames and start multi-object tracking!

\# loop over frames from the video stream
while True:
	# grab the current frame, then handle if we are using a
	# VideoStream or VideoCapture object
	frame = vs.read()
	frame = frame\[1\] if args.get("video", False) else frame

	# check to see if we have reached the end of the stream
	if frame is None:
		break

	# resize the frame (so we can process it faster)
	frame = imutils.resize(frame, width=600)

**Lines 45-50** handle grabbing a `frame` from the stream. We’ll break out of the loop if no more frames are available. Frames won’t be available at the end of a video file or if there is a problem with the webcam connection.

Then we `resize` the frame to a known dimension on **Line 53** — a smaller video frame yields faster FPS as there is less data to process.

From there, let’s `update` our `trackers` and draw bounding boxes around the objects:

	# grab the updated bounding box coordinates (if any) for each
	# object that is being tracked
	(success, boxes) = trackers.update(frame)

	# loop over the bounding boxes and draw then on the frame
	for box in boxes:
		(x, y, w, h) = \[int(v) for v in box\]
		cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)

Here we are updating all single object trackers inside `trackers` thereby obtaining multi-object tracking (**Line 57**).

For each tracked object there is an associated bounding box. The box is drawn on the frame via the `cv2.rectangle` drawing function on **Lines 60-62**.

Next, we’ll display the `frame` as well as select our objects to track:

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the 's' key is selected, we are going to "select" a bounding
	# box to track
	if key == ord("s"):
		# select the bounding box of the object we want to track (make
		# sure you press ENTER or SPACE after selecting the ROI)
		box = cv2.selectROI("Frame", frame, fromCenter=False,
			showCrosshair=True)

		# create a new object tracker for the bounding box and add it
		# to our multi-object tracker
		tracker = OPENCV\_OBJECT\_TRACKERS\[args\["tracker"\]\]()
		trackers.add(tracker, frame, box)

The `frame` is shown on the screen via **Line 65**.

A keypress is also captured on **Line 66**.

If the key pressed is an “s” for “select”, we’ll manually select bounding box of object to track using our mouse pointer via the `cv2.selectROI` function (**Lines 73 and 74**). If you’re unhappy with your selection you can press “ESCAPE” to reset the selection, otherwise hit “SPACE” or “ENTER” to begin the object tracking.

Once the selection has been made we add the `tracker` object to `trackers` (**Lines 78 and 79**).

_**Important:**_ _You’ll need to press “s” key and select each object we want to track individually._

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%200'%3E%3C/svg%3E)

This is of course just an example. If you were building a truly autonomous system, you would not select objects with your mouse. Instead, you would use an object detector (Haar Cascade, HOG + SVM, Faster R-CNN, MobileNet, YOLO, etc.). **I’ll be demonstrating how to do this process starting next week, so stay tuned!**

Let’s handle when the “q” key (“quit”) has been pressed (or if we’ve broken out of the loop due to reaching the end of our video file):

	# if the \`q\` key was pressed, break from the loop
	elif key == ord("q"):
		break

# if we are using a webcam, release the pointer
if not args.get("video", False):
	vs.stop()

# otherwise, release the file pointer
else:
	vs.release()

# close all windows
cv2.destroyAllWindows()

When the “q” key is pressed, we break out of the loop and perform cleanup. Cleanup involves releasing pointers and closing GUI windows.

### Multi-object tracking results

Head over the _**“Downloads”**_ section at the bottom of this post to grab the source code + video files.

From there, open up a terminal and execute the following command:

$ python multi\_object\_tracking.py --tracker csrt

Your webcam will be used by default since the `--video` switch was not listed. You may also leave off the `--tracker` argument and `kcf` will be used.

**Would you like to use one of the four supplied video files or a video file of your own?**

No problem. Just supply the `--video` [command line argument](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/) along with a path to a video file. Provided OpenCV can decode the video file, you can begin tracking multiple objects:

$ python multi\_object\_tracking.py --video videos/soccer\_01.mp4 --tracker csrt

You may also supply your desired tracking algorithm via the `--tracker`  command line argument (as shown).

Notice how we are able to:

*   Track multiple soccer players across the pitch
*   Track multiple race cars in a race
*   And track multiple vehicles as they are driving in a freeway

Be sure to give the code a try when you need to track multiple objects with OpenCV!

### Problems and limitations

There are two limitations that we can run into when performing multiple object tracking with OpenCV.

As my results from the previous section demonstrated, the first (and biggest) issue we ran into was that _**the more trackers we created, the slower our pipeline ran.**_

Keep in mind that we need to instantiate a _brand new_ OpenCV object tracker for each object we want to track — **we cannot use the same object tracker instance to track multiple objects.**

For example, suppose we have 10 objects in a video that we would like to track, implying that:

1.  We need to create 10 object tracker instances
2.  And therefore, we’ll see the frames per second throughput of our pipeline decrease by a factor of 10.

Object trackers are not “magic” and computationally “free” to apply — **each and every one of these object trackers requires computation to track an object, there is no way to get around that.**

_…or is there?_

If you open up your activity monitor on your system you’ll notice that when running today’s script that only _one_ processor core is being utilized.

Is there a way to distribute each of the object trackers to a separate process, thereby allowing us to utilize all cores of the processor for faster object tracking?

**The answer is yes, we can — and I’ll show you how to obtain faster multi-object tracking later in this series.**

### Video and Audio credits

To create the examples for this tutorial I needed to use clips from a number of different videos. A big thank you and credit to [Ben Sound](https://www.bensound.com/royalty-free-music/track/actionable), [Dash Cam Tours](https://www.youtube.com/watch?v=6nvEQ3WoMak), [NASCAR](https://www.youtube.com/watch?v=p6QmmFn48IE), and [FIFATV](https://www.youtube.com/watch?v=a7IMPsyQg6k).

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s tutorial, we learned how to perform multiple object tracking using OpenCV and Python.

To accomplish our multi-object tracking task, we leveraged OpenCV’s `cv2.MultiTracker_Create`  function.

This method allows us to instantiate _single_ object trackers ([just like we did in last week’s blog post](https://pyimagesearch.com/2018/07/30/opencv-object-tracking/)) and then add them to a class that updates the locations of objects for us.

Keep in mind though, the `cv2.MultiTracker`  class is a _convenience_ function — while it allows us to easily add and update objects locations, it’s not necessarily going to _improve_ (or even _maintain_) the speed of our object tracking pipeline.

To obtain _faster, more efficient_ object tracking we’ll need to leverage multiple processes and spread the computation burden across multiple cores of our processor — I’ll be showing you how to accomplish this task in a future post in this series.

**To download the source code to this post, and be notified when the next object tracking tutorial is published, _be sure to enter your email address in the form below!_**

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!