Title: Real-time panorama and image stitching with OpenCV - PyImageSearch

URL Source: https://pyimagesearch.com/2016/01/25/real-time-panorama-and-image-stitching-with-opencv/

Published Time: 2016-01-25T15:00:08+00:00

Markdown Content:
One of my favorite parts of running the PyImageSearch blog is a being able to _link together previous blog posts_ and _create a solution to a particular problem_ — _**in this case, real-time panorama and image stitching with Python and OpenCV.**_

[![Image 1: realtime_panorama_stitching_animation](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20313'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2016/01/realtime_panorama_stitching_animation.gif)

Over the past month and a half, we’ve learned how to [increase the FPS processing rate of builtin/USB webcams](https://pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/) and the [Raspberry Pi camera module](https://pyimagesearch.com/2015/12/28/increasing-raspberry-pi-fps-with-python-and-opencv/). We also learned how to [unify access to _**both**_ USB webcams and the Raspberry Pi camera into a _single class_](https://pyimagesearch.com/2016/01/04/unifying-picamera-and-cv2-videocapture-into-a-single-class-with-opencv/), making all video processing and examples on the PyImageSearch blog capable of running on both USB and Pi camera setups without having to modify a single line of code.

And just to weeks ago, we discussed how keypoint detection, local invariant descriptors, keypoint matching, and homography matrix estimation can be used to [construct panoramas and stitch images together](https://pyimagesearch.com/2016/01/11/opencv-panorama-stitching/).

Today we are going to link together the past 1.5 months worth of posts and use them to perform **[real-time panorama and image stitching](https://pyimagesearch.com/2016/01/11/opencv-panorama-stitching/)** using Python and OpenCV. Our solution will be able to run on both laptop/desktops systems, along with the Raspberry Pi.

**Furthermore, we’ll also apply our _basic motion detection_ implementation from [last week’s post](https://pyimagesearch.com/2016/01/18/multiple-cameras-with-the-raspberry-pi-and-opencv/) to perform motion detection on the panorama image.**

This solution is especially useful in situations where you want to survey a wide area for motion, but don’t want “blind spots” in your camera view.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

Keep reading to learn more…

As I mentioned in the introduction to this post, we’ll be linking together concepts we have learned in the previous 1.5 months of PyImageSearch posts and:

1.  Use our improved FPS processing rate Python classes to access our builtin/USB webcams and/or the Raspberry Pi camera module.
2.  Access _multiple_ camera streams at once.
3.  Apply image stitching and panorama construction to the frames from these video streams.
4.  Perform motion detection in the panorama image.

Again, the benefit of performing motion detection in the _panorama image_ versus _two separate frames_ is that we won’t have any “blind spots” in our field of view.

### Hardware setup

For this project, I’ll be using my Raspberry Pi 2, although you could certainly use your laptop or desktop system instead. I simply went with the Pi 2 for it’s small form factor and ease of maneuvering in space constrained places.

I’ll also be using my [Logitech C920 webcam](http://www.amazon.com/gp/product/B006JH8T3S/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B006JH8T3S&linkCode=as2&tag=trndingcom-20&linkId=BWJJLCCXZCBM465P) (that is plug-and-play compatible with the Raspberry Pi) along with the [Raspberry Pi camera module](http://www.amazon.com/gp/product/B00E1GGE40/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00E1GGE40&linkCode=as2&tag=trndingcom-20&linkId=XF5KMO3TGBUENU5T). Again, if you decide to use your laptop/desktop system, you can simply hook-up multiple webcams to your machine — the same concepts discussed in this post still apply.

Below you can see my setup:

[![Image 3: Figure 1: My the Raspberry Pi 2 + USB webcam + Pi camera module setup.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2016/01/realtime_panorama_setup_02.jpg)

**Figure 1:** My the Raspberry Pi 2 + USB webcam + Pi camera module setup.

Here is another angle looking up at the setup:

[![Image 4: Figure 2: Placing my setup on top of a bookcase so it has a good viewing angle of my apartment.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2016/01/realtime_panorama_setup_03.jpg)

**Figure 2:** Placing my setup on top of a bookcase so it has a good viewing angle of my apartment.

The setup is pointing towards my front door, kitchen, and hallway, giving me a full view of what’s going on inside my apartment:

[![Image 5: Figure 3: Getting ready for real-time panorama construction.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2016/01/realtime_panorama_setup_01.jpg)

**Figure 3:** Getting ready for real-time panorama construction.

The goal is to take frames captured from both my video streams, stitch them together, and then perform motion detection in the panorama image.

Constructing a panorama, rather than [using multiple cameras and performing motion detection independently in each stream](https://pyimagesearch.com/2016/01/18/multiple-cameras-with-the-raspberry-pi-and-opencv/) ensures that I don’t have any “blind spots” in my field of view.

### Project structure

Before we get started, let’s look at our project structure:

|--- pyimagesearch
|    |---- \_\_init\_\_.py
|    |--- basicmotiondetector.py
|    |--- panorama.py
|--- realtime\_stitching.py

As you can see, we have defined a `pyimagesearch` module for organizational purposes. We then have the `basicmotiondetector.py` implementation from last week’s post on [accessing multiple cameras with Python and OpenCV](https://pyimagesearch.com/2016/01/18/multiple-cameras-with-the-raspberry-pi-and-opencv/). This class hasn’t changed at all, so we _won’t_ be reviewing the implementation in this post. For a thorough review of the basic motion detector, be sure to read [last week’s post](https://pyimagesearch.com/2016/01/18/multiple-cameras-with-the-raspberry-pi-and-opencv/).

We then have our `panorama.py` file which defines the `Stitcher` class used to stitch images together. We initially used this class in the [OpenCV panorama stitching tutorial](https://pyimagesearch.com/2016/01/11/opencv-panorama-stitching/).

However, as we’ll see later in this post, I have made a _slight_ modifications to the constructor and `stitch` methods to facilitate real-time panorama construction — we’ll learn more about these slight modifications later in this post.

Finally, the `realtime_stitching.py` file is our main Python driver script that will access the multiple video streams (in an efficient, threaded manner of course), stitch the frames together, and then perform motion detection on the panorama image.

### Updating the image stitcher

In order to (1) create a real-time image stitcher and (2) perform motion detection on the panorama image, we’ll assume that both cameras are _fixed_ and _non-moving_, like in **Figure 1** above.

Why is the fixed and non-moving assumption so important?

Well, remember back to our lesson on [panorama and image stitching](https://pyimagesearch.com/2016/01/11/opencv-panorama-stitching/).

Performing keypoint detection, local invariant description, keypoint matching, and homography estimation is a _computationally expensive_ task. If we were to use our previous implementation, we would have to perform stitching on _each_ set of frames, making it near impossible to run in real-time (especially for resource constrained hardware such as the Raspberry Pi).

However, if we assume that the cameras are fixed, _**we only have to perform the homography matrix estimation once!**_

After the initial homography estimation, we can use the same matrix to transform and warp the images to construct the final panorama — doing this enables us to skip the computationally expensive steps of keypoint detection, local invariant feature extraction, and keypoint matching in each set of frames.

Below I have provided the relevant updates to the `Sticher` class to facilitate a cached homography matrix:

\# import the necessary packages
import numpy as np
import imutils
import cv2

class Stitcher:
	def \_\_init\_\_(self):
		# determine if we are using OpenCV v3.X and initialize the
		# cached homography matrix
		self.isv3 = imutils.is\_cv3()
		self.cachedH = None

The only addition here is on **Line 11** were I define `cachedH` , the cached homography matrix.

We also need to update the `stitch` method to cache the homography matrix after it is computed:

	def stitch(self, images, ratio=0.75, reprojThresh=4.0):
		# unpack the images
		(imageB, imageA) = images

		# if the cached homography matrix is None, then we need to
		# apply keypoint matching to construct it
		if self.cachedH is None:
			# detect keypoints and extract
			(kpsA, featuresA) = self.detectAndDescribe(imageA)
			(kpsB, featuresB) = self.detectAndDescribe(imageB)

			# match features between the two images
			M = self.matchKeypoints(kpsA, kpsB,
				featuresA, featuresB, ratio, reprojThresh)

			# if the match is None, then there aren't enough matched
			# keypoints to create a panorama
			if M is None:
				return None

			# cache the homography matrix
			self.cachedH = M\[1\]

		# apply a perspective transform to stitch the images together
		# using the cached homography matrix
		result = cv2.warpPerspective(imageA, self.cachedH,
			(imageA.shape\[1\] + imageB.shape\[1\], imageA.shape\[0\]))
		result\[0:imageB.shape\[0\], 0:imageB.shape\[1\]\] = imageB

		# return the stitched image
		return result

On **Line 19** we make a check to see if the homography matrix has been computed before. If not, we detect keypoints and extract local invariant descriptors from the two images, followed by applying keypoint matching. We then cache the homography matrix on **Line 34**.

Subsequent calls to `stitch` will use this cached matrix, allowing us to sidestep detecting keypoints, extracting features, and performing keypoint matching on _every_ set of frames.

For the rest of the source code to `panorama.py` , please see the [image stitching tutorial](https://pyimagesearch.com/2016/01/11/opencv-panorama-stitching/) or use the form at the bottom of this post to download the source code.

### Performing real-time panorama stitching

Now that our `Stitcher` class has been updated, let’s move on to to the `realtime_stitching.py` driver script:

\# import the necessary packages
from \_\_future\_\_ import print\_function
from pyimagesearch.basicmotiondetector import BasicMotionDetector
from pyimagesearch.panorama import Stitcher
from imutils.video import VideoStream
import numpy as np
import datetime
import imutils
import time
import cv2

# initialize the video streams and allow them to warmup
print("\[INFO\] starting cameras...")
leftStream = VideoStream(src=0).start()
rightStream = VideoStream(usePiCamera=True).start()
time.sleep(2.0)

We start off by importing our required Python packages. The `BasicMotionDetector` and `Stitcher` classes are imported from the `pyimagesearch` module. We’ll also need the `VideoStream` class from the [imutils](https://github.com/jrosebr1/imutils) package.

If you don’t already have `imutils` installed on your system, you can install it using:

$ pip install imutils

If you do already have it installed, make sure you have upgraded to the latest version (which has added Python 3 support to the `video` sub-module):

$ pip install --upgrade imutils

**Lines 14 and 15** then initialize our two `VideoStream` classes. Here I assume that `leftStream` is a USB camera and `rightStream` is a Raspberry Pi camera (indicated by `usePiCamera=True` ).

If you wanted to use _two USB cameras_, you would simply have to update the stream initializations to:

leftStream = VideoStream(src=0).start()
rightStream = VideoStream(src=1).start()

The `src` parameter controls the _index_ of the camera on your system.

Again, it’s **_imperative_** that you initialize `leftStream` and `rightStream` correctly. When standing _behind_ the cameras, the `leftStream` should be the camera to your lefthand side and the `rightStream` should be the camera to your righthand side.

**Failure to set these stream variables correctly will result in a “panorama” that contains only _one_ of the _two_ frames.**

From here, let’s initialize the image stitcher and motion detector:

\# initialize the image stitcher, motion detector, and total
# number of frames read
stitcher = Stitcher()
motion = BasicMotionDetector(minArea=500)
total = 0

Now we come to the main loop of our driver script where we loop over frames infinitely until instructed to exit the program:

\# loop over frames from the video streams
while True:
	# grab the frames from their respective video streams
	left = leftStream.read()
	right = rightStream.read()

	# resize the frames
	left = imutils.resize(left, width=400)
	right = imutils.resize(right, width=400)

	# stitch the frames together to form the panorama
	# IMPORTANT: you might have to change this line of code
	# depending on how your cameras are oriented; frames
	# should be supplied in left-to-right order
	result = stitcher.stitch(\[left, right\])

	# no homograpy could be computed
	if result is None:
		print("\[INFO\] homography could not be computed")
		break

	# convert the panorama to grayscale, blur it slightly, update
	# the motion detector
	gray = cv2.cvtColor(result, cv2.COLOR\_BGR2GRAY)
	gray = cv2.GaussianBlur(gray, (21, 21), 0)
	locs = motion.update(gray)

**Lines 27 and 28** read the `left` and `right` frames from their respective video streams. We then resize the frames to have a width of 400 pixels, followed by stitching them together to form the panorama. Remember, frames supplied to the `stitch` method need to be supplied in _left-to-right order!_

In the case that the images cannot be stitched (i.e., a homography matrix could not be computed), we break from the loop (**Lines 41-43**).

Provided that the panorama could be constructed, we then process it by converting it to grayscale and blurring it slightly (**Lines 47 and 48**). The processed panorama is then passed into the motion detector (**Line 49**).

However, before we can detect any motion, we first need to allow the motion detector to “run” for a bit to obtain an accurate running average of the background model:

	# only process the panorama for motion if a nice average has
	# been built up
	if total > 32 and len(locs) > 0:
		# initialize the minimum and maximum (x, y)-coordinates,
		# respectively
		(minX, minY) = (np.inf, np.inf)
		(maxX, maxY) = (-np.inf, -np.inf)

		# loop over the locations of motion and accumulate the
		# minimum and maximum locations of the bounding boxes
		for l in locs:
			(x, y, w, h) = cv2.boundingRect(l)
			(minX, maxX) = (min(minX, x), max(maxX, x + w))
			(minY, maxY) = (min(minY, y), max(maxY, y + h))

		# draw the bounding box
		cv2.rectangle(result, (minX, minY), (maxX, maxY),
			(0, 0, 255), 3)

We use the first 32 frames of the initial video streams as an estimation of the background — during these 32 frames _no_ motion should be taking place.

Otherwise, provided that we have processed the 32 initial frames for the background model initialization, we can check the `len` of `locs` to see if it is greater than zero. If it is, then we can assume “motion” is taking place in the panorama image.

We then initialize the minimum and maximum _(x, y)-_coordinates associated with the locations containing motion. Given this list (i.e., `locs` ), we loop over the contour regions individually, compute the bounding box, and determine the smallest region encompassing all contours. This bounding box is then drawn on the panorama image.

As mentioned in [last week’s post](https://pyimagesearch.com/2016/01/18/multiple-cameras-with-the-raspberry-pi-and-opencv/), the motion detector we use assumes there is only _one_ object/person moving at a time. For _multiple objects_, a more advanced algorithm is required (which we will cover in a future PyImageSearch post).

Finally, the last step is to draw the timestamp on panorama and show the output images:

	# increment the total number of frames read and draw the 
	# timestamp on the image
	total += 1
	timestamp = datetime.datetime.now()
	ts = timestamp.strftime("%A %d %B %Y %I:%M:%S%p")
	cv2.putText(result, ts, (10, result.shape\[0\] - 10),
		cv2.FONT\_HERSHEY\_SIMPLEX, 0.35, (0, 0, 255), 1)

	# show the output images
	cv2.imshow("Result", result)
	cv2.imshow("Left Frame", left)
	cv2.imshow("Right Frame", right)
	key = cv2.waitKey(1) & 0xFF

	# if the \`q\` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
print("\[INFO\] cleaning up...")
cv2.destroyAllWindows()
leftStream.stop()
rightStream.stop()

**Lines 82-86** make a check to see if the `q` key is pressed. If it is, we break from the video stream loop and do a bit of cleanup.

### Running our panorama builder + motion detector

To execute our script, just issue the following command:

$ python realtime\_stitching.py

Below you can find an example GIF of my results:

[![Image 6: Figure 4: Applying motion detection on a panorama constructed from multiple cameras on the Raspberry Pi, using Python + OpenCV.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20313'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2016/01/realtime_panorama_stitching_animation.gif)

**Figure 4:** Applying motion detection on a panorama constructed from multiple cameras on the Raspberry Pi, using Python + OpenCV.

On the _top-left_ we have the **left video stream**. And on the _top-right_ we have the **right video stream**. **On the _bottom_, we can see that both frames have been stitched together into a single panorama.** Motion detection is then performed on the panorama image and a bounding box drawn around the motion region.

The full video demo can be seen below:

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this blog post, we combined our knowledge over the past 1.5 months of tutorials and:

1.  Increased FPS processing rate using threadng.
2.  Accessed _multiple_ video streams at once.
3.  Performed image stitching and panorama construction from these video streams.
4.  And applied motion detection on the panorama image.

Overall, we were able to easily accomplish all of this on the Raspberry Pi. We can expect _even faster_ performance on a modern laptop or desktop system.

See you next week!

**If you enjoyed this post, _please be sure to signup for the PyImageSearch Newsletter using the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!