Title: OpenCV GrabCut: Foreground Segmentation and Extraction - PyImageSearch

URL Source: https://pyimagesearch.com/2020/07/27/opencv-grabcut-foreground-segmentation-and-extraction/

Published Time: 2020-07-27T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to use OpenCV and GrabCut to perform foreground segmentation and extraction.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

Prior to deep learning and instance/semantic segmentation networks such as Mask R-CNN, U-Net, etc., **GrabCut was _the method_ to accurately segment the foreground of an image from the background.**

The GrabCut algorithm works by:

*   Accepting an input image with _either_ (1) a **bounding box** that specified the location of the object in the image we wanted to segment or (2) a **mask** that _approximated_ the segmentation
*   Iteratively performing the following steps:
    *   **Step #1:** Estimating the color distribution of the foreground and background via a Gaussian Mixture Model (GMM)
    *   **Step #2:** Constructing a Markov random field over the pixels labels (i.e., foreground vs. background)
    *   **Step #3:** Applying a graph cut optimization to arrive at the final segmentation

Sounds complicated, doesn’t it?

Luckily, OpenCV has an implementation of GrabCut via the `cv2.grabCut` function that makes applying GrabCut a breeze (once you know the parameters to the function and how to tweak them, of course).

But before you go saying:

> _Hey Adrian, isn’t the GrabCut algorithm old news?_
> 
> _Shouldn’t we just be applying Mask R-CNN, U-Net, or other image segmentation networks to segment background and foreground instead?_

The above is the perfect example of how deep learning and traditional computer vision are being blended together.

If you’ve ever used Mask R-CNN or U-Net before, you know these deep neural networks are _super powerful_, but the masks are not always perfect. **In practice, you can actually use GrabCut to clean up these segmentation masks** (and I’ll be showing you how to do that in a future post).

But in the meantime, let’s learn about the fundamentals of GrabCut.

**To learn how to use OpenCV and GrabCut for foreground segmentation, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, we’ll discuss GrabCut, its implementation in OpenCV via the `cv2.grabCut` function, and its associated parameters.

From there, we’ll learn how to implement GrabCut with OpenCV via both:

1.  GrabCut initialization with bounding boxes
2.  GrabCut initialization with mask approximations

Afterward, we’ll apply GrabCut and review our results.

### GrabCut in OpenCV

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20203'%3E%3C/svg%3E)

**Figure 1:** A selection of methods for performing foreground segmentation. _Column f_ shows GrabCut results; compared to the other methodologies, GrabCut results in a high quality output segmentation. In today’s tutorial, we’ll apply GrabCut with OpenCV for foreground and background segmentation and extraction. (image source: Figure 2 from [Kolmogorov and Blake, 2004](https://cvg.ethz.ch/teaching/cvl/2012/grabcut-siggraph04.pdf))

The `cv2.grabCut` function has the following signature:

grabCut(img, mask, rect, bgdModel, fgdModel, iterCount\[, mode\]) ->
	mask, bgdModel, fgdModel

**To obtain a complete understanding of the implementation, let’s review each of these parameters:**

*   `` `img` ``: The input image, which GrabCut assumes to be an 8-bit, 3-channel image (i.e., unsigned 8-bit integer in BGR channel ordering).
*   `` `mask` ``: The input/output mask. This mask is assumed to be a single-channel image with an unsigned 8-bit integer data type. This mask is initialized automatically if you use bounding box initialization (i.e., `cv2.GC_INIT_WITH_RECT`); otherwise, GrabCut assumes you are performing mask initialization (`cv2.GC_INIT_WITH_MASK`).
*   `` `rect` ``: The bounding box rectangle that contains the region that we want to segment. This parameter is only used when you set the `mode` to `cv2.GC_INIT_WITH_MASK`).
*   `bgModel`: Temporary array used by GrabCut internally when modeling the _background_.
*   `` `fgModel` ``: Temporary array used by GrabCut when modeling the _foreground._
*   `` `iterCount` ``: Number of iterations GrabCut will perform when modeling the foreground versus background. The more iterations, the longer GrabCut will run, and ideally the results will be better.
*   `` `mode` ``: Either `cv2.GC_INIT_WITH_RECT` or `cv2.GC_INIT_WITH_MASK`, depending on whether you are initializing GrabCut with a bounding box or a mask, respectively.

OpenCV’s GrabCut implementation returns a 3-tuple of:

*   `` `mask` ``: The output mask after applying GrabCut
*   `bgModel`: The temporary array used to model the background (you can ignore this value)
*   `` `fgModel` ``: The temporary array for the foreground (again, you can ignore this value)

Now that we have an understanding of the `cv2.grabCut` function including its parameters and the values that it returns, let’s move on to applying GrabCut to an example computer vision project.

### Configuring your development environment

You can set up your system today with a Python virtual environment containing OpenCV by following my _[pip install opencv](https://pyimagesearch.com/2018/09/19/pip-install-opencv/)_ tutorial (instructions included for Ubuntu, macOS, and Raspbian).

Please note that [PyImageSearch does not recommend or support Windows for computer vision and deep learning development](https://pyimagesearch.com/faqs/single-faq/can-you-help-me-do-___-on-windows/).

### Project structure

Before we move on, use the **_“Downloads”_** section of today’s tutorial to grab the .zip associated with this blog post. From there, let’s inspect the layout of the files and folders directly in our terminal with the `tree` command:

$ tree --dirsfirst
.
├── images
│   ├── adrian.jpg
│   ├── lighthouse.png
│   └── lighthouse\_mask.png
├── grabcut\_bbox.py
└── grabcut\_mask.py

1 directory, 5 files

Our project today consists of one folder of `images/` and two Python scripts:

*   `` `images/` ``: Two input photos and one _manually_ created approximation mask image
*   `` `grabcut_bbox.py` ``: A script that accomplishes GrabCut by means of bounding box initialization
*   `grabcut_mask.py`: Performs GrabCut via mask initialization

Using both of the Python scripts, we are going to learn how to perform GrabCut using two methods (bounding box initialization vs. mask initialization). We’ll begin with the bounding box approach in the next section.

### GrabCut with OpenCV: Initialization with bounding boxes

Let’s get started implementing GrabCut with OpenCV — we’ll start by reviewing the bounding box implementation method.

Here, we’ll specify the bounding box of the object we want to segment in the image. The bounding box could be generated by:

*   Manually examining the image and labeling the _(x, y)_\-coordinates of the bounding box
*   Applying a Haar cascade
*   Using HOG + Linear SVM to detect the object
*   Utilizing deep learning-based object detectors such as Faster R-CNN, SSDs, YOLO, etc.

**As long as the algorithm generates a bounding box, you can use it in conjunction with GrabCut.**

For the purposes of our demo script today, we will **_manually_** define the bounding box _(x, y)_\-coordinates (i.e., rather than applying an automated object detector).

Let’s take a look at the bounding box initialization method of GrabCut now.

Open up a new file, name it `grabcut_bbox.py`, and insert the following code:

\# import the necessary packages
import numpy as np
import argparse
import time
import cv2
import os

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--image", type=str,
	default=os.path.sep.join(\["images", "adrian.jpg"\]),
	help="path to input image that we'll apply GrabCut to")
ap.add\_argument("-c", "--iter", type=int, default=10,
	help="# of GrabCut iterations (larger value => slower runtime)")
args = vars(ap.parse\_args())

We begin this script with a selection of imports, namely OpenCV and NumPy (the rest are built into Python). Please refer to the _“Configuring your development environment”_ section above to install Python, OpenCV, and associated software on your system.

Our script handles two [command line arguments](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/):

*   `` `--image` ``: The path to your input image. By `default`, we’ll use the `adrian.jpg` image in the `images/` directory.
*   `--iter`: The number of GrabCut iterations to perform, where smaller values lead to faster overall time and larger values result in a slower runtime (but ideally better segmentation results)

Let’s go ahead and load our input `--image` and allocate space for an equivalently sized mask:

\# load the input image from disk and then allocate memory for the
# output mask generated by GrabCut -- this mask should hae the same
# spatial dimensions as the input image
image = cv2.imread(args\["image"\])
mask = np.zeros(image.shape\[:2\], dtype="uint8")

Here, **Line 20** loads your input `--image` from disk and **Line 21** creates a `mask` (i.e., empty image) with the same dimensions. The `mask` will soon be populated with the results of the GrabCut algorithm.

Next, we will **_manually_** define the coordinates of the face in the `adrian.jpg` image:

\# define the bounding box coordinates that approximately define my
# face and neck region (i.e., all visible skin)
rect = (151, 43, 236, 368)

**Line 25** defines the bounding box coordinates of the face in the image. These _(x, y)_\-coordinates were determined **_manually_** by means of a mouse hovering over pixels in the image and me jotting them down. You can accomplish this with most photo editing software including Photoshop or free alternatives such as GIMP and other apps you find online.

It is important to note here that while these face `rect` coordinates were determined manually, any object detector could do the job. Given that our first example is a face, you could have chosen a Haar, HOG, or [DL-based face detector](https://pyimagesearch.com/2018/02/26/face-detection-with-opencv-and-deep-learning/) to find the bounding box coordinates of the face (substitute a different object detector for different types of objects).

In this next code block, we’ll will **execute the GrabCut algorithm with bounding box initialization** on our input:

\# allocate memory for two arrays that the GrabCut algorithm internally
# uses when segmenting the foreground from the background
fgModel = np.zeros((1, 65), dtype="float")
bgModel = np.zeros((1, 65), dtype="float")

# apply GrabCut using the the bounding box segmentation method
start = time.time()
(mask, bgModel, fgModel) = cv2.grabCut(image, mask, rect, bgModel,
	fgModel, iterCount=args\["iter"\], mode=cv2.GC\_INIT\_WITH\_RECT)
end = time.time()
print("\[INFO\] applying GrabCut took {:.2f} seconds".format(end - start))

Before we perform the GrabCut computation, we need two empty arrays for GrabCut to use internally when segmenting the foreground from the background (`fgModel` and `bgModel`). **Lines 29 and 30** generate both arrays with NumPy’s `zeros` method.

From there, **Lines 34 and 35** apply GrabCut (timestamps are collected before/after the operation), and the elapsed time is printed via **Line 37**.

GrabCut returns our populated `mask` as well as two arrays that we can ignore. If you need a **review of the GrabCut method signature** including the input parameters and return values, please refer to the _“GrabCut in OpenCV”_ section above.

Let’s go ahead and post-process our `mask`:

\# the output mask has for possible output values, marking each pixel
# in the mask as (1) definite background, (2) definite foreground,
# (3) probable background, and (4) probable foreground
values = (
	("Definite Background", cv2.GC\_BGD),
	("Probable Background", cv2.GC\_PR\_BGD),
	("Definite Foreground", cv2.GC\_FGD),
	("Probable Foreground", cv2.GC\_PR\_FGD),
)

# loop over the possible GrabCut mask values
for (name, value) in values:
	# construct a mask that for the current value
	print("\[INFO\] showing mask for '{}'".format(name))
	valueMask = (mask == value).astype("uint8") \* 255

	# display the mask so we can visualize it
	cv2.imshow(name, valueMask)
	cv2.waitKey(0)

**Lines 42-47** define possible `values` in the output GrabCut `mask` including our definite/probable backgrounds and foregrounds.

We then proceed to loop over these `values` so that we can visualize each. Inside the loop (**Lines 50-57**), we (1) construct a mask for the current `value` and (2) display it until any key is pressed.

After each of our definite/probable backgrounds and foregrounds have been displayed, our code will begin generating an `outputMask` and an `output` image:

\# we'll set all definite background and probable background pixels
# to 0 while definite foreground and probable foreground pixels are
# set to 1
outputMask = np.where((mask == cv2.GC\_BGD) | (mask == cv2.GC\_PR\_BGD),
	0, 1)

# scale the mask from the range \[0, 1\] to \[0, 255\]
outputMask = (outputMask \* 255).astype("uint8")

# apply a bitwise AND to the image using our mask generated by
# GrabCut to generate our final output image
output = cv2.bitwise\_and(image, image, mask=outputMask)

Here we produce two visualizations:

1.  GrabCut output mask
2.  Output image (with the background masked out)

To produce our GrabCut `outputMask`, **Lines 62 and 63** find all pixels that are either definite background or probable background and set them to `0` — all other pixels should be marked as `1` (i.e., foreground). Notice how we take advantage of NumPy’s `where` function while OR-ing each mask and setting the values to `0` and `1` accordingly. Then, **Line 66** scales the `outputMask` from the range _\[0, 1\]_ to _\[0, 255\]_.

We then generate our `output` image with the background masked out by means of a `bitwise_and` operation and pass the `outputMask` as the `mask` parameter (**Line 70**).

At this point, we have:

*   Prepared inputs to the `grabCut` function including our input `image`, `mask`, `rect` coordinates, and `fgModel` and `bgModel` zero arrays. Note that the `rect` coordinates were determined _manually_.
*   Executed the GrabCut algorithm.
*   Generated and visualized our definite/probable background and foreground masks.
*   Generated our (1) GrabCut output mask (`outputMask`) and our (2) output image with the background masked out (`output`).

Let’s go ahead and display our final results:

\# show the input image followed by the mask and output generated by
# GrabCut and bitwise masking
cv2.imshow("Input", image)
cv2.imshow("GrabCut Mask", outputMask)
cv2.imshow("GrabCut Output", output)
cv2.waitKey(0)

To wrap up, we show each of the following in separate windows:

*   `` `image` ``: Our original input `--image`
*   `` `outputMask` ``: The GrabCut mask
*   `` `output` ``: The results of our hard work — only the foreground from our original image (i.e., the background has been masked out by means of GrabCut)

Now that GrabCut with bounding box initialization has been implemented, let’s move on to applying it to our input images.

### Bounding box GrabCut results

Start by using the **_“Downloads”_** section of this blog post to download the source code and example image.

From there, open up a terminal, and execute the following command:

$ python grabcut\_bbox.py
\[INFO\] applying GrabCut took 1.08 seconds
\[INFO\] showing mask for 'Definite Background'
\[INFO\] showing mask for 'Probable Background'
\[INFO\] showing mask for 'Definite Foreground'
\[INFO\] showing mask for 'Probable Foreground'

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20301'%3E%3C/svg%3E)

**Figure 2:** The GrabCut bounding box initialization method requires that the bounding box coordinates be provided as input to the algorithm. Here, I’ve _manually_ found the coordinates of the bounding box; however, you could apply any type of object detector to grab the _(x, y)_\-coordinates. Either way, you’ll be able to apply GrabCut with OpenCV to perform foreground segmentation and extraction.

On the _left,_ you can see the original input image, while on the _right,_ you can see the same face with a bounding box drawn around the face/neck region (this bounding box corresponds to the `rect` variable in the `grabcut_bbox.py` script).

Our goal here is to automatically segment the face and neck region from the above image using GrabCut and OpenCV.

Next, you can see our output from **Lines 45-60** where we visualize the definite and probable background and foreground segmentations:

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

**Figure 3:** The various GrabCut masks (bounding box initialization) visualized with OpenCV. _Top-left:_ Definite background. _Top-right:_ Probable background. _Bottom-left:_ Definite foreground. _Bottom-right:_ Probable foreground.

These values map to:

1.  **Definite background** _(top-left):_ `cv2.GC_BGD`
2.  **Probable background** _(top-right)_: `` `cv2.GC_PR_BGD` ``
3.  **Definite foreground** _(bottom-left)_: `cv2.GC_FGD`
4.  **Probable foreground** _(bottom-right)_: `cv2.GC_PR_FGD`

Finally, we have the output of GrabCut itself:

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)

**Figure 4:** _Left:_ Our original input image of me. _Right:_ GrabCut mask via bounding box initialization. _Bottom:_ Our output image where the foreground is segmented from the background via GrabCut masking. Each of these images was generated by means of OpenCV and applying GrabCut for foreground segmentation and extraction.

On the _left,_ we have our original input image.

The _right_ shows the output mask generated by GrabCut, while the _bottom_ shows the output of applying the mask to the input image — **notice how my face and neck region is cleanly segmented and extracted via GrabCut.**

### GrabCut with OpenCV: Initialization with masks

Previously, we learned how to initialize OpenCV’s GrabCut using bounding boxes — but there’s actually a _second_ method to initialize GrabCut.

**Using masks, we can supply the _approximate_ segmentation of the object in the image.** GrabCut can then iteratively apply graph cuts to improve the segmentation and extract the foreground from the image.

These masks could be generated by:

*   Manually creating them in photo editing software such as Photoshop, GIMP, etc.
*   Applying basic image processing operations such as thresholding, edge detection, contour filtering, etc.
*   Utilizing deep learning-based segmentation networks (ex., Mask R-CNN and U-Net)

**How the mask is generated is irrelevant to GrabCut.** As long as you have a mask that approximates the segmentation of the object in an image, _you can use GrabCut to further improve the segmentation._

Let’s see how GrabCut with mask initialization works.

Open up the `grabcut_mask.py` file in your project directory structure, and insert the following code:

\# import the necessary packages
import numpy as np
import argparse
import time
import cv2
import os

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--image", type=str,
	default=os.path.sep.join(\["images", "lighthouse.png"\]),
	help="path to input image that we'll apply GrabCut to")
ap.add\_argument("-mask", "--mask", type=str,
	default=os.path.sep.join(\["images", "lighthouse\_mask.png"\]),
	help="path to input mask")
ap.add\_argument("-c", "--iter", type=int, default=10,
	help="# of GrabCut iterations (larger value => slower runtime)")
args = vars(ap.parse\_args())

Again, our most notable imports are OpenCV and NumPy. Please follow the _“Configuring your development environment”_ section instructions if you need to set up your system to perform GrabCut with mask initialization.

*   `--image`: The path to your input image. This time, by `default`, we’ll use the `lighthouse.png` photo available in the `images/` directory.
*   `` `--mask` ``: The path to input approximation mask associated with the input image. Again, you could create this mask in a number of ways listed at the top of this section, but for the sake of this example, I _manually_ created the mask.
*   `--iter`: The number of GrabCut iterations to perform, where smaller values lead to faster overall time and larger values result in a slower runtime (but ideally better segmentation results)

Now that our imports and command line arguments are taken care of, let’s go ahead and load our input `--image` and input `--mask`:

\# load the input image and associated mask from disk
image = cv2.imread(args\["image"\])
mask = cv2.imread(args\["mask"\], cv2.IMREAD\_GRAYSCALE)

# apply a bitwise mask to show what the rough, approximate mask would
# give us
roughOutput = cv2.bitwise\_and(image, image, mask=mask)

# show the rough, approximated output
cv2.imshow("Rough Output", roughOutput)
cv2.waitKey(0)

Before we get into the weeds of this second GrabCut method, we need to load our input `--image` and `--mask` from disk (**Lines 21 and 22**).

Please note that our rough mask was **_manually_** generated for the sake of this example (using Photoshop/GIMP); however, in a future post we’ll be showing you how to _automatically_ generate the mask via a deep learning Mask R-CNN.

**Line 26** applies a bitwise AND to the `image` using the `mask`, resulting in our rough approximation of our foreground segmentation. Subsequently **Lines 29 and 30** display the approximation until any key is pressed.

From here, we’ll set our probable/definite foreground values into the `mask` array:

\# any mask values greater than zero should be set to probable
# foreground
mask\[mask > 0\] = cv2.GC\_PR\_FGD
mask\[mask == 0\] = cv2.GC\_BGD

Any pixel values in the `mask` greater than zero are set to probable foreground (**Line 34**); all other pixel values are set to definite background (**Line 35**).

We’re now ready to **apply GrabCut with mask initialization:**

\# allocate memory for two arrays that the GrabCut algorithm internally
# uses when segmenting the foreground from the background
fgModel = np.zeros((1, 65), dtype="float")
bgModel = np.zeros((1, 65), dtype="float")

# apply GrabCut using the the mask segmentation method
start = time.time()
(mask, bgModel, fgModel) = cv2.grabCut(image, mask, None, bgModel,
	fgModel, iterCount=args\["iter"\], mode=cv2.GC\_INIT\_WITH\_MASK)
end = time.time()
print("\[INFO\] applying GrabCut took {:.2f} seconds".format(end - start))

Again, we allocate memory for the foreground and background models of GrabCut (**Lines 39 and 40**).

And then we execute GrabCut on the image using the approximate mask segmentation (**Lines 44 and 45**). Note how the `rect` parameter is set to `None` (we don’t need it for this method), unlike the first bounding box-based method described in this blog post.

Moving on, we’ll post-process the results:

\# the output mask has for possible output values, marking each pixel
# in the mask as (1) definite background, (2) definite foreground,
# (3) probable background, and (4) probable foreground
values = (
	("Definite Background", cv2.GC\_BGD),
	("Probable Background", cv2.GC\_PR\_BGD),
	("Definite Foreground", cv2.GC\_FGD),
	("Probable Foreground", cv2.GC\_PR\_FGD),
)

# loop over the possible GrabCut mask values
for (name, value) in values:
	# construct a mask that for the current value
	print("\[INFO\] showing mask for '{}'".format(name))
	valueMask = (mask == value).astype("uint8") \* 255

	# display the mask so we can visualize it
	cv2.imshow(name, valueMask)
	cv2.waitKey(0)

This block should look especially familiar. In fact, it is identical to a block in our first GrabCut method code walkthrough.

Again, we define definite/probable foreground and background `values` (**Lines 52-57**) and display each of the resulting `valueMask` images (**Lines 60-67**).

Next, we’ll prepare our GrabCut mask and output image with the background removed:

\# set all definite background and probable background pixels to 0
# while definite foreground and probable foreground pixels are set
# to 1, then scale teh mask from the range \[0, 1\] to \[0, 255\]
outputMask = np.where((mask == cv2.GC\_BGD) | (mask == cv2.GC\_PR\_BGD),
	0, 1)
outputMask = (outputMask \* 255).astype("uint8")

# apply a bitwise AND to the image using our mask generated by
# GrabCut to generate our final output image
output = cv2.bitwise\_and(image, image, mask=outputMask)

Again, this code on **Lines 72-78** should be familiar at this point (they are identical to the previous script).

Here, we find all pixels that are either definite background or probable background and set them to `0`; all other pixels are marked as `1` (i.e., foreground). We then scale the mask to the range _\[0, 255\]_.

We then apply a bitwise AND operation to the input `image` using the `outputMask`, resulting in the background being removed (masked out).

And finally we display the results on screen:

\# show the input image followed by the mask and output generated by
# GrabCut and bitwise masking
cv2.imshow("Input", image)
cv2.imshow("GrabCut Mask", outputMask)
cv2.imshow("GrabCut Output", output)
cv2.waitKey(0)

Again, to conclude our script, we show the input `image`, GrabCut `outputMask`, and `output` of GrabCut after applying the mask.

With GrabCut mask initialization now implemented, let’s move on to testing it with our own example images.

### Mask GrabCut results

We are now ready to use OpenCV and GrabCut to segment an image via mask initialization.

Start by using the **_“Downloads”_** section of this tutorial to download the source code and example images.

From there, open up a terminal, and execute the following command:

$ python grabcut\_mask.py
\[INFO\] applying GrabCut took 0.56 seconds
\[INFO\] showing mask for 'Definite Background'
\[INFO\] showing mask for 'Probable Background'
\[INFO\] showing mask for 'Definite Foreground'
\[INFO\] showing mask for 'Probable Foreground'

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20206'%3E%3C/svg%3E)

**Figure 5:** _Left:_ Our original photo of a lighthouse. _Right:_ The output of applying GrabCut via mask initialization.

On the _left,_ you can see our original input image. On the _right_ you can see the output of applying GrabCut via mask initialization.

The image on the _right_ shows the mask associated with the lighthouse. For the sake of this blog post/example I manually created this mask in Photoshop; however any algorithm capable of producing a mask could be used here (ex., basic image processing via thresholding, edge detection, contours; deep learning-based segmentation; etc.) **Notice how the mask/segmentation isn’t very “clean” — we can easily see the blue sky of the background “leaking” into our mask.**

From there, we can visualize our definite and probable masks for the background and foreground, respectively:

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20412'%3E%3C/svg%3E)

**Figure 6:** The various GrabCut masks (mask initialization) visualized with OpenCV. _Top-left:_ Definite background. _Top-right:_ Probable background. _Bottom-left:_ Definite foreground. _Bottom-right:_ Probable foreground.

These values map to:

1.  **Definite background** _(top-left):_ `cv2.GC_BGD`
2.  **Probable background** _(top-right)_: `cv2.GC_PR_BGD`
3.  **Definite foreground** _(bottom-left)_: `cv2.GC_FGD`
4.  **Probable foreground** _(bottom-right)_: `cv2.GC_PR_FGD`

And finally, we have the output of OpenCV’s GrabCut with mask initialization:

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20412'%3E%3C/svg%3E)

**Figure 7:** _Left:_ Our original input image of a lighthouse. _Right:_ GrabCut mask via mask initialization. _Bottom:_ Our output image where the foreground segmented from the background via GrabCut masking. Each of these images was generated by means of OpenCV and applying GrabCut for foreground segmentation and extraction.

For reference, the _left_ displays our input image.

The _right_ shows our output mask generated by GrabCut, while the _bottom_ displays the output of applying the mask created by GrabCut to the original input image.

**Notice that we have cleaned up our segmentation** — the blue background from the sky has been removed, while the lighthouse is left as the foreground.

The only problem is that the area where the actual spotlight sits in the lighthouse has been marked as background:

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20412'%3E%3C/svg%3E)

**Figure 8:** Click to view this in your browser with the option to zoom in (`ctrl + "+"`). As you can observe, the results of GrabCut with mask initialization aren’t perfect. I suggest you use the _definite background_ mask `value` result rather than both the _definite/probable foreground_ masks in this specific case. You will need to invert the definite background mask image using your OpenCV/NumPy knowledge. From there, your GrabCut mask initialization method will produce a better foreground segmentation.

The problem here is that the area where the light sits in the lighthouse is more-or-less transparent, causing the blue sky background to shine through, thereby causing GrabCut to mark this area as background.

You could fix this problem by updating your mask to use the definite background (i.e., `cv.GC_BGD`) when loading your mask from disk. I will leave this as an exercise to you, the reader, to implement.

### Why GrabCut is good, but not perfect

GrabCut is one of my favorite computer vision algorithms ever invented, but it’s not perfect.

Furthermore, deep learning-based segmentation networks such as Faster R-CNN and U-Net can _automatically_ generate masks that can segment objects (foreground) from their backgrounds — **does that mean that GrabCut is irrelevant in the age of deep learning?**

_Actually, far from it._

While Faster R-CNN and U-Net are super powerful methods, they can result in masks that are a bit messy. We can use GrabCut to help clean up these masks. I’ll be showing you how to do exactly that in a future blog post.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this tutorial, you learned how to use OpenCV and the GrabCut algorithm to perform foreground segmentation and extraction.

The GrabCut algorithm is implemented in OpenCV via the `cv2.grabCut` function and can be initialized via either:

1.  A bounding box that specifies the location of the object you want to segment in the input image
2.  A mask that approximates the pixel-wise location of the object in the image

The GrabCut algorithm takes the bounding box/mask and then iteratively approximates the foreground and background.

While deep learning-based image segmentation networks (ex., Mask R-CNN and U-Net) tend to be more powerful in actually _detecting_ and _approximating the mask_ of objects in an image, we know that these masks can be less than perfect — **we can actually use GrabCut to clean up “messy” masks returned by these segmentation networks!**

In a future tutorial, I’ll show you how to do exactly that.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!