Title: Image inpainting with OpenCV and Python - PyImageSearch

URL Source: https://pyimagesearch.com/2020/05/18/image-inpainting-with-opencv-and-python/

Published Time: 2020-05-18T14:00:00+00:00

Markdown Content:
In this tutorial, you will learn how to perform image inpainting with OpenCV and Python.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20450'%3E%3C/svg%3E)

Image inpainting is a form of **image conservation** and **image restoration**, dating back to the 1700s when [Pietro Edwards](https://it.wikipedia.org/wiki/Pietro_Edwards), director of the Restoration of the Public Pictures in Venice, Italy, applied this scientific methodology to restore and conserve famous works ([source](https://en.wikipedia.org/wiki/Inpainting)).

Technology has advanced image painting significantly, allowing us to:

*   Restore old, degraded photos
*   Repair photos with missing areas due to damage and aging
*   Mask out and remove particular objects from an image (and do so in an aesthetically pleasing way)

Today, we’ll be looking at two image inpainting algorithms that OpenCV ships “out-of-the-box.”

**To learn how to perform image inpainting with OpenCV and Python, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, you’ll learn about OpenCV’s inpainting algorithms.

From there, we’ll implement an inpainting demo using OpenCV’s built-in algorithms, and then apply inpainting to a set of images.

Finally, we’ll review the results and discuss the next steps.

I’ll also be upfront and say that this tutorial is an _introduction_ to inpainting including its basics, how it works, and what kind of results we can expect.

While this tutorial doesn’t necessarily “break new ground” in terms of inpainting results, it is an essential prerequisite to future tutorials because:

1.  It shows you how to use inpainting with OpenCV
2.  It provides you with a baseline that we can improve on
3.  It shows some of the _manual input_ required by traditional inpainting algorithms, which deep learning methods can now _automate_

### **OpenCV’s inpainting algorithms**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20624%20332'%3E%3C/svg%3E)

**Figure 1:** An example of image inpainting with OpenCV and Python. ([image source](https://heartbeat.fritz.ai/guide-to-image-inpainting-using-machine-learning-to-edit-and-correct-defects-in-photos-3c1b0e13bbd0))

The OpenCV library ships with two inpainting algorithms:

1.  `` `cv2.INPAINT_TELEA` ``: _[An image inpainting technique based on the fast marching method](https://www.researchgate.net/publication/238183352_An_Image_Inpainting_Technique_Based_on_the_Fast_Marching_Method)_ (Telea, 2004)
2.  `cv2.INPAINT_NS`: _[Navier-stokes, Fluid dynamics, and image and video inpainting](https://www.math.ucla.edu/~bertozzi/papers/cvpr01.pdf)_ (Bertalmío et al., 2001)

To quote the [OpenCV documentation,](https://docs.opencv.org/master/df/d3d/tutorial_py_inpainting.html) the Telea method:

> _… is based on Fast Marching Method. Consider a region in the image to be inpainted. Algorithm starts from the boundary of this region and goes inside the region gradually filling everything in the boundary first. It takes a small neighbourhood around the pixel on the neighbourhood to be inpainted. This pixel is replaced by normalized weighted sum of all the known pixels in the neighbourhood. Selection of the weights is an important matter. More weightage is given to those pixels lying near to the point, near to the normal of the boundary and those lying on the boundary contours. Once a pixel is inpainted, it moves to next nearest pixel using Fast Marching Method. FMM ensures those pixels near the known pixels are inpainted first, so that it just works like a manual heuristic operation._

The second method, Navier-Stokes, is based on fluid dynamics.

Again, quoting the OpenCV documentation:

> _This algorithm is based on fluid dynamics and utilizes partial differential equations. Basic principle is heurisitic \[sic\]. It first travels along the edges from known regions to unknown regions (because edges are meant to be continuous). It continues isophotes (lines joining points with same intensity, just like contours joins points with same elevation) while matching gradient vectors at the boundary of the inpainting region. For this, some methods from fluid dynamics are used. Once they are obtained, color is filled to reduce minimum variance in that area._

In the rest of this tutorial you will learn how to apply both the `cv2.INPAINT_TELEA` and `cv2.INPAINT_NS` methods using OpenCV.

### **How does inpainting work with OpenCV?**

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20550%20442'%3E%3C/svg%3E)

**Figure 2:** Photograph restoration via OpenCV, Python, and image inpainting.

When applying inpainting with OpenCV, **we need to provide two images:**

1.  **The input image we wish to inpaint and restore.** Presumably, this image is “damaged” in some manner, and we need to apply inpainting algorithms to fix it
2.  **The mask image, which indicates _where_ in the image the damage is.** This image should have the same spatial dimensions (width and height) as the input image. Non-zero pixels correspond to areas that should be inpainted (i.e., fixed), while zero pixels are considered “normal” and do not need inpainting

An example of these images can be seen in **Figure 2** above.

The image on the _left_ is our original input image. Notice how this image is old, faded, and damaged/ripped.

The image on the _right_ is our mask image. Notice how white pixels in the mask mark where the damage is in the input image _(left)._

Finally, on the _bottom,_ we have our output image after applying inpainting with OpenCV. Our old, faded, damaged image has now been partially restored.

### **How do we create the mask for inpainting with OpenCV?**

At this point, the big question is:

> _“Adrian, how did you create the mask? Was that created programmatically? Or did you manually create it?”_

For **Figure 2** above (in the previous section), I had to _manually_ create the mask. To do so, I opened up Photoshop ([GIMP](https://www.gimp.org/) or another photo editing/manipulation tool would work just as well), and then used the _Magic Wand_ tool and manual selection tool to select the damaged areas of the image.

I then flood-filled the selection area with _white_, left the background as black, and saved the mask to disk.

**Doing so was a manual, tedious process** — you _may_ be able to programmatically define masks for your own images using image processing techniques such as thresholding, edge detection, and contours to mark damaged reasons, but realistically, there will likely be some sort of manual intervention.

**The manual intervention is one of the _primary limitations_ of using OpenCV’s built-in inpainting algorithms.**

I discuss how we can improve upon OpenCV’s inpainting algorithms, including deep learning-based methods, in the _“How can we improve OpenCV inpainting results?”_ section later in this tutorial.

### **Project structure**

Scroll to the **_“Downloads”_** section of this tutorial and grab the `.zip` containing our code and images. The files are organized as follows:

$ tree --dirsfirst
.
├── examples
│   ├── example01.png
│   ├── example02.png
│   ├── example03.png
│   ├── mask01.png
│   ├── mask02.png
│   └── mask03.png
└── opencv\_inpainting.py

1 directory, 7 files

We have a number of `examples/` including damaged photographs and masks. The mask indicates where in the photo there is damage. Be sure to open each of these files on your machine to become familiar with them.

Our sole Python script for today’s tutorial is `opencv_inpainting.py`. Inside this script, we have our method for repairing our damaged photographs with inpainting techniques.

### **Implementing inpainting with OpenCV and Python**

Let’s learn how to implement inpainting with OpenCV and Python.

Open up a new file, name it `opencv_inpainting.py`, and insert the following code:

\# import the necessary packages
import argparse
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--image", type=str, required=True,
	help="path input image on which we'll perform inpainting")
ap.add\_argument("-m", "--mask", type=str, required=True,
	help="path input mask which corresponds to damaged areas")
ap.add\_argument("-a", "--method", type=str, default="telea",
	choices=\["telea", "ns"\],
	help="inpainting algorithm to use")
ap.add\_argument("-r", "--radius", type=int, default=3,
	help="inpainting radius")
args = vars(ap.parse\_args())

We begin by importing OpenCV and `argparse`. If you do not have OpenCV installed in a virtual environment on your computer, follow my _[pip install opencv](https://pyimagesearch.com/2018/09/19/pip-install-opencv/)_ tutorial to get up and running.

*   `` `--image` ``: The path to the damaged photograph upon which we’ll perform inpainting
*   `` `--mask` ``: The path to the mask, which corresponds to the damaged areas in the photograph
*   `` `--method` ``: Either the `"telea"` or `"ns"` algorithm `choices` are valid inpaining methods for OpenCV and this Python script. By default (i.e., if this argument is not provided via the terminal), the Telea et al. method is chosen
*   `--radius`: The inpainting radius is set to `3` pixels by default; you can adjust this value to see how it affects the results of image restoration

Next, let’s proceed to select our inpaining `--method`:

\# initialize the inpainting algorithm to be the Telea et al. method
flags = cv2.INPAINT\_TELEA

# check to see if we should be using the Navier-Stokes (i.e., Bertalmio
# et al.) method for inpainting
if args\["method"\] == "ns":
	flags = cv2.INPAINT\_NS

Notice that **Line 19** sets our default inpainting method (Telea’s method). If the Navier-Stokes method is going to be applied, the `flags` value is subsequently overridden (**Lines 23 and 24**).

From here, we’ll load our `--image` and `--mask`:

\# load the (1) input image (i.e., the image we're going to perform
# inpainting on) and (2) the  mask which should have the same input
# dimensions as the input image -- zero pixels correspond to areas
# that \*will not\* be inpainted while non-zero pixels correspond to
# "damaged" areas that inpainting will try to correct
image = cv2.imread(args\["image"\])
mask = cv2.imread(args\["mask"\])
mask = cv2.cvtColor(mask, cv2.COLOR\_BGR2GRAY)

Both our `image` and `mask` are loaded into memory via OpenCV’s `imread` function (**Lines 31 and 32**). We require for our `mask` to be a single-channel grayscale image, so a quick conversion takes place on **Line 33**.

We’re now ready to perform inpainting with OpenCV to restore our damaged photograph!

\# perform inpainting using OpenCV
output = cv2.inpaint(image, mask, args\["radius"\], flags=flags)

Inpainting with OpenCV couldn’t be any easier — simply call the built-in `inpaint` function while passing the following parameters:

*   `image`: The damaged photograph
*   `mask`: The single-channel grayscale mask, which highlights the corresponding damaged areas of the photograph
*   `` `inpaintRadius` ``: The radius in pixels is the “circular neighborhood of each point inpainted that is considered by the algorithm” ([OpenCV docs](https://docs.opencv.org/4.3.0/d7/d8b/group__photo__inpaint.html#gaedd30dfa0214fec4c88138b51d678085)); in our case, it comes directly from the `--radius` command line argument
*   `` `flags` ``: Holds the inpainting method (either `cv2.INPAINT_TELEA` or `cv2.INPAINT_NS`)

The return value is the restored photograph (`output`).

Let’s display the results on our screen to see how it works!

\# show the original input image, mask, and output image after
# applying inpainting
cv2.imshow("Image", image)
cv2.imshow("Mask", mask)
cv2.imshow("Output", output)
cv2.waitKey(0)

We display three images on-screen: (1) our original _damaged_ photograph, (2) our `mask` which highlights the damaged areas, and (3) the inpainted (i.e., restored) `output` photograph. Each of these images will remain on your screen until any key is pressed while one of the GUI windows is in focus.

### **OpenCV inpainting results**

We are now ready to apply inpainting using OpenCV.

Make sure you have used the **_“Downloads”_** section of this tutorial to download the source code and example images.

From there, open a terminal, and execute the following command:

$ python opencv\_inpainting.py --image examples/example01.png \\
	--mask examples/mask01.png

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20800'%3E%3C/svg%3E)

**Figure 3:** Our Python-based OpenCV inpainting tutorial is a success — the image that I intentionally damaged with red markings is fully restored.

On the _left,_ you can see the original input image of my dog Janie, sporting an ultra punk/ska jean jacket.

I have _purposely_ added the text _“Adrian wuz here”_ to the image, the mask of which is shown in the _middle_.

The _bottom_ image shows the results of applying the `cv2.INPAINTING_TELEA` fast marching method. The text has been successfully removed, but you can see a number of image artifacts, _especially_ in high-texture areas, such as the concrete sidewalk and the leash.

Let’s try a different image, this time using the Navier-Stokes method:

$ python opencv\_inpainting.py --image examples/example02.png \\
	--mask examples/mask02.png --method ns

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20550%20442'%3E%3C/svg%3E)

**Figure 4:** Successful damaged-photograph restoration via Python and OpenCV using the Navier-Stokes inpainting technique.

On the _top,_ you can see an old photograph, which has been damaged. I then manually created a mask for the damaged areas on the _right_ (using Photoshop as explained in the _“How do we create the mask for inpainting with OpenCV?”_ section).

The _bottom_ shows the output of the Navier-Stokes inpainting method. By applying this method of OpenCV inpainting, we have been able to partially repair the old, damaged photo.

Let’s try one final image:

$ python opencv\_inpainting.py --image examples/example03.png \\
	--mask examples/mask03.png

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20337'%3E%3C/svg%3E)

**Figure 5:** Image inpainting with OpenCV and Python has successfully removed the watermark in the lower-right and a tree in the lower-left.

On the _left,_ we have the original image, while on the _right,_ we have the corresponding mask.

Notice that the mask has two areas that we’ll be trying to “repair”:

1.  The watermark on the bottom-right
2.  The circular area corresponds to one of the trees

In this example, we’re treating OpenCV inpainting as a method of _removing_ objects from an image, the results of which can be seen on the _bottom._

Unfortunately, results are not as good as we would have hoped for. The tree we wish to have removed appears as a circular blur, while the watermark is blurry as well.

That begs the question — **_what can we do to improve our results?_**

### **How can we improve OpenCV inpainting results?**

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20234'%3E%3C/svg%3E)

**Figure 6:** Deep learning-based image inpainting restoration techniques will be discussed in a future PyImageSearch tutorial. ([image source](http://iizuka.cs.tsukuba.ac.jp/projects/completion/data/completion_sig2017.pdf))

One of the biggest problems with OpenCV’s built-in inpainting algorithms is that they require _manual intervention_, meaning that we have to manually supply the masked region we wish to fix and restore.

**Manually supplying the mask is tedious — _isn’t there a better way?_**

In fact, there is.

Using deep learning-based approaches, including fully-convolutional neural networks and Generative Adversarial Networks (GANs), we can “learn to inpaint.”

These networks:

*   Require _zero manual intervention_
*   Can _generate their own training data_
*   Generate results that _are more aesthetically pleasing_ than traditional computer vision inpainting algorithms

Deep learning-based inpainting algorithms are outside the scope of this tutorial but will be covered in a future blog post.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, you learned how to perform inpainting with OpenCV.

The OpenCV library ships with two inpainting algorithms:

1.  `` `cv2.INPAINT_TELEA` ``: _[An image inpainting technique based on the fast marching method](https://www.researchgate.net/publication/238183352_An_Image_Inpainting_Technique_Based_on_the_Fast_Marching_Method)_ (Telea, 2004)
2.  `cv2.INPAINT_NS`: _[Navier-stokes, Fluid dynamics, and image and video inpainting](https://www.math.ucla.edu/~bertozzi/papers/cvpr01.pdf)_ (Bertalmío et al., 2001)

These methods are _traditional_ computer vision algorithms and do not rely on deep learning, making them easy and efficient to utilize.

However, while these algorithms are easy to use (since they are baked into OpenCV), they leave a lot to be desired in terms of accuracy.

Not to mention, having to _manually_ supply the mask image, marking the damaged areas of the original photograph, is quite tedious.

In a future tutorial, we’ll look at deep learning-based inpainting algorithms — these methods require more computation and are a bit harder to code, but ultimately lead to better results (plus, there’s no mask image requirement).

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _just enter your email address in the form below!_**

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!