Title: Inpaint Images: AI Photo Restoration - PyImageSearch

URL Source: https://pyimagesearch.com/2024/03/11/understanding-tasks-in-diffusers-part-2/

Published Time: 2024-03-11T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Understanding Tasks in Diffusers: Part 2](#h2-understanding-tasks-in-diffusers-part-2)

*   [Configuring Your Development Environment](#h3-configuring-your-development-environment)
*   [What Is Inpainting?](#h3-what-is-inpainting)
*   [Setup and Imports](#h3-setup-and-imports)
*   [Loading the Image and Masks](#h3-loading-the-image-and-masks)
*   [An Inpainting Demo](#h3-inpainting-demo)
*   [Blurring the Masked Area](#h3-blurring-masked-area)
*   [SDXL for Inpainting](#h3-sdxl-for-inpainting)
*   [Preserving the Unmasked Area](#h3-preserving-unmasked-area)
*   [Pipeline Parameters](#h3-pipeline-parameters)
*   [Chained Inpainting (Text2Img to Inpainting)](#h3-chained-inpainting-text2img-to-inpainting)
*   [Chained Inpainting (Inpainting to Img2Img)](#h3-chained-inpainting-inpainting-to-img2img)
*   [Controlling Image Generation](#h3-controlling-image-generation)

*   [Using a Control Image](#h4-using-control-image)
*   [Using Another Refiner Model](#h4-using-another-refiner-model)

*   [Summary](#h2-summary)

*   [Citation Information](#h3-citation)

* * *

In this tutorial, you will learn how to do image inpainting with Diffusers.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20603%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2024/03/understanding-tasks-in-diffusers-part2-featured.png)

This lesson is the 2nd in a 3-part series on **Understanding Tasks in Diffusers**:

1.  [_Understanding Tasks in Diffusers: Part 1_](https://pyimg.co/5azv1)
2.  [**_Understanding Tasks in Diffusers: Part 2_**](https://pyimg.co/ga0jk) **(this tutorial)**
3.  _Understanding Tasks in Diffusers: Part 3_

**To learn how to perform image inpainting and control image generation,** **_just keep reading._**

* * *

[**Understanding Tasks in Diffusers Part 2**](#TOC-h2-understanding-tasks-in-diffusers-part-2)
----------------------------------------------------------------------------------------------

Welcome to Part 2 of the Understanding Tasks in Diffusers tutorial series.

This tutorial is inspired by the [beautiful documentation](https://huggingface.co/docs/diffusers/en/using-diffusers/inpaint?regular-specific=runwayml%2Fstable-diffusion-v1-5#inpainting) at Hugging Face Diffusers.

* * *

### [**Configuring Your Development Environment**](#TOC-h3-configuring-your-development-environment)

To follow this guide, you need to have the `diffusers` and `accelerate` libraries installed on your system.

Luckily, `diffusers` is pip-installable:

$ pip install diffusers
$ pip install accelerate

* * *

### [**What Is Inpainting?**](#TOC-h3-what-is-inpainting)

Image inpainting refers to the process of repairing the missing or damaged parts of images, as shown in **Figure 1**.

Traditionally, it has been widely used to restore aged or deteriorated photographs, effectively removing imperfections (e.g., cracks, scratches, dust spots, and red eyes), thus preserving the integrity of visual memories.

Diffusion models allow us to take a more creative approach. These models enable users to selectively mask regions within an image and then guide the restoration or alteration process through textual prompts. This technique represents a specialized application of the img2img (image-to-image) transformation, where only a designated portion of the image is modified in accordance with the directions provided by the prompt.

So effectively, image inpainting through diffusion:

*   Fills specific areas in images
*   Repairs defects and artifacts
*   Replaces areas with entirely new content
*   Uses a mask (white=fill, black=keep)
*   Guided by a text prompt

Let us begin by importing the necessary libraries required for diffusers.

* * *

### [**Setup and Imports**](#TOC-h3-setup-and-imports)

import torch

from diffusers import (
    AutoPipelineForText2Image,
    AutoPipelineForInpainting,
    ControlNetModel,
    StableDiffusionControlNetInpaintPipeline,
    AutoPipelineForImage2Image,
)
from diffusers.utils import load\_image, make\_image\_grid

* * *

### [**Loading the Image and Masks**](#TOC-h3-loading-the-image-and-masks)

Next, we want to:

*   Define the device type (please note that a gpu, preferably A100, is needed for running most of the examples).
*   Define a generator (optional). This is to make the images reproducible.
*   Load the image and masks. You can create your own masks from images from [Hugging Face Spaces](https://huggingface.co/spaces/ritwikraha/mask-maker).

We display the image and the mask in **Figure 2**.

DEVICE = "cuda" if torch.cuda.is\_available() else "cpu"
GENERATOR = torch.Generator(DEVICE).manual\_seed(31)
IMAGE\_SIZE = (1024, 1024)

initial\_image = load\_image("https://i.imgur.com/UjAIycp.png").resize(IMAGE\_SIZE)
initial\_mask\_image = load\_image("https://i.imgur.com/JMHtoZE.png").resize(IMAGE\_SIZE)

make\_image\_grid(\[initial\_image, initial\_mask\_image\], rows=1, cols=2)

[![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20350'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/y3mkXEeTF08tpmSsQUyBsff6ztHHDdlBqHc-Kp0pgYemQDPtSrmL9FcSj8FIrtXz7xKHp2JSe30w-wY8l9FBaJWySO1tkpHXLjB06zdJpiNFijTzkBMuNOCIcnhSQq_JPO5FRQ_0kUh1GIvuspxbBKc)

**Figure 2:** Original Image and the Mask (source: image by the authors).

* * *

### [**An Inpainting Demo**](#TOC-h3-inpainting-demo)

Now, let’s look at a demo of inpainting with the above mask and image.

The model we are using here is: `runwayml/stable-diffusion-v1-5`.

Stable Diffusion Inpainting, Stable Diffusion XL (SDXL) Inpainting, and Kandinsky 2.2 Inpainting are the most popular models for inpainting.

SDXL is capable of producing higher resolution images, but the `init_image` for SDXL must be `1024x1024`.

What is Stable Diffusion Inpainting?

The short answer is that it’s a latent diffusion model fine-tuned on `512x512` images on inpainting. The architecture resembles a standard image-to-image diffusion pipeline, where the denoising is guided by both a text prompt and is limited to a particular region.

Do you want to learn more about Stable Diffusion and the Inpainting Architecture? Please complete the survey [**here**](https://docs.google.com/forms/d/e/1FAIpQLSd1Jof_Q_kWzX8IxpYN-VxIl5Gm8NAEki31Y6FtsHuiyMQODw/viewform).

We create the pipeline by fetching the required model using `AutoPipelineForInpainting` and supply it with a `prompt`, `negative_prompt`, original image, and mask image.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"

model\_id = "runwayml/stable-diffusion-v1-5"

pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
    use\_safetensors=True,
).to(DEVICE)

output = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[0\],
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    generator=GENERATOR,
).images\[0\]

Finally, the rendered image is shown using the `make_image_grid` functionality in **Figure 4**.

make\_image\_grid(\[initial\_image, initial\_mask\_image, output\], rows=1, cols=3)

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20234'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/ytLAsjPec3ytZe_bZ7x3CUR2r3nVPuSNoGO9Cmz2QmaAz7YHgLMNdACothikHhwhUpAyeYTKTwWHziP4r2yDo-kgMvwsloDjwLbjdNhDHVMXuLRs5uS07k9QUxIr7atqbjVyMxOjibv_cln1fZn1d1s)

**Figure 4:** Original Image, Mask, and Rendered Image (source: image by the authors).

* * *

### [**Blurring the Masked Area**](#TOC-h3-blurring-masked-area)

Sometimes, a very strict mask can be constricting to the inpainting model. A little rough/ blurred or dilated mask is often seen to give better outputs.

This functionality is provided by the `VaeImageProcessor.blur` method.

A higher `blur_factor` makes the edges more blurry, while a lower value retains the sharper edges.

Let’s see how we can use this:

blurred\_mask = pipeline.mask\_processor.blur(
    initial\_mask\_image,
    blur\_factor=50,
)

Let us visualize the blurred mask. The blurred mask is shown in **Figure 5**.

make\_image\_grid(\[blurred\_mask\], rows=1, cols=1)

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/f14FtuDa1oBgLs5SOler46WGGVOCHP8WYkdAUYALMnFb6HNDuAeaU1FOG4WA8Y0e7YlnfoLqbZhjTPicCoZtK26ld9nt43v6C2G36wuGdnAkUpxGJvfnpBBk3hMrAzhbsVoFkoiovjlPuY9WB0lL9r4)

**Figure 5:** Blurred mask (source: image by the authors).

output\_blurred = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[0\],
    image=initial\_image,
    mask\_image=blurred\_mask,
    generator=GENERATOR
).images\[0\]

And finally, look at the generated image (shown in **Figure 6**) with the blurred mask.

make\_image\_grid(\[initial\_image, initial\_mask\_image, output, output\_blurred\], rows=2, cols=2)

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/cYztGes6rBUAyehswMR50wCBlntFQwOlF830kYwDyFcgMlx8s111xjs9v2khTAuBeKSBR6bCB8D8ox2Ud15NKPUXSxy6TJ4QDx6Ijkbrdf6R03rZOOEXKUU-vzNAQyKQ9btz5b_QH_KPuLsF4Fjm1KE)

**Figure 6:** Original image, mask, generated image (without mask), and generated image (with mask) (source: image by the authors).

* * *

### [**SDXL for Inpainting**](#TOC-h3-sdxl-for-inpainting)

Let us now focus on how to use Stable Diffusion XL for inpainting. The model name is: `diffusers/stable-diffusion-xl-1.0-inpainting-0.1`.

What is SDXL Image Inpainting?

SDXL is a larger and more powerful version of Stable Diffusion v1.5. It generally follows a two-stage process, where the base model first produces an image, and then the refiner model polishes it. However, each model can be used separately as well. We can see sdxl inpainting in work in **Figure 7**.

Let us see how we can use it. We load the pipeline using `AutoPipelineForInpainting` and pass in the `prompt`, `negative_prompt`, and the image dimensions and the original and mask images.

In **Figure 8**, we see the final result.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"

model\_id = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"

pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
    use\_safetensors=True,
).to(DEVICE)

output = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[0\],
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[initial\_image, initial\_mask\_image, output\], rows=1, cols=3)

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20234'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/0wgqsgN68XLc09R5S36sA55mYPIJ_Ff3iRF5kb_sBGrtHAIMajFTwXdAx27zo12lgwoYIDwAnihVpvAsM6CapjAB0EyXFAT0CwIvCAF4tgHQVPsi31pURwi3oFzLZEE4Nh_zffqNaWHk5KKR5DJlA-4)

**Figure 8:** Final result (source: image by the authors).

* * *

### [**Preserving the Unmasked Area**](#TOC-h3-preserving-unmasked-area)

A frequent problem that arises is how to preserve the unmasked area while doing generations,

Again, we can make use of the `VaeImageProcessor` or the `VaeImageProcessor.apply_overlay` method specifically.

This forces the unmasked area of the original image to remain the same, even though this can create some artifacts between the masked and unmasked area in the final generation.

Let us see how we can achieve this using a previously used model: `runwayml/stable-diffusion-inpainting`.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"

model\_id = "runwayml/stable-diffusion-inpainting"

pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)

Here, we first create the `inpainted_image` exactly as we had done before.

inpainted\_image = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[0\],
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    generator=GENERATOR,
).images\[0\]

Now, we apply the `apply_overlay`, which allows us to paste the unmasked region of the original image back on the generated `inpainted_image`.

unmasked\_unchanged\_image = pipeline.image\_processor.apply\_overlay(
    initial\_mask\_image,
    initial\_image,
    inpainted\_image,
)

Finally, we visualize the images using the `make_image_grid` utility in **Figure 9**.

make\_image\_grid(\[initial\_image, initial\_mask\_image, inpainted\_image, unmasked\_unchanged\_image\], rows=2, cols=2)

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/9KXCCw19AXmx8Wozxp9GOf3_llYDdcosAVt-pl0lKradOUxzNPrwjNWQ1ti3dsgBT7jlJ8qtoZ11p0UCt4AYuTYTe8Zzrk44QrlcINLmf4IJfiJDA-hQyj3CwFY1loY1xzDtKSU8QRKmJ2T4CJMzXUk)

**Figure 9:** The final result (source: image by the authors).

* * *

### [**Pipeline Parameters**](#TOC-h3-pipeline-parameters)

As with the previous tutorial, it is important to understand the various pipeline parameters to:

*   `strength`: controls noise added to the base image.
    *   **High strength:** More noise, longer processing. Higher quality, more different from base.
    *   **Low strength:** Less noise, faster processing. Lower quality, closer to base.

*   `guidance_scale`: describes how aligned the text prompt and generated image are.
    *   **High:** Stricter interpretation of prompt (less creativity).
    *   **Low:** More varied output (more creative).
    *   **High strength & High** `guidance_scale`**:** Maximum creative freedom within prompt restrictions.

*   `negative_prompt`: It has the opposite role of a prompt.
    *   Guides the model away from unwanted visuals.
    *   It can be used effectively to improve image quality without much effort.

*   `padding_mask_crop`: This crops the masked area with some user-specified padding, and it’ll also crop the same area from the original image.
    *   Crops masked area + padding from image and mask.
    *   Upscales cropped image and mask.
    *   Overlays upscaled on the original image.
    *   Boosts quality without complex pipelines.

Let us see how we can effectively play with these parameters using a previously defined model: `runwayml/stable-diffusion-inpainting`.

Parameters to tune:

*   **Strength:** defines how much the inpainted region should resemble the original region. 1 = no trace left of original, 0 = original region reconstructed.
*   **Guidance Scale:** how well aligned the generation and the prompt are. Higher = strictly aligned, lower = loosely aligned.
*   **Negative Prompt:** what visuals and imageries to stay away from when generating the image.
*   **Padding Mask Crop:** crop parts of the mask and the original image for a more informed image generation.

model\_id = "runwayml/stable-diffusion-inpainting"

pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)

Feel free to change these parameters and watch the generated image change.

output = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    strength=1,
    guidance\_scale=10.5,
    padding\_mask\_crop=32
).images\[0\]

Visualize the final image using the `make_image_grid` utility in **Figure 10**.

make\_image\_grid(\[initial\_image, initial\_mask\_image, output\], rows=1, cols=3)

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20234'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/DCDREWF7oEZvzfZwGydXRwAnJTbnromB17LEK6ECAza90jgYqwMz7aNoN4RI4sHG8DHFq4o_ON6hE9xvaHUJpSyAVF7B9eVi6Lb6l18GJmrt5WNZpe_mIdTExdmAbsTRZ-r1cL7VBAnmGAm-cppF9Ho)

**Figure 10:** Final result (source: image by the authors).

* * *

### [**Chained Inpainting (Text2Img to Inpainting)**](#TOC-h3-chained-inpainting-text2img-to-inpainting)

`AutoPipelineForInpainting` can be chained with other Diffusers pipelines.

This can improve the quality of generations and also remove any unnecessary artifacts that may crop up during image inpainting or other generations.

Chaining a text-to-image and inpainting pipeline allows us to inpaint the generated image.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"

model\_id = "runwayml/stable-diffusion-v1-5"

pipeline = AutoPipelineForText2Image.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
    use\_safetensors=True,
).to(DEVICE)

First, we create the text2image pipeline for generating images based on a prompt.

text2image\_output = pipeline(
    prompt,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]

We can visualize the image using the `make_image_grid` utility, as shown in **Figure 11**.

make\_image\_grid(\[text2image\_output\], rows=1, cols=1)

[![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/xIZkwF4KY7qrtDn_EQRrqGtAMLBkmLQXU8ZFhDXd4b8Gb_S7IY94BzmPxPQdfISEAjy8ufb7NCJe3tiR6w5yK8JTwAxfL8CFZghAqKfLdrnS-1njXEBgpHILKP_0Lm3JfFVZWaCAIZyzYTaLKuJbRUg)

**Figure 11:** Text 2 Image output (source: image by the authors).

Next, we load an `arbitrary_mask`, which has an arbitrary shape to be inpainted. We visualize the arbitrarily shaped mask in **Figure 12**.

arbitrary\_mask = load\_image("https://i.imgur.com/X4yzBR7.png").resize(IMAGE\_SIZE)
make\_image\_grid(\[arbitrary\_mask\], rows=1, cols=1)

[![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/Cf1mz7o6ajp64hwr8cc5IjGOJdUjAJqrAAWCqwy-MdFmjyCa0whFAtkPrMSbhMxyD6WXUJKuaI3sBAbu84-eYu6Pqfp92GQ4D8ssVk0-WMHU22cOxxK0T5Yg8KPhZ6XLOteD_vi-4kestzu2_juBOiw)

**Figure 12:** A mask (source: image by the authors).

model\_id = "kandinsky-community/kandinsky-2-2-decoder-inpaint"

pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)

We supply this pipeline with a new prompt and use it to create the inpainted generation. Let’s visualize the result in **Figure 13**.

arbitrary\_mask\_text2image\_output = pipeline(
    prompt="a blossom of cherry flowers, beautiful, cute, disney, pixar, 8k",
    image=text2image\_output,
    mask\_image=arbitrary\_mask,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[text2image\_output, arbitrary\_mask, arbitrary\_mask\_text2image\_output\], rows=1, cols=3)

[![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20234'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/PLN140k79rEib7Dq8jV5EyxT2gM6_VOnyaKSu5J81Z9yU8zMDTacVUnAD4j8MatNc5cifV3I7jlGzX_h8WEaJVNJh518MNNGwiZNsr6qQCvzL0n6ISO6aXARaWrpadIKD6f1kNE_F5eW6P1HCYRLSEQ)

**Figure 13:** The text to image with arbitrary mask (source: image by the authors).

* * *

### [**Chained Inpainting (Inpainting to Img2Img)**](#TOC-h3-chained-inpainting-inpainting-to-img2img)

We can also chain the inpainting pipeline with another image-to-image pipeline to refine its quality.

The model we use for the initial inpainting is: `runwayml/stable-diffusion-inpainting`, and the model used for image-to-image is: `stabilityai/stable-diffusion-xl-refiner-1.0`. Let’s view the end product in **Figure 14**.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"


model\_id = "runwayml/stable-diffusion-inpainting"


pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)


image\_inpainting = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[image\_inpainting\], rows=1, cols=1)

[![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/_WQv16Wit7H4t_o1sgB9OL6KTDTm7R3tU6V4ChgCIF_r8vwiEU025Ms8-MLq-3Sep3x7wohIU9Kr99knCpprVTS618LrfHkWdhfN9PhIvAAnM7RaagYYxeFKJ1TcWKJBc_4UVSoWelit0WsHlOb0nxs)

**Figure 14:** Inpainted image (source: image by the authors).

model\_id = "stabilityai/stable-diffusion-xl-refiner-1.0"
pipeline = AutoPipelineForInpainting.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)


refined\_inpaint\_image = pipeline(
    prompt=prompt,
    image=image\_inpainting,
    mask\_image=initial\_mask\_image,
    output\_type="latent",
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


pipeline = AutoPipelineForImage2Image.from\_pipe(pipeline).to(DEVICE)


image = pipeline(
    prompt=prompt,
    image=refined\_inpaint\_image,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[initial\_image, initial\_mask\_image, image\_inpainting, image\], rows=2, cols=2)

Here, we use the `AutoPipelineForImage2Image` to create another image from the inpainted image shown in **Figure 15.**

[![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/nS0RoiV97jhcx9D7451Ixf1AmH3tm2gL5kL327eJK_HhGfeCG7NphGh8l7XRH_vwTvSsNjBqkolBlQZlP8C5ARLZZaFDtzjcdCLI1cZtgPvqrk7fEfIY4naCU1EDCDOWUgkTLLRROwA6mHEkFrsIG_k)

**Figure 15:** Image grid with the original image, mask image, inpainted image, and the image-to-image generation (source: image by the authors).

* * *

### [**Controlling Image Generation**](#TOC-h3-controlling-image-generation)

Denoising a noisy image to get a generated image is a stochastic process. We can play around with the pipeline parameters, but only to a certain extent. Fortunately, there are more definitive ways to control image generation.

Let us learn how to do this with the following models:

*   `"lllyasviel/control_v11p_sd15_inpaint"`
*   `runwayml/stable-diffusion-inpainting`

* * *

#### [**Using a Control Image**](#TOC-h4-using-control-image)

ControlNet models can be used with other models from the Diffusers library.

*   It works by accepting a conditioning image input.
*   This guides the diffusion model to preserve the features in it.

Let’s see how we can achieve this.

prompt = "ilustration of a banyan tree with lush green leafs, high resolution, disney, pixar, 8k"
negative\_prompt = "bad anatomy, deformed, ugly, disfigured"

First, we create two pipelines:

*   A ControlNet pipeline
*   A StableDiffusion pipeline

controlnet\_model\_id = "lllyasviel/control\_v11p\_sd15\_inpaint"
sd\_model\_id = "runwayml/stable-diffusion-inpainting"

# load ControlNet
controlnet = ControlNetModel.from\_pretrained(controlnet\_model\_id,
                                             torch\_dtype=torch.float16,
                                             variant="fp16").to(device)
# pass ControlNet to the pipeline
pipeline = StableDiffusionControlNetInpaintPipeline.from\_pretrained(
    sd\_model\_id,
    controlnet=controlnet,
    torch\_dtype=torch.float16,
    variant="fp16"
).to(device)

Next, we create a function named `make_inpaint_condition` that prepares a “control image” for inpainting based on an input image and its corresponding mask image. Its purpose is:

*   Marking the masked regions in the original image for the inpainting algorithm to target.
*   Ensuring the image and mask have matching dimensions for proper processing.
*   Converting the data to a format compatible with commonly used inpainting libraries.

\# prepare control image
def make\_inpaint\_condition(init\_image, mask\_image):
    init\_image = np.array(init\_image.convert("RGB")).astype(np.float32) / 255.0
    mask\_image = np.array(mask\_image.convert("L")).astype(np.float32) / 255.0

    assert init\_image.shape\[0:1\] == mask\_image.shape\[0:1\], "image and image\_mask must have the same image size"
    init\_image\[mask\_image > 0.5\] = -1.0  # set as masked pixel
    init\_image = np.expand\_dims(init\_image, 0).transpose(0, 3, 1, 2)
    init\_image = torch.from\_numpy(init\_image)
    return init\_image

control\_image = make\_inpaint\_condition(init\_image, mask\_image)

We can visualize the control image using the `make_image_grid` functionality, as shown in **Figure 16**.

make\_image\_grid(\[control\_image\], rows=1, cols=1)

[![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/n28pMRq5JTiZbPooG469mfL8y-B7fecsASuybPxGqNIUcAYFnrdsewme1MINxWOhr6yy-qi38p9dva8TDmRKRPEe9gJOm2VI3B0pxBO3VNy0DxtVCeED8X5rzGvfe4wKy-P22V0I6UICGvt6p4qMiuk)

**Figure 16:** The control image for our pipeline (source: image by the authors).

We can chain this with an image-to-image pipeline to get a more refined image.

Now, we generate the final image using the initial image, mask image, and control image, as shown in **Figure 17.**

image = pipeline(
    prompt=prompt,
    negative\_prompt=negative\_prompt,
    image=initial\_image,
    mask\_image=initial\_mask\_image,
    control\_image=control\_image,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[initial\_image, initial\_mask\_image, Image.fromarray(np.uint8(control\_image\[0\]\[0\])).convert('RGB'), image\], rows=2, cols=2)

[![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/wvRALwBeZ9KDLUlJ-NB7r5sPBmG4eqBqBOZkyT0Xt_n-M0lFtu1kenRYuOksDoGgPl06E2g4vcuPExqn8tfa8wTh9YZ5onD_SJytAIGSKwxCVEUDXFEGZhZc_1-YJNAFxkxifWSq7AizMtgYjLkoZH4)

**Figure 17:** Generating images using a control image (source: image by the authors).

* * *

#### [**Using Another Refiner Model**](#TOC-h4-using-another-refiner-model)

We can go a step further by using the `stabilityai/stable-diffusion-xl-refiner-1.0` model.

This will allow us to generate the image in the custom style we want. The final output is shown in **Figure 18**.

prompt = "zack snyder style"
negative\_prompt = "bad architecture, deformed, disfigured, poor details"


model\_id = "stabilityai/stable-diffusion-xl-refiner-1.0"
pipeline = AutoPipelineForImage2Image.from\_pretrained(
    model\_id,
    low\_cpu\_mem\_usage=True,
).to(DEVICE)


refined\_image = pipeline(
    prompt,
    negative\_prompt=negative\_prompt,
    image=image,
    height=IMAGE\_SIZE\[0\],
    width=IMAGE\_SIZE\[1\],
    generator=GENERATOR,
).images\[0\]


make\_image\_grid(\[initial\_image, initial\_mask\_image, image, refined\_image\], rows=2, cols=2)

[![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/xsEkrz1N_74-3eGG-LZpo0ApckcMhL5cs-8ElNOLsBNGwL_qqG5KCQUlGKU_sLpoMFC2ZiSZVgqaFGUMyfhP_aG_5zLIGzu8r6U3MZUqu1k2P3qXSqlQlAOsbaRGqnY_bh3RYG5jaIHi4M3QbtltXKw)

**Figure 18:** Refining the generated image in a custom style (source: image by the authors).

* * *

[**Summary**](#TOC-h2-summary)
------------------------------

This tutorial delves into image inpainting with the Diffusers library, showcasing its versatility for both basic edits and elaborate artistic endeavors.

Here are the following things we learned:

*   Inpainting Demo
    *   Simple example
    *   Blurring the mask area
*   SDXL for Image Inpainting
*   Preserving the Unmasked area
*   Chained Image Inpainting
    *   Text2Img to Image Inpainting
    *   Inpainting to Img2Img
*   Controlling Image Generation
    *   Using a control image
    *   Using a refiner model

Image inpainting can be really simple, and it can also be really complex, depending upon the size of the masked region, the specific shape of the region, and the prompt from the user.

By leveraging these methods, you’ll gain greater control over the image generation process, minimizing the randomness introduced by denoising and getting closer to your ideal visual representation.

A number of web applications have started internet businesses founded on the principle of basic image inpainting.

It is important to see how inpainting works and how to easily get your own results using the Diffusers library from Hugging Face.

In the next part of the tutorial, we will discuss some advanced workflows like Depth2Img and more.

* * *

### [**Citation Information**](#TOC-h3-citation)

**A. R. Gosthipaty and R. Raha.** “Understanding Tasks in Diffusers: Part 2,” _PyImageSearch_, P. Chugh, S. Huot, and K. Kidriavsteva, eds., 2024, [https://pyimg.co/ga0jk](https://pyimg.co/ga0jk)

@incollection{ARG-RR\_2024\_Understanding-Tasks-in-Diffusers-Part-2,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Understanding Tasks in Diffusers: Part 2},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva},
  year = {2024},
  url = {https://pyimg.co/ga0jk},
}

* * *

![Image 17: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.