Title: Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1 - PyImageSearch

URL Source: https://pyimagesearch.com/2024/03/18/step-by-step-guide-to-open-source-implementation-of-generative-fill-part-1/

Published Time: 2024-03-18T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1](#h2-step-by-step-guide-to-open-source-implementation-of-generative-fill-part-1)

*   [Understanding Open-Source Generative Fill](#h3-understanding-open-source-generative-fill)
*   [Configuring Your Development Environment for Generative Fill](#h3-configuring-your-development-environment-for-generative-fill)
*   [Essential Setup and Imports](#h3-essential-setup-and-imports)
*   [Loading and Preparing Input Images for Editing](#h3-loading-and-preparing-input-images-for-editing)
*   [Generating Captions for Image Understanding](#h3-generating-captions-for-image-understanding)

*   [Summary](#h2-summary)

*   [Next Steps: Building an End-to-End Workflow for Image Editing](#h3-next-steps-building-an-end-to-end-workflow-for-image-editing)
*   [Citation Information](#h3-citation-information)

* * *

In this tutorial, you will learn to edit text-based images using Open-Source models like Segment Anything (SAM), OWL-ViT (Vision Transformer for Open-World Localization), and SDXL (Stable Diffusion XL) inpainting.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20603%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2024/03/open-source-implementation-generative-fill-part1-featured.png)

This lesson is the 1st of a 2-part series on **Open-Sourcing Generative Fill**:

1.  [**_Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1_**](https://pyimg.co/jbsn9) **(this tutorial)**
2.  _Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2_

**To learn how to create your own text-based image editing product,** **_just keep reading._**

* * *

[**Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1**](#TOC-h2-step-by-step-guide-to-open-source-implementation-of-generative-fill-part-1)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

Have you ever wished you could edit an image simply by describing the changes you want? Well, you can now, and with open-source deep learning models. In this two-part blog series, we’ll explore the world of “Open-Source Generative Fill,” where text prompts transform your images in ways once strictly limited to professional photo editing software.

Here, we will harness the power of image captioning models, large language models (LLMs), segmentation techniques, and image inpainting. We’re building an open-source pipeline to give you unprecedented control over image manipulation, all driven by the power of your words.

* * *

### [**Understanding Open-Source Generative Fill**](#TOC-h3-understanding-open-source-generative-fill)

Generative Fill is a proprietary technology from Adobe that allows one to replace or change any part of the image with a custom image generated through the prompt, as shown in **Figure 1.**

This is patented and not something one can open up and inspect to understand how it works. However, with the boom of Open-Source models and their heightened capabilities, we can take a look at how to create the same functionality with these models.

What we aim to achieve here is:

*   An edit prompt or instruction laying down what needs to be changed
*   An input picture of where the change needs to happen
*   Edit the image based on the initial edit prompt

How do we propose to do this?

*   Accept an `edit_prompt` and an image as input.
*   Use a vision model to caption the image.
*   Pass the `edit_prompt` through a language model to extract the source entity.
*   Create a `replacement_caption` where the source entity of the original image is swapped with the target entity in the `edit_prompt`.
*   Use the source entity to create a segmentation mask using OWL-VIT and SAM.
*   Use the mask and the `replacement_caption` for image inpainting.

* * *

### [**Configuring Your Development Environment for Generative Fill**](#TOC-h3-configuring-your-development-environment-for-generative-fill)

To follow this guide, you need to have some Open-Source libraries installed on your system.

Luckily, these are all pip-installable:

$ pip install diffusers
$ pip install accelerate
$ pip install transformers

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in minutes.

* * *

### [**Essential Setup and Imports**](#TOC-h3-essential-setup-and-imports)

import torch
from PIL import Image
import requests
from matplotlib import pyplot as plt
from transformers import (
    BlipForConditionalGeneration,
    BlipProcessor,
)

Here, we import essential libraries for working with images and advanced natural language processing. The core libraries provide tools for loading, manipulating, and displaying images. The `transformers` library offers a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model and its processor, which specializes in generating text descriptions of images.

IMAGE\_URL = "https://huggingface.co/datasets/pyimagesearch/blog-post-images/resolve/main/bottle.jpeg"
EDIT\_PROMPT = "change the bottle to a wine"
SEED\_VALUE = 41
﻿
# Set the device
DEVICE = "cuda" if torch.cuda.is\_available() else "cpu"
GENERATOR = torch.Generator().manual\_seed(SEED\_VALUE)

The `IMAGE_URL` stores the address of an image to be used, while `EDIT_PROMPT` provides instructions to modify the image. The `SEED_VALUE` ensures that any random elements involved in image editing will be reproducible. Here, we also check for a GPU (CUDA) to accelerate image processing, falling back to the CPU if a GPU isn’t available.

class Config:
    IMAGE\_SIZE = (512, 512)
    LANGUAGE\_MODEL\_ID = "Qwen/Qwen1.5-0.5B-Chat"
    CAPTION\_MODEL\_ID = "Salesforce/blip-image-captioning-base"
    DETECTION\_MODEL\_ID = "google/owlv2-base-patch16-ensemble"
    SEGMENTATION\_MODEL\_ID = "facebook/sam-vit-base"
    INPAINTING\_MODEL\_ID = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"
﻿
config = Config()

We define our configuration inside a class. This class acts as a central place to store various important settings and parameters that will be used throughout our image editing or generation process.

def delete\_model(model: torch.nn.Module):
    model.to("cpu")
    del model
    torch.cuda.empty\_cache()

The `delete_model` function aims to thoroughly remove a PyTorch model (`torch.nn.Module`) from GPU memory, ensuring resources are freed up for other tasks.

* * *

### [**Loading and Preparing Input Images for Editing**](#TOC-h3-loading-and-preparing-input-images-for-editing)

def load\_image(image\_url, image\_size = (512, 512)):
    raw\_image = Image.open(requests.get(image\_url, stream=True).raw).convert("RGB")
﻿
    image = raw\_image.resize(image\_size)
    return image

The `load_image` function downloads an image from a web address and prepares it for use in your image processing tasks. It first fetches the image using the requests library and opens it with the PIL (`Image`) library. The image is then converted to standard RGB format (if necessary) for compatibility. Finally, it’s resized according to the specified `image_size` parameter and returned, ready to be used by other parts of your code.

\# Load the image from the url and get the text prompt
image = load\_image(image\_url=IMAGE\_URL, image\_size=config.IMAGE\_SIZE)

Let’s load the image and visualize it. The image is shown in **Figure 2**.

plt.figure(figsize=(5, 5))
plt.imshow(image)
plt.axis("off")
plt.show()

[![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20500'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/eHYwcAqKd87BD88iwZFJATkLoMO1z5YveIavSFw3H2V4NBO0OyCJI-rbhx5qnUTbc1XwgUEifZDRYzWGfl5MLGqU82HEU5wgrdMlMa0pXoU1wdQ3PQnFJHt3ZFGCZR3c1GyX9Ny90KG_eoGVa6pDU5o)

**Figure 2:** The original image (source: image by the authors).

* * *

### [**Generating Captions for Image Understanding**](#TOC-h3-generating-captions-for-image-understanding)

def run\_caption\_model(model\_id: str, image: Image, device: str = "cuda"):
    caption\_model = BlipForConditionalGeneration.from\_pretrained(model\_id).to(device)
    caption\_processor = BlipProcessor.from\_pretrained(model\_id)

    inputs = caption\_processor(image, return\_tensors="pt").to(device)
    with torch.no\_grad():
        outputs = caption\_model.generate(\*\*inputs, max\_new\_tokens=50)

    caption = caption\_processor.decode(outputs\[0\], skip\_special\_tokens=True)
    delete\_model(caption\_model)
    return caption

The `run_caption_model` function takes a model identifier, an image, and a device preference (“cuda” for GPU or “cpu”). It uses the Hugging Face `transformers` library to load a pre-trained BLIP image captioning model specified by the `model_id`. The `BlipProcessor` prepares the image into a format the model expects. The model then generates a caption (up to 50 tokens) describing the image. Finally, to conserve memory, the captioning model is deleted. The generated caption is returned as the result.

\# Image captioning
caption = run\_caption\_model(
    model\_id=config.CAPTION\_MODEL\_ID, image=image, device=DEVICE
)

Let’s run the caption model and print the caption of the image.

print("#"\*100)
print(f"Using caption model: {config.CAPTION\_MODEL\_ID}")
print(f"Caption: {caption}")
print("#"\*100)

#################################################
Using caption model: Salesforce/blip-image-captioning-base
Caption: a bottle of beer sitting on a bed
#################################################

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC-h2-summary)
------------------------------

In this first installment, we laid the foundation for our image-editing adventure. We learned the importance of image captioning for understanding the content of our images. We also understood how the end-to-end workflow must be structured to achieve something like this.

To recap, we would need the following:

*   Vision model for image captioning
*   Language model for entity extraction from the prompt
*   Object detection model for detecting the entity in the image
*   Segmentation model for creating the mask
*   Diffusion model for performing the image inpainting

To recreate generative fill, we would need about five open-source models. In this tutorial, we saw the segment about creating the image-captioning workflow. The image captioning workflow is one of the most crucial steps for the entire pipeline. Whether the image caption is good or not will determine how well the image inpainting works.

We used the `Salesforce/blip-image-captioning-base` from the Hugging Face Hub since it is readily usable in a free-tier Colab notebook.

* * *

### [**Next Steps: Building an End-to-End Workflow for Image Editing**](#TOC-h3-next-steps-building-an-end-to-end-workflow-for-image-editing)

In the next part of this tutorial, we will look at the end-to-end workflow and understand how we can stitch together different models to create a pipeline where we can input a text and an image and edit the image based on the text.

* * *

### [**Citation Information**](#TOC-h3-citation-information)

**A. R. Gosthipaty and R. Raha.** “Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1,” _PyImageSearch_, P. Chugh, S. Huot, and K. Kidriavsteva, eds., 2024, [https://pyimg.co/jbsn9](https://pyimg.co/jbsn9)

@incollection{ARG-RR\_2024\_Open-Source-Implementation-Generative-Fill-part1,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva},
  year = {2024},
  url = {https://pyimg.co/jbsn9},
}

* * *

![Image 3: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.