Title: Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2 - PyImageSearch

URL Source: https://pyimagesearch.com/2024/03/25/step-by-step-guide-to-open-source-implementation-of-generative-fill-part-2/

Published Time: 2024-03-25T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2](#h2-step-by-step-guide-to-open-source-implementation-of-generative-fill-part-2)

*   [Configuring Your Development Environment](#h3-configuring-your-development-environment)
*   [Need Help Configuring Your Development Environment?](#h3-need-help-configuring-your-development-environment)
*   [Implementation](#h3-implementation)
*   [Essential Setup: Libraries and Imports for Generative Image Editing](#h3-essential-setup-libraries-and-imports-for-generative-image-editing)
*   [How to Prompt Language Models for Image Editing](#h3-how-to-prompt-language-models-for-image-editing)

*   [Purpose](#h4-purpose)
*   [Initial Setup](#h4-initial-setup)
*   [Output from the Language Model](#h4-output-from-the-language-model)
*   [Optimizing Memory Management](#h4-optimizing-memory-management)
*   [Testing the Language Model](#h4-testing-the-language-model)

*   [Image Segmentation with SAM (Segment Anything Model)](#h3-image-segmentation-with-sam-segment-anything-model)

*   [Object Detection with OWL-ViT (Vision Transformer for Open-World Localization)](#h4-object-detection-with-owl-vit-vision-transformer-for-open-world-localization)
*   [Image Segmentation from Object Detection Bounding Boxes](#h4-image-segmentation-from-object-detection-bounding-boxes)

*   [Inpaint the Image Using a Segmentation Mask and Diffusers](#h3-inpaint-the-image-using-a-segmentation-mask-and-diffusers)

*   [Preparing for Inpainting: Setup and Techniques](#h4-preparing-for-inpainting-setup-and-techniques)
*   [Executing Image Inpainting](#h4-executing-image-inpainting)
*   [Revealing the Final Image: Output and Cleanup](#h4-revealing-the-final-image-output-and-cleanup)

*   [Summary](#h2-summary)

*   [Citation Information](#h3-citation-information)

* * *

In this tutorial, you will learn to edit text-based images using Open-Source models like Segment Anything (SAM), OWL-ViT (Vision Transformer for Open-World Localization), and SDXL (Stable Diffusion XL) inpainting.

[![Image 1: open-source-implementation-generative-fill-part2-featured.png](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20603%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2024/03/open-source-implementation-generative-fill-part2-featured.png)

This lesson is the last of a 2-part series on **Open-Sourcing Generative Fill**:

1.  [_Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 1_](https://pyimg.co/jbsn9)
2.  [**_Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2_**](https://pyimg.co/5h9t1) **(this tutorial)**

**To learn how to create your own text-based image editing product, just keep reading.**

* * *

[**Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2**](#TOC-h2-step-by-step-guide-to-open-source-implementation-of-generative-fill-part-2)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

Welcome back! In Part 1, we prepared our image for editing. Now, we’ll breathe life into our text-based image manipulation pipeline. We’ll explore how to use large language models to intelligently modify your edit prompts, creating instructions that directly target image content.

We’ll then delve into segmentation techniques, learning how to isolate the specific parts of the image you want to change. Finally, we’ll create the inpainting pipeline, where our model seamlessly updates the image based on our enhanced prompts and the identified segments.

* * *

### [**Configuring Your Development Environment**](#TOC-h3-configuring-your-development-environment)

To follow this guide, you need to have the `diffusers`, `accelerate`, and `transformers` libraries installed on your system.

Luckily, these are all pip-installable:

$ pip install --upgrade -qq diffusers accelerate transformers

* * *

### [**Need Help Configuring Your Development Environment?**](#TOC-h3-need-help-configuring-your-development-environment)

[![Image 2: Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join PyImageSearch University — you’ll be up and running with this tutorial in minutes.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_immediately_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation is required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Implementation**](#TOC-h3-implementation)

We will learn about stitching together all the parts and creating an end-to-end pipeline for image editing.

We will do this by:

*   Accepting an `edit_prompt` and an image as input
*   Using a Vision Model to caption the image
*   Passing the `edit_prompt` through a language model to extract the source entity
*   Creating a `replacement_caption` where the source entity of the original image is swapped with the target entity in the `edit_prompt`
*   Using the source entity to create a segmentation mask using `OWL-VIT` and `SAM`
*   Using the mask and the `replacement_caption` for image inpainting

* * *

### [**Essential Setup: Libraries and Imports for Generative Image Editing**](#TOC-h3-essential-setup-libraries-and-imports-for-generative-image-editing)

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    Owlv2ForObjectDetection,
    Owlv2Processor,
    SamModel,
    SamProcessor,
)
from diffusers import AutoPipelineForInpainting

Let’s import our essential packages and deep dive into the code.

* * *

### [**How to Prompt Language Models for Image Editing**](#TOC-h3-how-to-prompt-language-models-for-image-editing)

def run\_lm\_model(model\_id: str, caption: str, edit\_prompt: str, device: str = "cuda"):
    language\_model = AutoModelForCausalLM.from\_pretrained(model\_id, device\_map="auto")
    tokenizer = AutoTokenizer.from\_pretrained(model\_id)
﻿
    messages = \[
        {
            "role": "system",
            "content": "Follow the examples and return the expected output",
        },
        {
            "role": "user",
            "content": "Caption: a blue sky with fluffy clouds\\nQuery: Make the sky stormy",
        },
        {
            "role": "assistant",
            "content": "A: sky\\nB: a stormy sky with heavy gray clouds, torrential rain, gloomy, overcast",
        },
        {
            "role": "user",
            "content": "Caption: a cat sleeping on a sofa\\nQuery: Change the cat to a dog",
        },
        {
            "role": "assistant",
            "content": "A: cat\\nB: a dog sleeping on a sofa, cozy and comfortable, snuggled up in a warm blanket, peaceful",
        },
        {
            "role": "user",
            "content": "Caption: a snowy mountain peak\\nQuery: Replace the snow with greenery",
        },
        {
            "role": "assistant",
            "content": "A: snow\\nB: a lush green mountain peak in summer, clear blue skies, birds flying overhead, serene and majestic",
        },
        {
            "role": "user",
            "content": "Caption: a vintage car parked by the roadside\\nQuery: Change the car to a modern electric vehicle",
        },
        {
            "role": "assistant",
            "content": "A: car\\nB: a sleek modern electric vehicle parked by the roadside, cutting-edge design, environmentally friendly, silent and powerful",
        },
        {
            "role": "user",
            "content": "Caption: a wooden bridge over a river\\nQuery: Make the bridge stone",
        },
        {
            "role": "assistant",
            "content": "A: bridge\\nB: an ancient stone bridge over a river, moss-covered, sturdy and timeless, with clear waters flowing beneath",
        },
        {
            "role": "user",
            "content": "Caption: a bowl of salad on the table\\nQuery: Replace salad with soup",
        },
        {
            "role": "assistant",
            "content": "A: bowl\\nB: a bowl of steaming hot soup on the table, scrumptious, with garnishing",
        },
        {
            "role": "user",
            "content": "Caption: a book on a desk surrounded by stationery\\nQuery: Remove all stationery, add a laptop",
        },
        {
            "role": "assistant",
            "content": "A: stationery\\nB: a book on a desk with a laptop next to it, modern study setup, focused and productive, technology and education combined",
        },
        {
            "role": "user",
            "content": "Caption: a cup of coffee on a wooden table\\nQuery: Change coffee to tea",
        },
        {
            "role": "assistant",
            "content": "A: cup\\nB: a steaming cup of tea on a wooden table, calming and aromatic, with a slice of lemon on the side, inviting",
        },
        {
            "role": "user",
            "content": "Caption: a small pen on a white table\\nQuery: Change the pen to an elaborate fountain pen",
        },
        {
            "role": "assistant",
            "content": "A: pen\\nB: an elaborate fountain pen on a white table, sleek and elegant, with intricate designs, ready for writing",
        },
        {
            "role": "user",
            "content": "Caption: a plain notebook on a desk\\nQuery: Replace the notebook with a journal",
        },
        {
            "role": "assistant",
            "content": "A: notebook\\nB: an artistically decorated journal on a desk, vibrant cover, filled with creativity, inspiring and personalized",
        },
        {"role": "user", "content": f"Caption: {caption}\\nQuery: {edit\_prompt}"},
    \]
﻿
    text = tokenizer.apply\_chat\_template(
        messages, tokenize=False, add\_generation\_prompt=True
    )
    model\_inputs = tokenizer(\[text\], return\_tensors="pt").to(device)
    with torch.no\_grad():
        generated\_ids = language\_model.generate(
            model\_inputs.input\_ids, max\_new\_tokens=512, temperature=0.0, do\_sample=False
        )
﻿
    generated\_ids = \[
        output\_ids\[len(input\_ids) :\]
        for input\_ids, output\_ids in zip(model\_inputs.input\_ids, generated\_ids)
    \]
﻿
    output\_generation = tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)\[
        0
    \]
﻿
    output\_generation\_a, output\_generation\_b = output\_generation.split("\\n")
    to\_replace = output\_generation\_a\[2:\].strip()
    replaced\_caption = output\_generation\_b\[2:\].strip()
﻿
    delete\_model(language\_model)
    return to\_replace, replaced\_caption

Here, we describe our `run_lm_model` function.

* * *

#### [**Purpose**](#TOC-h4-purpose)

This function uses a large language model (LLM) to intelligently edit an existing image caption based on a text-based editing instruction (the `edit_prompt`).

* * *

#### [**Initial Setup**](#TOC-h4-initial-setup)

*   It loads a language model `AutoModelForCausalLM` and its tokenizer `AutoTokenizer` from the Hugging Face repository using the specified `model_id`.
*   A structured list of messages is created that simulates a conversation. Examples of captions and edits are provided to guide the model in performing the desired transformation.

* * *

#### [**Output from the Language Model**](#TOC-h4-output-from-the-language-model)

*   The current caption and edit prompt are added to the example conversation. This complete conversation history is formatted and tokenized to be fed into the language model.
*   The language model generates a response, which is expected to contain two parts:
    *   The text to be replaced in the original caption.
    *   The replacement text.
*   The model’s output is cleaned up to extract these two elements.

The technique of prompting used here is called few-shot prompting, where a few high-quality examples are passed to the prompt along with the query to help the model understand what kind of response to provide.

In this case, we are prompting the model with examples that show how it should return the response. For example, when the user types in:

Caption: a small pen on a white table\\nQuery: Change the pen to an elaborate fountain pen

The model is supposed to return:

A: pen\\nB: an elaborate fountain pen on a white table, sleek and elegant, with intricate designs, ready for writing

This means the original object to be replaced will be identified and returned after the prefix “**A:**” and the changed prompt with the original object replaced with the new object will be returned after the prefix “**B:**”

The prefixes allow us to segregate the response and break it using simple string manipulation techniques.

* * *

#### [**Optimizing Memory Management**](#TOC-h4-optimizing-memory-management)

As before, the language model is deleted using `delete_model` to conserve resources.

* * *

#### [**Testing the Language Model**](#TOC-h4-testing-the-language-model)

Now, let’s call the function to see if it returns the text to be replaced in the original caption and the suggested replacement.

\# Language model
to\_replace, replaced\_caption = run\_lm\_model(
    model\_id=config.LANGUAGE\_MODEL\_ID,
    caption=caption,
    edit\_prompt=EDIT\_PROMPT,
    device=DEVICE,
)

Let’s print the output.

print("#"\*100)
print(f"Using llm: {config.LANGUAGE\_MODEL\_ID}")
print(f"Edit Prompt: {EDIT\_PROMPT}")
print(f"Object to replace: {to\_replace}")
print(f"Caption for inpaint: {replaced\_caption}")
print("#"\*100)

We see the language model-id, the edit-prompt, and the extracted entities.

####################################
Using llm: Qwen/Qwen1.5-0.5B-Chat
Edit Prompt: change the bottle to a wine
Object to replace: bottle
Caption for inpaint: a wine glass sitting on a bed, elegant and sophisticated, with a glass of red wine on top, a touch of elegance and sophistication.
####################################

* * *

### [**Image Segmentation with SAM (Segment Anything Model)**](#TOC-h3-image-segmentation-with-sam-segment-anything-model)

As we mentioned previously, we would need to create a mask of the object for the inpainting pipeline to work. To create a mask, we first need to create a segmentation mask of this object inside the image. Here, we use the Segment Anything Model from Meta.

The [Segment Anything Model (SAM)](https://segment-anything.com/), developed by Meta AI’s FAIR (Fundamental AI Research) lab, is a recent breakthrough in computer vision. This state-of-the-art model excels at segmenting objects in images, identifying their precise locations with exceptional detail. Unlike traditional segmentation models that require specific training for each object, SAM boasts “zero-shot performance,” meaning it can [tackle various segmentation tasks](https://github.com/topics/segment-anything-model) without additional training data. SAM is open-source and available for experimentation

def run\_segmentation\_pipeline(
    detection\_model\_id: str,
    segmentation\_model\_id: str,
    to\_replace: str,
    image: Image,
    device: str = "cuda",
):
    processor = Owlv2Processor.from\_pretrained(detection\_model\_id)
    od\_model = Owlv2ForObjectDetection.from\_pretrained(detection\_model\_id).to(device)
    text\_queries = \[to\_replace\]
    inputs = processor(text=text\_queries, images=image, return\_tensors="pt").to(device)
﻿
    with torch.no\_grad():
        outputs = od\_model(\*\*inputs)
        target\_sizes = torch.tensor(\[image.size\]).to(device)
        results = processor.post\_process\_object\_detection(
            outputs, threshold=0.1, target\_sizes=target\_sizes
        )\[0\]
﻿
    boxes = results\["boxes"\].tolist()
    delete\_model(od\_model)
﻿
    seg\_model = SamModel.from\_pretrained(segmentation\_model\_id).to(device)
    processor = SamProcessor.from\_pretrained(segmentation\_model\_id)
﻿
    input\_boxes = \[boxes\]
    inputs = processor(image, input\_boxes=input\_boxes, return\_tensors="pt").to(device)
﻿
    with torch.no\_grad():
        outputs = seg\_model(\*\*inputs)
﻿
    mask = processor.image\_processor.post\_process\_masks(
        outputs.pred\_masks.cpu(),
        inputs\["original\_sizes"\].cpu(),
        inputs\["reshaped\_input\_sizes"\].cpu(),
    )\[0\]
﻿
    mask = torch.max(mask\[:, 0, ...\], dim=0, keepdim=False).values
    segmentation\_mask = Image.fromarray(mask.numpy())
﻿
    delete\_model(seg\_model)
    return segmentation\_mask

The `run_segmentation_pipeline` function isolates the specific part of an image that needs to be modified based on the text you provide in `to_replace`.

However, SAM isn’t capable of isolating or segmenting parts of an image based on textual prompts (yet). So, we need a way to ground SAM with an object detection model that can take in a text input and provide bounding boxes. These bounding boxes are then passed as inputs to SAM for segmentation.

We will use the OWLv2 model for this. OWLv2 was proposed in [Scaling Open-Vocabulary Object Detection](https://arxiv.org/abs/2306.09683). OWLv2 scales up OWL-ViT (Vision Transformer for Open-World Localization) using self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. This results in large gains over the previous state-of-the-art for zero-shot object detection.

* * *

#### [**Object Detection with OWL-ViT (Vision Transformer for Open-World Localization)**](#TOC-h4-object-detection-with-owl-vit-vision-transformer-for-open-world-localization)

First, specialized object detection models (`Owlv2Processor` and `Owlv2ForObjectDetection`) are loaded.

The object detection model analyzes the image along with the text description of what needs to be replaced (`to_replace`). This helps it identify the relevant region within the image.

The model outputs bounding boxes that enclose the object to be edited.

* * *

#### [**Image Segmentation from Object Detection Bounding Boxes**](#TOC-h4-image-segmentation-from-object-detection-bounding-boxes)

Next, an image segmentation model `SamModel` with its `SamProcessor` is loaded.

The bounding boxes determined earlier are used to focus the segmentation model only on the relevant part of the image.

The segmentation model generates a detailed mask that outlines the exact pixels belonging to the object that needs to be changed.

The function returns this `segmentation_mask`. This mask can be used later to isolate the object within the image for editing or replacement.

\# Segmentation pipeline
segmentation\_mask = run\_segmentation\_pipeline(
    detection\_model\_id=config.DETECTION\_MODEL\_ID,
    segmentation\_model\_id=config.SEGMENTATION\_MODEL\_ID,
    to\_replace=to\_replace,
    image=image,
    device=DEVICE,
)

Let’s run the segmentation pipeline and visualize the outputs. The segmentation mask is shown in **Figure 1**.

print("#"\*100)
print(f"Using detection model: {config.DETECTION\_MODEL\_ID}")
print(f"Using segmentation model: {config.SEGMENTATION\_MODEL\_ID}")
print("#"\*100)

##################################################
Using detection model: google/owlv2-base-patch16-ensemble
Using segmentation model: facebook/sam-vit-base
##################################################

plt.figure(figsize=(5, 5))
plt.imshow(segmentation\_mask)
plt.axis("off")
plt.show()

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20450%20450'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/H744oepBNrwPQ_4u8OgSGavCPcXlatWbZwiL2Nor6E_eZ7259kSBrAMRk0Ta9QWMUUCy6yEzVoO0lpiaz6Lh9I_LLGKxJ7TlxOzEjKTUfm56DTf7mRAk1YqaLG_PtE7268FsV_-BG6s-ZKDQZ3p1JPk)

**Figure 1:** SAM-generated segmentation map (source: image by the authors).

* * *

### [**Inpaint the Image Using a Segmentation Mask and Diffusers**](#TOC-h3-inpaint-the-image-using-a-segmentation-mask-and-diffusers)

Finally, we arrive at the junction where we have all the pieces needed to run our inpainting pipeline.

Just to be clear, let us revise what these are:

*   An input image where something needs to be replaced (inpainted)
*   A mask of the area that needs to be replaced (inpainted)
*   A prompt describing what needs to be inpainted in

The user supplied the input image. We have created the mask image using a combination of an Object Detection and a Segmentation Model. The Language Model supplies the prompt describing what needs to be inpainted. Let us see how we can stitch these together in an inpainting pipeline for Diffusers.

If you are new to image inpainting or want to learn more about these pipelines, check out our blog post on Image Inpainting with Diffusers: [Inpaint Images: AI Photo Restoration](https://pyimagesearch.com/2024/03/11/understanding-tasks-in-diffusers-part-2/).

def run\_inpainting\_pipeline(
    inpainting\_model\_id: str,
    image: Image,
    mask: Image,
    replaced\_caption: str,
    image\_size: Tuple\[int, int\],
    generator: torch.Generator,
    device: str = "cuda",
):
    pipeline = AutoPipelineForInpainting.from\_pretrained(
        inpainting\_model\_id,
        torch\_dtype=torch.float16,
        variant="fp16",
    ).to(device)

    prompt = replaced\_caption

    negative\_prompt = """lowres, bad anatomy, bad hands,
    text, error, missing fingers, extra digit, fewer digits,
    cropped, worst quality, low quality"""

    output = pipeline(
        prompt=prompt,
        image=image,
        height=image\_size\[0\],
        width=image\_size\[1\],
        mask\_image=mask,
        negative\_prompt=negative\_prompt,
        guidance\_scale=7.5,
        strength=1.0,
        generator=generator,
    ).images\[0\]
    delete\_model(pipeline)
    return output

The `run_inpainting_pipeline` function takes an image, a segmentation mask (isolating the area to change), and a new text description (`replaced_caption`), and aims to modify the specified part of the image accordingly.

* * *

#### [**Preparing for Inpainting: Setup and Techniques**](#TOC-h4-preparing-for-inpainting-setup-and-techniques)

*   An image inpainting pipeline (`AutoPipelineForInpainting`) is loaded from the Hugging Face model repository. This pipeline is optimized for speed (using `torch_dtype=torch.float16` and `variant="fp16"`).
*   The `prompt` is set to `replaced_caption`, providing the textual description of the desired modification.
*   A `negative_prompt` is included to discourage the model from generating undesirable results (e.g., bad anatomy, text, etc.).

* * *

#### [**Executing Image Inpainting**](#TOC-h4-executing-image-inpainting)

*   The core of the process happens with the `pipeline` call. It takes the image, the mask (to focus on the correct region), the prompts, and a few parameters to guide the modification process:
    *   `guidance_scale`: controls how closely the model adheres to the text prompt.
    *   `strength`: determines how much of the masked area should be newly generated versus blended from the original image.
*   The pipeline produces a modified image.

* * *

#### [**Revealing the Final Image: Output and Cleanup**](#TOC-h4-revealing-the-final-image-output-and-cleanup)

The generated image is returned, and the pipeline is deleted (`delete_model`) to free up memory.

\# Inpainting pipeline
output = run\_inpainting\_pipeline(
    inpainting\_model\_id=config.INPAINTING\_MODEL\_ID,
    image=image,
    mask=segmentation\_mask,
    replaced\_caption=replaced\_caption,
    image\_size=config.IMAGE\_SIZE,
    generator=GENERATOR,
    device=DEVICE,
)

It is time to see the final result. The final image is shown in **Figure 2**.

print("#"\*100)
print(f"Using diffusion model: {config.INPAINTING\_MODEL\_ID}")
print("#"\*100)

########################################################
Using diffusion model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
########################################################

plt.figure(figsize=(5, 5))
plt.imshow(output)
plt.axis("off")
plt.show()

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20400%20400'%3E%3C/svg%3E)](https://lh7-us.googleusercontent.com/xc398IZ47KdAhQheZY6LNVH2RIbyPoBJQCogjov-P_knZFss6frviT8eBJSCZ8ZaybYeaQd3w9YCJrB1iTaRQivBCi_JZrc3P78pUgM6bEIIeKjzHPihdcbQwo_ZO1mnKMnCKUKnvVjMEwX3M9wauVk)

**Figure 2:** The changed image (source: image by the authors).

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC-h2-summary)
------------------------------

This second part brought our open-source generative fill journey to an exciting climax. Over the past year, we have seen a number of Open-Source Models become relevant and extremely useful for the community. Here, we understand how to combine several of these models and use their inference capabilities to create an end-to-end pipeline for something useful.

Let us wrap up by recalling what we achieved. We took an edit prompt and an image as inputs and created an edited image based on the prompt. Underneath this end-to-end pipeline, we used a Language Model, an Object Detection Model, a Segmentation Model, and finally, a Diffusion Model for inpainting.

The most interesting part, and probably the most overlooked aspect, is that we can choose to plug and play our own models into this pipeline or create a fine-tuned version of models capable of achieving a more focused output.

What other fun implementations would you like to see? Let us know on our socials @pyimagesearch

* * *

### [**Citation Information**](#TOC-h3-citation-information)

**A. R. Gosthipaty and R. Raha.** “Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2,” _PyImageSearch_, P. Chugh, S. Huot, and K. Kidriavsteva, eds., 2024, [https://pyimg.co/5h9t1](https://pyimg.co/5h9t1)

@incollection{ARG-RR\_2024\_Open-Source-Implementation-Generative-Fill-part2,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Step-by-Step Guide to Open-Source Implementation of Generative Fill: Part 2},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva},
  year = {2024},
  url = {https://pyimg.co/5h9t1},
}

* * *

![Image 5: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.