TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is <class 'NoneType'>

Dmitr: <p>Hello!<br>
I’m trying to replace the background of a picture using Stable Diffusion.<br>
But I have the following problem:<br>
<code>TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is &lt;class 'NoneType'&gt;</code><br>
Code</p>
<pre><code class="lang-auto">import os
from dotenv import load_dotenv
import torch
import numpy as np
from diffusers import AutoPipelineForInpainting, EulerDiscreteScheduler, ControlNetModel, AutoencoderKL
from PIL import Image, ImageOps
import rembg

load_dotenv()
SDV5_MODEL_PATH = os.getenv('SDV5_MODEL_PATH')
SAVE_PATH = os.getenv('SAVE_PATH')
KANDINSKY_MODEL_PATH=os.getenv('KANDINSKY_MODEL_PATH')
REV_ANIMATED_MODEL_PATH=os.getenv('REV_ANIMATED_MODEL_PATH')
CONTROLNET_MODEL_PATH=os.getenv('CONTROLNET_MODEL_PATH')
MMIX_MODEL_PATH=os.getenv('MMIX_MODEL_PATH')
SAM_MODEL_PATH = os.getenv('SAM_MODEL_PATH')
VAE_MODEL_PATH = os.getenv('VAE_MODEL_PATH')
CANNY_MODEL_PATH=os.getenv('CANNY_MODEL')

prompt = 'best quality, highres, high definition masterpiece, photorealistic, foggy winter forest'
negative_prompt = 'nsfw, people, worst quality, low quality, normal quality, lowres,watermark, monochrome, low resolutio, extra limbs'
num_inference_steps = 40
base_img = 'C:\\Users\\dovsy\\Downloads\\cat.png'
num_of_img_per_prompt = 1

controlnet = ControlNetModel.from_pretrained(
    CANNY_MODEL_PATH,
    torch_dtype=torch.float32,
    varient="fp32"
)

def binary_mask(init_image):
    #init_image = load_image(url).resize((512, 768))
    input_array = np.array(init_image)
    mask_array = rembg.remove(input_array, only_mask=True)
    mask_image = Image.fromarray(mask_array)
    mask_image = ImageOps.invert(mask_image)
    return mask_image

def toPill(image):
    minv = np.amin(image)
    maxv = np.amax(image)
    image = image - minv
    image = image / (maxv - minv)
    image = image * 255
    img = Image.fromarray(image.astype(np.uint8))
    return img

if __name__ == '__main__':
pipeline = AutoPipelineForInpainting.from_pretrained(
        REV_ANIMATED_MODEL_PATH,
        controlnet=controlnet,
torch_dtype=torch.float32,
        vae=AutoencoderKL.from_pretrained(
                 VAE_MODEL_PATH,
                 subfolder=None)
).to('cpu')

size = (1024, 704)
init_image = Image.open(base_img)
init_image = toPill(init_image)
bin_mask=binary_mask(init_image)

print(type(init_image))
    print(type(bin_mask))
    print(init_image.size)
    print(bin_mask.size)
image = pipeline(
                     prompt=prompt,
                     scheduler=EulerDiscreteScheduler.from_config(pipeline.scheduler.config),
                     negative_prompt=negative_prompt,
                     width=1024,
                     height=704,
                     num_inference_steps=70,
                     image=init_image,
                     mask_image=bin_mask,
                     #control_image=control_image,
                     guidance_scale=15,
                     strength=0.8
                     #generator=torch.manual_seed(189123)
                     ).images[0]

    image.save('test1.jpg')
</code></pre>
<p>My error:</p>
<pre><code class="lang-auto">&lt;class 'PIL.Image.Image'&gt;
&lt;class 'PIL.Image.Image'&gt;
(1024, 704)
(1024, 704)
Traceback (most recent call last):
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\background_change.py", line 155, in &lt;module&gt;
    image = pipeline(
            ^^^^^^^^^
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 1174, in __call__
    self.check_inputs(
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 673, in check_inputs
    self.check_image(image, prompt, prompt_embeds)
  File "C:\Users\dovsy\Documents\pythonProject\stable_diffusion\venv\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_inpaint.py", line 775, in check_image
    raise TypeError(
TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is &lt;class 'NoneType'&gt;

Process finished with exit code 1
</code></pre>
<p>Photo that i used:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d94d143cd391652dec01e7c92914a51636c389c3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d94d143cd391652dec01e7c92914a51636c389c3" title="cat"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d94d143cd391652dec01e7c92914a51636c389c3.jpeg" alt="cat" data-base62-sha1="v0kKDrE791sQ7I7ao62DI7EvpKj" width="690" height="474" data-dominant-color="668671"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cat</span><span class="informations">1024×704 28.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance for your reply!</p>

KFrank: <p>Hi Dmitr!</p>
<aside class="quote no-group quote-modified" data-username="Dmitr" data-post="1" data-topic="217159">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/13edae/48.png" class="avatar"> Dmitr:</div>
<blockquote>
<p><code>TypeError: image must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is &lt;class 'NoneType'&gt;</code><br>
Code</p>
<pre><code class="lang-auto">controlnet = ControlNetModel.from_pretrained(
...
pipeline = AutoPipelineForInpainting.from_pretrained(
        REV_ANIMATED_MODEL_PATH,
        controlnet=controlnet,
...
image = pipeline(
...
                     #control_image=control_image,
</code></pre>
</blockquote>
</aside>
<p>Speculation on my part:</p>
<p>It looks like you are building <code>pipeline</code> with an explicit <code>controlnet</code>, but then calling<br>
<code>pipeline</code> without an explicit <code>control_image</code> argument.  I imagine that when you<br>
don’t specify the <code>control_image</code> argument, it defaults to <code>None</code>.  But then <code>controlnet</code><br>
in <code>pipeline</code> gets <code>None</code> passed on to it as its <code>control_image</code> and raises the <code>NoneType</code><br>
error.</p>
<p>To test this theory, try passing in an actual image, e.g., <code>control_image = init_image</code>,<br>
and see if the error goes away, or try passing in some non-image object, e.g.,<br>
<code>control_image = scheduler</code>, and see if you get an analogous wrong-type error,<br>
but with some actual object type, rather than <code>NoneType</code>.</p>
<p>If this is what’s going on, you either have to pass in an appropriate <code>control_image</code> or<br>
build a <code>pipeline</code> that doesn’t make use of a <code>controlnet</code>.</p>
<p>Good luck!</p>
<p>K. Frank</p>