Repository: huggingface/diffusers
Pull request: [IP Adapter] Fix object has no attribute with image encoder
# What does this PR do?

IP Adapter FaceID model don't need the image encoder so it's not loaded and the image encoder its used to get the clip size for the Kolors version, this produces an error with FaceID models. This PR sets back the default 224 size for these models.

IMO it would be better to not load the `feature extractor` altogether if it's not used (not sure) but this was in the original implementation.

Fixes #9187

## Who can review?

@sayakpaul @detkov @fabiorigano 


Problem statement:
IP Adapter is not working with SDXL when using `image_encoder_folder=None` after Kolors update
### Describe the bug

When loading IP Adapter into SDXL Text2Image pipeline with `image_encoder_folder=None`, I encounter the following error:
```
clip_image_size = self.image_encoder.config.image_size
AttributeError: 'NoneType' object has no attribute 'config'
```

As I can see from the code changes in the `ip_adapter.py` file, the bug was introduced when implementing Kolors support. 

So I hope that @asomoza can fix that issue having his experience with Kolors.

### Reproduction

```python
import torch
from diffusers import StableDiffusionXLPipeline


pipeline = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter-FaceID", subfolder=None, weight_name="ip-adapter-faceid-plusv2_sdxl.bin", image_encoder_folder=None)
```

### Logs

```shell
{
	"name": "AttributeError",
	"message": "'NoneType' object has no attribute 'config'",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 9
      2 from diffusers import StableDiffusionXLPipeline
      5 pipeline = StableDiffusionXLPipeline.from_pretrained(
      6     \"stabilityai/stable-diffusion-xl-base-1.0\",
      7     torch_dtype=torch.float16,
      8 ).to(\"cuda\")
----> 9 pipeline.load_ip_adapter(\"h94/IP-Adapter-FaceID\", subfolder=None, weight_name=\"ip-adapter-faceid-plusv2_sdxl.bin\", image_encoder_folder=None)

File /opt/.cache/virtualenvs/diffusion-exps-R-NqVyK3-py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:114, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    111 if check_use_auth_token:
    112     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 114 return fn(*args, **kwargs)

File /opt/.cache/virtualenvs/diffusion-exps-R-NqVyK3-py3.10/lib/python3.10/site-packages/diffusers/loaders/ip_adapter.py:225, in IPAdapterMixin.load_ip_adapter(self, pretrained_model_name_or_path_or_dict, subfolder, weight_name, image_encoder_folder, **kwargs)
    223 # create feature extractor if it has not been registered to the pipeline yet
    224 if hasattr(self, \"feature_extractor\") and getattr(self, \"feature_extractor\", None) is None:
--> 225     clip_image_size = self.image_encoder.config.image_size
    226     feature_extractor = CLIPImageProcessor(size=clip_image_size, crop_size=clip_image_size)
    227     self.register_modules(feature_extractor=feature_extractor)

AttributeError: 'NoneType' object has no attribute 'config'"
}
```


### System Info

- 🤗 Diffusers version: 0.30.0
- Platform: Linux-5.15.0-1055-aws-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.14
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.24.5
- Transformers version: 4.40.1
- Accelerate version: 0.31.0
- PEFT version: 0.10.0
- Bitsandbytes version: 0.43.1
- Safetensors version: 0.4.3
- xFormers version: not installed
- Accelerator: NVIDIA A10G, 23028 MiB
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help?

@asomoza

Hints from issue discussion:
[The related docs page](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters#ip-adapter-face-id-models) says plus versions need to be passed `image_encoder`.
@tolgacangoz this is not the case. Please read https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#face-model
@detkov  thanks for reporting, this is a bug, I'll open a PR to fix it.

Commit message: fix

Patch to review:
diff --git a/src/diffusers/loaders/ip_adapter.py b/src/diffusers/loaders/ip_adapter.py
index 44c8c0a5181c..855b7765bd3e 100644
--- a/src/diffusers/loaders/ip_adapter.py
+++ b/src/diffusers/loaders/ip_adapter.py
@@ -222,7 +222,7 @@ def load_ip_adapter(
 
             # create feature extractor if it has not been registered to the pipeline yet
             if hasattr(self, "feature_extractor") and getattr(self, "feature_extractor", None) is None:
-                clip_image_size = self.image_encoder.config.image_size
+                clip_image_size = self.image_encoder.config.image_size if self.image_encoder is not None else 224
                 feature_extractor = CLIPImageProcessor(size=clip_image_size, crop_size=clip_image_size)
                 self.register_modules(feature_extractor=feature_extractor)
 


Code review comments:
diff_hunk: @@ -222,7 +222,7 @@ def load_ip_adapter(
 
             # create feature extractor if it has not been registered to the pipeline yet
             if hasattr(self, "feature_extractor") and getattr(self, "feature_extractor", None) is None:
-                clip_image_size = self.image_encoder.config.image_size
+                clip_image_size = self.image_encoder.config.image_size if self.image_encoder is not None else 224
path: src/diffusers/loaders/ip_adapter.py
text: @user1:
Let's put 224 in a variable name and use it? Additionally, a comment to explain why would be nice. 

I guess we don't need to test this? If not, then let's also add a test? 

@author:
sure, I'll add the test, if we had it this wouldn't have passed without notice.

@author:
changed my mind for at least this PR, we don't have a test for the image encoder or feature extractor, the tests just load the ip adapter weights and use `ip_adapter_image_embeds` . The whole `load_ip_adapter` function isn't covered in the tests.

I'll add in my TODO list to make a complete test for this later.

@user1:
Okay with that

@user2:
hi, thanks for adding me here, the change looks good to me

I just leave one comment: the `load_ip_adapter` function is covered in the slow tests in tests/pipelines/ip_adapters/test_ip_adapter_stable_diffusion.py. the same for the  `unload_ip_adapter` function. maybe it is better to test all of them in the fast tests (tests/pipelines/test_pipelines_common.py)?

@author:
oh yeah missed that one, you're correct, the only caveat is that in the slow tests we don't test the FaceID one for SDXL.

> maybe it is better to test all of them in the fast tests

IMO yes, we can use tiny versions like the other tests and use those functions instead of just using the `ip_adapter_image_embeds`  so we can catch any loading errors in the fast tests too.

@user2:
oh sorry, but I see the bug is on a different model, the Face ID Plus, that always requires the CLIP image encoder [docs - see the last code snippet](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters#ip-adapter-face-id-models).


@author:
what do you mean? In my understanding this only happens with the FaceID ones and not the FaceID Plus ones, if there's a image encoder present, the code from before this fix runs ok. 

@author:
I see what you mean, the original issue was with a Plus Model, but still this fix is needed when using the normal FaceID one right?

@user3:
> but still this fix is needed when using the normal FaceID one right

Sure

@user2:
> I see what you mean, the original issue was with a Plus Model, but still this fix is needed when using the normal FaceID one right?

exactly

@user3:
> Face ID Plus, that always requires the CLIP image encoder

Duplicating the answer on this: https://github.com/huggingface/diffusers/issues/9187#issuecomment-2293214533

So it's not actually strictly required and one can use the normal FaceID or FaceID Plus without `image_encoder`