Repository: huggingface/diffusers
Pull request: Support for manual CLIP loading in StableDiffusionPipeline - txt2img.
# What does this PR do?
Fixes #3822 

This pull request introduces a feature that enhances the loading mechanism of the **`CLIP` model** when used in conjunction with `StableDiffusionPipeline.from_ckpt()`. This PR affects **only the `txt2img`** part of the mentioned pipeline. Users now have the flexibility to manually load the CLIP model and tokenizer, thereby bypassing the force loading behavior. This PR resolves the challenge of creating a fully portable solution, as the CLIP model and tokenizer would previously end up in the cache.
With this enhancement, users can now specify their desired CLIP model and tokenizer location as follows:
```python
# Users have the option to choose the official Hugging Face repository: `openai/clip-vit-large-patch14`,
# or provide the local path to load CLIP from.
clip_text_model = CLIPTextModel.from_pretrained("repo/id/or/path/to/local/folder")
clip_tokenizer = CLIPTokenizer.from_pretrained("repo/id/or/path/to/local/folder")

# Subsequently, these parameters are passed into the `StableDiffusionPipeline` function.
pipeline = StableDiffusionPipeline.from_ckpt("path/to/single/safetensors/or/bin/file", 
                                              clip_text_model=clip_text_model,
                                              clip_tokenizer=clip_tokenizer)
```
It is important to note that this pull request does not impact the behavior if users choose not to provide any parameters for `clip_text_model` or `clip_tokenizer`. In such cases, the code will behave as it did before the PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [x] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. => Discussed in #3822 
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten 
@sayakpaul 

Problem statement:
StableDiffusionPipeline() and CLIP "cooperation"
Hello!
First of all, amazing work, guys! Absolutely love the whole `transformers`, `diffusers` and others for its simplicity! ❤️ 

I would suggest an improvement to a `StableDiffusionPipeline()` function to allow for portable & offline execution on any machine if the required files are already in specified locations.
Right now, when running the following code, `StableDiffusionPipeline()`, it downloads an additional model: `openai/clip-vit-large-patch14`. 
```python
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_ckpt("path/to/single/safetensors/or/bin/file")
```
The problem is that the CLIP model downloads to cache, so it is impossible to create a truly portable & offline solution.
I suggest 2 solutions:
1. We load CLIP model ourselves and pass it as an argument to the `StableDiffusionPipeline`. If nothing is being passed, we go to current behavior. I definitely prefer this solution, as it does not break any previous code.
2. The `cache_dir` that we pass to `StableDiffusionPipeline` will also get passed to the CLIP model when it is called inside of `StableDiffusionPipeline`. This one, I am not so sure about, as there might be some edge cases, where this actually breaks smth.

Hints from issue discussion:
Hey @WadRex,

Very much agree, it would be better to not **force load** the `openai/clip-vit-large-patch14` checkpoint. Think ideally we should actually convert the text encoder completely from the checkpoint file to be sure we have 1-to-1 the same weights. Currently I'm not sure I'll find time to handle this though. 

cc'ing @sayakpaul here as well 
Thanks for your suggestions. 

> We load CLIP model ourselves and pass it as an argument to the StableDiffusionPipeline. If nothing is being passed, we go to current behavior. I definitely prefer this solution, as it does not break any previous code.

Sounds good to me! We're happy to accept your PRs and guide you :)

Commit message: Support for manual CLIP loading in StableDiffusionPipeline - txt2img.

Patch to review:
diff --git a/src/diffusers/loaders.py b/src/diffusers/loaders.py
index 5cddb7690e52..1e18c99ddd98 100644
--- a/src/diffusers/loaders.py
+++ b/src/diffusers/loaders.py
@@ -1394,6 +1394,10 @@ def from_ckpt(cls, pretrained_model_link_or_path, **kwargs):
                 "ddim"]`.
             load_safety_checker (`bool`, *optional*, defaults to `True`):
                 Whether to load the safety checker or not. Defaults to `True`.
+            clip_text_model (`transformers.models.clip.modeling_clip.CLIPTextModel`, *optional*, defaults to `None`):
+                An instance of `CLIPTextModel` to use. If this parameter is `None`, the function will load a new instance of `CLIPTextModel`, if needed.
+            clip_tokenizer (`transformers.models.clip.tokenization_clip.CLIPTokenizer`, *optional*, defaults to `None`):
+                An instance of `CLIPTokenizer` to use. If this parameter is `None`, the function will load a new instance of `CLIPTokenizer`, if needed.
             kwargs (remaining dictionary of keyword arguments, *optional*):
                 Can be used to overwrite load - and saveable variables - *i.e.* the pipeline components - of the
                 specific pipeline class. The overwritten components are then directly passed to the pipelines
@@ -1438,6 +1442,8 @@ def from_ckpt(cls, pretrained_model_link_or_path, **kwargs):
         upcast_attention = kwargs.pop("upcast_attention", None)
         load_safety_checker = kwargs.pop("load_safety_checker", True)
         prediction_type = kwargs.pop("prediction_type", None)
+        clip_text_model = kwargs.pop("clip_text_model", None)
+        clip_tokenizer = kwargs.pop("clip_tokenizer", None)
 
         torch_dtype = kwargs.pop("torch_dtype", None)
 
@@ -1518,6 +1524,8 @@ def from_ckpt(cls, pretrained_model_link_or_path, **kwargs):
             upcast_attention=upcast_attention,
             load_safety_checker=load_safety_checker,
             prediction_type=prediction_type,
+            clip_text_model=clip_text_model,
+            clip_tokenizer=clip_tokenizer,
         )
 
         if torch_dtype is not None:
diff --git a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
index e59b91e486f5..2bf5c57d179b 100644
--- a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
+++ b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
@@ -732,8 +732,8 @@ def _copy_layers(hf_layers, pt_layers):
     return hf_model
 
 
-def convert_ldm_clip_checkpoint(checkpoint, local_files_only=False):
-    text_model = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14", local_files_only=local_files_only)
+def convert_ldm_clip_checkpoint(checkpoint, local_files_only=False, clip_text_model=None):
+    text_model = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14", local_files_only=local_files_only) if clip_text_model is None else clip_text_model
 
     keys = list(checkpoint.keys())
 
@@ -1022,7 +1022,9 @@ def download_from_original_stable_diffusion_ckpt(
     controlnet: Optional[bool] = None,
     load_safety_checker: bool = True,
     pipeline_class: DiffusionPipeline = None,
-    local_files_only=False,
+    local_files_only = False,
+    clip_text_model = None,
+    clip_tokenizer = None,
 ) -> DiffusionPipeline:
     """
     Load a Stable Diffusion pipeline object from a CompVis-style `.ckpt`/`.safetensors` file and (ideally) a `.yaml`
@@ -1070,6 +1072,10 @@ def download_from_original_stable_diffusion_ckpt(
             The pipeline class to use. Pass `None` to determine automatically.
         local_files_only (`bool`, *optional*, defaults to `False`):
             Whether or not to only look at local files (i.e., do not try to download the model).
+        clip_text_model (`transformers.models.clip.modeling_clip.CLIPTextModel`, *optional*, defaults to `None`):
+            An instance of `CLIPTextModel` to use. If this parameter is `None`, the function will load a new instance of `CLIPTextModel`, if needed.
+        clip_tokenizer (`transformers.models.clip.tokenization_clip.CLIPTokenizer`, *optional*, defaults to `None`):
+            An instance of `CLIPTokenizer` to use. If this parameter is `None`, the function will load a new instance of `CLIPTokenizer`, if needed.
         return: A StableDiffusionPipeline object representing the passed-in `.ckpt`/`.safetensors` file.
     """
 
@@ -1325,8 +1331,8 @@ def download_from_original_stable_diffusion_ckpt(
             feature_extractor=feature_extractor,
         )
     elif model_type == "FrozenCLIPEmbedder":
-        text_model = convert_ldm_clip_checkpoint(checkpoint, local_files_only=local_files_only)
-        tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
+        text_model = convert_ldm_clip_checkpoint(checkpoint, local_files_only=local_files_only, clip_text_model=clip_text_model)
+        tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14") if clip_tokenizer is None else clip_tokenizer
 
         if load_safety_checker:
             safety_checker = StableDiffusionSafetyChecker.from_pretrained("CompVis/stable-diffusion-safety-checker")


Code review comments:
diff_hunk: @@ -1022,7 +1022,9 @@ def download_from_original_stable_diffusion_ckpt(
     controlnet: Optional[bool] = None,
     load_safety_checker: bool = True,
     pipeline_class: DiffusionPipeline = None,
-    local_files_only=False,
+    local_files_only = False,
+    clip_text_model = None,
+    clip_tokenizer = None,
path: src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
text: @user1:
```suggestion
    local_files_only=False,
    clip_text_model=None,
    clip_tokenizer=None,
```