Repository: huggingface/diffusers
Pull request: Check correct model type is passed to `from_pretrained`
# What does this PR do?

Example
```python
from diffusers import DiffusionPipeline 
from transformers import T5EncoderModel 
import torch 

repo_id = "black-forest-labs/FLUX.1-dev"
text_encoder_2 = T5EncoderModel.from_pretrained(repo_id, subfolder="text_encoder_2")
pipe = DiffusionPipeline.from_pretrained(
    repo_id, text_encoder=text_encoder_2, torch_dtype=torch.bfloat16
)
```
```python
ValueError: Expected CLIPTextModel for text_encoder, got T5EncoderModel.
```

Fixes #10093


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @sayakpaul 


Problem statement:
[pipelines] add better checking when a wrong model is passed when initializing a pipeline
With this code:

```py
from diffusers import DiffusionPipeline 
from transformers import T5EncoderModel 
import torch 

repo_id = "black-forest-labs/FLUX.1-dev"
text_encoder_2 = T5EncoderModel.from_pretrained(repo_id, subfolder="text_encoder_2")
pipe = DiffusionPipeline.from_pretrained(
    repo_id, text_encoder=text_encoder_2, torch_dtype=torch.bfloat16
).to("cuda")

image = pipe("A dog").images[0]
```

where we're doing `text_encoder=text_encoder_2` (should have been `text_encoder_2=text_encoder_2`), we get an extremely uninformative error:

<details>
<summary>Error</summary>

```bash
...
... [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/home/sayak/diffusers/check_incompatible_flux.py", line 11, in <module>
    image = pipe("A dog").images[0]
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 677, in __call__
    ) = self.encode_prompt(
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 353, in encode_prompt
    pooled_prompt_embeds = self._get_clip_prompt_embeds(
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 289, in _get_clip_prompt_embeds
    prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=False)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1972, in forward
    encoder_outputs = self.encoder(
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1029, in forward
    attention_mask = torch.ones(batch_size, mask_seq_length, device=inputs_embeds.device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

</details>

I know it's on the users to ensure that they're initializing a pipeline properly with the right components. However, the thrown error makes it worse to do any further debugging on the user side. IMO. 

It would be greatly helpful from the perspective of a user if we could throw a better warning/error instead. As soon as we detect wrong types (I think `model_index.json` info can be used in this case), we throw a warning/error. Is this something doable?

Cc: @DN6 @yiyixuxu 
Ccing @hlky as well in case you're interested.

Hints from issue discussion:
@sayakpaul 
1. adding a `validate_pipeline()` method in pipeline loading class if not validated throw simple yet understandable errors like these not sure how good these are:
```
`ValueError: Mismatched model type for component 'text_encoder_2'. 
Expected types: ['T5EncoderModel'], but got: 'T5Model' 
```
Ensure the component matches the expected type as specified in the `model_index.json``

2. While using `model_index.json() `If a mismatch is detected, throw a clear and actionable error, if strict validation isn’t ideal, show warnings instead of hard errors.

Commit message: add LCMScheduler

Patch to review:
diff --git a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
index 471db61556f5..c912344bd609 100644
--- a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
+++ b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
@@ -17,7 +17,7 @@
 import torch
 
 from ...models import UNet2DConditionModel, VQModel
-from ...schedulers import DDPMScheduler
+from ...schedulers import DDPMScheduler, UnCLIPScheduler
 from ...utils import deprecate, logging, replace_example_docstring
 from ...utils.torch_utils import randn_tensor
 from ..pipeline_utils import DiffusionPipeline, ImagePipelineOutput
@@ -83,7 +83,7 @@ class KandinskyV22Pipeline(DiffusionPipeline):
     def __init__(
         self,
         unet: UNet2DConditionModel,
-        scheduler: DDPMScheduler,
+        scheduler: Union[DDPMScheduler, UnCLIPScheduler],
         movq: VQModel,
     ):
         super().__init__()
diff --git a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_combined.py b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_combined.py
index 68334fef3811..9f054bd51198 100644
--- a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_combined.py
+++ b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_combined.py
@@ -140,13 +140,13 @@ class KandinskyV22CombinedPipeline(DiffusionPipeline):
     def __init__(
         self,
         unet: UNet2DConditionModel,
-        scheduler: DDPMScheduler,
+        scheduler: Union[DDPMScheduler, UnCLIPScheduler],
         movq: VQModel,
         prior_prior: PriorTransformer,
         prior_image_encoder: CLIPVisionModelWithProjection,
         prior_text_encoder: CLIPTextModelWithProjection,
         prior_tokenizer: CLIPTokenizer,
-        prior_scheduler: UnCLIPScheduler,
+        prior_scheduler: Union[DDPMScheduler, UnCLIPScheduler],
         prior_image_processor: CLIPImageProcessor,
     ):
         super().__init__()
diff --git a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
index f2134b22b40b..ec1057500e21 100644
--- a/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
+++ b/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
@@ -5,7 +5,7 @@
 from transformers import CLIPImageProcessor, CLIPTextModelWithProjection, CLIPTokenizer, CLIPVisionModelWithProjection
 
 from ...models import PriorTransformer
-from ...schedulers import UnCLIPScheduler
+from ...schedulers import DDPMScheduler, UnCLIPScheduler
 from ...utils import (
     logging,
     replace_example_docstring,
@@ -114,7 +114,7 @@ def __init__(
         image_encoder: CLIPVisionModelWithProjection,
         text_encoder: CLIPTextModelWithProjection,
         tokenizer: CLIPTokenizer,
-        scheduler: UnCLIPScheduler,
+        scheduler: Union[DDPMScheduler, UnCLIPScheduler],
         image_processor: CLIPImageProcessor,
     ):
         super().__init__()
diff --git a/src/diffusers/pipelines/pipeline_utils.py b/src/diffusers/pipelines/pipeline_utils.py
index a504184ea2f2..5fe0a33dcdcf 100644
--- a/src/diffusers/pipelines/pipeline_utils.py
+++ b/src/diffusers/pipelines/pipeline_utils.py
@@ -13,6 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import enum
 import fnmatch
 import importlib
 import inspect
@@ -811,6 +812,7 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
         # in this case they are already instantiated in `kwargs`
         # extract them here
         expected_modules, optional_kwargs = cls._get_signature_keys(pipeline_class)
+        expected_types = pipeline_class._get_signature_types()
         passed_class_obj = {k: kwargs.pop(k) for k in expected_modules if k in kwargs}
         passed_pipe_kwargs = {k: kwargs.pop(k) for k in optional_kwargs if k in kwargs}
         init_dict, unused_kwargs, _ = pipeline_class.extract_init_dict(config_dict, **kwargs)
@@ -832,6 +834,28 @@ def load_module(name, value):
             return True
 
         init_dict = {k: v for k, v in init_dict.items() if load_module(k, v)}
+        scheduler_types = None
+        if "scheduler" in expected_types:
+            scheduler_types = []
+            for scheduler_type in expected_types["scheduler"]:
+                if isinstance(scheduler_type, enum.EnumMeta):
+                    scheduler_types.extend(list(scheduler_type))
+                else:
+                    scheduler_types.extend([str(scheduler_type)])
+            scheduler_types = [str(scheduler).split(".")[-1].strip("'>") for scheduler in scheduler_types]
+
+        for key, (_, expected_class_name) in zip(init_dict.keys(), init_dict.values()):
+            if key not in passed_class_obj:
+                continue
+            class_name = passed_class_obj[key].__class__.__name__
+            class_name = class_name[4:] if class_name.startswith("Flax") else class_name
+            expected_class_name = (
+                expected_class_name[4:] if expected_class_name.startswith("Flax") else expected_class_name
+            )
+            if key == "scheduler" and scheduler_types is not None and class_name not in scheduler_types:
+                raise ValueError(f"Expected {scheduler_types} for {key}, got {class_name}.")
+            elif key != "scheduler" and class_name != expected_class_name:
+                raise ValueError(f"Expected {expected_class_name} for {key}, got {class_name}.")
 
         # Special case: safety_checker must be loaded separately when using `from_flax`
         if from_flax and "safety_checker" in init_dict and "safety_checker" not in passed_class_obj:
diff --git a/src/diffusers/schedulers/scheduling_utils.py b/src/diffusers/schedulers/scheduling_utils.py
index f20224b19009..ce6d04a1add2 100644
--- a/src/diffusers/schedulers/scheduling_utils.py
+++ b/src/diffusers/schedulers/scheduling_utils.py
@@ -46,6 +46,7 @@ class KarrasDiffusionSchedulers(Enum):
     UniPCMultistepScheduler = 13
     DPMSolverSDEScheduler = 14
     EDMEulerScheduler = 15
+    LCMScheduler = 16
 
 
 AysSchedules = {
diff --git a/tests/pipelines/test_pipelines.py b/tests/pipelines/test_pipelines.py
index 43b01c40f5bb..1b48dee59b16 100644
--- a/tests/pipelines/test_pipelines.py
+++ b/tests/pipelines/test_pipelines.py
@@ -1802,6 +1802,17 @@ def test_pipe_same_device_id_offload(self):
         sd.maybe_free_model_hooks()
         assert sd._offload_gpu_id == 5
 
+    def test_wrong_model(self):
+        tokenizer = CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip")
+        with self.assertRaises(ValueError) as error_context:
+            _ = StableDiffusionPipeline.from_pretrained(
+                "hf-internal-testing/diffusers-stable-diffusion-tiny-all", text_encoder=tokenizer
+            )
+
+        assert "Expected" in str(error_context.exception)
+        assert "text_encoder" in str(error_context.exception)
+        assert "CLIPTokenizer" in str(error_context.exception)
+
 
 @slow
 @require_torch_gpu


Code review comments:
diff_hunk: @@ -1802,6 +1802,17 @@ def test_pipe_same_device_id_offload(self):
         sd.maybe_free_model_hooks()
         assert sd._offload_gpu_id == 5
 
+    def test_wrong_model(self):
+        tokenizer = CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip")
+        with self.assertRaises(ValueError) as error_context:
+            _ = StableDiffusionPipeline.from_pretrained(
+                "hf-internal-testing/diffusers-stable-diffusion-tiny-all", text_encoder=tokenizer
+            )
+
+        assert "Expected" in str(error_context.exception)
+        assert "text_encoder" in str(error_context.exception)
+        assert "CLIPTokenizer" in str(error_context.exception)
path: tests/pipelines/test_pipelines.py
text: @author:
Using `f"{tokenizer.__class__.__name}"` failed with `type object 'CLIPTokenizer' has no attribute '_PipelineFastTests__name'`

@user1:
Sorry for that.