Repository: huggingface/diffusers

Issue:
[Bug][StableDiffusion] Allow loading models with Float16 in `load_pipeline_from_original_stable_diffusion_ckpt`
### Describe the bug

Models loaded from .ckpt / .safetensors file are using Float32 automatically, with no obvious way to change this behavior.

### Reproduction

- Import `load_pipeline_from_original_stable_diffusion_ckpt` from `diffusers.pipelines.stable_diffusion.convert_from_ckpt`
- Load model from .ckpt file
- Observe extreme VRAM usage compared to Float16

### Logs

_No response_

### System Info

Diffusers: 0.14.0
OS: Linux (Kubuntu 22.04)
Python 3.10.6


Discussion:
On a second look, it seems that an option to disable the safety checker is missing as well.

EDIT: I was able to manually edit that function to cast all required components by calling .half() on them, I will try to create a PR fixing these two issues. Can any maintainer tell me if there is a utility function that is used for this purpose, or should I just write one from scratch?
Thanks for your report! It'd be helpful to provide a minimally reproducible snippet as well. 

While instantiating a pipeline, you can specify `safety_checker=None` to disable it. 

Also, note that while instantiating a pipeline, you can specify the `torch_dtype` argument to be `torch.float16`.  

Resolving patch:
diff --git a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
--- a/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
+++ b/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py
@@ -989,6 +989,7 @@ def download_from_original_stable_diffusion_ckpt(
     stable_unclip_prior: Optional[str] = None,
     clip_stats_path: Optional[str] = None,
     controlnet: Optional[bool] = None,
+    load_safety_checker: bool = True,
 ) -> StableDiffusionPipeline:
     """
     Load a Stable Diffusion pipeline object from a CompVis-style `.ckpt`/`.safetensors` file and (ideally) a `.yaml`
@@ -1028,6 +1029,8 @@ def download_from_original_stable_diffusion_ckpt(
             The device to use. Pass `None` to determine automatically. :param from_safetensors: If `checkpoint_path` is
             in `safetensors` format, load checkpoint with safetensors instead of PyTorch. :return: A
             StableDiffusionPipeline object representing the passed-in `.ckpt`/`.safetensors` file.
+        load_safety_checker (`bool`, *optional*, defaults to `True`):
+            Whether to load the safety checker or not. Defaults to `True`.
     """
     if prediction_type == "v-prediction":
         prediction_type = "v_prediction"
@@ -1270,8 +1273,13 @@ def download_from_original_stable_diffusion_ckpt(
     elif model_type == "FrozenCLIPEmbedder":
         text_model = convert_ldm_clip_checkpoint(checkpoint)
         tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
-        safety_checker = StableDiffusionSafetyChecker.from_pretrained("CompVis/stable-diffusion-safety-checker")
-        feature_extractor = AutoFeatureExtractor.from_pretrained("CompVis/stable-diffusion-safety-checker")
+
+        if load_safety_checker:
+            safety_checker = StableDiffusionSafetyChecker.from_pretrained("CompVis/stable-diffusion-safety-checker")
+            feature_extractor = AutoFeatureExtractor.from_pretrained("CompVis/stable-diffusion-safety-checker")
+        else:
+            safety_checker = None
+            feature_extractor = None
 
         if controlnet:
             pipe = StableDiffusionControlNetPipeline(


Test patch: