Repository: huggingface/diffusers

Issue:
Running revision="fp16", torch_dtype=torch.float16 on mps M1
### Describe the bug

I'm using the following code:

```
!pip install diffusers
!pip install transformers scipy ftfy

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True)

prompt = "Pineapple on a white table"
pipe.to("mps")

# Found docs suggested this to get around a bug
with autocast('mps'):
    _ = pipe(prompt, num_inference_steps=1)

with autocast('mps'):
    image = pipe(prompt).images[0]
image.save("hmmm.png")

```

But I always get the error: `RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'`

Is there a way around this error? Seems like the values aren't getting cast right, or this layer isn't getting the message.

### Reproduction

```
!pip install diffusers
!pip install transformers scipy ftfy

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True)

prompt = "Very good prompt"
pipe.to("mps")

# Found docs suggested this to get around a bug
with autocast('mps'):
    _ = pipe(prompt, num_inference_steps=1)

with autocast('mps'):
    image = pipe(prompt).images[0]
image.save("hmmm.png")

```

### Logs

```shell
Traceback (most recent call last):
  File "/Users/administrator/Desktop/stable_diffusion/main.py", line 258, in <module>
    image = pipe(prompt, num_inference_steps=21, latents=latents).images[0]
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 182, in __call__
    text_embeddings = self.text_encoder(text_input.input_ids.to(self.device))[0]
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 731, in forward
    return self.text_model(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 649, in forward
    encoder_outputs = self.encoder(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 578, in forward
    layer_outputs = encoder_layer(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 320, in forward
    hidden_states = self.layer_norm1(hidden_states)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/functional.py", line 2511, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'
```


### System Info

python 3.9
torch==1.13.0.dev20220917
diffusers==0.3.0


Discussion:

Resolving patch:
diff --git a/src/diffusers/models/attention.py b/src/diffusers/models/attention.py
--- a/src/diffusers/models/attention.py
+++ b/src/diffusers/models/attention.py
@@ -376,6 +376,12 @@ def __init__(self, dim_in: int, dim_out: int):
         super().__init__()
         self.proj = nn.Linear(dim_in, dim_out * 2)
 
+    def gelu(self, gate):
+        if gate.device.type != "mps":
+            return F.gelu(gate)
+        # mps: gelu is not implemented for float16
+        return F.gelu(gate.to(dtype=torch.float32)).to(dtype=gate.dtype)
+
     def forward(self, hidden_states):
         hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1)
-        return hidden_states * F.gelu(gate)
+        return hidden_states * self.gelu(gate)


Test patch: