Repository: oobabooga/text-generation-webui

Issue:
Error with images using LLaVA
### Describe the bug

RuntimeError when LLaVA tries to answer to an image.

Different errors depending on the settings(clip+projector), but always related to "Half", 16bit.

Only working combinations are:
* clip+projector 16bit on gpu
* clip 32bit cpu/gpu, projector 16bit gpu

I think the projector only works with 16 bit when using sdp attention.

The weird thing is that this worked witrhout any hassle before LLaVA was merged(using the pr branch).

If these are not directly fixable, maybe some settings should be overridden if sdp attention is used(if that is the cause) and warned about conflicting settings. And for me at least 16 bit does not work on cpu at all.

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

1. LLaVA settings anything with projector in 32 bit
2. Start webui with LLaVA and sdp attention
3. In chat add an image to your message
4. Image gets embedded, but when trying to give it to the model it'll error

### Screenshot

_No response_

### Logs

```shell
Traceback (most recent call last):
  File "/home/laaza/ooba/text-generation-webui/modules/callbacks.py", line 66, in gentask
    ret = self.mfunc(callback=_callback, **self.kwargs)
  File "/home/laaza/ooba/text-generation-webui/modules/text_generation.py", line 257, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1559, in generate
    return self.sample(
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2598, in sample
    outputs = self(
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 688, in forward
    outputs = self.model(
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 578, in forward
    layer_outputs = decoder_layer(
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 293, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/laaza/ooba/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/laaza/ooba/text-generation-webui/modules/llama_attn_hijack.py", line 142, in sdp_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(query_states, key_states, value_states, attn_mask=attention_mask, is_causal=False)
RuntimeError: Expected attn_mask dtype to be bool or to match query dtype, but got attn_mask.dtype: float and  query.dtype: c10::Half instead.
```


### System Info

```shell
WSL2 Ubuntu-22.04
GPTQ-for-LLaMA triton
RTX 3080 10 GB
Ryzen 5800X
24 GB RAM

webui 2f4f124
model: wojtab/llava-13b-v0-4bit-128g (fully on gpu)
run with LLaVA and sdp attention
```



Discussion:
ok, can confirm, I'll fix it

Resolving patch:
diff --git a/extensions/llava/script.py b/extensions/llava/script.py
--- a/extensions/llava/script.py
+++ b/extensions/llava/script.py
@@ -245,7 +245,9 @@ def tokenizer_modifier(state, prompt, input_ids, input_embeds):
 
     prompt, input_ids, input_embeds, total_embedded = llava_embedder.forward(prompt, images, state)
     print(f'LLaVA - Embedded {total_embedded} image(s) in {time.time()-start_ts:.2f}s')
-    return prompt, input_ids.unsqueeze(0).to(shared.model.device), input_embeds.unsqueeze(0).to(shared.model.device)
+    return (prompt,
+        input_ids.unsqueeze(0).to(shared.model.device, dtype=torch.int64),
+        input_embeds.unsqueeze(0).to(shared.model.device, dtype=shared.model.dtype))
 
 
 def ui():


Test patch: