Repository: oobabooga/text-generation-webui

Issue:
exllama gibberish output
### Describe the bug

I'm running into issues using exllama with the textgen UI. I have exllama running well using its webui, but when I load the same model into textgen UI, I only get gibberish as output. Llama.cpp works fine as a backend. I looked at the output in the console with --verbose and the input prompt looks fine. I tried multiple different sampler presets with no change. I'm using an AMD Vega 56 and I talked with another person on Discord with the same problem on an AMD card so it is probably an AMD-specific issue. I followed the oobabooga ROCm rentry guide in the README to set up textgen. One thing I noticed is the bitsandbytes package in the rocm fork is a little outdated, so could that be the issue?

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Set up textgen webUI using instructions in the README and the ROCm installation guide in Arch Linux on a system with an AMD card. Clone exllama into the repositories folder and install the requirements. Test that exllama works with its on WebGUI. Load a model using exllama in textgen webUI, then generate output from any prompt. 

### Screenshot

_No response_

### Logs

```shell
No log available
```


### System Info

```shell
Arch Linux
AMD 5600X
AMD Vega 56
```



Discussion:
I also have this issue, using RX 6700 XT on EndeavourOS (Arch).
I have the exact same issue too. Running on Arch with a 5600X as well and an RX 6800.

Tried out a variety of models to no avail, however the provided exllama web UI works wonderfully and so does the CLI example chatbot, no matter which model I throw at it. Leads me to think oobabooga is the point of failure. The most I saw this discussed was the fellow in #\750 having something similar and resolving it with a driver update. However, that was on an Nvidia card, so I'm not sure it applies to us.

If it helps, here's an example of said gibberish output I receive:
```Common sense questions and answers

Question: Hello, what does one plus one equal to?
Factual answer:FDrnrn Beaubourg Burgrnлия Sud TournConstraint StockrutMatrixrou tématu Wallsko stick Matrixagentлияobre Starлия Reserve hus BrunoAAStarAngle Bourrou titles Burgiler respect lo sortsDIR Bourbourg Außoin stick Sud Beaubourg Beauroubourg Fourier Sudrou Beau Beaurournobreлия frame stick stick Blaignerutmatrix stick consp Beaurn Burg stickobre Sud Sud Bourлияobreoin Stock Burg ace stickrut Beau Sud Burgbourg Beau Stockлия substitute StockrnFD Stockbourg Assume Beau Beau stickrou Beau lyrrourouShiftMDb SudrutSortobrern stick SudrouEventListeneroin prayerrn Bourbourg Sud Bruno Beau Bruno BrunoFDrnMDb Bruno Bour autonom stickFDFDrutobreobreFD Stock Burgoin FinлияFD Burgoinoinobrehing stickrou stickrou Beaurou Bour Sudrou Beau Burg Sudobre BurgoinFDFDbourgiatbourg stickbourgobre Beau Burg Beau Burg substitutionrnbourgrouлияFD BeaulaimFD BurgMDbrutFDлия Burg Reserve Reserve stickbourg Brunoobre```
Running a 6800xt and have the same problem. I tried update to ROCm 5.5.1 and PyTorch nightly preview version. Didn't help. Exllama's own webui and CLI example chatbot work fine. But oobabooga webui throws gibberish at me :( Please help us!

Resolving patch:
diff --git a/modules/exllama.py b/modules/exllama.py
--- a/modules/exllama.py
+++ b/modules/exllama.py
@@ -1,6 +1,8 @@
 import sys
 from pathlib import Path
 
+from torch import version as torch_version
+
 from modules import shared
 from modules.logging_colors import logger
 
@@ -51,6 +53,12 @@ def from_pretrained(self, path_to_model):
         if shared.args.gpu_split:
             config.set_auto_map(shared.args.gpu_split)
             config.gpu_peer_fix = True
+        if torch_version.hip:
+            config.rmsnorm_no_half2 = True
+            config.rope_no_half2 = True
+            config.matmul_no_half2 = True
+            config.silu_no_half2 = True
+
 
         model = ExLlama(config)
         tokenizer = ExLlamaTokenizer(str(tokenizer_model_path))
diff --git a/modules/exllama_hf.py b/modules/exllama_hf.py
--- a/modules/exllama_hf.py
+++ b/modules/exllama_hf.py
@@ -97,6 +97,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
         if shared.args.gpu_split:
             config.set_auto_map(shared.args.gpu_split)
             config.gpu_peer_fix = True
+        if torch.version.hip:
+            config.rmsnorm_no_half2 = True
+            config.rope_no_half2 = True
+            config.matmul_no_half2 = True
+            config.silu_no_half2 = True
 
         # This slowes down a bit but align better with autogptq generation.
         # TODO: Should give user choice to tune the exllama config


Test patch: