MohamedRashad
/

AceGPT-7B-chat-AWQ

Text Generation

text-generation-inference

4-bit precision

Model card Files Files and versions Community

MohamedRashad commited on Nov 18, 2023

Commit

9db1926

•

1 Parent(s): c3fcc0e

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -96,7 +96,7 @@ model_name_or_path = "MohamedRashad/AceGPT-7B-chat-AWQ"
 tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, padding_side="right")
 model = AutoModelForCausalLM.from_pretrained(
     model_name_or_path,
-    use_flash_attention_2=True,
     torch_dtype=torch.float16,
     low_cpu_mem_usage=True,
     device_map="auto"

 tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, padding_side="right")
 model = AutoModelForCausalLM.from_pretrained(
     model_name_or_path,
+    use_flash_attention_2=True, # disable if you have problems with flash attention 2
     torch_dtype=torch.float16,
     low_cpu_mem_usage=True,
     device_map="auto"