arnavgrg
/

llama-2-7b-chat-nf4-fp16-upscaled

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

arnavgrg commited on Dec 5, 2023

Commit

e94da9a

•

1 Parent(s): aa9b295

Update README.md

Files changed (1) hide show

README.md +5 -1

README.md CHANGED Viewed

@@ -15,5 +15,9 @@ To use this model, you can just load it via `transformers` in fp16:
 import torch
 from transformers import AutoModelForCausalLM
-model = AutoModelForCausalLM.from_pretrained("arnavgrg/llama-2-7b-chat-nf4-fp16-upscaled", torch_dtype=torch.float16)
 ```

 import torch
 from transformers import AutoModelForCausalLM
+model = AutoModelForCausalLM.from_pretrained(
+  "arnavgrg/llama-2-7b-chat-nf4-fp16-upscaled",
+  device_map="auto",
+  torch_dtype=torch.float16
+)
 ```