OrionStarAI

/

Orion-14B-Chat-Int4

like 28

是不是下载的模型需要手工量化一次？晕，部署指导文档太少！

系统环境：
(Orion) PS D:\Huggin face\Orion-14B-App-Demo-CN\demo> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

demo.py

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("\Huggin face\Orion-14B-Chat-Int4", trust_remote_code=True,use_safetensors=True)
model = AutoModelForCausalLM.from_pretrained("\Huggin face\Orion-14B-Chat-Int4", torch_dtype=torch.bfloat16,device_map="auto", trust_remote_code=True,use_safetensors=True)

messages = [{"role": "user", "content": "hi,who are you?"}]
response = model.chat(tokenizer, messages, streaming=False)
print(response)

---

(Orion) PS D:\Huggin face\Orion-14B-App-Demo-CN\demo> python demo.py
bin D:\Users\Administrator\anaconda3\envs\Orion\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.dll
鲯榅鲯鲯榅 mathemat鲯鲯榅榅鲯鲯鲯鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯榅榅榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯榅鲯鲯榅鲯榅鲯榅鲯鲯榅榅榅榅鲯榅鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅榅鲯鲯鲯鲯鲯榅鲯鲯鲯榅 鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯榅鲯鲯榅鲯鲯鲯鲯鲯榅鲯榅榅鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯榅榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯 鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯榅鲯鲯鲯榅鲯鲯鲯榅榅鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅榅鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯 鲯鲯鲯鲯榅鲯榅榅榅鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯榅鲯鲯 鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯榅鲯鲯榅鲯鲯鲯鲯榅榅鲯鲯鲯鲯榅鲯榅鲯鲯鲯榅鲯鲯鲯榅榅鲯鲯榅鲯鲯鲯鲯鲯榅鲯榅鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯榅 鲯鲯鲯榅鲯鲯榅鲯鲯鲯榅鲯榅榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯 鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅榅鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯榅鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯 鲯鲯鲯鲯榅鲯鲯榅鲯鲯鲯鲯榅鲯榅鲯鲯鲯榅鲯鲯榅鲯榅鲯鲯榅鲯榅鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯榅鲯鲯榅鲯鲯榅鲯鲯 鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯榅榅榅鲯榅鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅榅鲯鲯榅榅鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯榅榅鲯榅榅榅鲯鲯鲯 鲯鲯榅鲯鲯榅榅鲯鲯榅鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅榅鲯榅鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯 榅榅鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯鲯榅鲯鲯鲯鲯鲯鲯鲯鲯

what's the version of your transformers ?
please try 4.36.2.

我使用的就是transformers 4.36.2，还有可能的其它原因吗？

找到原因了，从hugginface库下载过程有断点，模型文件出错了，重新下载后运行成功 ，谢谢！

这个回答很有趣