can this model run on cpu, I had a error when I test it on cpu, this with a error about should install flash_attn
test code:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
#model = AutoModelForCausalLM.from_pretrained("./model", low_cpu_mem_usage=True, device_map="cpu", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("./model", torch_dtype=torch.float16, device_map="cpu", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("./model").to("cpu")
x
= tok.encode("The mistral wind in is a phenomenon ", return_tensors="pt")
x = model.generate(x, max_new_tokens=128)
print(tok.batch_decode(x))
error message:
/home//.local/lib/python3.10/site-packages/torch/onnx/_internal/_beartype.py:30: UserWarning: module 'beartype.roar' has no attribute 'BeartypeDecorHintPep585DeprecationWarning'
warnings.warn(f"{e}")
Traceback (most recent call last):
File "/home//04_files/Model/mixtral/./test0.py", line 5, in
model = AutoModelForCausalLM.from_pretrained("./model", torch_dtype=torch.float16, device_map="cpu", trust_remote_code=True)
File "/home//.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 553, in from_pretrained
model_class = get_class_from_dynamic_module(
File "/home//.local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 488, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/home//.local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 315, in get_cached_module_file
modules_needed = check_imports(resolved_module_file)
File "/home//.local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 180, in check_imports
raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run pip install flash_attn
I tried to install flash_attn ,but it cannot install succ without a cuda driver, so this model cannot work on cpu now?