Sample Code Not Working
Am I the only one who doesn't have any luck with the provided sample code?
import torch
from transformers import pipeline
pipe = pipeline("text-generation", model="HuggingFaceH4/zephyr-7b-alpha", torch_dtype=torch.bfloat16, device_map="auto")
# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
{
"role": "system",
"content": "You are a friendly chatbot who always responds in the style of a pirate",
},
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
# <|system|>
# You are a friendly chatbot who always responds in the style of a pirate.</s>
# <|user|>
# How many helicopters can a human eat in one sitting?</s>
# <|assistant|>
# Ah, me hearty matey! But yer question be a puzzler! A human cannot eat a helicopter in one sitting, as helicopters are not edible. They be made of metal, plastic, and other materials, not food!
I get these errors:
ValueError: The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder` for them. Alternatively, make sure you have `safetensors` installed if the model you are using offers the weights in this format.
same for me
Judging from the error message it seems you need to have safetensors installed since we provide these weights - you also need to have accelerate and the main version of transformers installed to run the example (I'll add a note to the README)
This example is for local LLM so you will need to download the LLM.
@lewtun thanks for the reply.
I do have each of those installed though:
accelerate==0.23.0
safetensors==0.4.0
transformers==4.35.0.dev0
When you say the main version of transformers, I am working on the assumption that you mean the version of transformers that is installed with pip install git+https://github.com/huggingface/transformers.git
It works, now that it downloads the safetensor files.