Salesforce

/

codegen-16B-multi

like 120

The model card says that the code

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-16B-multi")
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen-16B-multi")

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

is used, but it doesnt work

ValueError: Tokenizer class CodeGenTokenizer does not exist or is not currently imported.