NousResearch

/

Yarn-Llama-2-7b-64k

like 23

I ran the code:

import torch
from transformers import AutoTokenizer, LlamaConfig, LlamaForCausalLM

# YaRN: Efficient Context Window Extension of Large Language Models: https://github.com/jquesnelle/yarn
# https://huggingface.co/NousResearch/Yarn-Llama-2-7b-64k
model_path = "NousResearch/Yarn-Llama-2-7b-64k"

tokenizer = AutoTokenizer.from_pretrained(model_path)
config = LlamaConfig.from_pretrained(model_path, trust_remote_code=True)
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto", config=config, trust_remote_code=True)

Then, I got the error:

The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-4-cb4c3c69fc36> in <cell line: 9>()
      7 
      8 tokenizer = AutoTokenizer.from_pretrained(model_path)
----> 9 config = LlamaConfig.from_pretrained(model_path, trust_remote_code=True)
     10 model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto", config=config, trust_remote_code=True)

3 frames

/usr/local/lib/python3.10/dist-packages/transformers/models/llama/configuration_llama.py in _rope_scaling_validation(self)
    174 
    175         if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
--> 176             raise ValueError(
    177                 "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
    178                 f"got {self.rope_scaling}"

ValueError: `rope_scaling` must be a dictionary with with two fields, `type` and `factor`, got {'factor': 16.0, 'original_max_position_embeddings': 4096, 'type': 'yarn', 'finetuned': True}

Thanks in advance for your help.