Add support for yarn
Author: RonanKMcGovern
Pull request: no

### System Info

transformers 4.33 (unreleased).

```
model_id = "TheBloke/Yarn-Llama-2-7B-128K-GPTQ"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    revision="main")
```
leading to:
```
ValueError                                Traceback (most recent call last)
[<ipython-input-19-5c42e5a41528>](https://localhost:8080/#) in <cell line: 1>()
----> 1 model = AutoModelForCausalLM.from_pretrained(
      2     model_id,
      3     torch_dtype=torch.float16,
      4     device_map="auto",
      5     revision="main")

4 frames
[/usr/local/lib/python3.10/dist-packages/transformers/models/llama/configuration_llama.py](https://localhost:8080/#) in _rope_scaling_validation(self)
    165 
    166         if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
--> 167             raise ValueError(
    168                 "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
    169                 f"got {self.rope_scaling}"

ValueError: `rope_scaling` must be a dictionary with with two fields, `type` and `factor`, got {'factor': 32.0, 'original_max_position_embeddings': 4096, 'type': 'yarn', 'finetuned': True}
```

### Who can help?

@SunMarc @PanQiWei

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Reproduction

See above. Also, this issue on [HuggingFace](https://huggingface.co/TheBloke/Yarn-Llama-2-13B-128K-GPTQ/discussions/1)

### Expected behavior

To support yarn, it seems that transformers would need to support added parameters for RoPE scaling.

Comments:
The error is not really giving you any clues as to what went wrong here in my opinion. You should be able to load the model if you pass `trust_remote_code=True`. 

```python
model_id = "TheBloke/Yarn-Llama-2-7B-128K-GPTQ"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    revision="main",
    trust_remote_code=True
)
```

Many thanks @casper-hansen , I did that and it was helpful.

I'm now getting:
```
>>>> Flash Attention installed
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[~/.cache/huggingface/modules/transformers_modules/TheBloke/Yarn-Llama-2-7B-128K-GPTQ/8cfa601c38543979a34fb31cbcd1d25682d020c4/modeling_llama_together_yarn.py](https://localhost:8080/#) in <module>
     51 try:
---> 52     from flash_attn.layers.rotary import apply_rotary_emb_func
     53     flash_rope_installed = True

12 frames
ModuleNotFoundError: No module named 'flash_attn.ops.triton'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
[~/.cache/huggingface/modules/transformers_modules/TheBloke/Yarn-Llama-2-7B-128K-GPTQ/8cfa601c38543979a34fb31cbcd1d25682d020c4/modeling_llama_together_yarn.py](https://localhost:8080/#) in <module>
     55 except ImportError:
     56     flash_rope_installed = False
---> 57     raise ImportError('Please install RoPE kernels: `pip install git+[https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary`](https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary%60)')
     58 
     59 

ImportError: Please install RoPE kernels: `pip install git+[https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary`](https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary%60)
```
Note that I have installed flash-attn (and the csrc/rotary package):
```
pip install flash-attn --no-build-isolation
pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
```

> ```
> pip install flash-attn --no-build-isolation
> pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
> ```

There are many "gotchas" with this new repo. Try creating a new environment / uninstall your already installed pip packages.

I have not confirmed if it is the order of installation or if it is only flash attention v2.1.1 that works. But this worked for me after running into the same issue as you

```
pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
pip install flash-attn==2.1.1 --no-build-isolation
```

Yeah that worked. Thanks!

I'm facing an issue now with invalid probabilities. I'll aim to replicate exactly what's recommended in the morning and revert back here.

```
RuntimeError                              Traceback (most recent call last)
[<ipython-input-42-081a24bc1ede>](https://localhost:8080/#) in <cell line: 1>()
----> 1 stream(f'howdy')

3 frames
[/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py](https://localhost:8080/#) in sample(self, input_ids, logits_processor, stopping_criteria, logits_warper, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
   2764             # sample
   2765             probs = nn.functional.softmax(next_token_scores, dim=-1)
-> 2766             next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
   2767 
   2768             # finished sentences should have their next token be a padding token

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```
from:
```
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

# Define a stream *without* function calling capabilities
def stream(user_prompt):

    prompt = f"{user_prompt.strip()}\n\n"

    inputs = tokenizer([prompt], return_tensors="pt").to(runtimeFlag)

    # with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    streamer = TextStreamer(tokenizer)

    # Despite returning the usual output, the streamer will also print the generated text to stdout.
    _ = model.generate(**inputs, streamer=streamer, max_new_tokens=500)
```

> I'm facing an issue now with invalid probabilities.

Ok, this issue is with the TheBloke/Yarn-Llama-2-7B-128K-GPTQ, not with the 13B version.

Here's the working code:
```
!pip3 install git+https://github.com/huggingface/transformers.git
!pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
!pip3 install git+https://github.com/huggingface/optimum.git
!pip3 install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
!pip3 install flash-attn==2.1.1 --no-build-isolation

import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "TheBloke/Yarn-Llama-2-13B-128K-GPTQ"
# To use a different branch, change revision
# For example: revision="gptq-4bit-32g-actorder_True"
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             torch_dtype=torch.float16,
                                             device_map="auto",
                                             revision="main",
                                             trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
```

Just an added note that I don't recommend trying this because - even at 8,000 token context length - the output quality is poor and repetitive. Further, on an A100 - 40 GB, the memory usage was spiking to use up nearly all of the memory even on 8k tokens.

> Just an added note that I don't recommend trying this because - even at 8,000 token context length - the output quality is poor and repetitive. Further, on an A100 - 40 GB, the memory usage was spiking to use up nearly all of the memory even on 8k tokens.

I suspect it needs fine-tuning as this model is only pre trained and not ready for real usage.

> > I'm facing an issue now with invalid probabilities.
> 
> Ok, this issue is with the TheBloke/Yarn-Llama-2-7B-128K-GPTQ, not with the 13B version.
> 
> Here's the working code:
> 
...
> 
> Just an added note that I don't recommend trying this because - even at 8,000 token context length - the output quality is poor and repetitive. Further, on an A100 - 40 GB, the memory usage was spiking to use up nearly all of the memory even on 8k tokens.

> I suspect it needs fine-tuning as this model is only pre trained and not ready for real usage.

Thanks for the note, i can loaded [TheBloke/Yarn-Llama-2-13B-128K-GPTQ:gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/Yarn-Llama-2-13B-128K-GPTQ) with ooba exllama successfully. (10240 token)

But yes, the output quality is poor. I will try [TheBloke/Airoboros-L2-13B-2_1-YaRN-64K-GPTQ](https://huggingface.co/TheBloke/Airoboros-L2-13B-2_1-YaRN-64K-GPTQ) to see if i can have better result.

Update: Airoboros-L2-13B-2_1-YaRN-64K-GPTQ can output meaningful response, but i feel no good as [TheBloke/OpenAssistant-Llama2-13B-Orca-8K-3319-GPTQ]. Hope YaRN can have more fine-tuning in the future.

I recommend using CodeLlama, and using AWQ if you want to quantize. It stretches out automatically fairly well to 32k context. Vid here: https://youtu.be/ELax81LjFhU

Actually I think that LongLoRA is probably now the best performing long context model. I'm working on making an AWQ for that.