`KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'`
Author: MohamedAliRashad
Pull request: no

### System Info

- `transformers` version: 4.36.0
- Platform: Linux-5.15.0-70-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.3
- Accelerate version: 0.25.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

long_text = # ...

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id, use_flash_attention_2=True, torch_dtype=torch.float16, device_map="auto")

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

messages = [
    {"role": "user", "content": f"Summarize the following:\n{long_text}"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)

outputs = model.generate(inputs, max_new_tokens=8192, do_sample=True, streamer=streamer)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Expected behavior

Expected it to work or at least give me a cuda error.

Comments:
Hi @MohamedAliRashad, thanks for raising this issue! 

Could you share the error encountered including its full traceback as well? 

@amyeroberts 
You are unable to reproduce ?

Hi! 
I'm Having the same Error: 
only happens when the token length is greater than the sliding window size
(I do not have the error with transformers version 4.34.0, but when I upgrade to 4.36.0 I get the error)

Thanks! :) 

- transformers version: 4.36.0
- Platform:  Linux 5.10.0-26-cloud-amd64
- Python version: 3.10.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/opt/conda/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/mmockus/dev/chatbot/rGPT/host.py", line 301, in benchmark_model
    response, time, previous_prompt = rgpt(
  File "/home/mmockus/dev/chatbot/rGPT/RGPT.py", line 384, in __call__
    output = self.__generate_text(final_prompt)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1140, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1147, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/mmockus/dev/chatbot/rGPT/instruct_pipeline.py", line 60, in _forward
    generated_sequence = self.model.generate(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1764, in generate
    return self.sample(
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2861, in sample
    outputs = self(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mixtral/modeling_mixtral.py", line 1212, in forward
    outputs = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mixtral/modeling_mixtral.py", line 1080, in forward
    layer_outputs = decoder_layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mixtral/modeling_mixtral.py", line 796, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mixtral/modeling_mixtral.py", line 441, in forward
    past_key = past_key_value[0]
  File "/opt/conda/lib/python3.10/site-packages/transformers/cache_utils.py", line 78, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

@manueljmockus Thanks for providing these details! As Mixtral was only part of the most recent release - does this mean this error message was generated loading and using a different model than in the issue example? 

> @amyeroberts You are unable to reproduce ?

@MohamedAliRashad  We ask that anyone posting an issue posts the full traceback. We have many issues opened and PRs to review a day; to help get your issue resolved as soon as possible, you need to help us help you. With partial information we can't know if we're trying to solve the right issue - or we might be able to reply with a solution from just seeing the stack trace. 

cc @tomaarsen as this seems like a cache issue 

@amyeroberts 
I understand.

I am asking if you were unable to produce the issue with the code i provided because i think it is an OOM probelm (the cached layers can't be accessed because the input has overwritten them).

@amyeroberts 
Hi, sorry if I was not clear. I was Using "mistralai/Mistral-7B-Instruct-v0.1" with transformers version 4.34.0 without an issue. 
Yesterday after  upgrading to 4.36.0 to test mixtral I am getting the error mentioned, Both with "mistralai/Mixtral-8x7B-Instruct-v0.1" and "mistralai/Mistral-7B-Instruct-v0.1" models. Here are the logs from the mistral execution:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/opt/conda/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/mmockus/dev/chatbot/rGPT/host.py", line 302, in benchmark_model
    response, time, previous_prompt = rgpt(
  File "/home/mmockus/dev/chatbot/rGPT/RGPT.py", line 386, in __call__
    output = self.__generate_text(final_prompt)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1140, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1147, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/mmockus/dev/chatbot/rGPT/instruct_pipeline.py", line 60, in _forward
    generated_sequence = self.model.generate(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1764, in generate
    return self.sample(
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2861, in sample
    outputs = self(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1044, in forward
    outputs = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 929, in forward
    layer_outputs = decoder_layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 654, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 391, in forward
    past_key = past_key_value[0]
  File "/opt/conda/lib/python3.10/site-packages/transformers/cache_utils.py", line 78, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

Thanks for yout help!

I am facing the same problem while using the official transformers examples (pytorch/run_clm with trainer). 

Specifically, I only get this exact error when 1. training context window is larger than sliding window (4096), and 2. I am using zero3/fsdp. When context window is less than 4096, I get a different error, and when using zero1/2, training works as expected.

This indeed seems like a caching issue. cc @gante
It seems like this snippet was not updated to work with the new Cache class: https://github.com/huggingface/transformers/blob/2788f8d8d5f9cee2fe33a9292b0f3570bd566a6d/src/transformers/models/mistral/modeling_mistral.py#L388-L407

I suspect that not using Flash Attention 2 may solve the issue in the meantime.

- Tom Aarsen

> This indeed seems like a caching issue. cc @gante It seems like this snippet was not updated to work with the new Cache class:
> 
> https://github.com/huggingface/transformers/blob/2788f8d8d5f9cee2fe33a9292b0f3570bd566a6d/src/transformers/models/mistral/modeling_mistral.py#L388-L407
> 
> I suspect that not using Flash Attention 2 may solve the issue in the meantime.
> 
> * Tom Aarsen

Hi, Thanks for the quick response. I can confirm that removing flash attention solves this issue. However removing it means that long contexts take up too much memory and i get a Cuda OOM.

Opening a PR to fix it :)

FYI, this seems to be a distributed error: in a single GPU device, the following script runs with no exceptions

**EDIT**: it doesn't fail because `mistralai/Mistral-7B-Instruct-v0.2` doesn't use a sliding window 👀 

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

long_text = "Foo Bar " * 5000

model_id = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id, use_flash_attention_2=True, torch_dtype=torch.float16, device_map="auto")

messages = [
    {"role": "user", "content": f"Summarize the following:\n{long_text}"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
print(inputs.shape)

outputs = model.generate(inputs, max_new_tokens=8192, do_sample=True)
print(outputs.shape)
```

I can confirm, this lines up with my experience since zero1/2 works but zero3/fsdp does not.

The PR linked above fixes it :)

Thank you so much! I can confirm training works now :)

Hi @gante , my transformers version is 4.36.2 （having viewed this issue I pip -U immediatly ） but still face the same question above when using standard llama2-7b-hf.
When loading model, I get a warning 

> 'Instantiating LlamaAttentionLayerBetterTransformer without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.'

Eventually

> KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

`self.model = AutoModelForCausalLM.from_pretrained(model_name,torch_dtype='auto',low_cpu_mem_usage=True).to_bettertransformer()`
I dive into the code about modeling_llama.py 
![image](https://github.com/huggingface/transformers/assets/88258534/b8180aa0-7ef1-40b5-aeb3-6a849bbd254f)
So I check my config.json also, it seems nothing wrong with num_hidden_layers.
`print(self.model.config.num_hidden_layers)` ->32
is there any possible reason cause this? (btw since this issue is closed, shall I open a new one? : | Thanks for your help in advance! 


Hey @rangehow 👋 

From your error, it seems that you are using BetterTransformer + Llama, which is deprecated. See [here](https://github.com/huggingface/optimum/releases/tag/v1.16.1) for more information (and how to update it)

> > This indeed seems like a caching issue. cc @gante It seems like this snippet was not updated to work with the new Cache class:
> > https://github.com/huggingface/transformers/blob/2788f8d8d5f9cee2fe33a9292b0f3570bd566a6d/src/transformers/models/mistral/modeling_mistral.py#L388-L407
> > 
> > I suspect that not using Flash Attention 2 may solve the issue in the meantime.
> > 
> > * Tom Aarsen
> 
> Hi, Thanks for the quick response. I can confirm that removing flash attention solves this issue. However removing it means that long contexts take up too much memory and i get a Cuda OOM.



Do you know if have same issue?  I am using TheBloke/Llama-2-13B-chat-GPTQ with GPU A100 80GB? if yes, please help me which file to change or update?


 File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 798, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/auto_gptq/nn_modules/fused_llama_attn.py", line 62, in forward
    kv_seq_len += past_key_value[0].shape[-2]
                  ~~~~~~~~~~~~~~^^^
  File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/transformers/cache_utils.py", line 78, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

@bp020108 make sure you're running the latest `transformers` version. If the issue persists after updating `transformers` to `v4.38`, we'll need a script to reproduce your issue 🤗 

looks like I have 4.37.2. 

-vm:~/miniconda3/LLAMA/localchat$ pip3.11 list | grep transf
sentence-transformers       2.2.2
transformers                4.37.2

you want  me to upgrade this?
pip3.11 install transformers==4.38?


@bp020108 My bad, I've missed a key point in your stacktrace the first time I saw it -- the issue seems to be in the `autogptq` library.

copying the last line:
> File "/home//miniconda3/envs/NetAI_B/lib/python3.11/site-packages/auto_gptq/nn_modules/fused_llama_attn.py", line 62, in forward
kv_seq_len += past_key_value[0].shape[-2]

Try updating `autogptq` to the latest version and, if the issue persists, open an issue there 🤗 

@gante it still does not work, in the generation code:
```
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = config.model_name,
    max_seq_length = config.max_seq_length,
    dtype = config.dtype,
    load_in_4bit = config.load_in_4bit,
)
inputs = tokenizer(
    [
        alpaca_prompt.format(
            instruction, # instruction
            input, # input
            output, # output - leave this blank for generation!
        )
    ], return_tensors = "pt").to(config.device)


    text_streamer = TextStreamer(tokenizer)
    model.generate(**inputs, streamer=text_streamer, max_new_tokens=64)
```
```
  File "/data/user/miniconda3/envs/llm/lib/python3.10/site-packages/unsloth/models/llama.py", line 733, in LlamaModel_fast_forward_inference
    seq_len = past_key_values[0][0].shape[-2]
  File "/data/user/miniconda3/envs/llm/lib/python3.10/site-packages/transformers/cache_utils.py", line 314, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'
```

I use `transformers-4.42.3`, `auto-gptq-0.7.1`

Some news about that issue? I am obtaing exctly the same errror.

**When using unsloth, im also getting the same error.**
```
FastLanguageModel.for_inference(model) 
outputs = model.generate(**inputs, max_new_tokens = 4096, use_cache = False)
```

```
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Traceback (most recent call last):
  File "/home/adrin/PycharmProjects/unsloth1/1_testInference.py", line 77, in <module>
    outputs = model.generate(**inputs, max_new_tokens = 4096, use_cache = False)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 988, in _fast_generate
    output = generate(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/peft/peft_model.py", line 1491, in generate
    outputs = self.base_model.generate(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1914, in generate
    result = self._sample(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2651, in _sample
    outputs = self(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 796, in _CausalLM_fast_forward
    outputs = fast_forward_inference(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 733, in LlamaModel_fast_forward_inference
    seq_len = past_key_values[0][0].shape[-2]
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/transformers/cache_utils.py", line 314, in __getitem__
    raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: **'Cache only has 0 layers, attempted to access layer with index 0'**
```

I solved it doing this:
`pip uninstall transformers -y`
`pip install transformers==4.38.0`

> I solved it doing this: `pip uninstall transformers -y` `pip install transformers==4.38.0`

downgrading the library is throwing entirely different set of errors.

```
/tmp/tmp7h24mgeg/main.c: In function ‘list_to_cuuint64_array’:
/tmp/tmp7h24mgeg/main.c:354:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (Py_ssize_t i = 0; i < len; i++) {
   ^
/tmp/tmp7h24mgeg/main.c:354:3: note: use option -std=c99 or -std=gnu99 to compile your code
/tmp/tmp7h24mgeg/main.c: In function ‘list_to_cuuint32_array’:
/tmp/tmp7h24mgeg/main.c:365:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (Py_ssize_t i = 0; i < len; i++) {
   ^
Traceback (most recent call last):
  File "/home/adrin/PycharmProjects/unsloth1/1_testInference.py", line 77, in <module>
    outputs = model.generate(**inputs, max_new_tokens = 4096, use_cache = False)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 988, in _fast_generate
    output = generate(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/peft/peft_model.py", line 1491, in generate
    outputs = self.base_model.generate(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1592, in generate
    return self.sample(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2696, in sample
    outputs = self(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 815, in _CausalLM_fast_forward
    outputs = self.model(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 682, in LlamaModel_fast_forward
    layer_outputs = decoder_layer(
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 434, in LlamaDecoderLayer_fast_forward
    hidden_states = fast_rms_layernorm(self.input_layernorm, hidden_states)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/kernels/rms_layernorm.py", line 190, in fast_rms_layernorm
    out = Fast_RMS_Layernorm.apply(X, W, eps, gemma)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/unsloth/kernels/rms_layernorm.py", line 144, in forward
    fx[(n_rows,)](
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/jit.py", line 363, in run
    device = driver.get_current_device()
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/driver.py", line 209, in __getattr__
    self._initialize_obj()
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/driver.py", line 206, in _initialize_obj
    self._obj = self._init_fn()
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/driver.py", line 239, in initialize_driver
    return CudaDriver()
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/driver.py", line 102, in __init__
    self.utils = CudaUtils()
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/runtime/driver.py", line 49, in __init__
    so = _build("cuda_utils", src_path, tmpdir)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/common/build.py", line 106, in _build
    ret = subprocess.check_call(cc_cmd)
  File "/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmp7h24mgeg/main.c', '-O3', '-I/home/adrin/anaconda3/envs/unsloth_env/lib/python3.10/site-packages/triton/common/../third_party/cuda/include', '-I/home/adrin/anaconda3/envs/unsloth_env/include/python3.10', '-I/tmp/tmp7h24mgeg', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmp7h24mgeg/cuda_utils.cpython-310-x86_64-linux-gnu.so', '-L/lib64', '-L/lib64']' returned non-zero exit status 1.

```

Here all the instruction for the installation:
```python
conda create --name unsloth_env \
    python=3.10 \
    pytorch-cuda=12.1 \
    pytorch cudatoolkit xformers -c pytorch -c nvidia -c xformers \
    -y
conda activate unsloth_env
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps "trl<0.9.0" peft accelerate bitsandbytes
pip uninstall transformers -y
pip install transformers==4.38.0 
```
I have a Testal T4 with cuda version 12.2

> Here all the instruction for the installation:
> 
> ```python
> conda create --name unsloth_env \
>     python=3.10 \
>     pytorch-cuda=12.1 \
>     pytorch cudatoolkit xformers -c pytorch -c nvidia -c xformers \
>     -y
> conda activate unsloth_env
> pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
> pip install --no-deps "trl<0.9.0" peft accelerate bitsandbytes
> pip uninstall transformers -y
> pip install transformers==4.38.0 
> ```
> 
> I have a Testal T4 with cuda version 12.2

Thank you. But it did not work. 
I followed https://github.com/unslothai/unsloth?tab=readme-ov-file#-installation-instructions instructions just as you wrote.
As you suggested, now i modified transformers to 4.38 but still the above error only coming
 

@baluraju15 try creating a new environment with my requirement [here](https://gist.github.com/binhna/6e5b383c8eae2acf4ac9dc68eaa5692d). It solves my problem

> I solved it doing this: `pip uninstall transformers -y` `pip install transformers==4.38.0`

this is solved my issue too. thanks :)

This error is coming up again when using unsloth on the latest version of transformers. Downgrading it is causing the `tokenizer.apply_chat_template()` to fail. It may be because of the tokenizers library but since 4.38.0 is dependant on 0.15.2 of tokenizers, you can't upgrade for this fix. Here is the code snippet:

```
from unsloth import FastLanguageModel, is_bfloat16_supported

model, tokenizer = FastLanguageModel.from_pretrained(
    # model_name = f"{model_folder}/lora_model", # YOUR MODEL YOU USED FOR TRAINING
    model_name = f"NousResearch/Hermes-2-Pro-Llama-3-8B", # YOUR MODEL YOU USED FOR TRAINING
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = False, # variable is defined up already if using 4bit model
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference


# from transformers import AutoTokenizer

# tokenizer = AutoTokenizer.from_pretrained('NousResearch/Hermes-2-Pro-Llama-3-8B')
messages=[
        {"role": "system", "content": "you are an AI"},
        {"role": "user", "content": "what do you like?"},
        ]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True, # Must add for generation
    return_tensors = "pt",
)#.to("cuda")
print(inputs.shape)
```

This throws the error:
```
Traceback (most recent call last):
  File "/home/ubuntu/llm-finetuning-us-east-1/unsloth/inference.py", line 50, in <module>
    inputs = tokenizer.apply_chat_template(
  File "/home/ubuntu/llm-finetuning-us-east-1/unsloth/unsloth_env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1743, in apply_chat_template
    compiled_template = self._compile_jinja_template(chat_template)
TypeError: unhashable type: 'list'
```

cc @Rocketknight1 specifically for the previous comment

Hi @AliButtar, yes, the chat templates on `Hermes-2-Pro-Llama-3-8B` are in the newer format (separate templates for normal chats and tool use). Older versions of `transformers` won't understand this, and will throw an error.

As a temporary workaround, you can do this:

`tokenizer.chat_template = tokenizer.chat_template["default"]`

However, ideally we'd resolve the actual issue that's forcing you to downgrade instead, which has nothing to do with chat templates!

I meet this error when I use `FastLanguageModel` to inference and I want to call model.generate, and I solve this problem by just add one line code:

```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
  model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
  max_seq_length = max_seq_length,
  dtype = dtype,
  load_in_4bit = load_in_4bit,
)
FastLanguageModel.for_inference(model) # just add this line and solve the error `Cache only has 0 layers...`
```

Hope it helpful :)

> I meet this error when I use `FastLanguageModel` to inference and I want to call model.generate, and I solve this problem by just add one line code:
> 
> ```python
> from unsloth import FastLanguageModel
> model, tokenizer = FastLanguageModel.from_pretrained(
>   model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
>   max_seq_length = max_seq_length,
>   dtype = dtype,
>   load_in_4bit = load_in_4bit,
> )
> FastLanguageModel.for_inference(model) # just add this line and solve the error `Cache only has 0 layers...`
> ```
> 
> Hope it helpful :)

It solved the problem. Thanks a million.

> I meet this error when I use `FastLanguageModel` to inference and I want to call model.generate, and I solve this problem by just add one line code:
> 
> ```python
> from unsloth import FastLanguageModel
> model, tokenizer = FastLanguageModel.from_pretrained(
>   model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
>   max_seq_length = max_seq_length,
>   dtype = dtype,
>   load_in_4bit = load_in_4bit,
> )
> FastLanguageModel.for_inference(model) # just add this line and solve the error `Cache only has 0 layers...`
> ```
> 
> Hope it helpful :)

This just helped me, thank you! I think that this step sets various vars on the model and wraps it with the correct dtype.

try close use_unsloth