`generate` method does not work with `generation_config.use_cache=False` in `Llama-2` model
Author: nurlanov-zh
Pull request: no
Labels: name: Generation

### System Info

- `transformers` version: 4.40.0.dev0
- Platform: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.8
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.29.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante @younesbelkada @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
from transformers import AutoTokenizer, LlamaForCausalLM
import torch
model_path = "meta-llama/Llama-2-7b-chat-hf"
device = "cuda:0"
torch_dtype = torch.float16
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch_dtype, local_files_only=True).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
model.config.use_cache = False
model.generation_config.use_cache = False

prompt = """
 
 
 There's a llama in my garden. What should I do? 
"""
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(input_ids, max_length=300)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False)[0]
print(output)
```


The output is:

> 
 
 
 There's a llama in my garden. What should I do? 
  Great! Here are some ideas for a fun a Here are some fun ideas for fun for fun ideas for fun ideas for fun and games for fun for kids for fun for kids and fun for kids for fun and games for kids for fun and fun for kids for fun and games for kids and fun for kids and fun for kids for fun and fun for kids for fun and games for fun and fun and games and fun and games and fun and games and fun and games and fun and fun and fun and games and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and fun and




### Expected behavior

Output similar to one when we do not set `model.generation_config.use_cache = False` (commenting out this line produces):

> 
 
 
 There's a llama in my garden. What should I do? 
  Oh no, a llama in your garden! That's quite a surprise. Here are some steps you can take to handle the situation:
> 1. Stay calm: Try to remain calm and composed. A llama in your garden can be a curious and unexpected occurrence, but it's important to handle it with care and caution.
> 2. Observe the llama from a safe distance: Keep a safe distance from the llama and observe its behavior. Look for any signs of aggression or discomfort. If the llama seems agitated or threatened, it's best to give it space and allow it to leave on its own.
> 3. Check for any injuries or signs of distress: If the llama appears to be injured or distressed, contact a local veterinarian or animal rescue organization for assistance.
> 4. Contact the llama's owner: If you are able to identify the llama's owner, contact them immediately to let them know where their llama is. They may be able to retrieve their llama and take it back home.
> 5. Provide food and water: If the llama is not injured or distressed, you can provide it with food and water to keep it comfortable while you wait for its

I believe the problem is that `prepare_inputs_for_generation` in `LlamaForCasualLM` always assumes that the cache is used (there are already several # TODO joao @gante ) in this function :)

Comments:
I have a fix, opening a PR right now 😉 

@ArthurZucker , @gante 
The issue still persists when `model.config._attn_implementation = "eager"` is used. Currently it works because the implementation relies on `sdpa` attention implementation, which does not use `causal_mask`. 
```py
class LlamaModel(LlamaPreTrainedModel):
    ...

    def _update_causal_mask(
        self,
        attention_mask: torch.Tensor,
        input_tensor: torch.Tensor,
        cache_position: torch.Tensor,
        past_seen_tokens: int,
    ):
        ...

        if self.config._attn_implementation == "sdpa":
            # For SDPA, when possible, we will rely on its `is_causal` argument instead of its `attn_mask` argument,
            # in order to dispatch on Flash Attention 2.
            if AttentionMaskConverter._ignore_causal_mask_sdpa(
                attention_mask, inputs_embeds=input_tensor, past_key_values_length=past_seen_tokens
            ):
                return None
```

Ideally, the code should work with any attention implementation. I believe the problem is that the `cache_position` is wrong when we use `model.generate` with `use_cache=False`. 

Can you provide a reproducer?