Llama-2-hf non stopping token generation.
Author: francescobodria
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-1035-azure-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

model_id = "/home/modelweights/llama2-hf-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  device_map='auto',
  load_in_8bit=True,
  )

pipe = pipeline(
    "text-generation", 
    model=model, 
    tokenizer=tokenizer,
    do_sample=True,
    top_k=50,
    top_p=0.95, 
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=256,
    temperature=0.1,
)

sequences = pipe(
    'Hello there! How are you doing?',
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")
```

### Expected behavior

Hi, 
My Llama 2 model is not generating the stopping tokens.
For example the reply of the question Hello there! How are you doing? is:
Result: Hello there! How are you doing? I hope you are doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing w
The reply only stops when the max token criteria is met
What am I doing wrong?

Comments:
cc @gante and @ArthurZucker 

Hey! This is expected, the llama model kind of rarely generates the `eos_token`. It was the same with Llama 1, and if you run your script with the original llama, you will get the same output:
```python
# Copyright (c) Meta Platforms, Inc. and affiliates.
# This software may be used and distributed according to the terms of the GNU General Public License version 3.

import fire

from llama import Llama


def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 512,
    max_gen_len: int = 256,
    max_batch_size: int = 8,
):
    generator = Llama.build(
        ckpt_dir=ckpt_dir,
        tokenizer_path=tokenizer_path,
        max_seq_len=max_seq_len,
        max_batch_size=max_batch_size,
    )


    results = generator.text_completion(
        ['Hello there! How are you doing?'],
        max_gen_len=max_gen_len,
        temperature=temperature,
        top_p=top_p,
    )

    print('Hello there! How are you doing?')
    print(f"> {results['generation']}")
    print("\n==================================\n")


if __name__ == "__main__":
    fire.Fire(main)
```
and 
```bash
torchrun --nproc_per_node=1  llama/example_text_completion.py  --ckpt_dir Llama-2-7b  --tokenizer_path Llama-2-7b/tokenizer.model --temperature 0.1 --top_p 0.95
```
will produce:
```txt
> I hope you are doing well. I am doing well. I am happy to be back here. I have been away for a while. I have been busy with my studies. I have been busy with my work. I have been busy with my life. I have been busy with my family. I have been busy with my friends. I have been busy with my hobbies. I have been busy with my interests. I have been busy with my passions. I have been busy with my dreams. I have been busy with my goals. I have been busy with my ambitions. I have been busy with my aspirations. I have been busy with my plans. I have been busy with my projects. I have been busy with my ideas. I have been busy with my thoughts. I have been busy with my feelings. I have been busy with my emotions. I have been busy with my mind. I have been busy with my heart. I have been busy with my soul. I have been busy with my spirit. I have been busy with my body. I have been busy with my mind. I have been busy with my soul. I have been busy with my spirit. I have been
```

Note that the setting you are providing (`temperature = 0.1`  etc)  can influence on the generation.

So what are the best practices currently known to reduce this random ramble?

You should probably ask this question on [the forum](https://discuss.huggingface.co/), using the default parameters should already help (`temperature=0.9`), you can try to use `LogitsProcessor` for length penalty to reduce potential hallucination. You can also change the sampling stategies, use `top_k`, `contrastive search` etc etc. @gante will have better solution!

@AnishAlapattu-GEP @ArthurZucker 

This is actually a hard problem to solve, and I have yet to see a solution that generalizes well! A few things that can be tried, ranked by implementation complexity:
1. In the prompt, mention that you want a short output (be specific if you can, like "Reply in 3 sentences or less");
2. Add a custom logits processor that increases the score of the eos token according to some rule (e.g. scaling with the generated length) *EDIT* it is implemented in [this logits processor](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.EncoderRepetitionPenaltyLogitsProcessor)!
3. Generate text in excess and have a post-processing step to crop unwanted text (e.g. based on the conditional probability of each sequence -- when the model starts rambling, I suspect there is a significant drop in the probability of the sentence given the past sentences)
4. Fine-tune the model 🙃 

Thanks @ArthurZucker and @gante!

The issue stems from using bare Llama-2 model, instead of `-chat` version, which is fine-tuned to follow instructions. 

Bare llama-2 model is trained to complete text, so if you include the beginning of the conversation in the prompt, you should expect the rest of the conversation to be predicted by such model. In contrast, -chat models are trained to be more aligned or to follow the human instructions. This idea is nicely presented by openai in InstructGPT [article](https://openai.com/research/instruction-following)

![2023-09-01_12-04](https://github.com/huggingface/transformers/assets/21311210/e5f0df11-2557-4f70-b3f7-d08e28401d08)



☝️ Precisely

A thing I forgot to mention in my answer above (which I've edited in case someone stumbles upon it): we have a way to add a soft cap on the generated text length. e.g. If you want your output to be about 100 tokens unless the remaining tokens are really important for the answer, you can do it through [this logits processor](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.EncoderRepetitionPenaltyLogitsProcessor).