LLaMA `generate` output changes depending on batch size
Author: ryan-caesar-ramos
Pull request: no

### System Info

```
- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.8 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
```

### Who can help?

@younesbelkada @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

These [cells](https://colab.research.google.com/drive/1nAz2MUphzg5ifWW3CfRCJdYUgSYN5ynH?usp=sharing) should reproduce the error.

### Expected behavior

**Concise version:** I was expecting the results to not change whether or not the inference was batched or not.

Long version: Basically when I run `generate` with just one tokenized sequence, I get a certain result, and when I process the same sequence but inside a batch instead, the result changes. To make sure it wasn't any tokenization shenanigans, I tokenized the whole batch and took just the parts that related to the weird sequence (basically just `{k: v[1:] for k, v in tokenized_inputs.items()}` where the batch is just two items and the weird one is the second item).

For clarity, when I pass just

```python3
"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.

### Input:
Is this a baseball game? no

### Response:
"""
```

to the generate function of a `LlamaForCausalLM` wrapped in a `PeftModel`, I get

```
This is not a baseball game.
```

However, if I put it in a two-item batch, the output for some reason is instead

```
A photo of people playing a game.
```

For more details, the base model was 8-bit quantized and the LoRA weights should be at half-precision. The weights were taken from `decapoda-research/llama-7b-hf` and `tloen/alpaca-lora-7b` respectively.

Edit: forgot to make the notebook public, should be fixed now.

Comments:
Hey @ryan-caesar-ramos 👋 

The particular base checkpoint you're using (`decapoda-research/llama-7b-hf`) is not compatible with transformers, so we do not provide support for related problems :) 

If you have access to the original Meta weights, you can use other checkpoints as a starting point (e.g. [these](https://huggingface.co/huggyllama)). If the issue you're seeing still persists after updating the checkpoint, I'd be happy to take a look!

Hi @gante ! Thanks, I'll try to look into this, but I'm unable to use repos like `huggyllama/llama-7b` that have shard sizes around 10GB since my CPU ram can't handle it. Any way I can tell if a checkpoint is supported or not? Maybe there's a sharded one out there that is compatible

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Same thing happens when I use the original meta llama2 models. When using batch, the answers are completely broken.

> Same thing happens when I use the original meta llama2 models. When using batch, the answers are completely broken.

Did you initialize your tokenizer with left padding?

> 

@zhilif No idea what you mean. this is how I load it:
```
generator = Llama.build(
    
    ckpt_dir="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct",
    tokenizer_path="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct/tokenizer.model",
    max_seq_len=max_seq_len,
    max_batch_size=max_batch_size,
    model_parallel_size = 1 # num of worlds/gpus
)
```


> > 
> 
> @zhilif No idea what you mean. this is how I load it:
> 
> ```
> generator = Llama.build(
>     
>     ckpt_dir="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct",
>     tokenizer_path="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct/tokenizer.model",
>     max_seq_len=max_seq_len,
>     max_batch_size=max_batch_size,
>     model_parallel_size = 1 # num of worlds/gpus
> )
> ```

Is this a huggingface interface? If so, can you point to me its doc page?

> > > 
> > 
> > 
> > @zhilif No idea what you mean. this is how I load it:
> > ```
> > generator = Llama.build(
> >     
> >     ckpt_dir="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct",
> >     tokenizer_path="C:/AI/LLaMA2_Docker_FileSystem/codellama/CodeLlama-7b-Instruct/tokenizer.model",
> >     max_seq_len=max_seq_len,
> >     max_batch_size=max_batch_size,
> >     model_parallel_size = 1 # num of worlds/gpus
> > )
> > ```
> 
> Is this a huggingface interface? If so, can you point to me its doc page?

@zhilif https://github.com/facebookresearch/codellama/blob/main/example_instructions.py

@realhaik 👋 As @zhilif pointed out, that is not a hugging face interface :) Have a look at this [blog post](https://huggingface.co/blog/codellama) for examples

>  out, that is not a hugging face interface :) Have a look at this [blog post](https://huggingface.co/blog/codellama) for examples

@gante  unfortunately all the hugging face llama2 models are trash, the answers are completely broken.
If you work with hugging face llama models, I feel really sorry for you.

> Same thing happens when I use the original meta llama2 models. When using batch, the answers are completely broken. 

I thought you said you had broken results with the original codebase as well? 

> > Same thing happens when I use the original meta llama2 models. When using batch, the answers are completely broken.
> 
> I thought you said you had broken results with the original codebase as well?

@ArthurZucker I am talking only about original, I don't use hugging face. 
When using batch in original llama2 models, the answers are broken.