Inconsistent in batch generation results
Author: qlwang25
Pull request: no

### System Info

load model code
```
model_path = "../../../pre-trained_models/h2oai-llama2-7b-chat"

tokenizer = LlamaTokenizer.from_pretrained(model_path)
config = LlamaConfig.from_pretrained(model_path)
config.max_length = 512

with init_empty_weights():
	model = LlamaForCausalLM._from_config(config, torch_dtype=torch.float16)
model.tie_weights()
model = load_checkpoint_and_dispatch(model, model_path, device_map="auto", no_split_module_classes=["LlamaDecoderLayer"], dtype=torch.float16)
model = model.eval()

model.generation_config = GenerationConfig.from_pretrained(pretrained_model_name=model_path, config_file_name='generation_config.json')
```

generate code
```
prompt1 = "Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Here are some examples: \n"
prompt1 += "Review: It is always reliable , never bugged and responds well ." + "\n"
prompt1 += "Label:[[responds, positive]]" + "\n"
prompt1 += "Review: The machine is slow to boot up and occasionally crashes completely ." + "\n"
prompt1 += "Label:[[boot up, negative]]" + "\n"
prompt1 += "Review: Enabling the battery timer is useless ." + "\n"
prompt1 += "Label:"

prompt2 = "Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Here are some examples: \n"
prompt2 += "Review: It rarely works and when it does it's incredibly slow ." + "\n"
prompt2 += "Label:[[works, negative]]" + "\n"
prompt2 += "Review: The machine is slow to boot up and occasionally crashes completely ." + "\n"
prompt2 += "Label:[[boot up, negative]]" + "\n"
prompt2 += "Review: Boot time is super fast , around anywhere from 35 seconds to 1 minute ." + "\n"
prompt2 += "Label:"

prompt3 = "Given a review, extract the aspect term(s) and determine their corresponding sentiment polarity. Here are some examples: \n"
prompt3 += "Review: It is always reliable , never bugged and responds well ." + "\n"
prompt3 += "Label:[[responds, positive]]" + "\n"
prompt3 += "Review: It rarely works and when it does it's incredibly slow ." + "\n"
prompt3 += "Label:[[works, negative]]" + "\n"
prompt3 += "Review: Boot time is super fast , around anywhere from 35 seconds to 1 minute ." + "\n"
prompt3 += "Label:"

tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer([prompt1, prompt2, prompt3], padding="longest", return_tensors="pt")
padding_len = inputs["input_ids"].size(1)
outputs = model.generate(**inputs, max_length=padding_len + 80, do_sample=False, num_beams=1)

for output in outputs:
    pred = tokenizer.decode(output[padding_len:], skip_special_tokens=True)
    pred = pred.split("\n")[0]
    print(pred)
```
---

When I use ```[prompt1, prompt2, prompt3]``` as input, the result is:
```
Љ [[battery timer, negative]]
Љ [[boot time, positive]]
[[fast, positive]]
```

When I use ```[prompt3, prompt2, prompt1]``` as input, the result is:
```
[[fast, positive]]
. [[boot time, positive]]
 [[useless, negative]]
```

Again, when I use ```[prompt3, prompt2, prompt1]``` as input, the result is: (the second result is empty)
```
[[fast, positive]]

Љ [[battery timer, negative]]
```

![image](https://github.com/huggingface/transformers/assets/38132016/2ccdcf52-ff80-482c-bfc7-ca3e47bd3640)


**Problem**
(1) Why does the prompt produce different results with different inputs (or the same input)?
(2) Why is this character (Љ) generated?
(3) Why do batch generation (batch_size=3) and individual generation (batch_size=1) have different results

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

please see the code above  

### Expected behavior

I expect the results by the batch generated to not be affected by the prompt order.
And,  each sample should be the same as the results generated separately (batch_size)

Comments:
Hi @qlwang25 👋 

Have a look at [this comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535), which I believe answers your question :) 

Thank you very much for your reply. 
The reason is ```torch type``` instead of ```padding```. Just changing it to ```torch.bfloat16``` (or ```torch.float32```) works.