Does the model.generate supports batch_size > 1 ?
Author: liuchengyuan123
Pull request: no

### System Info

Does the function `model.generate` supports the case when batch size of `input_ids` > 1?
It is required especially for evaluation!

The following bugs are reported when I call `model.generate` to generate 2 or more `input_ids`, where model is a`LlamaForCausalLM`:
```
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model = LlamaForCausalLM.from_pretrained('./chinese-alpaca-7b-merged', device_map="auto").half().cuda()
request_text = ["xxx", "yy"]
input_ids = tokenizer(request_text, return_tensors='pt', padding="longest", truncation=True, max_length=1024)
response = model.generate(input_ids=input_ids.input_ids.cuda(), max_new_tokens=1024, temperature=1,top_k=40,top_p=0.9,repetition_penalty=1.15)

### Expected behavior

```
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

Comments:
Yes, I'm specifically adding docs for it in #24432 

FYI this is the script you can use for batched generation:
```
from transformers import LlamaTokenizer, AutoModelForCausalLM
import torch

tokenizer = LlamaTokenizer.from_pretrained("openlm-research/open_llama_3b")
model = AutoModelForCausalLM.from_pretrained("openlm-research/open_llama_3b", torch_dtype=torch.float16, device_map="auto")

tokenizer.padding_side = "left"

# Define PAD Token = EOS Token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = model.config.eos_token_id

# use different length sentences to test batching
sentences = [
         "Hello, my dog is a little",
          "Today, I",
 ]

inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)

output_sequences = model.generate(**inputs, max_new_tokens=20)

print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
```

> FYI this is the script you can use for batched generation:
> 
> ```
> from transformers import LlamaTokenizer, AutoModelForCausalLM
> import torch
> 
> tokenizer = LlamaTokenizer.from_pretrained("openlm-research/open_llama_3b")
> model = AutoModelForCausalLM.from_pretrained("openlm-research/open_llama_3b", torch_dtype=torch.float16, device_map="auto")
> 
> tokenizer.padding_side = "left"
> 
> # Define PAD Token = EOS Token
> tokenizer.pad_token = tokenizer.eos_token
> model.config.pad_token_id = model.config.eos_token_id
> 
> # use different length sentences to test batching
> sentences = [
>          "Hello, my dog is a little",
>           "Today, I",
>  ]
> 
> inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)
> 
> output_sequences = model.generate(**inputs, max_new_tokens=20)
> 
> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
> ```

@NielsRogge hi, just curious about what `Define PAD Token = EOS Token` is for ?

By default, the padding token is not set in the tokenizer's config: https://huggingface.co/openlm-research/open_llama_3b/blob/main/tokenizer_config.json. So when you would pad you would get the following error:

```
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

> 

> FYI this is the script you can use for batched generation:
> 
> ```
> from transformers import LlamaTokenizer, AutoModelForCausalLM
> import torch
> 
> tokenizer = LlamaTokenizer.from_pretrained("openlm-research/open_llama_3b")
> model = AutoModelForCausalLM.from_pretrained("openlm-research/open_llama_3b", torch_dtype=torch.float16, device_map="auto")
> 
> tokenizer.padding_side = "left"
> 
> # Define PAD Token = EOS Token
> tokenizer.pad_token = tokenizer.eos_token
> model.config.pad_token_id = model.config.eos_token_id
> 
> # use different length sentences to test batching
> sentences = [
>          "Hello, my dog is a little",
>           "Today, I",
>  ]
> 
> inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)
> 
> output_sequences = model.generate(**inputs, max_new_tokens=20)
> 
> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
> ```

This also works fine for other 13b models based on Llama, Thanks a lot.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> FYI this is the script you can use for batched generation:
> 
> ```
> from transformers import LlamaTokenizer, AutoModelForCausalLM
> import torch
> 
> tokenizer = LlamaTokenizer.from_pretrained("openlm-research/open_llama_3b")
> model = AutoModelForCausalLM.from_pretrained("openlm-research/open_llama_3b", torch_dtype=torch.float16, device_map="auto")
> 
> tokenizer.padding_side = "left"
> 
> # Define PAD Token = EOS Token
> tokenizer.pad_token = tokenizer.eos_token
> model.config.pad_token_id = model.config.eos_token_id
> 
> # use different length sentences to test batching
> sentences = [
>          "Hello, my dog is a little",
>           "Today, I",
>  ]
> 
> inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)
> 
> output_sequences = model.generate(**inputs, max_new_tokens=20)
> 
> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
> ```

Just added
 model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    offload_folder="offload", offload_state_dict = True,
    torch_dtype=torch.float16,
    device_map="auto"
) 
and it worked for model ="llama/llama-2-7b-chat-hf" .
Thanks @NielsRogge for support.