How to generate texts in huggingface in a batch way?
Author: yananchen1116
Pull request: no

I am new to huggingface. My task is quite simple, where I want to generate contents based on the given titles. The below codes is of low efficiency, that the GPU Util is only about 15%. It seems that it makes generation one by one. How can I improve the code to process and generate the contents in a batch way?

```
df_test = pd.read_csv("./ag_news/test.csv").sample(frac=1)
from transformers import pipeline
text_generator = pipeline("text-generation")
rows = df_test.sample(1000)
titles = rows['title'].tolist()
contents = rows['content'].tolist()
generate_texts = text_generator(titles, max_length=40, do_sample=False)
```

Comments:
Looking at the [source code](https://github.com/huggingface/transformers/blob/4c32f9f26e6a84f0d9843fec8757e6ce640bb44e/src/transformers/pipelines/text_generation.py#L108) of the text-generation pipeline, it seems that the texts are indeed generated one by one, so it's not ideal for batch generation. 

In order to genere contents in a batch, you'll have to use GPT-2 (or another generation model from the hub) directly, like so (this is based on PR #7552):

```
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.padding_side = "left" 
tokenizer.pad_token = tokenizer.eos_token # to avoid an error
model = GPT2LMHeadModel.from_pretrained('gpt2')

device = 'cuda' if torch.cuda.is_available() else 'cpu'

texts = ["this is a first prompt", "this is a second prompt"]
encoding = tokenizer(texts, padding=True, return_tensors='pt').to(device)
with torch.no_grad():
    generated_ids = model.generate(**encoding)
generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

In my case, this prints out: 
`['this is a first prompt for the user to enter the password.\n\nThe password is a string', "this is a second prompt, but it's not a full-screen one.\n\nThe first"]`

> Looking at the [source code](https://github.com/huggingface/transformers/blob/4c32f9f26e6a84f0d9843fec8757e6ce640bb44e/src/transformers/pipelines/text_generation.py#L108) of the text-generation pipeline, it seems that the texts are indeed generated one by one, so it's not ideal for batch generation.
> 
> In order to genere contents in a batch, you'll have to use GPT-2 (or another generation model from the hub) directly, like so (this is based on PR #7552):
> 
> ```
> from transformers import GPT2Tokenizer, GPT2LMHeadModel
> import torch
> 
> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> tokenizer.padding_side = "left" 
> tokenizer.pad_token = tokenizer.eos_token # to avoid an error
> model = GPT2LMHeadModel.from_pretrained('gpt2')
> 
> device = 'cuda' if torch.cuda.is_available() else 'cpu'
> 
> texts = ["this is a first prompt", "this is a second prompt"]
> encoding = tokenizer(texts, return_tensors='pt').to(device)
> with torch.no_grad():
>     generated_ids = model.generate(**encoding)
> generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
> ```
> 
> In my case, this prints out:
> `['this is a first prompt for the user to enter the password.\n\nThe password is a string', "this is a second prompt, but it's not a full-screen one.\n\nThe first"]`

thanks. It seems that the standard workflow is to organize the components of `tokenizer`, `generate` and `batch_decode`
in a cascade way. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge  Is padding on the left still the way to go for batched generation? It seems odd to require a workaround for such a common feature

cc'ing @gante here, but I think yes you need to pad on the left during batched generation, simply because the model isn't trained with padding tokens on the right (you don't want the model to continue "hello world pad pad pad ....", so what you do is "pad pad pad hello world ...." in a batch).

👉 https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side

@NielsRogge I was wondering if it can infer multiple input sentences with beam search = 4 right now?

@NielsRogge to confirm - this has changed now and pipelines support batching, right?
https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/pipelines#pipeline-batching