GPT-Neo batch inferencing with sampling results unexpected output
Author: callzhang
Pull request: no

## Environment info

- `transformers` version: 4.13.0
- Platform: Linux-5.4.0-96-generic-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@cccntu @patil-suraj 

Models:
GPT-Neo

Library:
- Pipelines: @Narsil

## Information

I want to speed up the text generation work by using batching, and at the same time generate more text by using sampling.
But the result is abnormal


## To reproduce

Steps to reproduce the behavior:
```python
texts = [
    'Have a line of communication. You have two lines of communication.',
    'Wanting this is bad. Tell me to go ahead.',
    "I found a colony of bats in the steeple of St. Olaf's church while you were dating my brother.",
    "Fight so you don't have to do it again"
]
model_name = 'EleutherAI/gpt-neo-1.3B'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, pad_token_id=tokenizer.eos_token_id)
pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device=0)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = 'left'
def test_batch(texts, batch_size=1):
    results = pipe(
        texts, batch_size=batch_size, 
        max_length=50, 
        pad_token_id=tokenizer.eos_token_id,
        repetition_penalty=2.0,
        do_sample = True,
        num_return_sequences = 8,
        )
    results = [ri['generated_text'] for r in results for ri in r]
    return results
test_batch(texts, 4)
'''
['Have a line of communication. You have two lines of communication. One being when somebody is on the way to you or your home and another is over there, waiting for an answer. You want',
 'Wanting this is bad. Tell me to go ahead.o lines of communication. The first is the person who gave birth to you. The other is the person who has been listening to you and observing you for',
 "I found a colony of bats in the steeple of St. Olaf's church while you were dating my brother.can give you the results in your report. Second, you need to understand the results or we are done",
 'Fight so you don\'t have to do it again two lines of communication. The first is the "official"\nsocial media account and the other is a personal one. Most organizations, government entities,\n']
'''
```


## Expected behavior
Generate normal text.



Comments:
Hi @callzhang you are perfectly correct.

This issue is related to both arguments not working nice with each other, I created a PR to add 2 tests for both `text-generation` and `text2text-generation` pipelines (both were affected).

Thanks for reporting this ! 

@Narsil Thanks for the prompt reply and fix. Really appreciate that!

@callzhang Did this fix your issue?
I'm trying to understand how the `text-generation` pipeline works with inputting batches (as you properly set `tokenizer.padding_side = 'left'`) but the pipeline itself has `padding=False`:

https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/src/transformers/pipelines/text_generation.py#L175

Maybe @Narsil can provide further insight?

Thanks in advance.

BTW: I'm currently trying to do it with GPT-J6B without luck, `batch_size=1` works great, for larger batches I'm getting very weird output (seem like inputs are mixing together).