Output sequences are different between SummarizationPipeline and model.generate
Author: tagucci
Pull request: no

## Environment info
transformers version: 4.12.5
Platform: Linux
Python version: 3.8.5
PyTorch version (GPU?): 1.10 
Tensorflow version (GPU?): N/A
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @patil-suraj

## Information
Model I am using (Bert, XLNet ...): Bart (ConditionalGeneration)
The problem arises when using:
 - [ ] the official example scripts: (give details below)
 - [x] my own modified scripts: (give details below)

The tasks I am working on is:
 - [ ]  an official GLUE/SQUaD task: (give the name)
 - [x] my own task or dataset: (give details below)

When generating summaries using [`sshleifer/distilbart-cnn-12-6`](https://huggingface.co/sshleifer/distilbart-cnn-12-6), I encountered the following phenomenon. `SummarizationPipeline` and `model.generate` outputs are almost different even though the same parameters. 

Are there any solutions or tips to generate same summaries？

## To reproduce

sample code is below.

```python
from transformers import (
    BartTokenizer,
    BartForConditionalGeneration,
    SummarizationPipeline,
)

# sample document from https://huggingface.co/sshleifer/distilbart-cnn-12-6
doc = "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."

# load model & tokenizer
tokenizer = BartTokenizer.from_pretrained("sshleifer/distilbart-cnn-12-6")
model = BartForConditionalGeneration.from_pretrained("sshleifer/distilbart-cnn-12-6")
input_ids = tokenizer.encode(doc, return_tensors="pt").to("cuda:0")

# pipeline
pipe = SummarizationPipeline(model=model, tokenizer=tokenizer, device=0)

# generation params
params = model.config.task_specific_params["summarization"]
print(f"params: {params}")

# several num_beams
for num_beams in [1, 5, 10]:
    params["num_beams"] = num_beams
    print(f"num_beams: {num_beams}")
    pipeline_output = pipe(doc, **params, clean_up_tokenization_spaces=True)[0][
        "summary_text"
    ]
    output = model.generate(input_ids, **params)
    generate_output = tokenizer.batch_decode(output, skip_special_tokens=True)[0]
    print(f"SummarizationPipeline:\n{pipeline_output}")
    print(f"model.generate:\n{generate_output}")
    print(
        f"SummarizationPipeline == model.generate: {pipeline_output == generate_output}\n"
    )

print("=" * 50)
params["num_beams"] = 5
params["min_length"] = 10
# several max_length
for max_length in [20, 50, 100]:
    params["max_length"] = max_length
    print(f"max_length: {max_length}")
    pipeline_output = pipe(doc, **params, clean_up_tokenization_spaces=True)[0][
        "summary_text"
    ]
    output = model.generate(input_ids, **params)
    generate_output = tokenizer.batch_decode(output, skip_special_tokens=True)[0]
    print(f"SummarizationPipeline:\n{pipeline_output}")
    print(f"model.generate:\n{generate_output}")
    print(
        f"SummarizationPipeline == model.generate: {pipeline_output == generate_output}\n"
    )

```

output is below.

```
params: {'early_stopping': True, 'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'no_repeat_ngram_size': 3, 'num_beams': 4}
num_beams: 1
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It is the tallest structure in Paris and is the second tallest free-standing structure in France. It was the first structure to reach a height of 300 metres.
model.generate:
 The Eiffel Tower is 324 metres tall, about the same height as an 81-storey building. It is the tallest structure in Paris and is the second tallest free-standing structure in France. It was the first structure to reach a height of 300 metres.
SummarizationPipeline == model.generate: False

num_beams: 5
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. It is now taller than the Chrysler Building by 5.2 metres (17 ft) Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
model.generate:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
SummarizationPipeline == model.generate: False

num_beams: 10
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It is now taller than the Chrysler Building by 5.2 metres (17 ft) Excluding transmitters, it is the second tallest free-standing structure in France after the Millau Viaduct.
model.generate:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft) Excluding transmitters, it is the second tallest free-standing structure in France after the Millau Viaduct.
SummarizationPipeline == model.generate: False

==================================================
max_length: 20
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same
model.generate:
 The Eiffel Tower is 324 metres (1,063 ft) tall,
SummarizationPipeline == model.generate: False

max_length: 50
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. It is now taller than the Chrysler Building
model.generate:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. It is now taller than the Chrysler Building
SummarizationPipeline == model.generate: True

max_length: 100
SummarizationPipeline:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. It is now taller than the Chrysler Building by 5.2 metres (17 ft) Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France.
model.generate:
 The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. It was the first structure to reach a height of 300 metres. Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France.
SummarizationPipeline == model.generate: False
```

## Expected behavior

Same summaries are generated regardless of `SummarizationPipeline` or  `model.generate`.

Comments:
Hey @tagucci,

Good question! It took me some time to find the difference :D . The pipeline automatically takes the `model.config.prefix` parameter into account as well. So if you change your script as follows you should get the same results: 

```diff
from transformers import (
    BartTokenizer,
    BartForConditionalGeneration,
    SummarizationPipeline,
)

# sample document from https://huggingface.co/sshleifer/distilbart-cnn-12-6
doc = "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."

# load model & tokenizer
tokenizer = BartTokenizer.from_pretrained("sshleifer/distilbart-cnn-12-6")
model = BartForConditionalGeneration.from_pretrained("sshleifer/distilbart-cnn-12-6")
-input_ids = tokenizer.encode(doc, return_tensors="pt").to("cuda:0")
+input_ids = tokenizer.encode(model.config.prefix + doc, return_tensors="pt").to("cuda:0")

# pipeline
pipe = SummarizationPipeline(model=model, tokenizer=tokenizer, device=0)

# generation params
params = model.config.task_specific_params["summarization"]
print(f"params: {params}")

# several num_beams
for num_beams in [1, 5, 10]:
    params["num_beams"] = num_beams
    print(f"num_beams: {num_beams}")
    pipeline_output = pipe(doc, **params, clean_up_tokenization_spaces=True)[0][
        "summary_text"
    ]
    output = model.generate(input_ids, **params)
    generate_output = tokenizer.batch_decode(output, skip_special_tokens=True)[0]
    print(f"SummarizationPipeline:\n{pipeline_output}")
    print(f"model.generate:\n{generate_output}")
    print(
        f"SummarizationPipeline == model.generate: {pipeline_output == generate_output}\n"
    )

print("=" * 50)
params["num_beams"] = 5
params["min_length"] = 10
# several max_length
for max_length in [20, 50, 100]:
    params["max_length"] = max_length
    print(f"max_length: {max_length}")
    pipeline_output = pipe(doc, **params, clean_up_tokenization_spaces=True)[0][
        "summary_text"
    ]
    output = model.generate(input_ids, **params)
    generate_output = tokenizer.batch_decode(output, skip_special_tokens=True)[0]
    print(f"SummarizationPipeline:\n{pipeline_output}")
    print(f"model.generate:\n{generate_output}")
    print(
        f"SummarizationPipeline == model.generate: {pipeline_output == generate_output}\n"
    )
```

That's the line in question: https://github.com/huggingface/transformers/blob/8395f14de6068012787d83989c3627c3df6a252b/src/transformers/pipelines/text2text_generation.py#L88

@patrickvonplaten 
In my understanding, BART does not use prefix token unlike T5. So I had no idea to append prefix token.
I could generate same outputs both model.generate and pipeline:D
Thanks!