PEGASUS generation/decoding VERY Slow
Author: muggin
Pull request: no
Labels: name: wontfix

name: Summarization

## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.19.112+-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@sshleifer 
 
## Information

Model I am using: *PEGASUS-large*, *PEGASUS-cnn_dailymail*, *PEGASUS-xsum*

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I compared the generation/decoding performance of BART and PEGASUS both loaded via AutoModelSeq2SeqForLM fine-tuned on a custom dataset for the same amount of time. Generation was done using basic greedy decoding. PEGASUS models are anywhere between 5-15x slower than BART. Fine-tuning speed was on-par for both models.

## To reproduce

Steps to reproduce the behavior:
1. Load PEGASUS and BART from any of the mentioned checkpoints (using AutoModelSeq2SeqForLM)
1. Fine-tune models
2. Decode using greedy decoding
3. Compare fine-tuning performance with other Seq2Seq Model (BART-large)


## Expected behavior
Decoding performance on-par with BART-large?


Comments:
this is expected behavior. pegasus has more layers and higher num_beams than BART.
try using num_beams=2 for both models, and or using a distilled model for faster inference. BART also works in fp16.

I used greedy decoding when generating outputs, so the number of beams should not matter in this case.
Also, PEGASUS is 30% larger than BART, so why does it translate to at least 5x performance drop at inference?

I can't investigate deeply without a code snippet, but a 5X difference in greedy decoding time for roughly the same number of output characters is larger than I would have expected.

I'd try to verify if the finetuning is, in fact, a part of this problem or whether you're generation parameters/data can demonstrate the slowdown using public models, e.g.

I just ran:

```python
from transformers import *
import time 
torch_device = 'cuda'
for mname in ['facebook/bart-large-xsum', 'google/pegasus-xsum']:
    model = AutoModelForSeq2SeqLM.from_pretrained(mname).to(torch_device)
    tok = AutoTokenizer.from_pretrained(mname)
    batch = tok(['I am a small frog'], return_tensors='pt').to(torch_device)
    t0 = time.time()
    model.generate(**batch, min_length=0, max_length=10, num_beams=1)
    runtime = time.time() - t0
    print(f'{mname}: {runtime:.3f}')
```

and got
```
facebook/bart-large-xsum: 0.089
google/pegasus-xsum: 0.116
```
which is a < 30% difference.

In our case, the model is part of a larger system, so can't share the code I am using. 
However, I will create a minimal example that allows me to reproduce this issue (most likely by Monday).

One difference I already see is that in our code we work with long sequences that max out the model's limits both on the input and output side. I just started another batch of generation and on the exact same input data BART-large is projected to finish in 45 minutes, PEGASUS-large in 14 hours, both models running on A100 GPUs (one per model).


Edit:
@sshleifer running the script you provided gives the following (avg. across 5 runs):
```
facebook/bart-large-xsum: 0.570
google/pegasus-xsum: 0.952
```


Cool.
+ If you want smaller models you can checkout the distilbart/distill-pegasus variants [here](https://huggingface.co/models?search=sshleifer%2Fdistil) [Paper w Details](https://arxiv.org/abs/2010.13002)
+ `bart-large/pegasus-large` probably can't generate very well (unless you've fine-tuned them)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.