Replicating PEGASUS results on a benchmark dataset
Author: sajastu
Pull request: no

I'm trying to replicate the PEGASUS results on Reddit-TIFU dataset, but the scores I'm getting are a bit far from what has been reported in the main paper. I'm using the same test set as the one authors used in the main paper (80-10-10 splits based on `TensorflowDataset` according to their code-base). Would anyone have had similar experience of working with PEGASUS on either of reported datasets? Although I'm looking to replicate Reddit-TIFU results, but that would be also good to see if anyone could get the results replicated on either of the experimental datasets. 

It has to be mentioned that I'm using the finetuned checkpoint on the Reddit-TIFU dataset: `google/pegasus-reddit_tifu` without further fine-tuning (actually I don't need that) using the following script; `pegasus.sh`

```
CUDA_VISIBLE_DEVICES=0,1,2,3,4 python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path google/pegasus-reddit_tifu \
    --do_predict \
    --train_file $DS_BASE_DIR/train.json \
    --validation_file $DS_BASE_DIR/val.json \
    --test_file $DS_BASE_DIR/test.json \
    --output_dir /home/code-base/user_space/saved_models/pegasus/ \
    --per_device_train_batch_size=2 \
    --per_device_eval_batch_size=2 \
    --overwrite_output_dir \
    --predict_with_generate \
    --text_column text \
    --summary_column summary 
```

The scores I'm achieving: `do_predict` output:

> ***** predict metrics *****
>   predict_gen_len            =     40.294
>   predict_loss               =     3.9969
> **predict_rouge1             =    27.13
>   predict_rouge2             =     8.38
>   predict_rougeL             =    20.68**
>   predict_samples            =       4214


However, the (best) reported scores are:

> **predict_rouge1             =    26.63
>   predict_rouge2             =     9.01
>   predict_rougeL             =    21.60**


Even, assuming that `google/pegasus-reddit_tifu`'s pretraining is [improved](https://huggingface.co/google/pegasus-reddit_tifu) (Mixed & Stochastic), I can't reproduce the reported results on Reddit-TIFU, which are:  R-1: 27.99/       R-2:  9.81/       R-L: 22.94

## Environment info
- `transformers` version: 4.7.0 dev
- Platform: Linux Ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?): --
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes, I'm using four GPUs for prediction.

### Who can help
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @sshleifer, @patrickvonplaten, @patil-suraj


## Information

Model I am using (Bert, XLNet ...): PEGASUS

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ x] an official GLUE/SQUaD task: (give the name): Reddit-TIFU
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `bash pegasus.sh` _bash script is posted above_

## Expected behavior

I expect to be able to reproduce the official results reported in the main PEGASUS paper on Reddit-TIFU dataset; however, I'm getting higher Rouge-1 score, while lower Rouge-2 and Rouge-L scores.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.