Weird Behavior in Finetuning Pegasus on a Custom Dataset/Longer Summaries Generated
Author: bharathc346
Pull request: no

## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.4.0-186-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No

### Who can help
@sshleifer 
 
## Information

I am using Pegasus

The problem arises when using my own modified scripts

The tasks I am working on summarization on my own dataset:
I am finetuning `google/pegasus-cnn_dailymail` on my own dataset


## Problem 
I ran finetuning w/ `google/pegasus-cnn_dailymail` on my dataset about two weeks ago w/ similar code and got much better results. I have saved these checkpoints and will refer to them as the "old checkpoints"

Now I am running roughly the same script finetuning `google/pegasus-cnn_dailymail` on my dataset and for some reason Pegasus seems to produce a lot of irrelevant tokens (maybe doesn't know when to stop properly). I also saved these checkpoints and will refer to them as the "new checkpoints". 

**Example**
```
Source: 
"Yes, please confirm the medication above.
What do you think could be causing constipation? I eat well, exercise, drink a lot of water, etc."


Predicted Target (old checkpoint): "Thinks could be causing constipation. Eats well, drinks a lot of water, etc."

Predicted Target (new checkpoint): "Eats well. Exercised a lot of water above water. Constipation. Medications causing constipation. Is situated in the right-sided abdomen."
```

Both of the predicted targets were generated with the same decoding code so I do not think it is a problem there. Since the new checkpoint does not do as well as old I suspect I am doing something wrong in my training script.

Here is how I am doing my training step: 
```
    def _train_step(self, batch):
        outputs = self._step(batch)
        lm_logits = outputs.logits
        labels = batch["target_input_ids"].to(self.device)

        loss = F.cross_entropy(lm_logits.view(-1, lm_logits.shape[-1]),
            labels.view(-1), ignore_index=0)
        return loss

    def _step(self, batch): 
        pad_token_id = self.tokenizer.pad_token_id
        decoder_input_ids = shift_tokens_right(
            batch["target_input_ids"], pad_token_id).to(self.device)
        decoder_input_ids[:, 0] = self.tokenizer.pad_token_id

        return self.model(
            input_ids=batch["source_input_ids"].to(self.device),
            attention_mask=batch["source_attention_mask"].to(self.device),
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=batch["target_attention_mask"].to(
                self.device),
            use_cache=False,
            return_dict=True,
        )
```

I double checked this with code in `examples/seq2seq` and `modeling_bart` and it seems to be reasonable. Only difference is when I do the shift_tokens_right I make sure to use Pegasus's decoder_start_token_id of 0 = pad_token_id rather than eos. I tried w and w/o this and the results seem to be similar. 

Also I trained both checkpoints with batch size of 4 accumulating 64 batches so effective batch size is 256 as suggested in the paper. 

Any idea where I am going wrong with this? 

Comments:
Great Q!
we added `min_length=32` to  many pegasus configs. Set `min_length=0` to fallback to the old behavior.
You shouldn't need to re-train.