[Seq2Seq Trainer] Make sure padding is implemented for models without pad_token
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds padding for models without padding token as well. The logic is the following:

1) If model predicts `targets` < `max_length` => model has to have at least an `eos_token_id`. If model has no `config.pad_token_id` defined than the model simply uses the `config.eos_token_id` for padding.

2) If the model has no `config.eos_token_id`, => model cannot generate predictions shorter than `max_length`. In this case padding will never happen.

@sshleifer @patil-suraj - you guys were right -> the `Trainer` requires padding in any case (also if model has no padding token).
Could you guys review this PR and see if these fixes in Seq2Seq Trainer are ok for you?



Comments:
LGTM!

@patrickvonplaten , @sshleifer 
I am seeing a major slowdown on TPU V3-8,  
last time (9e68d075a4100906509170498480823e7e61874a) `sshleifer/student_marian_en_ro_6_3` finished 1 epoch in ~6 mins,
now on this branch it's showing ~1hr 20 mins

> @patrickvonplaten , @sshleifer
> I am seeing a major slowdown on TPU V3-8,
> last time ([9e68d07](https://github.com/huggingface/transformers/commit/9e68d075a4100906509170498480823e7e61874a)) `sshleifer/student_marian_en_ro_6_3` finished 1 epoch in ~6 mins,
> now on this branch it's showing ~1hr 20 mins

Ohoh :-/ can you narrow down the commit that caused the slow-down? I took a look again at https://github.com/huggingface/transformers/pull/7809/files and this line I added could be problematic `inputs = copy.deepcopy(inputs)`. 

> > @patrickvonplaten , @sshleifer
> > I am seeing a major slowdown on TPU V3-8,
> > last time ([9e68d07](https://github.com/huggingface/transformers/commit/9e68d075a4100906509170498480823e7e61874a)) `sshleifer/student_marian_en_ro_6_3` finished 1 epoch in ~6 mins,
> > now on this branch it's showing ~1hr 20 mins
> 
> Ohh, can you narrow down the commit that caused the slow-down? I took a look again at https://github.com/huggingface/transformers/pull/7809/files and this line I added could be problematic `inputs = copy.deepcopy(inputs)`.

Yeah this line is actually called at every step -> can you check whether removing the `copy` operation speeds the seq2seq trainer up again? I've been a bit sloppy there I think :-/ 

It's still very slow even after removing that line. I'll try to find the exact commit which is causing this slowdown.