Finetuned model generates incomprehensible text when used while in memory but works fine when loaded via saved checkpoints.
Author: sagewhocodes
Pull request: no

## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-5.4.0-1051-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten @LysandreJik @stas00

Models:
- gpt2-M, L, XL

Model I am using (Bert, XLNet ...): - GPT2-M, L, XL

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] Autoregressive text generation (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set the training arguments via `TrainingArguments` 
2. Load the pretrained model via `GPT2LMHeadModel.from_pretrained('gpt2')`
3. Initialize the `Trainer` with 'TrainingArguments`
4. run `Trainer.train()` 
5. After training has been completed, use the in-memory model to generate texts.

# Concerned snippets
```python 
self.training_args = TrainingArguments(
                                    "gpt2_model", deepspeed = self.ds_config,
                                    do_train = True,
                                    per_device_train_batch_size = 2,
                                    num_train_epochs = 10,
                                    logging_strategy = 'epoch',
                                    save_strategy = 'epoch', 
                                    fp16=True
                                    )
self.trainer = Trainer(
            model=self.model,
            args=self.training_args,
            train_dataset=self.dataset["train"],
            )
self.trainer.train()

def generate(self, prompt):
    ids = self.tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
    max_length = ids.shape[1] + 400
    gen_tokens = self.trainer.model.generate(
        ids,
        do_sample=True,
        min_length=max_length,
        max_length=max_length,
        temperature=0.9,
        use_cache=True,
    )
    gen_text = self.tokenizer.batch_decode(gen_tokens)[0]
    return gen_text
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The in-memory model obtained after training should generate text on the dataset it was finetuned on, but it generates incomprehensible text.
# Generated Output
![image](https://user-images.githubusercontent.com/54904272/124073253-8ac75680-da5f-11eb-9ff3-a14f57d31db3.png)


Comments:
Pinging @sgugger as using the `Trainer`

I think you forgot to put the model in evaluation mode maybe?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.