Converted TF model cannot generate line breaks anymore
Author: MaxGodTier
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.3
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.0 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

Models:

- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- TensorFlow: @Rocketknight1

Library:

- Text generation: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...):

GPT-2

The problem arises when using:
* [X] the official example scripts: (give details below)

When using the command [transformers-cli convert](https://huggingface.co/transformers/converting_tensorflow_models.html)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)
My dataset is structured like
```
[WP]Input data[RESPONSE]Output data
Output 1
Output 2
Output 3
...


```

## To reproduce

Steps to reproduce the behavior:

1. I converted my Tensorflow model to Pytorch by running `transformers-cli convert --model_type gpt2 --tf_checkpoint checkpoint/run1 --pytorch_dump_output pytorch --config checkpoint/run1/hparams.json`
2. The resulting "pytorch_model.bin" is completely unable to generate line breaks, my specific application needs them to operate correctly or the output will not make any sense at all.
3. For simplicity sake, let's say the Tensorflow model output looks like this:
```
123
456
789
```
The converted "pytorch_model.bin" output looks like:
`123456789`
There's no way to untangle this mess, not even manually, fixes or suggestions are most appreciated.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The Pytorch converted model output should contain line breaks just like the Tensorflow source model.
Expected output:
```
123
456
789
```
Incorrect output:
`123456789`

Comments:
Hi, this is an interesting issue, and I suspect it doesn't have much to do with either TF or PyTorch. The reason is that models like GPT2 output integer token IDs, and if you're getting the same text either way then it likely means your models are outputting the same tokens. The problem is therefore probably that your newline characters aren't getting printed properly. I suggest comparing both models to confirm they're outputting the same token IDs, and if so, following the code that decodes and prints those IDs as text to identify where the issue has arisen.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @MaxGodTier, 

How did you train your model? 

@patrickvonplaten 
My model has been trained with https://github.com/minimaxir/gpt-2-simple
I create a huge text file (ie. dataset.txt) structured like this:
```
[WP]Input data X[RESPONSE]Output data X
Output X1
Output X2
...


[WP]Input data Y[RESPONSE]Output data Y
Output Y1
Output Y2
...


```
Then I preencode and compress the dataset like this:
```
import gpt_2_simple as gpt2
import os
import requests
file_name = "dataset.txt"
gpt2.encode_dataset(file_name)
```
Now I have "dataset.npz", I download a pretrained model:
```
model_name = "355M"
if not os.path.isdir(os.path.join("models", model_name)):
    print(f"Downloading {model_name} model...")
    gpt2.download_gpt2(model_name=model_name)   # model is saved into current directory under /models/355M/
```
Then finetune it:
```
file_name = "dataset.npz"
sess = gpt2.start_tf_sess()
gpt2.finetune(sess,
              file_name,
              model_name=model_name,
              steps=250000)   # steps is max number of training steps
gpt2.generate(sess)
```
After training, the Tensorflow model can generate line breaks, but if I convert it to Pytorch with `transformers-cli convert --model_type gpt2 --tf_checkpoint checkpoint/run1 --pytorch_dump_output pytorch --config checkpoint/run1/hparams.json` the resulting "pytorch_model.bin" is completely unable to generate line breaks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry for bumping this, the problem still persists.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @MaxGodTier,

Sorry to be so incredibly slow here. I sadly won't find the time to look further into this. Especially since the model was trained with a library that is different to `transformers`.