GPT2 error when we try to run torch.jit.script 
Author: vdantu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.2
- Platform: Ubuntu 18.04
- Python version: Python3.6
- PyTorch version (GPU?): 1.9.0 GPU
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help 
@patrickvonplaten, @LysandreJik

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
I was trying to run torch.jit.script and I get the following error from JIT frontend 

```
 File "/home/ubuntu/anaconda3/envs/python3/lib/python3.6/site-packages/torch/jit/frontend.py", line 330, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported:
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.6/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 756
                # Ensure layer_past is on same device as hidden_states (might not be correct)
                if layer_past is not None:
                    layer_past = tuple(past_state.to(hidden_states.device) for past_state in layer_past)
                                       ~ <--- HERE
                # Ensure that attention_mask is always on the same device as hidden_states
                if attention_mask is not None:
```

I am curious to know if this is a known issue or learn if I am doing something wrong.
#### Sample Code:

```
from transformers import GPT2LMHeadModel, GPT2Config
import torch
configuration = GPT2Config(n_embd=1600, n_layer=48, n_head=25)
model = GPT2LMHeadModel(configuration)
script_model = torch.jit.script(model.base_model)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Can you try the following: 

```python
from transformers import GPT2LMHeadModel, GPT2Config
import torch
configuration = GPT2Config(n_embd=1600, n_layer=48, n_head=25, use_cache=False)
model = GPT2LMHeadModel(configuration)
script_model = torch.jit.script(model.base_model)
```

just to first see whether disabling the cache solves the problem

> Can you try the following:
> 
> ```python
> from transformers import GPT2LMHeadModel, GPT2Config
> import torch
> configuration = GPT2Config(n_embd=1600, n_layer=48, n_head=25, use_cache=False)
> model = GPT2LMHeadModel(configuration)
> script_model = torch.jit.script(model.base_model)
> ```
> 
> just to first see whether disabling the cache solves the problem

I still see the same error.
```
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported:
  File "/home/ubuntu/anaconda3/envs/python3/lib/python3.6/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 756
                # Ensure layer_past is on same device as hidden_states (might not be correct)
                if layer_past is not None:
                    layer_past = tuple(past_state.to(hidden_states.device) for past_state in layer_past)
                                       ~ <--- HERE
                # Ensure that attention_mask is always on the same device as hidden_states
                if attention_mask is not None:
```

This seems like GPT model as it is implemented today is not supported by `torch.jit.script` because the model uses generator expr and Script doesn't support it yet. Is this accurate or am I missing anything? Also, would `torch.jit.trace`ing the GPT2 model cause any correctness issues? 

Hello! The documentation regarding TorchScript can be found [here](https://huggingface.co/transformers/serialization.html#torchscript).

You're correct in your analysis: `torch.jit.script` isn't supported, while `torch.jit.trace` is supported!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.