Ensure OpenAI GPT position_ids is correctly initialized and registered at init.
Author: mfuntowicz
Pull request: yes

This will make it compatible with TorchScript export and avoid hardcoded `position_ids` tensor's device in the generated graph.

Solve the following issue: #5664 	

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
Fixes for the failing tests are not detected by Github while being present on the branch ... Lets see if it comes back to life in a while ...

CI failure has been fixed on master.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=h1) Report
> Merging [#5773](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/223bad242d0d64e20a39b956b73ab300231a9c70&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `90.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5773/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5773      +/-   ##
==========================================
- Coverage   78.67%   78.63%   -0.05%     
==========================================
  Files         146      146              
  Lines       26210    26206       -4     
==========================================
- Hits        20621    20606      -15     
- Misses       5589     5600      +11     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tb2JpbGViZXJ0LnB5) | `89.44% <66.66%> (-0.02%)` | :arrow_down: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.37% <100.00%> (-0.02%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.25% <100.00%> (-0.06%)` | :arrow_down: |
| [src/transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG0ucHk=) | `90.00% <100.00%> (-0.03%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.95% <0.00%> (-2.51%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5773/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `81.19% <0.00%> (-0.30%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=footer). Last update [b01a884...722824f](https://codecov.io/gh/huggingface/transformers/pull/5773?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sshleifer I've updated all the models with `position_ids` created in the forward pass as it seems to be the right design to allow people to export the bunch of models impacted. 

Also, it avoids allocating tensors at every forward call, so might reduce the pressure on PyTorch memory manager. 

Do we get the "Uninitialized parameters" warning because the position_ids are not in the `pytorch_model.bin` on S3?



I'll check the above, didn't catch at first 👌 

Could this be merged? Would love to test these changes after merging :)

@vdantu Now merged, sorry for the delay, many people are off these days, it slows down a little bit the merging process 😃. 

Let us know if you have any follow up issue(s) / question(s).

@mfuntowicz : Thanks for the push. Would we have to wait for a new release of `transformers` package to get these changes? 

@vdantu: I'll let @LysandreJik handle this one