Support various BERT relative position embeddings
Author: zhiheng-huang
Pull request: yes

# What does this PR do?

The default BERT model `bert-base-uncased` was pre-trained with absolute position embeddings. We provide three pre-trained models which were pre-trained on the same training data (BooksCorpus and English Wikipedia) as in the BERT model training, but with different relative position embeddings (Shaw et al., Self-Attention with Relative Position Representations, https://arxiv.org/abs/1803.02155 and Huang et al., Improve Transformer Models with Better Relative Position Embeddings, https://arxiv.org/abs/2009.13658, accepted in findings of EMNLP 2020). We show how to fine-tune these pre-trained models on SQuAD1.1 data set and we also report the EM and F1 score on SQuAD1.1 dev dataset. See examples/question-answering/README.md for more details.

Fixes # (issue)

N/A

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik @julien-c @patrickvonplaten



Comments:
Hey @zhiheng-huang,

Thanks for the PR!

In general I'm fine with this PR - think adding more types of position embeddings to a model is OK. 
Also can you rebase your PR to the most current version of `master` - I think you are working on a rather old version.



> Hey @zhiheng-huang,
> 
> Thanks for the PR!
> 
> In general I'm fine with this PR - think adding more types of position embeddings to a model is OK.
> Also can you rebase your PR to the most current version of `master` - I think you are working on a rather old version.

It was rebased to master on 10/27, will rebase again for the new revision.

@patrickvonplaten, I did a rebase and I am not sure this is the correct way to review commit "Address review comment". Please let me know if there is a better way to upload the new diff.

Hey @zhiheng-huang,

I think there was a problem with the rebase it seems like you added all commits on master on top of your PR. 
This happens from time to time sadly :-/ 

The way I'd fix it is to first save your changes of the last commit: https://github.com/huggingface/transformers/pull/8108/commits/ffe2e64c64f03c141cc085c8f3f509ae2e0992e2 somewhere (maybe a new branch).

Then correctly reset the head of your branch before all other commitns were falsely added:
```
git reset --hard 36729ee
```

Then add the single commit you saved in another branch


```
git cherry-pick ffe2e64
```

and finally either you correctly rebase OR the safer option here would probably be to merge the master into your branch

```
git fetch upstream/master
git merge upstream master
```

Hope this helps!

> Hey @zhiheng-huang,
> 
> I think there was a problem with the rebase it seems like you added all commits on master on top of your PR.
> This happens from time to time sadly :-/
> 
> The way I'd fix it is to first save your changes of the last commit: [ffe2e64](https://github.com/huggingface/transformers/commit/ffe2e64c64f03c141cc085c8f3f509ae2e0992e2) somewhere (maybe a new branch).
> 
> Then correctly reset the head of your branch before all other commitns were falsely added:
> 
> ```
> git reset --hard 36729ee
> ```
> 
> Then add the single commit you saved in another branch
> 
> ```
> git cherry-pick ffe2e64
> ```
> 
> and finally either you correctly rebase OR the safer option here would probably be to merge the master into your branch
> 
> ```
> git fetch upstream/master
> git merge upstream master
> ```
> 
> Hope this helps!

Thanks. @patrickvonplaten. this helps but I may have to revert the commits merged to zhiheng-huang:transformers-relative-embedding. I created a new PR at https://github.com/huggingface/transformers/pull/8276 to continue the review. Thanks!