Add TFDPR
Author: ratthachat
Pull request: yes

# What does this PR do?

Add `TFDPRContextEncoder, TFDPRQuestionEncoder` and `TFDPRReader` in `modeling_tf_dpr.py`, as well as other relevant files in the [checklist](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model) . 

Now the TF model works properly and can load Pytorch's weights successfully the same output as Pytorch's counterparts **except** small random noise (1e-5) which I suspect of some dtypes different , but I could not find the cause. 

We can try playing TFDPR models and compare to Pytorch's ones [here in Colab](https://colab.research.google.com/drive/1czS_m9zy5k-iSJbzA_DP1k1xAAC_sdkf?usp=sharing) 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to the it if that's the case.

Here: https://github.com/huggingface/transformers/issues/8171

- [X] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?
       Yes, I write simple test whether TF and Pytorch models with pretrained weights give the same output (except very small random noise). The test is in Colab (shared above).
       Also the model is passed all 27 tests in test_modeling_tf_dpr.py file (Please see the last cell in Colab above)

## Who can review?
@LysandreJik


# Details what were done according to Checklist

## Adding model/configuration/tokenization classes
Mostly done due to pre-existing of Pytorch's DPR.
- [X] Copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model
  name.
- [X] Edit the files to replace `XXX` (with various casing) with your model name.
- [X] Copy-paste or create a simple configuration class for your model in the `configuration_...` file.
- [X] Copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0).
- [X] Copy-paste or create a tokenizer class for your model in the `tokenization_...` file.

## Adding conversion scripts

- [ ] Copy the conversion script (`convert_...`) from the present folder to the main folder.
- [ ] Edit this script to convert your original checkpoint weights to the current pytorch ones.

Not sure what to do since there already exists DPR pretrained weights.

## Adding tests:

- [X] Copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main
  folder and rename them, replacing `xxx` with your model name.
- [X] Edit the tests files to replace `XXX` (with various casing) with your model name.
- [X] Edit the tests code as needed.

The model is passed all 27 tests in test_modeling_tf_dpr.py file (Please see the last cell in Colab above) -- This is my updated 4 days after made a 1st PR.

## Documenting your model:

- [X] Make sure all your arguments are properly documented in your configuration and tokenizer.
- [X] Most of the documentation of the models is automatically generated, you just have to make sure that
  `XXX_START_DOCSTRING` contains an introduction to the model you're adding and a link to the original
  article and that `XXX_INPUTS_DOCSTRING` contains all the inputs of your model.
- [X] Create a new page `xxx.rst` in the folder `docs/source/model_doc` and add this file in `docs/source/index.rst`.


## Final steps
(Note the Pytorch DPR was already existed, so I assume I should check as "Done" )
- [X] Add import for all the relevant classes in `__init__.py`.
- [X] Add your configuration in `configuration_auto.py`.
- [X] Add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`.
- [X] Add your tokenizer in `tokenization_auto.py`.
- [ ] Add a link to your conversion script in the main conversion utility (in `commands/convert.py`)
- [X] Edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py`
  file.
- [ ] Add a mention of your model in the doc: `README.md` and the documentation itself
  in `docs/source/pretrained_models.rst`. Rune `make fix-copies` to update `docs/source/index.rst` with your changes.
- [X] Upload the pretrained weights, configurations and vocabulary files.
- [ ] Create model card(s) for your models on huggingface.co. For those last two steps, check the
  [model sharing documentation](https://huggingface.co/transformers/model_sharing.html).


Comments:
Hi @LysandreJik , thanks for the great review! 
I will fix and test the code as suggested.

However, I have a very newbie question about style and document fixing. I admit that I can access only Colab (I have only windows PC) so I am really not sure how to do  `make fixup` and `make docs` (in my understanding, should run in linux environment) .  Could you please make some suggestions on this issue ?

Hmmm I think you could do the following in your colab environment:

```py
# Clone the repo or your fork, I assume this is your working directory
!git clone https://github.com/huggingface/transformers
!cd transformers

# Install all the dev dependencies (you've probably done that already)
!pip install -e .[dev]

# Then you should be able to run `make fixup`
!make fixup

# Same for the docs!
!make docs
```
Let me know if that works!

@LysandreJik , thanks again and I could run the two make commands.

1) `make docs` , produced the error message which I have no clue, so I still need suggestion on this issue, sorry:
```
cd docs && make html SPHINXOPTS="-W"
make[1]: Entering directory '/content/transformers/docs'
Running Sphinx v1.8.5

Extension error:
Could not import extension recommonmark (exception: No module named recommonmark)
Makefile:19: recipe for target 'html' failed
make[1]: *** [html] Error 2
make[1]: Leaving directory '/content/transformers/docs'
Makefile:68: recipe for target 'docs' failed
make: *** [docs] Error 2
```


2) `make fixup` , required `make fix-copies` and then  `make fixup` suggest me to add more tests on `test_modeling_tf_dpr.py` which I will investigate this issue and come back :D

Hey @ratthachat - great work! I helped you a bit with the docs and did some cleaning. I think we can merge the PR soon. It would be great if you could take a look at the comments above (a lot of them should already be resolved now) and also it would be awesome if you could add one integration test to both TF and PT (we forgot to do this originally for PyTorch).

An integration tests should be a slow test, where you can just statically type some input_ids vector and run it through one of the PyTorch pretrained models and test against its expected output. You should use the same input / expected output array then for Tensorflow, similar to how it's done here:

https://github.com/huggingface/transformers/blob/4185b115d4b3fd408265ffd91581698325652c47/tests/test_modeling_roberta.py#L423

Let me know if you have any questions!

Thanks very much for your great help, Patrick @patrickvonplaten !! I will get back to you guys as soon as possible.

Hi guys, with the great helps of Patrick, most comments of @LysandreJik were already dealt with. So I further addressed the rest as replied above. BTW, I did only very minimal and necessary changes, but many tests are now failed again , sorry.. I have no idea about this :( .

@patrickvonplaten I added one slow model integration test to `test_modeling_tf_dpr.py` . However, at the moment I still could not find a way to run original DPR repo to produce original output yet. So at the moment, the integration test is just a chek that TF and Pytorch `DPRQuestionEncoder` models produce the same output (with acceptable margin of error) -- i.e. the `tf.constant` expected slice comes from Pytorch's model.

(test can be played around here : https://colab.research.google.com/drive/1czS_m9zy5k-iSJbzA_DP1k1xAAC_sdkf?usp=sharing )

I will come back to add more model integration tests if I succeed to run the [original DPR](https://github.com/facebookresearch/DPR/blob/master/generate_dense_embeddings.py).

> Hi Patrick, I have a question. At the moment, we do not have native TF weights, so removing this is OK ?

I uploaded them a minute ago ;-) 

Thanks so much everyone. Very happy :D 👯 
See you guys again soon on TFRag (WIP)

@patrickvonplaten I think you didn't upload all of the weights on the model hub. I'm uploading the remaining weights now:

- `facebook/dpr-ctx_encoder-single-nq-base`
- `facebook/dpr-ctx_encoder-multiset-base`
- `facebook/dpr-question_encoder-multiset-base`
- `facebook/dpr-reader-single-nq-base`
- `facebook/dpr-reader-multiset-base`


(Current slow CI is failing because it's trying to load some of them)

They're all uploaded.