Add DPR model
Author: lhoestq
Pull request: yes

# Dense Passage Retrieval

## Intro

The Dense Passage Retrieval (DPR) model from facebook ([github](https://github.com/facebookresearch/DPR), [arxiv](https://arxiv.org/abs/2004.04906)).
It is used to do Open Domain Question Answering by extracting answer spans from a set of documents.
This model actually comes in three different parts:
- a context encoder
- a question encoder
- a reader for span prediction

I did a schema to show the roles and the pipeline that one could build with those parts.
The components in RED are the one in transformers.
You can use whatever you want for the retrieval part, but there will be a new [retrieval feature](https://github.com/huggingface/nlp/pull/298) in the 🤗nlp library soon that will make the use of models like DPR easier.
<img src="https://user-images.githubusercontent.com/42851186/85740169-b05da980-b701-11ea-975d-fbff4f368a5e.png" height="300">

## Implementation

- All three components share an encoding part with bert so I factorized this into the class `DprBertEncoder `.
- The reader has a `.generate` method that finds the best spans and return them, while the `.forward` method only returns the logits.
- In the config I allow to specify to load the weights from [files provided in the official repo](https://github.com/facebookresearch/DPR/blob/master/data/download_data.py). I've already added one pretrained weight file per component in S3.

## Things I'd like to improve:

- I think we can probably improve the tokenization step. Right now the reader inputs are currently two sets of input_ids. One for the question with text_pair=context_title, and one for the context_text (i.e. the content in which we are looking for answer spans). This is because they all need to be combined like
```
[CLS] <question_input_ids> [SEP] <context_title_input_ids> [SEP] <context_text_input_ids>
```
I was thinking of making a custom tokenizer just for the reader, let me know if it sounds reasonable.

## Example of usage (outdated)

Provided we have a retrieval module (here named`wiki`) we can do:
```python
tokenizer = DprTokenizer.from_pretrained('dpr-model-base')
ctx_encoder = DprContextEncoder.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base')
question_encoder = DprQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
reader = DprReader.from_pretrained('facebook/dpr-reader-single-nq-base')

# First step: retrieve the best wikipedia passages
question = 'Who created the Pokemon games ?'
question_emb = question_encoder(tokenizer.encode(question, return_tensors="pt")).numpy()
scores, passages = wiki.get_nearest_examples("embeddings", question_emb, k=10)
# Second step: Feed the reader with the question and the retrieved snippets
encoded_question_and_titles = [
    tokenizer.encode(question, text_pair=passage["title"], return_tensors="pt")
    for passage in passages]
encoded_texts = [
    tokenizer.encode(passage["text"], return_tensors="pt", add_special_tokens=False)
    for passage in passages]
predicted_spans = reader.generate(encoded_question_and_titles, encoded_texts)
# Last step: print the result
best_span = predicted_spans[0]
best_span_ids = encoded_texts[best_span.doc_id].numpy().flatten()
best_span_ids = best_span_ids[best_span.start_index:best_span.end_index + 1]
print(tokenizer.decode(best_span_ids))
# >>> satoshi tajiri
```
-----------------
I'd be very happy to have some feedbacks on this one, as it is my first contribution to the library :D



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=h1) Report
> Merging [#5279](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7c41057d5090f5e665f2404878369ecb13939def&el=desc) will **decrease** coverage by `0.80%`.
> The diff coverage is `37.29%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5279/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5279      +/-   ##
==========================================
- Coverage   78.34%   77.54%   -0.81%     
==========================================
  Files         138      141       +3     
  Lines       23841    24085     +244     
==========================================
- Hits        18679    18676       -3     
- Misses       5162     5409     +247     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kcHIucHk=) | `28.29% <28.29%> (ø)` | |
| [src/transformers/configuration\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2Rwci5weQ==) | `62.50% <62.50%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.23% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/tokenization\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZHByLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.66% <0.00%> (-28.03%)` | :arrow_down: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `89.95% <0.00%> (-0.92%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.12% <0.00%> (-0.89%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `76.42% <0.00%> (-0.39%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `91.50% <0.00%> (-0.32%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/huggingface/transformers/pull/5279/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=footer). Last update [7c41057...7a90958](https://codecov.io/gh/huggingface/transformers/pull/5279?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks very cool! 
I would have three general things:

1) For all parameter names I would try to stick to the ones we already have `config` instead of `cfg`, `pad_token_id` instead of `pad_id`. For me `Bert` is always the gold standard and I would try to name everything as it is named in Bert. All these `hidden_states`, `output_embeddings`, ...

2) I don't like single letter or very short variable names, it makes it very hard to understand it and impossible sometimes to run  select and replace commands in such files. I do not mind having long parameter names at all. But also not sure what other think here @LysandreJik 

3) I would always favor composition over inheritance (there was only one case which I would have changed) 

4) IMO, we should not introduce new class methods and in general new design choices that the user doesn't know from other models. The class method `init_encoder` is not needed here IMO. Also, all these `make it super easy for the user ` methods are not always good if it makes the model less easy to understand. Would always try to keep models as "thin" as possible and not add any "magic" methods if not needed

What do you think of having a special tokenizer for the `DPRReader` ? I find the current way to tokenize the inputs a bit weird.

I can have a custom `DPRReaderTokenizer` with a new `__call__` method like
```python
def __call__(self, question: str, titles: List[str], texts: List[str], ...):
```

Ok I think this one is ready to merge. Could you do a final pass to make sure everything is ok @LysandreJik @thomwolf ?
I'll do another PR about the tokenization stuff.

I did some changes @LysandreJik :
- have 1:1 correspondances betwen model + tokenizers 
- change `base_model_prefix` to be the attribute of the model the classes are wrapping
- remove the wrong `model_input_names `

Not sure why the CI doesn't pass. It seems related to `test_tokenization_bert_japanese ` though :/

I changed the tokenizers config names to match the pretrained model names.
About the `.generate` method that could be in the tokenizer I totally agree. But as it is linked to the way I'm going to change the reader's tokenizer, I will do the change at the same time in the next PR if it's good for you.

Is there anything else that needs to be improved ?

Feel free to do another pass on the PR @thomwolf @LysandreJik to make sure that all the model+tokenizers aspect are all good now :)
There is still room for some improvements but I keep them for the next PR:
- have a custom __call__ for the tokenizer of the reader
- move the deocing stuff of `.generate` to the tokenizer or the reader

Thanks for your comments @LysandreJik  :)
If there is a 3.0.1 that's going to be shipped I'd rather have everything in this PR then as there will be big breaking changes

Ok @LysandreJik I added the tests for the models and the tokenizers :)
I couldn't use all the tests of `ModelTesterMixin` as some of them don't apply here, but I used those that are relevant.

I merged your PR and updated the docs @LysandreJik 
Thanks for your help ;)

Very cool! Let's merge, thanks for iteration @lhoestq :)

Is there a short working example on the full end-to-end retrieval to reader pipeline?

If I may add to @weilin37 comment, fine tuning included