Non-BERT dpr tokenizers
Author: Zhylkaaa
Pull request: no

# 🚀 Feature request

At the moment `DPRContextReadertokenizer` is subclassing `BertTokenizer` which limits the model choice. Is it possible to use `PreTrainedTokenizer` instead?

## Motivation

We would like to contribute a non-english version of DPR and got stack because our base model uses `XLMTokenizer`. (model was trained on forked and modified version of original dpr script)

## Your contribution

I would love to open a PR with this modification, but really not sure where to start and what is the best choice for the solution.
Here is the link to hf forum is somebody is interested in discussion (I guess this is the best placce to discuss): https://discuss.huggingface.co/t/create-dpr-tokenizer-for-non-bert-model/9735/2


Comments:
cc @lhoestq 

The `DPRReaderTokenizer` is a custom tokenizer that is able to tokenize jointly the three DPR inputs (questions, context titles and context texts).
The custom part is implemented via `CustomDPRReaderTokenizerMixin`, so it should be possible to create a new tokenizer class that inherits from both `CustomDPRReaderTokenizerMixin` and the `XLMTokenizer`. Let me know if that helps

Does this require any change in the `transformers` codebase though ?

hello @lhoestq and thank you! I have tried inheriting `CustomDPRReaderTokenizerMixin` and our model's class, and it seems to work, but I believe it still isn't compatible with RAG model, as this requires `DPRContextEncoderTokenizerFast` (at least examples).

I think codebase would be necessary at some point, because adding new model will require adding +1 class with right inheritance or switching to `AutoTokenizer` as a base class (this works, but I think it will create some troubles with fast tokenizers)

What do you think about `AutoTokenizer` as a base class?

> hello @lhoestq and thank you! I have tried inheriting CustomDPRReaderTokenizerMixin and our model's class, and it seems to work, but I believe it still isn't compatible with RAG model, as this requires DPRContextEncoderTokenizerFast (at least examples).

For RAG you don't need the DPR Reader part. Which part of RAG doesn't work with your tokenizer ?

> I think codebase would be necessary at some point, because adding new model will require adding +1 class with right inheritance or switching to AutoTokenizer as a base class (this works, but I think it will create some troubles with fast tokenizers)

So if I understand correctly, the code of the tokenizer you would like to use is not in `transformers` since it's a mix of the XLMTokenizer and some custom code from the DPRReaderTokenizer.

Not sure why AutoTokenizer would work in this case ?

I am sorry, I made a few mistakes while writing previous post. (also sorry it took so long to reply)
So to make it all clear, I am trying to convert DPR retriever (question and context encoder) trained with original DPR script from Facebook. Base model for encoders is `Herbert` (this is basically `Bert` with `XLMTokenizer`). 

I was talking about editing (changing base class from `BertTokenizer`) `DPRContextEncoderTokenizer` and `DPRQuestionEncoderTokenizer` because those parts are needed to create [consolidated rag model](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/consolidate_rag_checkpoint.py) and  use it with [`finetune_rag.py`](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag) script.

(I actually, made it work just changing base classes to AutoModel, very dirty, but working solution. But I can't guarantee that this solution won't fire at me at some point, because I am struggling with distributed retriever part in this script and can't confirm that there aren't some pitfalls I am not ever of) 

> I actually, made it work just changing base classes to AutoModel

I think you did right and it should be fine :)

Ok, so should I open PR with this or just close this issue and work with my local brunch? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.