RAG: Do we need to pretrained the doc-encoder when using a custom dataset?
Author: shamanez
Pull request: no
Labels: name: wontfix

Now the RAG consists of a [script](https://github.com/huggingface/transformers/blob/master/examples/rag/use_own_knowledge_dataset.py) where we can use a custom dataset other than the wiki-dataset. 


Since, in the fine-tuning phase of the RAG, we do not update the doc-encoder (we update only BART and Question Encoder), what if our custom dataset consists of different distribution compared to the wiki dataset (Ex: medical records)? 

Will it still work?

Comments:
Hey @shamanez,

From the paper: https://arxiv.org/pdf/2005.11401.pdf (check part 2.2 - 2.5) it seems like the doc-encoder was never explicitely pre-trained for RAG, but the authors used a pre-trained retriever that "was trained to retrieve documents which
contain answers to TriviaQA [20] questions and Natural Questions [25]." => So you would have to see for yourself if the document encoder is sutiable for your task or not. If it is not suitable you will have to pre-train your own doc encoder and built the index using this document encoder. 

It's a good question though we should probably also put it on the discussion forum: http://discuss.huggingface.co/.

Also pinging @lhoestq  as he can probably provide you with a better answer than I can.


+1 for Patrick, and I confirm that RAG uses the pretrained doc encoder from DPR.
It would be very interesting to see if the doc encoder handles well documents from specific domains.
Let us know if you plan to test that :)

If it doesn't work for your case you will probably need to train DPR on your dataset for retrieval before using RAG

@patrickvonplaten  @lhoestq 

Ok, I will give it a try and do a comparison for my task. Just need to clarify my pre-training pipeline for DPR in case I need to pre-train the doc encoder.

1. Pre-train the DPR using the FacebookAI [repository](https://github.com/facebookresearch/DPR).
2. Use the custom checkpoint and load it [here](https://github.com/huggingface/transformers/blob/master/examples/rag/finetune.py#L105) (Is there any conversion I need to do before this step)

@lhoestq 

Sorry for spamming. Can you please let me know, whether I can directly use different DPR checkpoints that trained with Facebook repo? 

> @patrickvonplaten @lhoestq
> 
> Ok, I will give it a try and do a comparison for my task. Just need to clarify my pre-training pipeline for DPR in case I need to pre-train the doc encoder.
> 
> 1. Pre-train the DPR using the FacebookAI [repository](https://github.com/facebookresearch/DPR).
> 2. Use the custom checkpoint and load it [here](https://github.com/huggingface/transformers/blob/master/examples/rag/finetune.py#L105) (Is there any conversion I need to do before this step)

Yes that's it. 
To convert the DPR checkpoint from the original repo to transformers you can use the script `src/transformers/convert_dpr_original_checkpoint_to_pytorch.py`



Perfect. Thanks for the clarification  :).

On Thu, Oct 29, 2020, 22:32 Quentin Lhoest <notifications@github.com> wrote:

> @patrickvonplaten <https://github.com/patrickvonplaten> @lhoestq
> <https://github.com/lhoestq>
>
> Ok, I will give it a try and do a comparison for my task. Just need to
> clarify my pre-training pipeline for DPR in case I need to pre-train the
> doc encoder.
>
>    1. Pre-train the DPR using the FacebookAI repository
>    <https://github.com/facebookresearch/DPR>.
>    2. Use the custom checkpoint and load it here
>    <https://github.com/huggingface/transformers/blob/master/examples/rag/finetune.py#L105>
>    (Is there any conversion I need to do before this step)
>
> Yes that's it.
> To convert the DPR checkpoint from the original repo to transformers you
> can use the script
> src/transformers/convert_dpr_original_checkpoint_to_pytorch.py
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8037#issuecomment-718546578>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGQXQYM5S2CYMOQ3RWDSNEZCPANCNFSM4S6VBIUA>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.