[RAG] token discrepancy between question token which should be input to generator and the one actually encoded in postprocess_docs()
Author: 4kasha
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
```


### Who can help?

RAG, DPR: @patrickvonplaten, @lhoestq

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the [RAG implementation](https://github.com/huggingface/transformers/tree/v4.19.2/src/transformers/models/rag), especially in [here](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/rag/retrieval_rag.py#L466), input question and retrieved documents by DPR are concatenated in the form like 
` doc_title + / + doc_text + // + input_string (question)`,
in `postprocess_docs(docs, input_strings, ... )` and this is the input to the generator. (eg, BART, T5 ).
This postprocess_docs() receives input_strings decoded by question tokenizer at [here](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/rag/retrieval_rag.py#L610-L613).

However, this postprocess may cause token mismatch in the case that question and generator tokenizer are different.
To see this, I choose basic retriever and generator as follows (original paper setting).

```
from transformers import DPRQuestionEncoderTokenizer, BartTokenizer

dpr_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
bart_tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
```

and then, following the implementation, compare their token ids each other
```
def check_ids(q_tokenizer, g_tokenizer, text):
    print(f'>> {g_tokenizer.tokenize(text)}')
    ids = q_tokenizer(text)['input_ids']
    decode = q_tokenizer.decode(ids, skip_special_tokens=True)
    print(f'>> {g_tokenizer.tokenize(decode)}')
    _ids = g_tokenizer(decode)['input_ids']
    print(ids == _ids)
```
As a sample text, `text = "Don't you love Transformers? We sure do."` 

```
check_ids(dpr_tokenizer, bart_tokenizer, text)
>> ['Don', "'t", 'Ġyou', 'Ġlove', 'ĠTransformers', '?', 'ĠWe', 'Ġsure', 'Ġdo', '.']
>> ['don', "'t", 'Ġyou', 'Ġlove', 'Ġtransform', 'ers', '?', 'Ġwe', 'Ġsure', 'Ġdo', '.']
False
```

In conclusion, I would be wrong, but postprocess_docs() should be better if it receives the raw question string without tokenizer encode-decode process.


### Expected behavior

```shell
Question string part of the input to the generator is expected to be same with the question (input) to the RAG model.
```


Comments:
Interesting question! @ola13 do you have a good answer here maybe? 

Intuitively I would think that this shouldn't be a problem since the model was trained also this way, but think @ola13 knows best here :-) 

Hi, sorry for the delayed response! Thanks for your question @4kasha!

To answer your point here:

> In conclusion, I would be wrong, but postprocess_docs() should be better if it receives the raw question string without tokenizer encode-decode process.

This is in fact what happens - the `postprocess_docs` function does receivesthe question as a string and the documents as strings and only encodes them once with the generator's tokenizer to create inputs to the generator model. If we were passing tokens between the retriever and the generator we would indeed have a mismatch - we're not doing this though, and this is by design. Making a round-trip (first decoding retrieved docs tokens to pure strings and then encoding them with the generator tokenizer  gives us the flexibility to combine retrievers and generators which don't have matching tokenization schemes.

I hope this clarifies things, but feel free to re-open the issue if there's anything unclear still.